2024-09-21T09:40:03.181 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T09:40:03.211 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T09:40:03.313 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665 branch: main description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} email: ceph-qa@ceph.com first-half-sequence: - cephadm.shell: env: - sha1 mon.a: - ceph config set mgr mgr/cephadm/daemon_cache_timeout 60 - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps - echo wait for minority of mons to upgrade - while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error; do sleep 2 ; done - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for majority of mons to upgrade - 'while ! ceph mon versions | grep $sha1 | egrep '': [23]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for all mons to upgrade - 'while ! ceph mon versions | grep $sha1 | grep '': 3'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch ps - ceph orch upgrade pause - sleep 60 - ceph orch upgrade resume - echo wait for half of osds to upgrade - 'while ! ceph osd versions | grep $sha1 | egrep '': [45678]'' && ceph orch upgrade status | jq ''.in_progress'' | grep true && ! ceph orch upgrade status | jq ''.message'' | grep Error; do sleep 2 ; done' - ceph orch upgrade pause - ceph orch ps - ceph orch ps - ceph versions first-half-tasks: - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' first_in_suite: false job_id: '7913665' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'librbd C and C++ api tests ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' - desc: 'run basic cls tests for rbd ' - desc: 'run basic import/export cli tests for rbd ' - desc: 'librbd C and C++ api tests ' - desc: 'randomized correctness test for rados operations on a replicated pool, using only reads, writes, and deletes ' - desc: 'randomized correctness test for rados operations on a replicated pool with snapshot operations ' - desc: 'run randomized correctness test for rados operations generate write load with rados bench ' name: teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true create_rbd_pool: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_DEGRADED - Reduced data availability - Degraded data redundancy - OSDMAP_FLAGS - PG_AVAILABILITY sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 owner: scheduled_teuthology@teuthology priority: 850 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b second-half-sequence: sequential: - cephadm.shell: env: - sha1 mon.a: - ceph orch upgrade resume - sleep 60 - echo wait for upgrade to complete - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done - echo upgrade complete - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 second-half-tasks: - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' seed: 5357 sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 sleep_before_teardown: 0 stress-tasks: - thrashosds: aggressive_pg_num_changes: false chance_force_recovery: 0 chance_pgnum_grow: 1 chance_pgpnum_fix: 1 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_thrash_pg_upmap_items: 0 disable_objectstore_tool_tests: true timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - print: '**** done end radosbench.yaml' - workunit: branch: quincy clients: client.0: - cls/test_cls_rbd.sh - print: '**** done cls/test_cls_rbd.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - print: '**** done rbd/import_export.sh 5-workload' - workunit: branch: quincy clients: client.0: - rbd/test_librbd.sh env: RBD_FEATURES: '61' - print: '**** done rbd/test_librbd.sh 7-workload' - full_sequential: - rados: clients: - client.0 objects: 500 op_weights: delete: 10 read: 45 write: 45 ops: 4000 write_append_excl: false - print: '**** done rados/readwrite 5-workload' - full_sequential: - rados: clients: - client.0 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 write_append_excl: false - print: '**** done rados/snaps-few-objects 5-workload' subset: 31/32 suite: upgrade suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 targets: smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITN88S7UUJCLuB6FpSjBmZ3xTNZCcYzhwPvdt2xwO4cHxdClfACFXcVI3wQiZRr/1i+cSR+taEGN+BOKhw2dmM= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZRYzER2sEaNxtgEmItWEnKLSuLifDVQ4PVDJi2mHfxdPvJThlxBTRVZIkaD9Psd5fsAnFqsFwIslI2OLutwWY= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: branch: quincy exclude_packages: - ceph-volume - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy - cephadm.shell: mon.a: - ceph fs volume create foo - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph.healthy: null - print: '**** upgrading first half of cluster, with stress ****' - parallel: - first-half-tasks - first-half-sequence - print: '**** done upgrading first half of cluster ****' - ceph.healthy: null - print: '**** applying stress + thrashing to mixed-version cluster ****' - parallel: - stress-tasks - ceph.healthy: null - print: '**** finishing upgrade ****' - parallel: - second-half-tasks - second-half-sequence - ceph.healthy: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_03:08:26 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T09:40:03.314 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa; will attempt to use it 2024-09-21T09:40:03.314 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks 2024-09-21T09:40:03.315 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T09:40:03.316 INFO:teuthology.task.internal:Checking packages... 2024-09-21T09:40:03.336 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '62da54b4b83a96ca9976b7e1d0c8272a564a2208' 2024-09-21T09:40:03.336 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:40:03.336 INFO:teuthology.packaging:ref: None 2024-09-21T09:40:03.336 INFO:teuthology.packaging:tag: None 2024-09-21T09:40:03.336 INFO:teuthology.packaging:branch: main 2024-09-21T09:40:03.337 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:40:03.337 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-09-21T09:40:03.543 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5092.g62da54b4 2024-09-21T09:40:03.545 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T09:40:03.551 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T09:40:03.551 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T09:40:03.556 INFO:teuthology.task.internal:Saving configuration 2024-09-21T09:40:03.579 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T09:40:03.586 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T09:40:03.607 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:35:06.809119', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITN88S7UUJCLuB6FpSjBmZ3xTNZCcYzhwPvdt2xwO4cHxdClfACFXcVI3wQiZRr/1i+cSR+taEGN+BOKhw2dmM='} 2024-09-21T09:40:03.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 09:35:06.810940', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZRYzER2sEaNxtgEmItWEnKLSuLifDVQ4PVDJi2mHfxdPvJThlxBTRVZIkaD9Psd5fsAnFqsFwIslI2OLutwWY='} 2024-09-21T09:40:03.626 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T09:40:03.631 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T09:40:03.632 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-21T09:40:03.632 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T09:40:03.687 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f02c5318160>, signals=[15]) 2024-09-21T09:40:03.688 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T09:40:03.693 INFO:teuthology.task.internal:Opening connections... 2024-09-21T09:40:03.693 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-09-21T09:40:03.694 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:40:03.767 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-09-21T09:40:03.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:40:03.841 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T09:40:03.848 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-09-21T09:40:03.866 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-09-21T09:40:03.866 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-09-21T09:40:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:40:03.925 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:40:03.925 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:40:03.925 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:40:03.925 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:40:03.925 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:40:03.925 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-09-21T09:40:03.946 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-09-21T09:40:03.965 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-09-21T09:40:03.965 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-09-21T09:40:04.022 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2024-09-21T09:40:04.022 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="9" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="9" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el9" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:LOGO="fedora-logo-icon" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T09:40:04.023 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T09:40:04.024 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-09-21T09:40:04.046 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T09:40:04.056 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T09:40:04.060 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T09:40:04.060 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:40:04.063 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-09-21T09:40:04.079 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T09:40:04.084 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T09:40:04.084 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:40:04.120 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-09-21T09:40:04.234 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T09:40:04.288 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T09:40:04.300 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T09:40:04.300 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T09:40:04.300 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:40:04.301 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T09:40:04.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:40:04.317 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-21T09:40:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:40:04.318 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-09-21T09:40:04.374 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:40:04.374 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-503.el9.x86_64 2024-09-21T09:40:04.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-09-21T09:40:04.375 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-503.el9.x86_64 2024-09-21T09:40:04.376 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.14.0-503.el9.x86_64 2024-09-21T09:40:04.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2024-09-21T09:40:06.689 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-21T09:40:07.344 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.3 MB 00:01 2024-09-21T09:40:08.415 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 31 MB/s | 20 MB 00:00 2024-09-21T09:40:12.941 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.5 MB 00:00 2024-09-21T09:40:14.400 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 19 kB 00:00 2024-09-21T09:40:14.594 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 20 MB 00:06 2024-09-21T09:40:15.154 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-21T09:40:20.227 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-21T09:40:20.610 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 10 kB/s | 1.7 kB 00:00 2024-09-21T09:40:21.761 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 19 kB 00:00 2024-09-21T09:40:22.491 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:40:22.491 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:40:22.578 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:40:22.589 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:40:22.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:40:22.590 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:40:22.590 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:40:22.590 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-09-21T09:40:22.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:40:22.590 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 79 M 2024-09-21T09:40:22.591 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 126 M 2024-09-21T09:40:22.591 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:40:22.624 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-21T09:40:22.728 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 307 kB/s | 36 kB 00:00 2024-09-21T09:40:24.436 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 8.6 MB/s | 16 MB 00:01 2024-09-21T09:40:25.727 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 9.5 MB/s | 29 MB 00:02 2024-09-21T09:40:26.295 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 9.4 MB/s | 35 MB 00:03 2024-09-21T09:40:26.295 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:40:26.295 INFO:teuthology.orchestra.run.smithi119.stdout:Total 21 MB/s | 79 MB 00:03 2024-09-21T09:40:26.976 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:40:27.335 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:40:27.335 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:40:28.088 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T09:40:28.286 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:40:28.287 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:40:30.028 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T09:40:30.028 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T09:40:30.121 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-09-21T09:40:30.128 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:40:30.129 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 79 M 2024-09-21T09:40:30.129 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 126 M 2024-09-21T09:40:30.129 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:40:30.364 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 321 kB/s | 36 kB 00:00 2024-09-21T09:40:30.710 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:40:30.906 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 53 MB/s | 35 MB 00:00 2024-09-21T09:40:31.182 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 35 MB/s | 29 MB 00:00 2024-09-21T09:40:31.365 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:40:31.440 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-21T09:40:31.440 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:40:31.441 INFO:teuthology.orchestra.run.smithi145.stdout:Total 60 MB/s | 79 MB 00:01 2024-09-21T09:40:31.447 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:40:32.125 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:40:32.495 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:40:32.495 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:40:33.433 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:40:33.434 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:40:34.200 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:40:34.325 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:40:35.866 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:40:36.509 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:40:36.607 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:40:39.150 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:40:39.192 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:40:39.357 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:40:39.486 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:40:44.014 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:40:44.316 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:40:44.353 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:40:49.193 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:21.916 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:21.956 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:22.317 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:22.317 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:41:22.317 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:41:22.317 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:41:23.153 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:41:23.154 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:41:23.510 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:41:23.602 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:41:23.602 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:41:23.602 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:41:23.602 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:41:23.602 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:41:23.602 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T09:41:23.603 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:41:23.603 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:41:23.661 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:41:23.661 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:41:23.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 82.el9 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8274670 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:41:23.693 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:41:23.694 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:41:23.695 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:41:23.695 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:41:23.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:41:24.321 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-09-21T09:41:25.636 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:41:25.665 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-09-21T09:41:25.666 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-21T09:41:25.682 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.roFHsywXE8 2024-09-21T09:41:25.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.roFHsywXE8 2024-09-21T09:41:25.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.roFHsywXE8 2024-09-21T09:41:25.887 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.roFHsywXE8 is 6KB 2024-09-21T09:41:25.935 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.roFHsywXE8 2024-09-21T09:41:25.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:41:26.024 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:41:26.024 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:41:26.024 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:41:26.024 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:41:26.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:41:26.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-09-21T09:41:27.252 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:27.300 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:27.704 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:27.704 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T09:41:27.704 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T09:41:27.704 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T09:41:28.549 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T09:41:28.549 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:41:28.550 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:41:28.906 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:41:28.998 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:41:28.999 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:41:28.999 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T09:41:28.999 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T09:41:28.999 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T09:41:28.999 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T09:41:28.999 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T09:41:28.999 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T09:41:29.057 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T09:41:29.058 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T09:41:29.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.06 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 82.el9 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Group : Unspecified 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 8274670 2024-09-21T09:41:29.089 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : builder@centos.org 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T09:41:29.090 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T09:41:29.091 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2024-09-21T09:41:29.091 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:41:29.091 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T09:41:29.091 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T09:41:29.091 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 5.14.0-508.el9.x86_64 2024-09-21T09:41:29.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T09:41:29.728 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-09-21T09:41:30.948 INFO:teuthology.orchestra.run.smithi145.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T09:41:30.977 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-09-21T09:41:30.980 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-09-21T09:41:30.995 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.NLOCNURivC 2024-09-21T09:41:30.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NLOCNURivC 2024-09-21T09:41:31.066 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.NLOCNURivC 2024-09-21T09:41:31.216 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.NLOCNURivC is 6KB 2024-09-21T09:41:31.264 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.NLOCNURivC 2024-09-21T09:41:31.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T09:41:31.350 INFO:teuthology.orchestra.run.smithi145.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T09:41:31.350 INFO:teuthology.orchestra.run.smithi145.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T09:41:31.350 INFO:teuthology.orchestra.run.smithi145.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T09:41:31.351 INFO:teuthology.orchestra.run.smithi145.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T09:41:31.352 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T09:41:31.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-09-21T09:41:56.191 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T09:41:56.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-21T09:41:56.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:01.510 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T09:42:01.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:42:01.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:14.511 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-09-21T09:42:19.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-09-21T09:42:24.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-21T09:42:24.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:27.123 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-09-21T09:42:28.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:42:28.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:31.951 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-09-21T09:42:39.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-21T09:42:39.131 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:42:43.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:42:43.957 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:43:02.383 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-09-21T09:43:11.238 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-21T09:43:11.786 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-21T09:43:11.786 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:43:11.786 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-21T09:43:11.827 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:43:11.827 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:43:11.827 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:43:11.827 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:43:12.828 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T09:43:12.829 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:43:12.896 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-09-21T09:43:12.944 DEBUG:teuthology.parallel:result is None 2024-09-21T09:43:17.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:43:17.384 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:43:17.722 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-09-21T09:43:18.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:43:18.117 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T09:43:18.118 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-09-21T09:43:18.131 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-508.el9.x86_64 2024-09-21T09:43:18.132 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T09:43:18.132 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T09:43:18.132 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T09:43:19.133 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T09:43:19.133 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T09:43:19.235 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2024-09-21T09:43:19.257 DEBUG:teuthology.parallel:result is None 2024-09-21T09:43:19.257 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T09:43:19.271 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T09:43:19.271 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:43:19.274 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T09:43:19.290 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T09:43:19.307 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T09:43:19.322 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T09:43:19.322 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:43:19.333 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T09:43:19.374 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T09:43:19.380 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T09:43:19.381 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:43:19.412 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T09:43:19.436 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:43:19.444 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:43:19.446 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:43:19.455 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T09:43:19.457 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T09:43:19.463 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T09:43:19.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:43:19.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T09:43:19.542 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T09:43:19.550 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T09:43:19.550 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:43:19.567 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T09:43:19.600 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:43:19.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:43:19.710 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:43:19.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:43:19.833 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:43:19.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:43:19.897 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:43:19.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T09:43:19.990 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:43:20.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T09:43:20.175 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:43:20.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T09:43:20.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-09-21T09:43:20.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-09-21T09:43:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:43:20.383 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T09:43:20.793 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T09:43:20.800 INFO:teuthology.task.internal:Starting timer... 2024-09-21T09:43:20.800 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T09:43:20.809 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T09:43:20.818 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T09:43:20.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-09-21T09:43:20.867 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-09-21T09:43:20.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2024-09-21T09:43:20.970 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2024-09-21T09:43:20.977 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T09:43:20.977 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-09-21T09:43:21.009 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-09-21T09:43:21.010 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2024-09-21T09:43:21.034 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2024-09-21T09:43:21.034 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2024-09-21T09:43:21.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T09:43:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:43:21.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T09:43:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:43:21.103 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T09:43:21.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:43:21.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2024-09-21T09:43:21.185 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T09:43:21.195 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T09:43:21.197 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T09:43:21.305 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T09:43:21.315 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T09:43:21.316 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T09:47:02.689 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2024-09-21T09:47:02.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-21T09:47:02.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:47:02.763 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-21T09:47:02.844 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-21T09:47:02.844 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:47:02.845 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T09:47:02.933 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-09-21T09:47:03.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-21T09:47:03.014 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T09:47:03.023 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T09:47:03.023 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:47:03.024 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:47:03.026 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T09:47:03.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T09:47:03.055 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:47:03.069 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:47:03.082 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T09:47:03.094 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-09-21T09:47:03.096 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T09:47:03.105 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-09-21T09:47:03.119 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:47:03.120 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2024-09-21T09:47:03.131 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2024-09-21T09:47:03.133 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:47:03.146 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T09:47:03.159 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T09:47:03.187 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:47:03.191 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:47:03.191 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-21T09:47:03.191 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.191 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.191 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.191 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.215 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-09-21T09:47:03.219 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T09:47:03.219 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-09-21T09:47:03.219 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.219 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.219 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.219 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T09:47:03.220 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T09:47:03.229 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T09:47:03.230 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi119.front.sepia.ceph.com 2024-09-21T09:47:03.230 DEBUG:teuthology.orchestra.run.smithi119:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:47:03.230 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi145.front.sepia.ceph.com 2024-09-21T09:47:03.230 DEBUG:teuthology.orchestra.run.smithi145:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T09:47:03.581 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:Removing: 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:Removing unused dependencies: 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:Remove 2 Packages 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:03.590 INFO:teuthology.orchestra.run.smithi119.stdout:Freed space: 5.5 M 2024-09-21T09:47:03.591 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:47:03.594 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:47:03.594 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:47:03.608 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:Removing: 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:Removing unused dependencies: 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout:Remove 2 Packages 2024-09-21T09:47:03.622 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:03.623 INFO:teuthology.orchestra.run.smithi145.stdout:Freed space: 5.5 M 2024-09-21T09:47:03.623 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:47:03.626 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:47:03.626 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:47:03.663 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:47:03.663 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:47:03.690 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:47:03.691 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:47:03.828 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:47:03.843 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:47:03.909 INFO:teuthology.orchestra.run.smithi119.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:47:03.933 INFO:teuthology.orchestra.run.smithi145.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T09:47:03.969 INFO:teuthology.orchestra.run.smithi119.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:47:03.994 INFO:teuthology.orchestra.run.smithi145.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:47:04.489 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:47:04.489 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:47:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T09:47:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T09:47:04.762 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:47:04.762 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:04.762 INFO:teuthology.orchestra.run.smithi119.stdout:Removed: 2024-09-21T09:47:04.762 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:47:04.762 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:04.762 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:47:04.811 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T09:47:04.811 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:04.811 INFO:teuthology.orchestra.run.smithi145.stdout:Removed: 2024-09-21T09:47:04.811 INFO:teuthology.orchestra.run.smithi145.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:47:04.812 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:04.812 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:47:05.633 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Sep 2024 09:46:17 AM UTC. 2024-09-21T09:47:05.667 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:46:16 AM UTC. 2024-09-21T09:47:05.768 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:47:05.769 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:05.769 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:47:05.769 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:05.769 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout:Install 7 Packages 2024-09-21T09:47:05.770 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:05.783 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 5.2 M 2024-09-21T09:47:05.783 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 19 M 2024-09-21T09:47:05.783 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:47:05.809 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:47:05.810 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:05.810 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:05.811 INFO:teuthology.orchestra.run.smithi119.stdout:Install 7 Packages 2024-09-21T09:47:05.812 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:05.824 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 5.2 M 2024-09-21T09:47:05.824 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 19 M 2024-09-21T09:47:05.824 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:47:06.511 INFO:teuthology.orchestra.run.smithi145.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 120 kB/s | 44 kB 00:00 2024-09-21T09:47:06.602 INFO:teuthology.orchestra.run.smithi145.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 230 kB/s | 106 kB 00:00 2024-09-21T09:47:06.711 INFO:teuthology.orchestra.run.smithi145.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 358 kB/s | 72 kB 00:00 2024-09-21T09:47:06.736 INFO:teuthology.orchestra.run.smithi145.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 629 kB/s | 84 kB 00:00 2024-09-21T09:47:06.741 INFO:teuthology.orchestra.run.smithi119.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 70 kB/s | 44 kB 00:00 2024-09-21T09:47:06.775 INFO:teuthology.orchestra.run.smithi119.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 158 kB/s | 106 kB 00:00 2024-09-21T09:47:06.945 INFO:teuthology.orchestra.run.smithi145.stdout:(5/7): runc-1.1.13-4.el9.x86_64.rpm 15 MB/s | 3.2 MB 00:00 2024-09-21T09:47:06.970 INFO:teuthology.orchestra.run.smithi145.stdout:(6/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 1.1 MB/s | 906 kB 00:00 2024-09-21T09:47:07.017 INFO:teuthology.orchestra.run.smithi119.stdout:(3/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 992 kB/s | 906 kB 00:00 2024-09-21T09:47:07.179 INFO:teuthology.orchestra.run.smithi145.stdout:(7/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 1.8 MB/s | 837 kB 00:00 2024-09-21T09:47:07.179 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:07.179 INFO:teuthology.orchestra.run.smithi145.stdout:Total 3.7 MB/s | 5.2 MB 00:01 2024-09-21T09:47:07.288 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:47:07.302 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:47:07.302 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:47:07.367 INFO:teuthology.orchestra.run.smithi119.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 115 kB/s | 72 kB 00:00 2024-09-21T09:47:07.450 INFO:teuthology.orchestra.run.smithi119.stdout:(5/7): python3-kmod-0.9-32.el9.x86_64.rpm 125 kB/s | 84 kB 00:00 2024-09-21T09:47:07.611 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:47:07.611 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:47:08.009 INFO:teuthology.orchestra.run.smithi119.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 851 kB/s | 837 kB 00:00 2024-09-21T09:47:08.071 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:47:08.156 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T09:47:08.159 INFO:teuthology.orchestra.run.smithi119.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 4.0 MB/s | 3.2 MB 00:00 2024-09-21T09:47:08.159 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:08.160 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.2 MB/s | 5.2 MB 00:02 2024-09-21T09:47:08.191 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T09:47:08.219 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T09:47:08.276 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:47:08.292 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:47:08.292 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:47:08.440 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T09:47:08.467 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:47:08.601 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:47:08.601 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:47:08.909 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:47:08.937 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:47:09.050 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:47:09.088 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:47:09.115 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:47:09.170 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T09:47:09.206 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T09:47:09.233 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T09:47:09.439 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T09:47:09.472 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:47:09.916 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T09:47:09.943 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:47:10.055 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T09:47:10.087 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:47:10.087 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T09:47:10.087 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T09:47:10.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T09:47:10.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T09:47:10.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T09:47:10.088 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T09:47:10.112 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:47:10.474 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:47:10.474 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:10.475 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:47:10.844 DEBUG:teuthology.parallel:result is None 2024-09-21T09:47:11.082 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:47:11.082 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T09:47:11.082 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T09:47:11.082 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T09:47:11.082 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T09:47:11.082 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T09:47:11.083 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:11.425 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:47:11.836 DEBUG:teuthology.parallel:result is None 2024-09-21T09:47:11.836 INFO:teuthology.run_tasks:Running task install... 2024-09-21T09:47:11.849 DEBUG:teuthology.task.install:project ceph 2024-09-21T09:47:11.849 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T09:47:11.849 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:47:11.849 INFO:teuthology.task.install:Using flavor: default 2024-09-21T09:47:11.854 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T09:47:11.855 INFO:teuthology.task.install:extra packages: [] 2024-09-21T09:47:11.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:47:11.855 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:47:11.855 INFO:teuthology.packaging:ref: None 2024-09-21T09:47:11.855 INFO:teuthology.packaging:tag: None 2024-09-21T09:47:11.855 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:47:11.855 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:11.856 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:47:11.857 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208', 'tag': None, 'wait_for_package': False} 2024-09-21T09:47:11.857 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:47:11.857 INFO:teuthology.packaging:ref: None 2024-09-21T09:47:11.857 INFO:teuthology.packaging:tag: None 2024-09-21T09:47:11.857 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:47:11.857 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:11.857 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:47:12.100 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/ 2024-09-21T09:47:12.100 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1649.gfbd1a6ab 2024-09-21T09:47:12.101 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/ 2024-09-21T09:47:12.102 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1649.gfbd1a6ab 2024-09-21T09:47:12.213 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:47:12.214 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:47:12.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:47:12.224 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/fbd1a6ab0448f011c60f6376195d5c078b0cf20b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T09:47:12.225 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:47:12.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T09:47:12.245 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:47:12.245 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:47:12.245 INFO:teuthology.packaging:ref: None 2024-09-21T09:47:12.245 INFO:teuthology.packaging:tag: None 2024-09-21T09:47:12.245 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:47:12.245 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:12.245 DEBUG:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:47:12.250 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T09:47:12.250 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:47:12.250 INFO:teuthology.packaging:ref: None 2024-09-21T09:47:12.250 INFO:teuthology.packaging:tag: None 2024-09-21T09:47:12.250 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:47:12.250 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:47:12.250 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T09:47:12.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:47:12.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T09:47:12.395 DEBUG:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:47:12.406 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T09:47:12.430 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-09-21T09:47:12.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-09-21T09:47:12.462 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2024-09-21T09:47:12.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum clean all 2024-09-21T09:47:12.649 INFO:teuthology.orchestra.run.smithi119.stdout:45 files removed 2024-09-21T09:47:12.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-09-21T09:47:12.710 INFO:teuthology.orchestra.run.smithi145.stdout:45 files removed 2024-09-21T09:47:12.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw 2024-09-21T09:47:13.224 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 295 kB/s | 77 kB 00:00 2024-09-21T09:47:13.294 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 288 kB/s | 77 kB 00:00 2024-09-21T09:47:13.507 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-21T09:47:13.578 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-21T09:47:13.783 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 7.4 kB/s | 1.8 kB 00:00 2024-09-21T09:47:13.845 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 7.7 kB/s | 1.8 kB 00:00 2024-09-21T09:47:14.362 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.3 MB 00:00 2024-09-21T09:47:15.291 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.3 MB 00:01 2024-09-21T09:47:16.909 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 36 MB/s | 20 MB 00:00 2024-09-21T09:47:17.771 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 8.7 MB/s | 20 MB 00:02 2024-09-21T09:47:23.076 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.5 MB 00:02 2024-09-21T09:47:23.405 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-21T09:47:24.610 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 19 kB 00:00 2024-09-21T09:47:25.339 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 37 MB/s | 23 MB 00:00 2024-09-21T09:47:25.889 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-21T09:47:26.793 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T09:47:30.720 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-09-21T09:47:32.182 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-21T09:47:32.912 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 14 M 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.1 M 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:47:32.914 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 5.0 M 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 22 M 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 636 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 457 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 4.9 M 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:47:32.915 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 45 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 120 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 156 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 320 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 295 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 99 k 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout:Install 22 Packages 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-09-21T09:47:32.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:32.917 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 58 M 2024-09-21T09:47:32.917 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:47:33.721 INFO:teuthology.orchestra.run.smithi119.stdout:(1/24): ceph-base-17.2.7-1649.gfbd1a6ab.el9.x86 8.5 MB/s | 5.0 MB 00:00 2024-09-21T09:47:33.771 INFO:teuthology.orchestra.run.smithi119.stdout:(2/24): ceph-selinux-17.2.7-1649.gfbd1a6ab.el9. 506 kB/s | 25 kB 00:00 2024-09-21T09:47:33.847 INFO:teuthology.orchestra.run.smithi119.stdout:(3/24): libcephfs2-17.2.7-1649.gfbd1a6ab.el9.x8 8.3 MB/s | 636 kB 00:00 2024-09-21T09:47:33.922 INFO:teuthology.orchestra.run.smithi119.stdout:(4/24): libradosstriper1-17.2.7-1649.gfbd1a6ab. 6.0 MB/s | 457 kB 00:00 2024-09-21T09:47:34.155 INFO:teuthology.orchestra.run.smithi119.stdout:(5/24): ceph-radosgw-17.2.7-1649.gfbd1a6ab.el9. 14 MB/s | 14 MB 00:01 2024-09-21T09:47:34.214 INFO:teuthology.orchestra.run.smithi119.stdout:(6/24): python3-ceph-argparse-17.2.7-1649.gfbd1 776 kB/s | 45 kB 00:00 2024-09-21T09:47:34.281 INFO:teuthology.orchestra.run.smithi119.stdout:(7/24): librgw2-17.2.7-1649.gfbd1a6ab.el9.x86_6 14 MB/s | 4.9 MB 00:00 2024-09-21T09:47:34.306 INFO:teuthology.orchestra.run.smithi119.stdout:(8/24): python3-ceph-common-17.2.7-1649.gfbd1a6 1.3 MB/s | 120 kB 00:00 2024-09-21T09:47:34.323 INFO:teuthology.orchestra.run.smithi119.stdout:(9/24): python3-cephfs-17.2.7-1649.gfbd1a6ab.el 3.6 MB/s | 156 kB 00:00 2024-09-21T09:47:34.357 INFO:teuthology.orchestra.run.smithi119.stdout:(10/24): python3-rados-17.2.7-1649.gfbd1a6ab.el 6.2 MB/s | 320 kB 00:00 2024-09-21T09:47:34.424 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:47:34.426 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 14 M 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.1 M 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 5.0 M 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 22 M 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T09:47:34.427 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 636 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 457 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 4.9 M 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 45 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 120 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 156 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T09:47:34.428 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 320 k 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 295 k 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 99 k 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout:Install 22 Packages 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 2 Packages 2024-09-21T09:47:34.429 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:34.430 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 58 M 2024-09-21T09:47:34.430 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:47:34.566 INFO:teuthology.orchestra.run.smithi119.stdout:(11/24): ceph-common-17.2.7-1649.gfbd1a6ab.el9. 16 MB/s | 22 MB 00:01 2024-09-21T09:47:34.590 INFO:teuthology.orchestra.run.smithi119.stdout:(12/24): python3-rgw-17.2.7-1649.gfbd1a6ab.el9. 425 kB/s | 99 kB 00:00 2024-09-21T09:47:34.616 INFO:teuthology.orchestra.run.smithi119.stdout:(13/24): python3-rbd-17.2.7-1649.gfbd1a6ab.el9. 1.0 MB/s | 295 kB 00:00 2024-09-21T09:47:34.700 INFO:teuthology.orchestra.run.smithi119.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 416 kB/s | 45 kB 00:00 2024-09-21T09:47:34.750 INFO:teuthology.orchestra.run.smithi119.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-09-21T09:47:34.800 INFO:teuthology.orchestra.run.smithi119.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 848 kB/s | 42 kB 00:00 2024-09-21T09:47:34.834 INFO:teuthology.orchestra.run.smithi119.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-21T09:47:34.876 INFO:teuthology.orchestra.run.smithi119.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-21T09:47:34.909 INFO:teuthology.orchestra.run.smithi119.stdout:(19/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.2 MB/s | 662 kB 00:00 2024-09-21T09:47:34.934 INFO:teuthology.orchestra.run.smithi119.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 493 kB/s | 49 kB 00:00 2024-09-21T09:47:34.968 INFO:teuthology.orchestra.run.smithi119.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 745 kB/s | 67 kB 00:00 2024-09-21T09:47:35.018 INFO:teuthology.orchestra.run.smithi119.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T09:47:35.200 INFO:teuthology.orchestra.run.smithi145.stdout:(1/24): ceph-base-17.2.7-1649.gfbd1a6ab.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-21T09:47:35.283 INFO:teuthology.orchestra.run.smithi145.stdout:(2/24): ceph-selinux-17.2.7-1649.gfbd1a6ab.el9. 303 kB/s | 25 kB 00:00 2024-09-21T09:47:35.293 INFO:teuthology.orchestra.run.smithi119.stdout:(23/24): librados2-17.2.7-1649.gfbd1a6ab.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-09-21T09:47:35.384 INFO:teuthology.orchestra.run.smithi145.stdout:(3/24): libcephfs2-17.2.7-1649.gfbd1a6ab.el9.x8 6.2 MB/s | 636 kB 00:00 2024-09-21T09:47:35.484 INFO:teuthology.orchestra.run.smithi145.stdout:(4/24): libradosstriper1-17.2.7-1649.gfbd1a6ab. 4.5 MB/s | 457 kB 00:00 2024-09-21T09:47:35.843 INFO:teuthology.orchestra.run.smithi145.stdout:(5/24): ceph-radosgw-17.2.7-1649.gfbd1a6ab.el9. 13 MB/s | 14 MB 00:01 2024-09-21T09:47:35.926 INFO:teuthology.orchestra.run.smithi145.stdout:(6/24): librgw2-17.2.7-1649.gfbd1a6ab.el9.x86_6 11 MB/s | 4.9 MB 00:00 2024-09-21T09:47:35.951 INFO:teuthology.orchestra.run.smithi145.stdout:(7/24): python3-ceph-argparse-17.2.7-1649.gfbd1 417 kB/s | 45 kB 00:00 2024-09-21T09:47:36.168 INFO:teuthology.orchestra.run.smithi145.stdout:(8/24): ceph-common-17.2.7-1649.gfbd1a6ab.el9.x 15 MB/s | 22 MB 00:01 2024-09-21T09:47:36.193 INFO:teuthology.orchestra.run.smithi145.stdout:(9/24): python3-ceph-common-17.2.7-1649.gfbd1a6 450 kB/s | 120 kB 00:00 2024-09-21T09:47:36.194 INFO:teuthology.orchestra.run.smithi119.stdout:(24/24): librbd1-17.2.7-1649.gfbd1a6ab.el9.x86_ 2.4 MB/s | 2.9 MB 00:01 2024-09-21T09:47:36.199 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:36.199 INFO:teuthology.orchestra.run.smithi119.stdout:Total 18 MB/s | 58 MB 00:03 2024-09-21T09:47:36.219 INFO:teuthology.orchestra.run.smithi145.stdout:(10/24): python3-cephfs-17.2.7-1649.gfbd1a6ab.e 584 kB/s | 156 kB 00:00 2024-09-21T09:47:36.244 INFO:teuthology.orchestra.run.smithi145.stdout:(11/24): python3-rados-17.2.7-1649.gfbd1a6ab.el 4.2 MB/s | 320 kB 00:00 2024-09-21T09:47:36.265 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:47:36.269 INFO:teuthology.orchestra.run.smithi145.stdout:(12/24): python3-rbd-17.2.7-1649.gfbd1a6ab.el9. 3.8 MB/s | 295 kB 00:00 2024-09-21T09:47:36.297 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:47:36.297 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:47:36.303 INFO:teuthology.orchestra.run.smithi145.stdout:(13/24): python3-rgw-17.2.7-1649.gfbd1a6ab.el9. 1.2 MB/s | 99 kB 00:00 2024-09-21T09:47:36.395 INFO:teuthology.orchestra.run.smithi145.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 362 kB/s | 45 kB 00:00 2024-09-21T09:47:36.437 INFO:teuthology.orchestra.run.smithi145.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 832 kB/s | 160 kB 00:00 2024-09-21T09:47:36.487 INFO:teuthology.orchestra.run.smithi145.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 847 kB/s | 42 kB 00:00 2024-09-21T09:47:36.529 INFO:teuthology.orchestra.run.smithi145.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-09-21T09:47:36.563 INFO:teuthology.orchestra.run.smithi145.stdout:(18/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-21T09:47:36.588 INFO:teuthology.orchestra.run.smithi145.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-09-21T09:47:36.613 INFO:teuthology.orchestra.run.smithi145.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-09-21T09:47:36.638 INFO:teuthology.orchestra.run.smithi145.stdout:(21/24): liboath-2.6.7-2.el9.x86_64.rpm 446 kB/s | 49 kB 00:00 2024-09-21T09:47:36.689 INFO:teuthology.orchestra.run.smithi145.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-21T09:47:36.790 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:47:36.790 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:47:36.822 INFO:teuthology.orchestra.run.smithi145.stdout:(23/24): librados2-17.2.7-1649.gfbd1a6ab.el9.x8 15 MB/s | 3.1 MB 00:00 2024-09-21T09:47:36.873 INFO:teuthology.orchestra.run.smithi145.stdout:(24/24): librbd1-17.2.7-1649.gfbd1a6ab.el9.x86_ 13 MB/s | 2.9 MB 00:00 2024-09-21T09:47:36.877 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:47:36.878 INFO:teuthology.orchestra.run.smithi145.stdout:Total 24 MB/s | 58 MB 00:02 2024-09-21T09:47:36.943 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:47:36.976 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:47:36.976 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:47:37.450 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:47:37.450 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:47:38.007 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:47:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-21T09:47:38.462 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:47:38.553 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:47:38.631 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:47:38.663 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:47:38.664 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:47:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:47:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-21T09:47:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-21T09:47:39.050 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-21T09:47:39.109 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:47:39.201 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:47:39.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-21T09:47:39.278 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:47:39.292 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:47:39.301 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:47:39.363 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:47:39.390 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:47:39.391 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-21T09:47:39.439 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 10/26 2024-09-21T09:47:39.441 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-21T09:47:39.473 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-21T09:47:39.481 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:47:39.541 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:47:39.569 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:47:39.619 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:47:39.659 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-21T09:47:39.694 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-21T09:47:39.728 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-21T09:47:39.919 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-21T09:47:39.963 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:47:40.009 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 8/26 2024-09-21T09:47:40.036 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-21T09:47:40.046 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-21T09:47:40.073 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:47:40.092 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 10/26 2024-09-21T09:47:40.134 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:47:40.163 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:47:40.196 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:47:40.224 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:47:40.237 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 19/26 2024-09-21T09:47:40.267 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 20/26 2024-09-21T09:47:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 13/26 2024-09-21T09:47:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-21T09:47:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-21T09:47:40.391 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-21T09:47:40.684 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-21T09:47:40.728 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:47:40.817 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 18/26 2024-09-21T09:47:40.892 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 19/26 2024-09-21T09:47:40.922 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 20/26 2024-09-21T09:47:42.768 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:47:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:47:43.311 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:47:43.351 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:47:43.376 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:47:43.418 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:47:43.444 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:47:43.444 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:47:43.444 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:47:43.445 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:43.485 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:47:43.945 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 21/26 2024-09-21T09:47:43.988 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:47:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 22/26 2024-09-21T09:47:44.090 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T09:47:44.090 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T09:47:44.090 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:44.123 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:47:53.379 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:47:53.379 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-09-21T09:47:53.380 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:53.753 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:47:53.753 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2024-09-21T09:47:53.753 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2024-09-21T09:47:53.753 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2024-09-21T09:47:53.753 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2024-09-21T09:47:53.753 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2024-09-21T09:47:53.754 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /root 2024-09-21T09:47:53.754 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2024-09-21T09:47:53.754 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:54.723 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:47:54.796 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:47:54.797 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:47:54.797 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:47:54.797 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:47:54.797 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:47:54.797 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:54.798 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:47:54.871 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:47:54.871 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:47:55.090 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:47:55.168 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 24/26 2024-09-21T09:47:55.168 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:47:55.168 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T09:47:55.168 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:47:55.168 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T09:47:55.168 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:55.170 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:47:55.251 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-21T09:47:55.251 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 6/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 7/26 2024-09-21T09:47:56.075 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 8/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 9/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 10/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 13/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-21T09:47:56.076 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-21T09:47:56.077 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:47:56.077 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-21T09:47:56.077 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 25/26 2024-09-21T09:47:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:47:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/26 2024-09-21T09:47:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/26 2024-09-21T09:47:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/26 2024-09-21T09:47:56.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_6 6/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 7/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9. 8/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x8 9/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 10/26 2024-09-21T09:47:56.496 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 11/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 12/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 13/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-21T09:47:56.497 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-21T09:47:56.498 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-21T09:47:56.498 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-21T09:47:56.498 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-21T09:47:56.498 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 23/26 2024-09-21T09:47:56.498 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-21T09:47:56.498 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 25/26 2024-09-21T09:47:57.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:47:57.053 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:47:57.054 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:57.055 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:47:57.236 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.237 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:57.238 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:47:57.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-09-21T09:47:57.611 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-test 2024-09-21T09:47:57.970 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:47:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:47:58.099 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:58.099 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 48 M 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 163 k 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-09-21T09:47:58.100 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:47:58.101 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 48 M 2024-09-21T09:47:58.101 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 193 M 2024-09-21T09:47:58.101 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:47:58.199 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:26 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:47:58.330 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 48 M 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:47:58.331 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 163 k 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 48 M 2024-09-21T09:47:58.332 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 193 M 2024-09-21T09:47:58.333 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:47:58.574 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-21T09:47:58.591 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-09-21T09:47:58.599 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): libcephsqlite-17.2.7-1649.gfbd1a6ab.el9. 745 kB/s | 163 kB 00:00 2024-09-21T09:47:58.616 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): libcephsqlite-17.2.7-1649.gfbd1a6ab.el9. 650 kB/s | 163 kB 00:00 2024-09-21T09:47:58.624 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T09:47:58.641 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T09:48:00.626 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): ceph-test-17.2.7-1649.gfbd1a6ab.el9.x86_ 21 MB/s | 48 MB 00:02 2024-09-21T09:48:00.626 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:00.627 INFO:teuthology.orchestra.run.smithi145.stdout:Total 21 MB/s | 48 MB 00:02 2024-09-21T09:48:00.655 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:00.658 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): ceph-test-17.2.7-1649.gfbd1a6ab.el9.x86_ 21 MB/s | 48 MB 00:02 2024-09-21T09:48:00.660 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:00.661 INFO:teuthology.orchestra.run.smithi119.stdout:Total 19 MB/s | 48 MB 00:02 2024-09-21T09:48:00.674 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:00.674 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:00.688 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:00.707 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:00.707 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:01.041 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:01.041 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:01.075 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:01.075 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:01.559 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:01.608 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:01.642 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:48:01.680 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T09:48:01.682 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:48:01.709 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:48:01.719 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T09:48:01.746 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:48:06.136 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:48:06.164 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/4 2024-09-21T09:48:06.207 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:48:06.245 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:48:08.473 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:48:08.473 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/4 2024-09-21T09:48:08.474 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/4 2024-09-21T09:48:08.474 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:48:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/4 2024-09-21T09:48:08.557 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/4 2024-09-21T09:48:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/4 2024-09-21T09:48:08.558 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:08.782 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:08.872 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:08.873 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:09.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-09-21T09:48:09.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph 2024-09-21T09:48:09.673 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:39 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:09.811 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:37 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:09.813 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:09.815 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:09.815 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:48:09.815 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:09.815 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:48:09.815 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 6.5 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.1 M 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 1.4 M 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 240 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.9 M 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 15 M 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:48:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:48:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout:Install 34 Packages 2024-09-21T09:48:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:09.819 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 27 M 2024-09-21T09:48:09.819 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 96 M 2024-09-21T09:48:09.819 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:48:09.944 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:09.946 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:09.946 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:09.946 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:09.946 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 6.5 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.1 M 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 1.4 M 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 240 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 3.9 M 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 15 M 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T09:48:09.947 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T09:48:09.948 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout:Install 34 Packages 2024-09-21T09:48:09.949 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 27 M 2024-09-21T09:48:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 96 M 2024-09-21T09:48:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:10.157 INFO:teuthology.orchestra.run.smithi145.stdout:(1/34): ceph-17.2.7-1649.gfbd1a6ab.el9.x86_64.r 48 kB/s | 6.5 kB 00:00 2024-09-21T09:48:10.178 INFO:teuthology.orchestra.run.smithi119.stdout:(1/34): ceph-17.2.7-1649.gfbd1a6ab.el9.x86_64.r 55 kB/s | 6.5 kB 00:00 2024-09-21T09:48:10.341 INFO:teuthology.orchestra.run.smithi145.stdout:(2/34): ceph-mds-17.2.7-1649.gfbd1a6ab.el9.x86_ 6.5 MB/s | 2.1 MB 00:00 2024-09-21T09:48:10.375 INFO:teuthology.orchestra.run.smithi145.stdout:(3/34): ceph-mgr-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.0 MB/s | 1.4 MB 00:00 2024-09-21T09:48:10.475 INFO:teuthology.orchestra.run.smithi145.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1649.gfbd1 2.3 MB/s | 240 kB 00:00 2024-09-21T09:48:10.486 INFO:teuthology.orchestra.run.smithi119.stdout:(2/34): ceph-mds-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.9 MB/s | 2.1 MB 00:00 2024-09-21T09:48:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:(3/34): ceph-mgr-17.2.7-1649.gfbd1a6ab.el9.x86_ 3.1 MB/s | 1.4 MB 00:00 2024-09-21T09:48:10.634 INFO:teuthology.orchestra.run.smithi145.stdout:(5/34): ceph-mon-17.2.7-1649.gfbd1a6ab.el9.x86_ 8.2 MB/s | 3.9 MB 00:00 2024-09-21T09:48:10.637 INFO:teuthology.orchestra.run.smithi119.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1649.gfbd1 2.0 MB/s | 240 kB 00:00 2024-09-21T09:48:10.659 INFO:teuthology.orchestra.run.smithi145.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-09-21T09:48:10.701 INFO:teuthology.orchestra.run.smithi145.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T09:48:10.742 INFO:teuthology.orchestra.run.smithi145.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-09-21T09:48:10.785 INFO:teuthology.orchestra.run.smithi145.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-21T09:48:10.804 INFO:teuthology.orchestra.run.smithi119.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-21T09:48:10.913 INFO:teuthology.orchestra.run.smithi119.stdout:(6/34): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-21T09:48:10.943 INFO:teuthology.orchestra.run.smithi145.stdout:(10/34): ceph-osd-17.2.7-1649.gfbd1a6ab.el9.x86 25 MB/s | 15 MB 00:00 2024-09-21T09:48:10.957 INFO:teuthology.orchestra.run.smithi119.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 106 kB 00:00 2024-09-21T09:48:10.968 INFO:teuthology.orchestra.run.smithi145.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-09-21T09:48:11.002 INFO:teuthology.orchestra.run.smithi145.stdout:(12/34): python3-cryptography-36.0.1-4.el9.x86_ 3.3 MB/s | 1.2 MB 00:00 2024-09-21T09:48:11.015 INFO:teuthology.orchestra.run.smithi119.stdout:(8/34): ceph-mon-17.2.7-1649.gfbd1a6ab.el9.x86_ 4.7 MB/s | 3.9 MB 00:00 2024-09-21T09:48:11.041 INFO:teuthology.orchestra.run.smithi119.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.6 MB/s | 135 kB 00:00 2024-09-21T09:48:11.077 INFO:teuthology.orchestra.run.smithi145.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-21T09:48:11.091 INFO:teuthology.orchestra.run.smithi119.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-21T09:48:11.102 INFO:teuthology.orchestra.run.smithi145.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-21T09:48:11.128 INFO:teuthology.orchestra.run.smithi145.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 615 kB/s | 77 kB 00:00 2024-09-21T09:48:11.153 INFO:teuthology.orchestra.run.smithi145.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-21T09:48:11.158 INFO:teuthology.orchestra.run.smithi119.stdout:(11/34): python3-requests-2.25.1-8.el9.noarch.r 879 kB/s | 125 kB 00:00 2024-09-21T09:48:11.178 INFO:teuthology.orchestra.run.smithi145.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-21T09:48:11.262 INFO:teuthology.orchestra.run.smithi145.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2024-09-21T09:48:11.287 INFO:teuthology.orchestra.run.smithi145.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 458 kB/s | 11 kB 00:00 2024-09-21T09:48:11.312 INFO:teuthology.orchestra.run.smithi145.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-21T09:48:11.338 INFO:teuthology.orchestra.run.smithi145.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-21T09:48:11.372 INFO:teuthology.orchestra.run.smithi145.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 393 kB/s | 23 kB 00:00 2024-09-21T09:48:11.383 INFO:teuthology.orchestra.run.smithi119.stdout:(12/34): ceph-osd-17.2.7-1649.gfbd1a6ab.el9.x86 17 MB/s | 15 MB 00:00 2024-09-21T09:48:11.397 INFO:teuthology.orchestra.run.smithi145.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2024-09-21T09:48:11.422 INFO:teuthology.orchestra.run.smithi145.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-21T09:48:11.447 INFO:teuthology.orchestra.run.smithi145.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-09-21T09:48:11.473 INFO:teuthology.orchestra.run.smithi145.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T09:48:11.498 INFO:teuthology.orchestra.run.smithi145.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.1 MB/s | 358 kB 00:00 2024-09-21T09:48:11.523 INFO:teuthology.orchestra.run.smithi145.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-21T09:48:11.548 INFO:teuthology.orchestra.run.smithi145.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-21T09:48:11.574 INFO:teuthology.orchestra.run.smithi145.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-21T09:48:11.592 INFO:teuthology.orchestra.run.smithi119.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 80 kB/s | 35 kB 00:00 2024-09-21T09:48:11.599 INFO:teuthology.orchestra.run.smithi145.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T09:48:11.624 INFO:teuthology.orchestra.run.smithi145.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-21T09:48:11.658 INFO:teuthology.orchestra.run.smithi145.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-21T09:48:11.683 INFO:teuthology.orchestra.run.smithi145.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T09:48:11.687 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:11.687 INFO:teuthology.orchestra.run.smithi145.stdout:Total 16 MB/s | 27 MB 00:01 2024-09-21T09:48:11.725 INFO:teuthology.orchestra.run.smithi119.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 271 kB/s | 172 kB 00:00 2024-09-21T09:48:11.751 INFO:teuthology.orchestra.run.smithi119.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 210 kB/s | 77 kB 00:00 2024-09-21T09:48:11.793 INFO:teuthology.orchestra.run.smithi119.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 254 kB/s | 51 kB 00:00 2024-09-21T09:48:11.815 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:11.826 INFO:teuthology.orchestra.run.smithi119.stdout:(17/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-09-21T09:48:11.852 INFO:teuthology.orchestra.run.smithi119.stdout:(18/34): python3-toml-0.10.2-6.el9.noarch.rpm 331 kB/s | 42 kB 00:00 2024-09-21T09:48:11.852 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:11.852 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:11.885 INFO:teuthology.orchestra.run.smithi119.stdout:(19/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.9 MB/s | 172 kB 00:00 2024-09-21T09:48:11.910 INFO:teuthology.orchestra.run.smithi119.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 760 kB/s | 18 kB 00:00 2024-09-21T09:48:11.935 INFO:teuthology.orchestra.run.smithi119.stdout:(21/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-09-21T09:48:11.961 INFO:teuthology.orchestra.run.smithi119.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-21T09:48:11.986 INFO:teuthology.orchestra.run.smithi119.stdout:(23/34): python3-cherrypy-18.6.1-2.el9.noarch.r 2.2 MB/s | 358 kB 00:00 2024-09-21T09:48:12.011 INFO:teuthology.orchestra.run.smithi119.stdout:(24/34): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-09-21T09:48:12.037 INFO:teuthology.orchestra.run.smithi119.stdout:(25/34): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-09-21T09:48:12.062 INFO:teuthology.orchestra.run.smithi119.stdout:(26/34): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-09-21T09:48:12.079 INFO:teuthology.orchestra.run.smithi119.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-09-21T09:48:12.104 INFO:teuthology.orchestra.run.smithi119.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-09-21T09:48:12.129 INFO:teuthology.orchestra.run.smithi119.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T09:48:12.155 INFO:teuthology.orchestra.run.smithi119.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-21T09:48:12.180 INFO:teuthology.orchestra.run.smithi119.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-09-21T09:48:12.205 INFO:teuthology.orchestra.run.smithi119.stdout:(32/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T09:48:12.230 INFO:teuthology.orchestra.run.smithi119.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-21T09:48:12.253 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:12.253 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:12.256 INFO:teuthology.orchestra.run.smithi119.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-21T09:48:12.260 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:12.260 INFO:teuthology.orchestra.run.smithi119.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-21T09:48:12.394 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:12.430 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:12.430 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:12.822 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:12.822 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:13.517 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:13.586 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-21T09:48:13.626 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-21T09:48:13.660 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-21T09:48:13.760 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-21T09:48:13.838 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-21T09:48:13.879 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-21T09:48:13.914 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-21T09:48:13.943 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-21T09:48:13.978 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-21T09:48:14.011 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-21T09:48:14.052 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-21T09:48:14.079 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:14.087 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-21T09:48:14.129 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-21T09:48:14.142 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-21T09:48:14.174 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-21T09:48:14.178 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-21T09:48:14.207 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-21T09:48:14.210 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:48:14.282 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-21T09:48:14.299 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-21T09:48:14.378 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-21T09:48:14.422 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-21T09:48:14.426 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-21T09:48:14.461 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-21T09:48:14.491 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-21T09:48:14.501 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-21T09:48:14.526 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-21T09:48:14.559 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-21T09:48:14.562 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-21T09:48:14.599 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-21T09:48:14.609 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-21T09:48:14.634 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-21T09:48:14.659 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-21T09:48:14.694 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-21T09:48:14.714 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-21T09:48:14.742 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-21T09:48:14.782 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:48:14.857 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-21T09:48:14.902 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-21T09:48:14.955 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-21T09:48:15.004 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-21T09:48:15.025 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-21T09:48:15.099 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-21T09:48:15.114 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-21T09:48:15.152 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-21T09:48:15.154 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-21T09:48:15.191 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-21T09:48:15.248 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-21T09:48:15.269 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-21T09:48:15.303 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-21T09:48:15.418 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 29/34 2024-09-21T09:48:15.450 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:48:15.476 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:48:15.476 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:15.476 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:48:15.476 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:48:15.476 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:48:15.476 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:15.490 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-21T09:48:15.544 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-21T09:48:15.622 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-21T09:48:15.711 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-21T09:48:15.752 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-21T09:48:15.867 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-21T09:48:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 29/34 2024-09-21T09:48:16.054 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:48:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 30/34 2024-09-21T09:48:16.079 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:16.079 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T09:48:16.079 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:48:16.079 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T09:48:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:16.826 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:48:16.853 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:48:16.853 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:16.853 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:48:16.853 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:48:16.853 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:48:16.853 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:17.235 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:48:17.264 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:48:17.264 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:17.264 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:48:17.264 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:48:17.264 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:48:17.264 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:17.396 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:48:17.424 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 31/34 2024-09-21T09:48:17.425 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:17.425 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T09:48:17.425 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:48:17.425 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T09:48:17.425 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:17.477 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:48:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:48:17.502 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:17.502 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:48:17.502 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:48:17.502 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:48:17.502 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:17.566 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:48:17.789 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:48:17.812 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 32/34 2024-09-21T09:48:17.813 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:17.813 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T09:48:17.813 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:48:17.813 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T09:48:17.813 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:18.014 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:48:18.038 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 33/34 2024-09-21T09:48:18.038 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:18.038 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T09:48:18.038 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:48:18.039 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T09:48:18.039 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:18.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:48:19.306 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:48:19.306 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/34 2024-09-21T09:48:19.306 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/34 2024-09-21T09:48:19.306 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 6/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-21T09:48:19.307 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-21T09:48:19.308 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-21T09:48:19.309 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-21T09:48:19.309 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-21T09:48:19.310 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-21T09:48:19.310 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-21T09:48:19.310 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-21T09:48:19.310 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 34/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 3/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 4/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 5/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9. 6/34 2024-09-21T09:48:19.728 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-21T09:48:19.729 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-21T09:48:19.730 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-21T09:48:19.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-21T09:48:19.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-21T09:48:19.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-21T09:48:19.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-21T09:48:19.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-21T09:48:19.731 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.121 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:48:20.122 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:20.123 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:20.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-base 2024-09-21T09:48:20.600 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-21T09:48:20.600 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:20.600 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:48:20.600 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.600 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.600 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T09:48:20.601 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T09:48:20.602 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:20.603 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:20.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-09-21T09:48:21.106 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:21.196 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:48:21.238 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:21.239 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:48:21.239 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:21.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install cephadm 2024-09-21T09:48:21.557 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:21.646 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:48:21.687 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:21.688 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:48:21.688 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:21.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-09-21T09:48:21.889 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:22.021 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 83 k 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:22.022 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:22.023 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:22.023 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:48:22.023 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:22.023 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 83 k 2024-09-21T09:48:22.023 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 375 k 2024-09-21T09:48:22.023 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:22.235 INFO:teuthology.orchestra.run.smithi145.stdout:cephadm-17.2.7-1649.gfbd1a6ab.el9.noarch.rpm 391 kB/s | 83 kB 00:00 2024-09-21T09:48:22.235 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:22.236 INFO:teuthology.orchestra.run.smithi145.stdout:Total 388 kB/s | 83 kB 00:00 2024-09-21T09:48:22.236 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:22.241 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:22.241 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:22.243 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:22.243 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:22.317 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:22.317 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:22.449 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 83 k 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:22.450 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:48:22.451 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:22.451 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 83 k 2024-09-21T09:48:22.451 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 375 k 2024-09-21T09:48:22.451 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:48:22.654 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-17.2.7-1649.gfbd1a6ab.el9.noarch.rpm 407 kB/s | 83 kB 00:00 2024-09-21T09:48:22.654 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:22.654 INFO:teuthology.orchestra.run.smithi119.stdout:Total 404 kB/s | 83 kB 00:00 2024-09-21T09:48:22.655 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:22.660 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:22.660 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:22.661 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:22.662 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:22.729 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:22.750 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:22.837 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.146 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.234 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.240 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.495 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.495 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:23.495 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:23.495 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:23.495 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:23.496 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:23.625 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.714 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:48:23.886 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/1 2024-09-21T09:48:23.887 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:23.887 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:48:23.887 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:23.887 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:23.887 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:24.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T09:48:24.308 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:24.444 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 137 k 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:48:24.445 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:24.446 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 137 k 2024-09-21T09:48:24.446 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 379 k 2024-09-21T09:48:24.446 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:24.657 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-immutable-object-cache-17.2.7-1649.gfbd1a6 651 kB/s | 137 kB 00:00 2024-09-21T09:48:24.657 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:24.658 INFO:teuthology.orchestra.run.smithi145.stdout:Total 647 kB/s | 137 kB 00:00 2024-09-21T09:48:24.658 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:24.658 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:24.665 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:24.665 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:24.723 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:24.723 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:24.793 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 137 k 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:48:24.794 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:24.795 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 137 k 2024-09-21T09:48:24.795 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 379 k 2024-09-21T09:48:24.795 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:48:24.864 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:24.916 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:48:24.941 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:48:24.941 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:24.941 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:48:24.941 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:25.002 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-17.2.7-1649.gfbd1a6 664 kB/s | 137 kB 00:00 2024-09-21T09:48:25.002 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:25.002 INFO:teuthology.orchestra.run.smithi119.stdout:Total 660 kB/s | 137 kB 00:00 2024-09-21T09:48:25.003 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:25.010 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:25.010 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:25.067 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:25.068 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:25.212 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:25.269 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:48:25.293 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:48:25.293 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:48:25.293 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T09:48:25.293 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:48:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:25.887 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:25.887 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:25.887 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:26.111 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr 2024-09-21T09:48:26.226 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab. 1/1 2024-09-21T09:48:26.227 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:26.227 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:48:26.227 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:48:26.227 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:26.228 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:26.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-09-21T09:48:26.707 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:26.797 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:48:26.839 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:26.840 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:48:26.841 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:26.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:48:27.048 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:27.142 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:48:27.185 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:27.187 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:48:27.187 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:27.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T09:48:27.498 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:27.636 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:27.637 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:27.637 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:27.637 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:27.637 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:27.637 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 3.4 M 2024-09-21T09:48:27.637 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 24 k 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 15 k 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout:Installing weak dependencies: 2024-09-21T09:48:27.638 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-21T09:48:27.639 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:27.639 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:27.639 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:27.639 INFO:teuthology.orchestra.run.smithi145.stdout:Install 7 Packages 2024-09-21T09:48:27.639 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:27.640 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.7 M 2024-09-21T09:48:27.640 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 80 M 2024-09-21T09:48:27.640 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:27.828 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:27.843 INFO:teuthology.orchestra.run.smithi145.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1649.gfbd 171 kB/s | 24 kB 00:00 2024-09-21T09:48:27.860 INFO:teuthology.orchestra.run.smithi145.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1649.gfbd1 91 kB/s | 15 kB 00:00 2024-09-21T09:48:27.894 INFO:teuthology.orchestra.run.smithi145.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 180 kB/s | 9.0 kB 00:00 2024-09-21T09:48:27.910 INFO:teuthology.orchestra.run.smithi145.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-21T09:48:27.927 INFO:teuthology.orchestra.run.smithi145.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 609 kB/s | 41 kB 00:00 2024-09-21T09:48:27.953 INFO:teuthology.orchestra.run.smithi145.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-21T09:48:27.959 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:27.960 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 3.4 M 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 24 k 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 15 k 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:27.961 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:48:27.962 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:27.962 INFO:teuthology.orchestra.run.smithi119.stdout:Install 7 Packages 2024-09-21T09:48:27.962 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:27.962 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.7 M 2024-09-21T09:48:27.962 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 80 M 2024-09-21T09:48:27.962 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:48:28.086 INFO:teuthology.orchestra.run.smithi145.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1649.gfbd1a6ab 8.7 MB/s | 3.4 MB 00:00 2024-09-21T09:48:28.090 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:28.091 INFO:teuthology.orchestra.run.smithi145.stdout:Total 8.1 MB/s | 3.7 MB 00:00 2024-09-21T09:48:28.091 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:28.099 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:28.099 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:28.183 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:28.211 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:28.480 INFO:teuthology.orchestra.run.smithi119.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1649.gfbd1 99 kB/s | 15 kB 00:00 2024-09-21T09:48:28.488 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:28.505 INFO:teuthology.orchestra.run.smithi119.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1649.gfbd 142 kB/s | 24 kB 00:00 2024-09-21T09:48:28.539 INFO:teuthology.orchestra.run.smithi119.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 155 kB/s | 9.0 kB 00:00 2024-09-21T09:48:28.554 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-21T09:48:28.564 INFO:teuthology.orchestra.run.smithi119.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-21T09:48:28.581 INFO:teuthology.orchestra.run.smithi119.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 541 kB/s | 41 kB 00:00 2024-09-21T09:48:28.588 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-21T09:48:28.606 INFO:teuthology.orchestra.run.smithi119.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-09-21T09:48:28.643 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-21T09:48:28.675 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-21T09:48:28.704 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 5/7 2024-09-21T09:48:28.740 INFO:teuthology.orchestra.run.smithi119.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1649.gfbd1a6ab 8.2 MB/s | 3.4 MB 00:00 2024-09-21T09:48:28.743 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:28.743 INFO:teuthology.orchestra.run.smithi119.stdout:Total 4.7 MB/s | 3.7 MB 00:00 2024-09-21T09:48:28.744 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:28.752 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:28.752 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:28.833 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:28.833 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:29.140 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:29.200 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-21T09:48:29.234 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-21T09:48:29.280 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-21T09:48:29.321 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-21T09:48:29.349 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 5/7 2024-09-21T09:48:29.722 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 6/7 2024-09-21T09:48:29.758 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:48:30.344 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 6/7 2024-09-21T09:48:30.386 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:48:30.885 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:48:30.885 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 1/7 2024-09-21T09:48:30.886 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 2/7 2024-09-21T09:48:30.886 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 3/7 2024-09-21T09:48:30.886 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-21T09:48:30.886 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-21T09:48:30.886 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-21T09:48:31.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-21T09:48:31.176 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:31.176 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:31.176 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:31.176 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:31.176 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:31.177 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-21T09:48:31.177 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-21T09:48:31.177 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:48:31.177 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:48:31.177 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:31.177 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:31.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:48:31.452 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 7/7 2024-09-21T09:48:31.452 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9. 1/7 2024-09-21T09:48:31.452 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarc 2/7 2024-09-21T09:48:31.452 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.n 3/7 2024-09-21T09:48:31.452 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-21T09:48:31.452 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-21T09:48:31.453 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-21T09:48:31.727 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:31.728 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:31.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T09:48:31.999 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:32.131 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================= 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================= 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 7.4 M 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:48:32.132 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================= 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout:Install 12 Packages 2024-09-21T09:48:32.133 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:32.134 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 38 M 2024-09-21T09:48:32.134 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 212 M 2024-09-21T09:48:32.134 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:32.534 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:02 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:32.666 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:32.667 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-21T09:48:32.667 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:48:32.667 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-21T09:48:32.667 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:48:32.667 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 7.4 M 2024-09-21T09:48:32.667 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T09:48:32.668 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:32.669 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:48:32.669 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-21T09:48:32.669 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-09-21T09:48:32.669 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:32.669 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 38 M 2024-09-21T09:48:32.669 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 212 M 2024-09-21T09:48:32.670 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:48:32.745 INFO:teuthology.orchestra.run.smithi145.stdout:(1/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-21T09:48:32.811 INFO:teuthology.orchestra.run.smithi145.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 494 kB/s | 33 kB 00:00 2024-09-21T09:48:33.114 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T09:48:33.153 INFO:teuthology.orchestra.run.smithi145.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 216 kB/s | 191 kB 00:00 2024-09-21T09:48:33.195 INFO:teuthology.orchestra.run.smithi145.stdout:(4/12): libgfortran-11.5.0-2.el9.x86_64.rpm 865 kB/s | 801 kB 00:00 2024-09-21T09:48:33.253 INFO:teuthology.orchestra.run.smithi145.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 182 kB/s | 18 kB 00:00 2024-09-21T09:48:33.264 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 801 kB 00:00 2024-09-21T09:48:33.279 INFO:teuthology.orchestra.run.smithi145.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 462 kB/s | 38 kB 00:00 2024-09-21T09:48:33.354 INFO:teuthology.orchestra.run.smithi145.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-09-21T09:48:33.456 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-21T09:48:33.473 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 91 kB/s | 33 kB 00:00 2024-09-21T09:48:33.579 INFO:teuthology.orchestra.run.smithi145.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.9 MB/s | 3.0 MB 00:00 2024-09-21T09:48:33.680 INFO:teuthology.orchestra.run.smithi145.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.5 MB/s | 250 kB 00:00 2024-09-21T09:48:33.682 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 185 kB/s | 38 kB 00:00 2024-09-21T09:48:33.733 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 66 kB/s | 18 kB 00:00 2024-09-21T09:48:34.208 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 515 kB/s | 245 kB 00:00 2024-09-21T09:48:34.480 INFO:teuthology.orchestra.run.smithi145.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.0 MB/s | 4.9 MB 00:01 2024-09-21T09:48:34.639 INFO:teuthology.orchestra.run.smithi145.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.0 MB/s | 5.1 MB 00:01 2024-09-21T09:48:34.725 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-09-21T09:48:34.841 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-21T09:48:35.542 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.8 MB/s | 5.1 MB 00:01 2024-09-21T09:48:35.958 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.2 MB/s | 4.9 MB 00:02 2024-09-21T09:48:37.872 INFO:teuthology.orchestra.run.smithi145.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.9 MB/s | 16 MB 00:04 2024-09-21T09:48:37.873 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:37.873 INFO:teuthology.orchestra.run.smithi145.stdout:Total 6.7 MB/s | 38 MB 00:05 2024-09-21T09:48:38.233 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:38.260 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:38.260 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:38.778 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:38.779 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:39.536 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:39.611 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:48:39.653 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:48:39.751 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:48:39.796 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:48:39.979 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:48:40.011 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:48:40.092 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:48:40.743 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:48:40.907 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:48:42.161 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:48:43.194 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:48:43.218 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:48:45.804 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 1/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:48:45.805 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:48:46.210 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:48:46.210 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:46.210 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:46.210 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:46.210 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:48:46.210 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:46.211 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:46.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:48:46.824 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.4 MB/s | 16 MB 00:11 2024-09-21T09:48:46.825 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:46.825 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.7 MB/s | 38 MB 00:14 2024-09-21T09:48:47.178 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:47.204 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:47.204 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:47.252 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:15 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:47.393 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:47.394 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:47.394 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:47.394 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:47.394 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 47 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:48:47.395 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout:Install 13 Packages 2024-09-21T09:48:47.396 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:47.397 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 2.2 M 2024-09-21T09:48:47.397 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 26 M 2024-09-21T09:48:47.397 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:47.711 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:47.711 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:47.975 INFO:teuthology.orchestra.run.smithi145.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 278 kB/s | 26 kB 00:00 2024-09-21T09:48:48.001 INFO:teuthology.orchestra.run.smithi145.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 163 kB/s | 19 kB 00:00 2024-09-21T09:48:48.059 INFO:teuthology.orchestra.run.smithi145.stdout:(3/13): ceph-mgr-rook-17.2.7-1649.gfbd1a6ab.el9 269 kB/s | 47 kB 00:00 2024-09-21T09:48:48.084 INFO:teuthology.orchestra.run.smithi145.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-21T09:48:48.101 INFO:teuthology.orchestra.run.smithi145.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 159 kB 00:00 2024-09-21T09:48:48.135 INFO:teuthology.orchestra.run.smithi145.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-21T09:48:48.168 INFO:teuthology.orchestra.run.smithi145.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T09:48:48.194 INFO:teuthology.orchestra.run.smithi145.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-21T09:48:48.219 INFO:teuthology.orchestra.run.smithi145.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-21T09:48:48.244 INFO:teuthology.orchestra.run.smithi145.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-21T09:48:48.270 INFO:teuthology.orchestra.run.smithi145.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.3 MB/s | 279 kB 00:00 2024-09-21T09:48:48.295 INFO:teuthology.orchestra.run.smithi145.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-21T09:48:48.329 INFO:teuthology.orchestra.run.smithi145.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-09-21T09:48:48.331 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:48.331 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-09-21T09:48:48.394 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:48.410 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:48.410 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:48.466 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:48.499 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:48.499 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:48.560 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T09:48:48.605 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:48:48.703 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:48:48.740 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T09:48:48.923 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T09:48:48.955 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T09:48:49.013 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:49.036 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T09:48:49.185 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:48:49.239 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:48:49.294 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:48:49.322 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:48:49.364 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:48:49.475 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:48:49.585 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:48:49.632 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:48:49.689 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T09:48:49.808 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T09:48:50.348 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:48:50.396 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:48:50.430 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:48:50.475 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:48:50.499 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:48:51.024 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:48:51.289 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:48:51.290 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:48:51.290 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:48:51.624 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:51.625 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:51.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:48:52.050 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T09:48:52.079 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:48:52.512 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:20 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:52.643 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 124 k 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:48:52.644 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:52.645 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 7.0 M 2024-09-21T09:48:52.646 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 33 M 2024-09-21T09:48:52.646 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:52.905 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1649.gfbd1a6ab.e 594 kB/s | 124 kB 00:00 2024-09-21T09:48:52.972 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-09-21T09:48:53.006 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.1 MB/s | 58 kB 00:00 2024-09-21T09:48:53.031 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.0 MB/s | 74 kB 00:00 2024-09-21T09:48:53.489 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 313 kB/s | 248 kB 00:00 2024-09-21T09:48:53.890 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.0 MB/s | 6.0 MB 00:01 2024-09-21T09:48:53.969 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:53.969 INFO:teuthology.orchestra.run.smithi145.stdout:Total 5.6 MB/s | 7.0 MB 00:01 2024-09-21T09:48:53.969 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:53.976 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:53.977 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:54.076 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:54.077 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:54.383 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:54.547 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:48:54.591 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:48:54.681 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 12/12 2024-09-21T09:48:54.681 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a 1/12 2024-09-21T09:48:54.681 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T09:48:54.681 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T09:48:54.681 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T09:48:54.681 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T09:48:54.682 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T09:48:54.682 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T09:48:54.682 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T09:48:54.682 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T09:48:54.682 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T09:48:54.682 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T09:48:55.081 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:55.082 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:48:55.117 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:48:55.202 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:48:55.266 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:48:55.295 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:48:55.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-09-21T09:48:56.090 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:48:56.129 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:48:56.130 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/6 2024-09-21T09:48:56.130 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:48:56.130 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:48:56.130 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:48:56.130 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:48:56.223 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:48:56.224 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 47 k 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T09:48:56.225 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-09-21T09:48:56.226 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:48:56.227 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.2 M 2024-09-21T09:48:56.227 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-09-21T09:48:56.227 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:48:56.422 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:48:56.423 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:48:56.423 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:56.423 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:56.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-fuse 2024-09-21T09:48:56.756 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 134 kB/s | 26 kB 00:00 2024-09-21T09:48:56.781 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 88 kB/s | 19 kB 00:00 2024-09-21T09:48:56.806 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): ceph-mgr-rook-17.2.7-1649.gfbd1a6ab.el9 196 kB/s | 47 kB 00:00 2024-09-21T09:48:56.898 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-21T09:48:56.923 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-21T09:48:56.957 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-21T09:48:56.990 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-21T09:48:57.016 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-21T09:48:57.041 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 172 kB/s | 14 kB 00:00 2024-09-21T09:48:57.075 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-21T09:48:57.100 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-21T09:48:57.125 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-21T09:48:57.167 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-21T09:48:57.169 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:57.170 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-09-21T09:48:57.233 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:48:57.248 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:48:57.248 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:48:57.278 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:48:57.334 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:48:57.335 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:48:57.410 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 813 k 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:57.411 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:48:57.412 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:48:57.412 INFO:teuthology.orchestra.run.smithi145.stdout:Install 2 Packages 2024-09-21T09:48:57.412 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:57.412 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 893 k 2024-09-21T09:48:57.412 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 2.6 M 2024-09-21T09:48:57.412 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:48:57.724 INFO:teuthology.orchestra.run.smithi145.stdout:(1/2): ceph-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-21T09:48:57.839 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:48:57.925 INFO:teuthology.orchestra.run.smithi145.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 172 kB/s | 80 kB 00:00 2024-09-21T09:48:57.926 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:48:57.926 INFO:teuthology.orchestra.run.smithi145.stdout:Total 1.7 MB/s | 893 kB 00:00 2024-09-21T09:48:57.937 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:48:57.959 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:48:57.959 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:48:58.005 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T09:48:58.039 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:48:58.040 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:48:58.059 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T09:48:58.114 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T09:48:58.144 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T09:48:58.188 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T09:48:58.224 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:48:58.295 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T09:48:58.371 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:48:58.404 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T09:48:58.448 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:48:58.462 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T09:48:59.115 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:48:59.116 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/2 2024-09-21T09:48:59.175 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T09:48:59.227 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T09:48:59.260 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T09:48:59.325 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T09:48:59.346 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:48:59.425 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:48:59.425 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:59.425 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:48:59.425 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:48:59.425 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:48:59.425 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:48:59.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados-devel 2024-09-21T09:49:00.098 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 13/13 2024-09-21T09:49:00.098 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T09:49:00.099 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T09:49:00.315 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:00.451 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 126 k 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:49:00.452 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:49:00.453 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 126 k 2024-09-21T09:49:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 448 k 2024-09-21T09:49:00.453 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T09:49:00.464 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:00.465 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:00.655 INFO:teuthology.orchestra.run.smithi145.stdout:librados-devel-17.2.7-1649.gfbd1a6ab.el9.x86_64 625 kB/s | 126 kB 00:00 2024-09-21T09:49:00.655 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:00.656 INFO:teuthology.orchestra.run.smithi145.stdout:Total 621 kB/s | 126 kB 00:00 2024-09-21T09:49:00.656 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:49:00.661 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:49:00.661 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:49:00.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T09:49:00.724 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:49:00.725 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:49:00.873 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:49:00.976 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:01.294 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:01.411 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:01.426 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:01.427 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:01.427 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:49:01.427 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:01.427 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:01.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1649.gfbd1a6ab.el9 ceph-noarch 124 k 2024-09-21T09:49:01.427 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-09-21T09:49:01.428 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:01.429 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 7.0 M 2024-09-21T09:49:01.429 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 33 M 2024-09-21T09:49:01.429 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:01.679 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:01.680 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:01.680 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:49:01.680 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:01.680 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:01.680 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:01.752 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1649.gfbd1a6ab.e 641 kB/s | 124 kB 00:00 2024-09-21T09:49:01.802 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T09:49:01.828 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-21T09:49:01.844 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 4.4 MB/s | 74 kB 00:00 2024-09-21T09:49:01.870 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 798 kB/s | 248 kB 00:00 2024-09-21T09:49:01.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs2 2024-09-21T09:49:02.095 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-21T09:49:02.097 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:02.097 INFO:teuthology.orchestra.run.smithi119.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-21T09:49:02.167 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:02.178 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:02.178 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:02.278 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:02.278 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:02.514 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:02.578 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:02.605 INFO:teuthology.orchestra.run.smithi145.stdout:Package libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:02.650 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:02.651 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:02.651 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:02.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs-devel 2024-09-21T09:49:02.727 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T09:49:02.788 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T09:49:03.302 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:03.316 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T09:49:03.401 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T09:49:03.434 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:03.435 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:03.435 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-09-21T09:49:03.435 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 25 k 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 110 k 2024-09-21T09:49:03.436 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:49:03.457 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T09:49:03.484 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:49:03.591 INFO:teuthology.orchestra.run.smithi145.stdout:libcephfs-devel-17.2.7-1649.gfbd1a6ab.el9.x86_6 161 kB/s | 25 kB 00:00 2024-09-21T09:49:03.592 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:03.592 INFO:teuthology.orchestra.run.smithi145.stdout:Total 159 kB/s | 25 kB 00:00 2024-09-21T09:49:03.592 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:49:03.596 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:49:03.596 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:49:03.597 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:49:03.598 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:49:03.685 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:49:03.788 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:04.163 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:04.306 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 6/6 2024-09-21T09:49:04.306 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 1/6 2024-09-21T09:49:04.306 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T09:49:04.306 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T09:49:04.306 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T09:49:04.306 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T09:49:04.410 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:04.410 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:04.410 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:49:04.411 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:04.411 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:04.411 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:17.2.7-1649.gfbd1a6ab.el9.noarch 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T09:49:04.580 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T09:49:04.581 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T09:49:04.581 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:04.581 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:04.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados2 2024-09-21T09:49:04.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-09-21T09:49:05.201 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:05.292 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:05.335 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:05.336 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:05.336 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:05.404 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librbd1 2024-09-21T09:49:05.412 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 813 k 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-21T09:49:05.545 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T09:49:05.546 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:05.546 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:05.546 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:05.546 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-09-21T09:49:05.546 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:05.547 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 893 k 2024-09-21T09:49:05.547 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 2.6 M 2024-09-21T09:49:05.547 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:05.930 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 646 kB/s | 80 kB 00:00 2024-09-21T09:49:05.992 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:06.084 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:06.105 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-21T09:49:06.105 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:06.106 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-09-21T09:49:06.118 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:06.127 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:06.128 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:06.128 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:06.140 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:06.140 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:06.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rados 2024-09-21T09:49:06.221 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:06.221 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:06.403 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:06.565 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T09:49:06.630 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:49:06.789 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:06.877 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:06.920 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:06.921 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:06.921 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:06.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rgw 2024-09-21T09:49:07.272 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2/2 2024-09-21T09:49:07.273 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/2 2024-09-21T09:49:07.539 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T09:49:07.539 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:07.539 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:07.539 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T09:49:07.539 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:07.539 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:07.570 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:07.661 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:07.704 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:07.705 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:07.705 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:07.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-cephfs 2024-09-21T09:49:07.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-09-21T09:49:08.350 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:08.389 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:08.439 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:08.481 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:08.482 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:08.482 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:08.522 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 126 k 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:08.523 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:08.524 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:08.524 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:49:08.524 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:08.524 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 126 k 2024-09-21T09:49:08.524 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 448 k 2024-09-21T09:49:08.524 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:08.553 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rbd 2024-09-21T09:49:08.725 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-17.2.7-1649.gfbd1a6ab.el9.x86_64 626 kB/s | 126 kB 00:00 2024-09-21T09:49:08.726 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:08.726 INFO:teuthology.orchestra.run.smithi119.stdout:Total 621 kB/s | 126 kB 00:00 2024-09-21T09:49:08.727 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:08.731 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:08.732 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:08.795 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:08.795 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:08.943 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:09.047 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:09.141 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:09.231 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:09.274 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:09.275 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-09-21T09:49:09.275 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:09.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-fuse 2024-09-21T09:49:09.457 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:09.714 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:09.714 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:09.950 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:09.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-09-21T09:49:10.083 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:10.084 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:10.084 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 82 k 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 82 k 2024-09-21T09:49:10.085 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 222 k 2024-09-21T09:49:10.086 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:49:10.278 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 425 kB/s | 82 kB 00:00 2024-09-21T09:49:10.278 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:10.279 INFO:teuthology.orchestra.run.smithi145.stdout:Total 422 kB/s | 82 kB 00:00 2024-09-21T09:49:10.279 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:49:10.285 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:49:10.285 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:49:10.349 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:49:10.349 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:49:10.490 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:49:10.555 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:10.600 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:10.645 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:10.690 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:10.691 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:10.691 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:10.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-09-21T09:49:10.989 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:11.251 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:11.251 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:11.252 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:49:11.252 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:11.252 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:11.252 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:11.322 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:11.452 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 25 k 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:11.453 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 25 k 2024-09-21T09:49:11.454 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 110 k 2024-09-21T09:49:11.454 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:11.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-mirror 2024-09-21T09:49:11.612 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-17.2.7-1649.gfbd1a6ab.el9.x86_6 157 kB/s | 25 kB 00:00 2024-09-21T09:49:11.613 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:11.613 INFO:teuthology.orchestra.run.smithi119.stdout:Total 156 kB/s | 25 kB 00:00 2024-09-21T09:49:11.613 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:11.616 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:11.617 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:11.618 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:11.618 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:11.698 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:11.817 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:12.135 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:12.156 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:12.271 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:49:12.272 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:12.273 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 2.9 M 2024-09-21T09:49:12.273 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 12 M 2024-09-21T09:49:12.273 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:49:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:12.377 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:12.377 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:12.377 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:12.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-09-21T09:49:12.606 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-mirror-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 8.8 MB/s | 2.9 MB 00:00 2024-09-21T09:49:12.606 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:12.607 INFO:teuthology.orchestra.run.smithi145.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-21T09:49:12.607 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:49:12.616 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:49:12.616 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:49:12.697 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:49:12.697 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:49:13.111 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:49:13.147 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:13.181 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:13.207 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:13.207 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:49:13.207 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:49:13.207 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:49:13.207 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:49:13.207 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:13.239 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:13.281 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:13.283 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:13.283 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:13.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-09-21T09:49:13.923 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:14.011 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:14.055 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:14.056 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:14.056 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:14.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-09-21T09:49:14.352 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:14.352 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:14.352 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:49:14.352 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:14.352 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:14.352 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:14.625 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-nbd 2024-09-21T09:49:14.698 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:14.787 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:14.830 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:14.831 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:14.831 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:14.884 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-09-21T09:49:15.221 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 09:47:32 AM UTC. 2024-09-21T09:49:15.355 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-09-21T09:49:15.356 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:15.356 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-09-21T09:49:15.356 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:15.356 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-09-21T09:49:15.356 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 165 k 2024-09-21T09:49:15.356 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 165 k 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 477 k 2024-09-21T09:49:15.357 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-09-21T09:49:15.462 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:45 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:15.551 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:15.575 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-nbd-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 754 kB/s | 165 kB 00:00 2024-09-21T09:49:15.576 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:15.576 INFO:teuthology.orchestra.run.smithi145.stdout:Total 749 kB/s | 165 kB 00:00 2024-09-21T09:49:15.576 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-09-21T09:49:15.584 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-09-21T09:49:15.584 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-09-21T09:49:15.594 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:15.595 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:15.595 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:15.647 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-09-21T09:49:15.647 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-09-21T09:49:15.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-09-21T09:49:15.799 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-09-21T09:49:15.905 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:16.228 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:16.316 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:16.358 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:16.359 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:16.359 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:16.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-09-21T09:49:16.604 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:16.604 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:16.604 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-09-21T09:49:16.604 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:16.604 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:49:16.604 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-09-21T09:49:16.836 DEBUG:teuthology.parallel:result is None 2024-09-21T09:49:17.002 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:17.091 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 is already installed. 2024-09-21T09:49:17.133 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:17.134 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-21T09:49:17.134 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:17.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-09-21T09:49:17.781 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:47 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:17.914 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 82 k 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:17.915 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:17.916 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:17.916 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:49:17.916 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:17.916 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 82 k 2024-09-21T09:49:17.916 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 222 k 2024-09-21T09:49:17.916 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:18.107 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 428 kB/s | 82 kB 00:00 2024-09-21T09:49:18.107 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:18.108 INFO:teuthology.orchestra.run.smithi119.stdout:Total 425 kB/s | 82 kB 00:00 2024-09-21T09:49:18.108 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:18.114 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:18.114 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:18.176 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:18.176 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:18.322 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:18.446 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:18.821 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:19.081 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:19.081 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:19.081 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:19.081 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:19.081 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:19.082 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:19.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-09-21T09:49:19.901 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:20.034 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 2.9 M 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:20.035 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:49:20.036 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:20.036 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.9 M 2024-09-21T09:49:20.036 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 12 M 2024-09-21T09:49:20.036 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:20.376 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 8.6 MB/s | 2.9 MB 00:00 2024-09-21T09:49:20.377 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:20.377 INFO:teuthology.orchestra.run.smithi119.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-09-21T09:49:20.377 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:20.386 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:20.386 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:20.466 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:20.466 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:20.882 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:20.944 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:20.968 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:20.968 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T09:49:20.969 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T09:49:20.969 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:49:20.969 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T09:49:20.969 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:22.069 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:22.069 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:22.070 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:22.070 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:22.070 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:22.070 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:22.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-09-21T09:49:22.916 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:52 ago on Sat 21 Sep 2024 09:47:30 AM UTC. 2024-09-21T09:49:23.050 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:17.2.7-1649.gfbd1a6ab.el9 ceph 165 k 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-21T09:49:23.051 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:23.052 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 165 k 2024-09-21T09:49:23.052 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 477 k 2024-09-21T09:49:23.052 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-21T09:49:23.255 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-17.2.7-1649.gfbd1a6ab.el9.x86_64.rpm 814 kB/s | 165 kB 00:00 2024-09-21T09:49:23.255 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:23.256 INFO:teuthology.orchestra.run.smithi119.stdout:Total 808 kB/s | 165 kB 00:00 2024-09-21T09:49:23.256 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-21T09:49:23.263 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-21T09:49:23.263 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-21T09:49:23.326 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-21T09:49:23.327 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-21T09:49:23.481 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-21T09:49:23.577 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:23.961 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 1/1 2024-09-21T09:49:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:24.221 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-21T09:49:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:17.2.7-1649.gfbd1a6ab.el9.x86_64 2024-09-21T09:49:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:49:24.221 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-21T09:49:24.511 DEBUG:teuthology.parallel:result is None 2024-09-21T09:49:24.511 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:49:24.511 INFO:teuthology.packaging:ref: None 2024-09-21T09:49:24.511 INFO:teuthology.packaging:tag: None 2024-09-21T09:49:24.511 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:49:24.511 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:49:24.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:49:24.722 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:49:24.743 INFO:teuthology.orchestra.run.smithi119.stdout:17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:49:24.743 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:49:24.743 INFO:teuthology.task.install:The correct ceph version 17.2.7-1649.gfbd1a6ab is installed. 2024-09-21T09:49:24.745 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T09:49:24.745 INFO:teuthology.packaging:ref: None 2024-09-21T09:49:24.745 INFO:teuthology.packaging:tag: None 2024-09-21T09:49:24.745 INFO:teuthology.packaging:branch: quincy 2024-09-21T09:49:24.745 INFO:teuthology.packaging:sha1: 62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:49:24.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-09-21T09:49:24.935 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T09:49:24.961 INFO:teuthology.orchestra.run.smithi145.stdout:17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:49:24.961 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1649.gfbd1a6ab.el9 2024-09-21T09:49:24.961 INFO:teuthology.task.install:The correct ceph version 17.2.7-1649.gfbd1a6ab is installed. 2024-09-21T09:49:24.963 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T09:49:24.963 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:49:24.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:49:24.989 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:49:24.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T09:49:25.030 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T09:49:25.030 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:49:25.030 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:49:25.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:49:25.119 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:49:25.120 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T09:49:25.147 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T09:49:25.214 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T09:49:25.215 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:49:25.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:49:25.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:49:25.304 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:49:25.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T09:49:25.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T09:49:25.398 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T09:49:25.398 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:49:25.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:49:25.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:49:25.487 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:49:25.487 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T09:49:25.513 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T09:49:25.579 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T09:49:25.667 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}, 'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'} 2024-09-21T09:49:25.667 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:49:25.668 INFO:tasks.cephadm:Cluster fsid is c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:25.668 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T09:49:25.668 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.119', 'mon.c': '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]', 'mon.b': '172.21.15.145'} 2024-09-21T09:49:25.668 INFO:tasks.cephadm:First mon is mon.a on smithi119 2024-09-21T09:49:25.668 INFO:tasks.cephadm:First mgr is y 2024-09-21T09:49:25.668 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T09:49:25.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-09-21T09:49:25.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname $(hostname -s) 2024-09-21T09:49:25.719 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-09-21T09:49:25.720 DEBUG:teuthology.orchestra.run.smithi119:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:49:25.857 INFO:teuthology.orchestra.run.smithi119.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 21 09:49 /home/ubuntu/cephtest/cephadm 2024-09-21T09:49:25.858 DEBUG:teuthology.orchestra.run.smithi145:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T09:49:26.075 INFO:teuthology.orchestra.run.smithi145.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 21 09:49 /home/ubuntu/cephtest/cephadm 2024-09-21T09:49:26.075 DEBUG:teuthology.orchestra.run.smithi119:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:49:26.091 DEBUG:teuthology.orchestra.run.smithi145:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T09:49:26.117 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-09-21T09:49:26.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-21T09:49:26.134 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-09-21T09:49:27.749 INFO:teuthology.orchestra.run.smithi145.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:49:27.796 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:49:54.394 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T09:49:54.394 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)", 2024-09-21T09:49:54.395 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9", 2024-09-21T09:49:54.395 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-09-21T09:49:54.395 INFO:teuthology.orchestra.run.smithi119.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2" 2024-09-21T09:49:54.395 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T09:49:54.395 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T09:49:58.847 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T09:49:58.847 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)", 2024-09-21T09:49:58.847 INFO:teuthology.orchestra.run.smithi145.stdout: "image_id": "fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9", 2024-09-21T09:49:58.847 INFO:teuthology.orchestra.run.smithi145.stdout: "repo_digests": [ 2024-09-21T09:49:58.847 INFO:teuthology.orchestra.run.smithi145.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2" 2024-09-21T09:49:58.848 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T09:49:58.848 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T09:49:58.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-09-21T09:49:58.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph 2024-09-21T09:49:58.932 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-09-21T09:49:58.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /etc/ceph 2024-09-21T09:49:58.999 INFO:tasks.cephadm:Writing seed config... 2024-09-21T09:49:58.999 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T09:49:59.000 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T09:49:59.001 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:49:59.001 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T09:49:59.026 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c96f302a-77fe-11ef-baf6-efdc52797490 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T09:49:59.027 DEBUG:teuthology.orchestra.run.smithi119:mon.a> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T09:49:59.069 DEBUG:teuthology.orchestra.run.smithi119:mgr.y> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T09:49:59.111 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T09:49:59.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid c96f302a-77fe-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.119 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:49:59.312 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-21T09:49:59.312 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'c96f302a-77fe-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.119', '--skip-admin-label'] 2024-09-21T09:49:59.336 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:49:59.336 INFO:teuthology.orchestra.run.smithi119.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T09:49:59.336 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying podman|docker is present... 2024-09-21T09:49:59.359 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:49:59.359 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying lvm2 is present... 2024-09-21T09:49:59.359 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying time synchronization is in place... 2024-09-21T09:49:59.367 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:49:59.367 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:49:59.373 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:49:59.373 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-21T09:49:59.380 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout enabled 2024-09-21T09:49:59.387 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout active 2024-09-21T09:49:59.387 INFO:teuthology.orchestra.run.smithi119.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:49:59.387 INFO:teuthology.orchestra.run.smithi119.stdout:Repeating the final host check... 2024-09-21T09:49:59.409 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 5.2.2 2024-09-21T09:49:59.409 INFO:teuthology.orchestra.run.smithi119.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T09:49:59.409 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl is present 2024-09-21T09:49:59.409 INFO:teuthology.orchestra.run.smithi119.stdout:lvcreate is present 2024-09-21T09:49:59.416 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T09:49:59.416 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T09:49:59.423 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T09:49:59.423 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-21T09:49:59.430 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout enabled 2024-09-21T09:49:59.436 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout active 2024-09-21T09:49:59.437 INFO:teuthology.orchestra.run.smithi119.stdout:Unit chronyd.service is enabled and running 2024-09-21T09:49:59.437 INFO:teuthology.orchestra.run.smithi119.stdout:Host looks OK 2024-09-21T09:49:59.437 INFO:teuthology.orchestra.run.smithi119.stdout:Cluster fsid: c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:49:59.437 INFO:teuthology.orchestra.run.smithi119.stdout:Acquiring lock 139632194638752 on /run/cephadm/c96f302a-77fe-11ef-baf6-efdc52797490.lock 2024-09-21T09:49:59.437 INFO:teuthology.orchestra.run.smithi119.stdout:Lock 139632194638752 acquired on /run/cephadm/c96f302a-77fe-11ef-baf6-efdc52797490.lock 2024-09-21T09:49:59.437 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying IP 172.21.15.119 port 3300 ... 2024-09-21T09:49:59.438 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying IP 172.21.15.119 port 6789 ... 2024-09-21T09:49:59.438 INFO:teuthology.orchestra.run.smithi119.stdout:Base mon IP(s) is [172.21.15.119:3300, 172.21.15.119:6789], mon addrv is [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-09-21T09:49:59.441 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.119 metric 100 2024-09-21T09:49:59.441 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.119 metric 100 2024-09-21T09:49:59.444 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T09:49:59.444 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T09:49:59.444 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T09:49:59.447 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T09:49:59.447 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T09:49:59.447 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:49:59.447 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T09:49:59.447 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link noprefixroute 2024-09-21T09:49:59.447 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T09:49:59.448 INFO:teuthology.orchestra.run.smithi119.stdout:Mon IP `172.21.15.119` is in CIDR network `172.21.0.0/20` 2024-09-21T09:49:59.448 INFO:teuthology.orchestra.run.smithi119.stdout:Mon IP `172.21.15.119` is in CIDR network `172.21.0.0/20` 2024-09-21T09:49:59.448 INFO:teuthology.orchestra.run.smithi119.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T09:49:59.448 INFO:teuthology.orchestra.run.smithi119.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T09:49:59.448 INFO:teuthology.orchestra.run.smithi119.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying blob sha256:3ba00a516579693832719224d909a7cd5333511394d4f35f95e899e50f0bfaeb 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying config sha256:fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 2024-09-21T09:49:59.936 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T09:50:00.958 INFO:teuthology.orchestra.run.smithi119.stdout:ceph: stdout ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable) 2024-09-21T09:50:00.958 INFO:teuthology.orchestra.run.smithi119.stdout:Ceph version: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable) 2024-09-21T09:50:00.958 INFO:teuthology.orchestra.run.smithi119.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T09:50:01.989 INFO:teuthology.orchestra.run.smithi119.stdout:stat: stdout 167 167 2024-09-21T09:50:01.989 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial keys... 2024-09-21T09:50:03.035 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQDKlu5mvSQDExAATVisbR6TirQZjSGRTFe1hQ== 2024-09-21T09:50:03.958 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQDLlu5moNNTFRAAZ4Y3OovRiv5mIaxU/9ga1Q== 2024-09-21T09:50:04.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQDMlu5mhle0ERAAjAPKoz4aCvxwwiA9c2TuJA== 2024-09-21T09:50:04.839 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial monmap... 2024-09-21T09:50:05.903 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:50:05.903 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-21T09:50:05.903 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:05.903 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:50:05.903 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool for a [v2:172.21.15.119:3300,v1:172.21.15.119:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T09:50:05.903 INFO:teuthology.orchestra.run.smithi119.stdout:setting min_mon_release = octopus 2024-09-21T09:50:05.904 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: set fsid to c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:05.904 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T09:50:05.904 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:50:05.904 INFO:teuthology.orchestra.run.smithi119.stdout:Creating mon... 2024-09-21T09:50:06.885 INFO:teuthology.orchestra.run.smithi119.stdout:create mon.a on 2024-09-21T09:50:07.096 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T09:50:07.234 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T09:50:07.376 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c96f302a-77fe-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-c96f302a-77fe-11ef-baf6-efdc52797490.target. 2024-09-21T09:50:07.377 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c96f302a-77fe-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-c96f302a-77fe-11ef-baf6-efdc52797490.target. 2024-09-21T09:50:07.541 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a 2024-09-21T09:50:07.541 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T09:50:07.700 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c96f302a-77fe-11ef-baf6-efdc52797490.target.wants/ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-c96f302a-77fe-11ef-baf6-efdc52797490@.service. 2024-09-21T09:50:08.187 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:08 smithi119 ceph-mon[28900]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:50:08.187 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:08 smithi119 bash[28862]: 7f6d46db37129ff461bb0518738ae24c7f994c30f6ec87a430b0dece6b0176b0 2024-09-21T09:50:08.200 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:50:08.200 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-21T09:50:08.207 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:50:08.207 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-21T09:50:08.208 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-21T09:50:08.208 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:50:08.208 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mon to start... 2024-09-21T09:50:08.208 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mon... 2024-09-21T09:50:08.438 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:08 smithi119 systemd[1]: Started Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:50:09.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:09 smithi119 ceph-mon[28900]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:50:09.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:09 smithi119 ceph-mon[28900]: monmap e1: 1 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:50:09.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:09 smithi119 ceph-mon[28900]: fsmap 2024-09-21T09:50:09.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:09 smithi119 ceph-mon[28900]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:50:09.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:09 smithi119 ceph-mon[28900]: mgrmap e1: no daemons active 2024-09-21T09:50:09.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:09 smithi119 ceph-mon[28900]: from='client.? 172.21.15.119:0/2012796464' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout id: c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout services: 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.588077s) 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout data: 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T09:50:09.306 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T09:50:09.307 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T09:50:09.307 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:09.307 INFO:teuthology.orchestra.run.smithi119.stdout:mon is available 2024-09-21T09:50:09.307 INFO:teuthology.orchestra.run.smithi119.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T09:50:10.327 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout fsid = c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:10.329 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:50:10.330 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:50:10.330 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:50:10.330 INFO:teuthology.orchestra.run.smithi119.stdout:Generating new minimal ceph.conf... 2024-09-21T09:50:10.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:10 smithi119 ceph-mon[28900]: from='client.? 172.21.15.119:0/1636845729' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:50:10.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:10 smithi119 ceph-mon[28900]: from='client.? 172.21.15.119:0/1636845729' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T09:50:11.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:11 smithi119 ceph-mon[28900]: from='client.? 172.21.15.119:0/3412036105' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:50:11.375 INFO:teuthology.orchestra.run.smithi119.stdout:Restarting the monitor... 2024-09-21T09:50:11.637 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:11 smithi119 systemd[1]: Stopping Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:50:12.093 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[28877]: 2024-09-21T09:50:11.634+0000 7f898162a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T09:50:12.093 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[28877]: 2024-09-21T09:50:11.634+0000 7f898162a640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T09:50:12.093 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:11 smithi119 podman[29216]: 2024-09-21 09:50:11.82936961 +0000 UTC m=+0.298589108 container died 7f6d46db37129ff461bb0518738ae24c7f994c30f6ec87a430b0dece6b0176b0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a) 2024-09-21T09:50:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 podman[29216]: 2024-09-21 09:50:12.190959464 +0000 UTC m=+0.660178974 container cleanup 7f6d46db37129ff461bb0518738ae24c7f994c30f6ec87a430b0dece6b0176b0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731) 2024-09-21T09:50:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 bash[29216]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T09:50:12.811 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 podman[29235]: 2024-09-21 09:50:12.442024179 +0000 UTC m=+0.616648940 container remove 7f6d46db37129ff461bb0518738ae24c7f994c30f6ec87a430b0dece6b0176b0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T09:50:12.811 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T09:50:12.811 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 systemd[1]: Stopped Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:50:12.811 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 systemd[1]: Starting Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:50:13.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 podman[29329]: 2024-09-21 09:50:12.809547467 +0000 UTC m=+0.097511943 container create 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a) 2024-09-21T09:50:13.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:12 smithi119 podman[29329]: 2024-09-21 09:50:12.727697077 +0000 UTC m=+0.015661521 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 podman[29329]: 2024-09-21 09:50:13.02025768 +0000 UTC m=+0.308222128 container init 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 podman[29329]: 2024-09-21 09:50:13.024955935 +0000 UTC m=+0.312920369 container start 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 7 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: pidfile_write: ignore empty --pid-file 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: load: jerasure load: lrc 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: DB SUMMARY 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: DB Session ID: ZU9BR6FW7K36TDAJZRGW 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:50:13.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83718 ; 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.env: 0x55b2aa8a3440 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.fs: Posix File System 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.info_log: 0x55b2ac180ec0 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.statistics: (nil) 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.use_fsync: 0 2024-09-21T09:50:13.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.db_log_dir: 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:50:13.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.write_buffer_manager: 0x55b2ac23e780 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:50:13.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:50:13.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:50:13.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.unordered_write: 0 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.row_cache: None 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.wal_filter: None 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:50:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:50:13.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_open_files: -1 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:50:13.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Compression algorithms supported: 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kZSTD supported: 0 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:50:13.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.merge_operator: 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_filter: None 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b2ac127a38) 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_type: 0 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_index_type: 0 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_shortening: 1 2024-09-21T09:50:13.121 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: hash_index_allow_collision: 1 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: checksum: 1 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: no_block_cache: 0 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache: 0x55b2ac15d1f0 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_options: 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: capacity : 536870912 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: num_shard_bits : 4 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: strict_capacity_limit : 0 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_compressed: (nil) 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: persistent_cache: (nil) 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size: 4096 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size_deviation: 10 2024-09-21T09:50:13.122 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_restart_interval: 16 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_block_restart_interval: 1 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: metadata_block_size: 4096 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: partition_filters: 0 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: use_delta_encoding: 1 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: whole_key_filtering: 1 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: verify_compression: 0 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: format_version: 4 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_index_compression: 1 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_align: 0 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression: NoCompression 2024-09-21T09:50:13.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.num_levels: 7 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:50:13.124 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:50:13.125 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:50:13.126 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:50:13.127 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.ttl: 2592000 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:50:13.128 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912213048013, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912213048683, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80291, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78533, "index_size": 220, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10130, "raw_average_key_size": 47, "raw_value_size": 72746, "raw_average_value_size": 341, "num_data_blocks": 10, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726912213, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "34a153c9-dbdd-4107-a831-a2e275cf1a06", "db_session_id": "ZU9BR6FW7K36TDAJZRGW"}} 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912213049005, "job": 1, "event": "recovery_finished"} 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55b2ac190700 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: DB pointer 0x55b2ac160000 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:50:13.129 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** DB Stats ** 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.130 INFO:journalctl@ceph.mon.a.smithi119.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative compaction: 0.00 GB write, 26.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:50:13.131 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval compaction: 0.00 GB write, 26.46 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: L0 2/0 80.19 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: Sum 2/0 80.19 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:50:13.132 INFO:journalctl@ceph.mon.a.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 175.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative compaction: 0.00 GB write, 26.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: starting mon.a rank 0 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:13.133 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???) e1 preinit fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).mds e1 new map 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).mds e1 print_map 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout: e1 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout: legacy client fscid: -1 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout: No filesystems configured 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:50:13.134 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T09:50:13.135 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T09:50:13.135 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: monmap e1: 1 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:50:13.135 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: fsmap 2024-09-21T09:50:13.135 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T09:50:13.135 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 ceph-mon[29367]: mgrmap e1: no daemons active 2024-09-21T09:50:13.138 INFO:teuthology.orchestra.run.smithi119.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T09:50:13.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 bash[29329]: 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 2024-09-21T09:50:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:13 smithi119 systemd[1]: Started Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:50:14.214 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T09:50:14.214 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:50:14.215 INFO:teuthology.orchestra.run.smithi119.stdout:Creating mgr... 2024-09-21T09:50:14.215 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T09:50:14.215 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T09:50:14.386 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y 2024-09-21T09:50:14.386 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T09:50:14.534 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c96f302a-77fe-11ef-baf6-efdc52797490.target.wants/ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-c96f302a-77fe-11ef-baf6-efdc52797490@.service. 2024-09-21T09:50:14.860 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:14 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/4252000611' entity='client.admin' 2024-09-21T09:50:14.860 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:14 smithi119 podman[29603]: 2024-09-21 09:50:14.662726273 +0000 UTC m=+0.015555725 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph:quincy 2024-09-21T09:50:15.097 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:50:15.097 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-21T09:50:15.106 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:50:15.106 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-21T09:50:15.106 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-21T09:50:15.106 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T09:50:15.116 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:50:15.116 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-21T09:50:15.126 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:50:15.126 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-21T09:50:15.126 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-21T09:50:15.126 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T09:50:15.126 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr to start... 2024-09-21T09:50:15.126 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr... 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:14 smithi119 podman[29603]: 2024-09-21 09:50:14.972160395 +0000 UTC m=+0.324989853 container init 34c990954610e68d3a5d0f687cf3d0c1975cb0ad0f42790682f146d9b05840f5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:14 smithi119 podman[29603]: 2024-09-21 09:50:14.977101304 +0000 UTC m=+0.329930754 container start 34c990954610e68d3a5d0f687cf3d0c1975cb0ad0f42790682f146d9b05840f5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True) 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 bash[29603]: 34c990954610e68d3a5d0f687cf3d0c1975cb0ad0f42790682f146d9b05840f5 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 systemd[1]: Started Ceph mgr.y for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.090+0000 7f383bde3ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.159+0000 7f383bde3ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:15.227 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.224+0000 7f383bde3ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:50:15.493 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.350+0000 7f383bde3ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:15.493 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.490+0000 7f383bde3ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:50:15.762 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1269920999' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:50:16.012 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.760+0000 7f383bde3ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:50:16.013 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.821+0000 7f383bde3ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:50:16.013 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:15 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:15.986+0000 7f383bde3ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:50:16.249 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:16.256 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:50:16.256 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "c96f302a-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:50:16.256 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:50:16.256 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:50:16.256 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:50:16.256 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:50:16.257 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:50:16.258 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:16.259 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:16.260 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:50:08.134196+0000", 2024-09-21T09:50:16.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:50:16.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:16.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:50:16.261 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:50:16.261 INFO:teuthology.orchestra.run.smithi119.stdout:mgr not available, waiting (1/15)... 2024-09-21T09:50:17.116 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:16.832+0000 7f383bde3ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:50:17.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:16.896+0000 7f383bde3ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:50:17.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:17.598+0000 7f383bde3ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:50:17.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:17.707+0000 7f383bde3ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:50:17.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:17.825+0000 7f383bde3ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:18.137 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:17.885+0000 7f383bde3ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:50:18.138 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.008+0000 7f383bde3ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:50:18.138 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.076+0000 7f383bde3ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:50:18.138 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.135+0000 7f383bde3ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:50:18.407 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.275+0000 7f383bde3ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:50:18.408 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.340+0000 7f383bde3ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:50:18.408 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.405+0000 7f383bde3ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:50:18.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.487+0000 7f383bde3ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:18.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.545+0000 7f383bde3ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:50:18.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:18 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:18.600+0000 7f383bde3ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:19.348+0000 7f383bde3ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: Activating manager daemon y 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: mgrmap e2: y(active, starting, since 0.00308633s) 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: Manager daemon y is now available 2024-09-21T09:50:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:50:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:50:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' 2024-09-21T09:50:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' 2024-09-21T09:50:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:19 smithi119 ceph-mon[29367]: from='mgr.14100 172.21.15.119:0/4023790176' entity='mgr.y' 2024-09-21T09:50:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:21 smithi119 ceph-mon[29367]: mgrmap e3: y(active, since 1.00718s) 2024-09-21T09:50:22.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:22 smithi119 ceph-mon[29367]: mgrmap e4: y(active, since 2s) 2024-09-21T09:50:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:22 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/4258191668' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "c96f302a-77fe-11ef-baf6-efdc52797490", 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T09:50:28.145 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:28.146 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T09:50:28.147 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T09:50:28.150 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.151 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.152 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T09:50:08.134196+0000", 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:50:28.153 INFO:teuthology.orchestra.run.smithi119.stdout:mgr is available 2024-09-21T09:50:30.788 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:30 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/177946517' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T09:50:31.084 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:31.084 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout fsid = c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T09:50:31.085 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling cephadm module... 2024-09-21T09:50:32.078 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:31 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2228772281' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-21T09:50:32.079 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setuser ceph since I am not root 2024-09-21T09:50:32.079 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setgroup ceph since I am not root 2024-09-21T09:50:32.079 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:31.902+0000 7ffb0d7d9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:50:32.079 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:31.956+0000 7ffb0d7d9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:32.079 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:32.025+0000 7ffb0d7d9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:50:32.328 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:32.149+0000 7ffb0d7d9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:32.329 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:32.285+0000 7ffb0d7d9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:50:32.588 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:32.547+0000 7ffb0d7d9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:50:33.018 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2228772281' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T09:50:33.019 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-mon[29367]: mgrmap e5: y(active, since 12s) 2024-09-21T09:50:33.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:32.627+0000 7ffb0d7d9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:50:33.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:32.792+0000 7ffb0d7d9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:50:33.530 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for the mgr to restart... 2024-09-21T09:50:33.531 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr epoch 5... 2024-09-21T09:50:33.946 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:33 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3309841685' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:50:33.946 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:33.660+0000 7ffb0d7d9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:50:33.946 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:33.726+0000 7ffb0d7d9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:50:34.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.435+0000 7ffb0d7d9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:50:34.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.546+0000 7ffb0d7d9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:50:34.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.666+0000 7ffb0d7d9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:34.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.724+0000 7ffb0d7d9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:50:35.110 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.848+0000 7ffb0d7d9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:50:35.110 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.913+0000 7ffb0d7d9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:50:35.110 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:34.972+0000 7ffb0d7d9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:50:35.110 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:35 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:35.106+0000 7ffb0d7d9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:50:35.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:35 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:35.169+0000 7ffb0d7d9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:50:35.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:35 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:35.234+0000 7ffb0d7d9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:50:35.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:35 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:35.314+0000 7ffb0d7d9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:35.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:35 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:35.372+0000 7ffb0d7d9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:50:35.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:35 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:35.429+0000 7ffb0d7d9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:36.168+0000 7ffb0d7d9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: Active manager daemon y restarted 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: Activating manager daemon y 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: mgrmap e6: y(active, starting, since 0.00282123s) 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:50:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: Manager daemon y is now available 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:50:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:36 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:50:37.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:37 smithi119 ceph-mon[29367]: Found migration_current of "None". Setting to last migration. 2024-09-21T09:50:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:37 smithi119 ceph-mon[29367]: mgrmap e7: y(active, since 1.00501s) 2024-09-21T09:50:37.752 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:50:37.753 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T09:50:37.753 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:50:37.753 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:50:37.753 INFO:teuthology.orchestra.run.smithi119.stdout:mgr epoch 5 is available 2024-09-21T09:50:37.753 INFO:teuthology.orchestra.run.smithi119.stdout:Setting orchestrator backend to cephadm... 2024-09-21T09:50:38.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-mon[29367]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-mon[29367]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:38.301+0000 7ffabfbc8640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:50:38] ENGINE Error in HTTPServer.serve 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: Traceback (most recent call last): 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._connections.run(self.expiration_interval) 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._run(expiration_interval) 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:50:38.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: s = self.context.wrap_socket( 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: return self.sslsocket_class._create( 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self.do_handshake() 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._sslobj.do_handshake() 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:50:38.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:39.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:38] ENGINE Bus STARTING 2024-09-21T09:50:39.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: mgrmap e8: y(active, since 2s) 2024-09-21T09:50:39.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:38] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T09:50:39.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:38] ENGINE Bus STARTED 2024-09-21T09:50:39.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:38] ENGINE Error in HTTPServer.serve 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:50:39.561 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:39.562 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:39 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:40.082 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T09:50:40.082 INFO:teuthology.orchestra.run.smithi119.stdout:Generating ssh key... 2024-09-21T09:50:40.454 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-mon[29367]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.161 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: Generating public/private rsa key pair. 2024-09-21T09:50:41.161 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: Your identification has been saved in /tmp/tmpyu3rugby/key 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: Your public key has been saved in /tmp/tmpyu3rugby/key.pub 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: The key fingerprint is: 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: SHA256:SpllkWyOcHNg1VweR0zeEdQUGIIe2QdSYfhDAim6GcA ceph-c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: The key's randomart image is: 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: +---[RSA 3072]----+ 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: |. o===*B==B*=| 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | E .oo.=B=+.*..o| 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | . .o.*+ = o . .| 2024-09-21T09:50:41.162 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | o .=.. o | 2024-09-21T09:50:41.163 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | + + S . | 2024-09-21T09:50:41.163 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | o . . | 2024-09-21T09:50:41.163 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | . | 2024-09-21T09:50:41.163 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | | 2024-09-21T09:50:41.163 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: | | 2024-09-21T09:50:41.163 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: +----[SHA256]-----+ 2024-09-21T09:50:41.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:41 smithi119 ceph-mon[29367]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:41.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:41 smithi119 ceph-mon[29367]: Generating ssh key... 2024-09-21T09:50:41.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:41 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:41.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:41 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:42.537 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaMKrC1ucrYaGjtWJGG8TqrLkcwtQXW1NgUfmTp6wX9WCB285u5DzTKtcKqmvqurqnpjRvafL8KZEmlCbNJ+u2mB/zfibG+zbi0/AuI2Xb/fbelfGfEEbELRDhJiBkQ+jK9gpxlp5b5gDoJpkRc1iYQyCsnA8Tqp8qBLDTOJsee49PVW8a7/C4/rWI70piyf+TgS7VUEmnSxzx9v16taRyZul9yYA9kvleAukomNVAjlNjxxsVd5qpbzYvj4nrZ4LQslKkjOmF2f5TcB0I4zQ69u7FdMAux3segRaNtDW0e3KZDyDDEsDfuXnTryMuI7zfv5jGk6dSmqnPSFnEq31kEKux8dqS+vJQvH26Y3XbLBrsd3JrRdtKp5NXOtXXnM6Lg9fxb8nGsRBR458qUs/ZXYGBjZTCN119J2EqFIVWiE6fBh5xeABTvE+pXKCIa6/H9pMnBUUQBnzeJkdIE59YQJmPfAih71x1b75TZb6EFwO6WYk1GZhSaBfMjhFmpc0= ceph-c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:50:42.537 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T09:50:42.537 INFO:teuthology.orchestra.run.smithi119.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T09:50:42.540 INFO:teuthology.orchestra.run.smithi119.stdout:Adding host smithi119... 2024-09-21T09:50:42.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:42 smithi119 ceph-mon[29367]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:43.905 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:43 smithi119 ceph-mon[29367]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:45.272 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:44 smithi119 ceph-mon[29367]: Deploying cephadm binary to smithi119 2024-09-21T09:50:46.337 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Added host 'smithi119' with addr '172.21.15.119' 2024-09-21T09:50:46.337 INFO:teuthology.orchestra.run.smithi119.stdout:Deploying unmanaged mon service... 2024-09-21T09:50:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:46 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:46 smithi119 ceph-mon[29367]: Added host smithi119 2024-09-21T09:50:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:46 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:47.797 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T09:50:47.797 INFO:teuthology.orchestra.run.smithi119.stdout:Deploying unmanaged mgr service... 2024-09-21T09:50:48.051 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:47 smithi119 ceph-mon[29367]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:48.052 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:47 smithi119 ceph-mon[29367]: Saving service mon spec with placement count:5 2024-09-21T09:50:48.052 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:47 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:49.816 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T09:50:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:49 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:49 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:49 smithi119 ceph-mon[29367]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:50:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:49 smithi119 ceph-mon[29367]: Saving service mgr spec with placement count:2 2024-09-21T09:50:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:49 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:51 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/288299675' entity='client.admin' 2024-09-21T09:50:52.800 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling the dashboard module... 2024-09-21T09:50:53.051 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:52 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:53.051 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:52 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1414217047' entity='client.admin' 2024-09-21T09:50:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-mon[29367]: from='mgr.14116 172.21.15.119:0/1514150822' entity='mgr.y' 2024-09-21T09:50:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2319268706' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-21T09:50:54.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setuser ceph since I am not root 2024-09-21T09:50:54.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setgroup ceph since I am not root 2024-09-21T09:50:54.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.262+0000 7fa8f3faeec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:50:54.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.317+0000 7fa8f3faeec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:54.644 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.382+0000 7fa8f3faeec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:50:54.644 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.507+0000 7fa8f3faeec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:54.644 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.642+0000 7fa8f3faeec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:50:55.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:55 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2319268706' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T09:50:55.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:55 smithi119 ceph-mon[29367]: mgrmap e9: y(active, since 17s) 2024-09-21T09:50:55.223 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.901+0000 7fa8f3faeec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:50:55.223 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:54.962+0000 7fa8f3faeec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:50:55.223 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:55.131+0000 7fa8f3faeec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:50:56.267 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:55.999+0000 7fa8f3faeec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:50:56.267 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:56.063+0000 7fa8f3faeec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:50:56.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:56 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/301443951' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T09:50:56.270 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for the mgr to restart... 2024-09-21T09:50:56.271 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr epoch 9... 2024-09-21T09:50:57.041 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:56.796+0000 7fa8f3faeec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:50:57.041 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:56.912+0000 7fa8f3faeec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:50:57.302 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.039+0000 7fa8f3faeec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:50:57.302 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.100+0000 7fa8f3faeec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:50:57.303 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.232+0000 7fa8f3faeec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:50:57.303 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.300+0000 7fa8f3faeec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:50:57.563 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.359+0000 7fa8f3faeec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:50:57.564 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.496+0000 7fa8f3faeec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:50:57.564 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.561+0000 7fa8f3faeec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:50:57.830 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.628+0000 7fa8f3faeec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:50:57.830 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.710+0000 7fa8f3faeec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:50:57.830 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.770+0000 7fa8f3faeec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:50:57.830 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:57.828+0000 7fa8f3faeec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:50:58.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:58.565+0000 7fa8f3faeec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: Active manager daemon y restarted 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: Activating manager daemon y 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: mgrmap e10: y(active, starting, since 0.00289528s) 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:50:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:50:58.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: Manager daemon y is now available 2024-09-21T09:50:58.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:50:58.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:50:58.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:50:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:51:00.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:50:59.911+0000 7fa8a4b0c640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:50:59] ENGINE Error in HTTPServer.serve 2024-09-21T09:51:00.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: Traceback (most recent call last): 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._connections.run(self.expiration_interval) 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._run(expiration_interval) 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: s = self.context.wrap_socket( 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:51:00.118 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: return self.sslsocket_class._create( 2024-09-21T09:51:00.119 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:51:00.119 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self.do_handshake() 2024-09-21T09:51:00.119 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:51:00.119 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._sslobj.do_handshake() 2024-09-21T09:51:00.119 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:51:00.119 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:50:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:51:00.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-21T09:51:00.271 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T09:51:00.272 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T09:51:00.272 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-21T09:51:00.272 INFO:teuthology.orchestra.run.smithi119.stdout:mgr epoch 9 is available 2024-09-21T09:51:00.272 INFO:teuthology.orchestra.run.smithi119.stdout:Generating a dashboard self-signed certificate... 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: mgrmap e11: y(active, since 1.00558s) 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:59] ENGINE Bus STARTING 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:59] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:59] ENGINE Bus STARTED 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: [21/Sep/2024:09:50:59] ENGINE Error in HTTPServer.serve 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:51:00.789 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:51:00.790 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-21T09:51:00.791 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:51:00.791 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T09:51:00.791 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:51:00.791 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:00.791 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:01.661 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-21T09:51:01.662 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial admin user... 2024-09-21T09:51:01.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:01 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:01.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:01 smithi119 ceph-mon[29367]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:01.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:01 smithi119 ceph-mon[29367]: mgrmap e12: y(active, since 2s) 2024-09-21T09:51:01.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:01 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:01.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:01 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:03.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$sZrQ1E1lZ79YTO/ZVPZYu.fQ5K/Dt2M3K92oV.4emGD7e76DV.F8y", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726912263, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T09:51:03.700 INFO:teuthology.orchestra.run.smithi119.stdout:Fetching dashboard port number... 2024-09-21T09:51:04.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:04 smithi119 ceph-mon[29367]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:04.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:04.906 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T09:51:04.916 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T09:51:04.916 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-21T09:51:04.923 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T09:51:04.923 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-21T09:51:04.923 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-21T09:51:04.923 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T09:51:04.925 INFO:teuthology.orchestra.run.smithi119.stdout:Ceph Dashboard is now available at: 2024-09-21T09:51:04.925 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:04.925 INFO:teuthology.orchestra.run.smithi119.stdout: URL: https://smithi119.front.sepia.ceph.com:8443/ 2024-09-21T09:51:04.925 INFO:teuthology.orchestra.run.smithi119.stdout: User: admin 2024-09-21T09:51:04.925 INFO:teuthology.orchestra.run.smithi119.stdout: Password: 97j8iopd7p 2024-09-21T09:51:04.925 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:04.926 INFO:teuthology.orchestra.run.smithi119.stdout:Saving cluster configuration to /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config directory 2024-09-21T09:51:04.926 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling autotune for osd_memory_target 2024-09-21T09:51:05.299 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:05 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2346189895' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-21T09:51:06.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:06 smithi119 ceph-mon[29367]: mgrmap e13: y(active, since 6s) 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.319 INFO:teuthology.orchestra.run.smithi119.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: ceph telemetry on 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout:For more information see: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:07.320 INFO:teuthology.orchestra.run.smithi119.stdout:Bootstrap complete. 2024-09-21T09:51:07.358 INFO:tasks.cephadm:Fetching config... 2024-09-21T09:51:07.358 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:07.358 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T09:51:07.374 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T09:51:07.374 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:07.374 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T09:51:07.430 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T09:51:07.430 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:07.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T09:51:07.495 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T09:51:07.496 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:07.496 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T09:51:07.551 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T09:51:07.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaMKrC1ucrYaGjtWJGG8TqrLkcwtQXW1NgUfmTp6wX9WCB285u5DzTKtcKqmvqurqnpjRvafL8KZEmlCbNJ+u2mB/zfibG+zbi0/AuI2Xb/fbelfGfEEbELRDhJiBkQ+jK9gpxlp5b5gDoJpkRc1iYQyCsnA8Tqp8qBLDTOJsee49PVW8a7/C4/rWI70piyf+TgS7VUEmnSxzx9v16taRyZul9yYA9kvleAukomNVAjlNjxxsVd5qpbzYvj4nrZ4LQslKkjOmF2f5TcB0I4zQ69u7FdMAux3segRaNtDW0e3KZDyDDEsDfuXnTryMuI7zfv5jGk6dSmqnPSFnEq31kEKux8dqS+vJQvH26Y3XbLBrsd3JrRdtKp5NXOtXXnM6Lg9fxb8nGsRBR458qUs/ZXYGBjZTCN119J2EqFIVWiE6fBh5xeABTvE+pXKCIa6/H9pMnBUUQBnzeJkdIE59YQJmPfAih71x1b75TZb6EFwO6WYk1GZhSaBfMjhFmpc0= ceph-c96f302a-77fe-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:51:07.628 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaMKrC1ucrYaGjtWJGG8TqrLkcwtQXW1NgUfmTp6wX9WCB285u5DzTKtcKqmvqurqnpjRvafL8KZEmlCbNJ+u2mB/zfibG+zbi0/AuI2Xb/fbelfGfEEbELRDhJiBkQ+jK9gpxlp5b5gDoJpkRc1iYQyCsnA8Tqp8qBLDTOJsee49PVW8a7/C4/rWI70piyf+TgS7VUEmnSxzx9v16taRyZul9yYA9kvleAukomNVAjlNjxxsVd5qpbzYvj4nrZ4LQslKkjOmF2f5TcB0I4zQ69u7FdMAux3segRaNtDW0e3KZDyDDEsDfuXnTryMuI7zfv5jGk6dSmqnPSFnEq31kEKux8dqS+vJQvH26Y3XbLBrsd3JrRdtKp5NXOtXXnM6Lg9fxb8nGsRBR458qUs/ZXYGBjZTCN119J2EqFIVWiE6fBh5xeABTvE+pXKCIa6/H9pMnBUUQBnzeJkdIE59YQJmPfAih71x1b75TZb6EFwO6WYk1GZhSaBfMjhFmpc0= ceph-c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:07.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaMKrC1ucrYaGjtWJGG8TqrLkcwtQXW1NgUfmTp6wX9WCB285u5DzTKtcKqmvqurqnpjRvafL8KZEmlCbNJ+u2mB/zfibG+zbi0/AuI2Xb/fbelfGfEEbELRDhJiBkQ+jK9gpxlp5b5gDoJpkRc1iYQyCsnA8Tqp8qBLDTOJsee49PVW8a7/C4/rWI70piyf+TgS7VUEmnSxzx9v16taRyZul9yYA9kvleAukomNVAjlNjxxsVd5qpbzYvj4nrZ4LQslKkjOmF2f5TcB0I4zQ69u7FdMAux3segRaNtDW0e3KZDyDDEsDfuXnTryMuI7zfv5jGk6dSmqnPSFnEq31kEKux8dqS+vJQvH26Y3XbLBrsd3JrRdtKp5NXOtXXnM6Lg9fxb8nGsRBR458qUs/ZXYGBjZTCN119J2EqFIVWiE6fBh5xeABTvE+pXKCIa6/H9pMnBUUQBnzeJkdIE59YQJmPfAih71x1b75TZb6EFwO6WYk1GZhSaBfMjhFmpc0= ceph-c96f302a-77fe-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T09:51:07.775 INFO:teuthology.orchestra.run.smithi145.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCaMKrC1ucrYaGjtWJGG8TqrLkcwtQXW1NgUfmTp6wX9WCB285u5DzTKtcKqmvqurqnpjRvafL8KZEmlCbNJ+u2mB/zfibG+zbi0/AuI2Xb/fbelfGfEEbELRDhJiBkQ+jK9gpxlp5b5gDoJpkRc1iYQyCsnA8Tqp8qBLDTOJsee49PVW8a7/C4/rWI70piyf+TgS7VUEmnSxzx9v16taRyZul9yYA9kvleAukomNVAjlNjxxsVd5qpbzYvj4nrZ4LQslKkjOmF2f5TcB0I4zQ69u7FdMAux3segRaNtDW0e3KZDyDDEsDfuXnTryMuI7zfv5jGk6dSmqnPSFnEq31kEKux8dqS+vJQvH26Y3XbLBrsd3JrRdtKp5NXOtXXnM6Lg9fxb8nGsRBR458qUs/ZXYGBjZTCN119J2EqFIVWiE6fBh5xeABTvE+pXKCIa6/H9pMnBUUQBnzeJkdIE59YQJmPfAih71x1b75TZb6EFwO6WYk1GZhSaBfMjhFmpc0= ceph-c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:07.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T09:51:08.032 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:07 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3801620952' entity='client.admin' 2024-09-21T09:51:10.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:10.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:10.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:10.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:10.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:10.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:11.255 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T09:51:11.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T09:51:11.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:11 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3821816056' entity='client.admin' 2024-09-21T09:51:11.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:11.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:12.003 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:14.460 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi145 2024-09-21T09:51:14.461 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:51:14.461 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/etc/ceph/ceph.conf 2024-09-21T09:51:14.479 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:51:14.479 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:14.537 INFO:tasks.cephadm:Adding host smithi145 to orchestrator... 2024-09-21T09:51:14.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch host add smithi145 2024-09-21T09:51:14.772 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:14.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:14.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:14.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:14.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:14.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:15 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:17.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:17 smithi119 ceph-mon[29367]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi145", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:18 smithi119 ceph-mon[29367]: Deploying cephadm binary to smithi145 2024-09-21T09:51:19.128 INFO:teuthology.orchestra.run.smithi119.stdout:Added host 'smithi145' with addr '172.21.15.145' 2024-09-21T09:51:19.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T09:51:20.009 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:20.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:20 smithi119 ceph-mon[29367]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:20.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:20.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:20 smithi119 ceph-mon[29367]: Added host smithi145 2024-09-21T09:51:20.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:20.460 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:20.460 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:21.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:21 smithi119 ceph-mon[29367]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:21.737 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:51:21.737 INFO:teuthology.orchestra.run.smithi119.stdout:[{"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.145", "hostname": "smithi145", "labels": [], "status": ""}] 2024-09-21T09:51:22.443 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T09:51:22.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T09:51:22.680 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:22.800 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:22 smithi119 ceph-mon[29367]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:23.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:24 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:25.516 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2024-09-21T09:51:25.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:25 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2123906216' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T09:51:25.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:25 smithi119 ceph-mon[29367]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:26.142 INFO:tasks.cephadm:Adding mon.a on smithi119 2024-09-21T09:51:26.142 INFO:tasks.cephadm:Adding mon.c on smithi119 2024-09-21T09:51:26.142 INFO:tasks.cephadm:Adding mon.b on smithi145 2024-09-21T09:51:26.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b' 2024-09-21T09:51:26.384 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:26 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2123906216' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:51:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:26 smithi119 ceph-mon[29367]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:27 smithi119 ceph-mon[29367]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:27.987 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled mon update... 2024-09-21T09:51:28.569 DEBUG:teuthology.orchestra.run.smithi119:mon.c> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T09:51:28.572 DEBUG:teuthology.orchestra.run.smithi145:mon.b> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T09:51:28.574 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:51:28.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b;count:3 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:29.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:28 smithi119 ceph-mon[29367]: Deploying daemon mon.b on smithi145 2024-09-21T09:51:30.299 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:51:30.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:29 smithi119 ceph-mon[29367]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:31.740 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:31 smithi145 systemd[1]: Starting Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:51:32.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:31 smithi119 ceph-mon[29367]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:32.703 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:51:32.703 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:50:05.190645Z","created":"2024-09-21T09:50:05.190645Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T09:51:32.703 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-09-21T09:51:33.016 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:32 smithi145 podman[33147]: 2024-09-21 09:51:32.634098095 +0000 UTC m=+0.113886491 container create f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T09:51:33.016 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:32 smithi145 podman[33147]: 2024-09-21 09:51:32.539096034 +0000 UTC m=+0.018884432 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T09:51:33.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:32 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/489275773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 podman[33147]: 2024-09-21 09:51:33.381678856 +0000 UTC m=+0.861467249 container init f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 podman[33147]: 2024-09-21 09:51:33.386649068 +0000 UTC m=+0.866437463 container start f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a) 2024-09-21T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 7 2024-09-21T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: pidfile_write: ignore empty --pid-file 2024-09-21T09:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: load: jerasure load: lrc 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: DB SUMMARY 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: DB Session ID: 0F8FJK38X1G5HYHEJX91 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:51:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.env: 0x556895e9e440 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.fs: Posix File System 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.info_log: 0x55689801eec0 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.statistics: (nil) 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.use_fsync: 0 2024-09-21T09:51:33.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:51:33.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:51:33.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:51:33.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:51:33.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.db_log_dir: 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:51:33.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.write_buffer_manager: 0x5568980dc780 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.unordered_write: 0 2024-09-21T09:51:33.549 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.row_cache: None 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.wal_filter: None 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:51:33.550 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:51:33.551 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_open_files: -1 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Compression algorithms supported: 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kZSTD supported: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:51:33.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.merge_operator: 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_filter: None 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556897fc5a38) 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:51:33.553 INFO:journalctl@ceph.mon.b.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: index_type: 0 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: data_block_index_type: 0 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: index_shortening: 1 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: hash_index_allow_collision: 1 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: checksum: 1 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: no_block_cache: 0 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache: 0x556897ffb1f0 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache_options: 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: capacity : 536870912 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: num_shard_bits : 4 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: strict_capacity_limit : 0 2024-09-21T09:51:33.554 INFO:journalctl@ceph.mon.b.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache_compressed: (nil) 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: persistent_cache: (nil) 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_size: 4096 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_size_deviation: 10 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_restart_interval: 16 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: index_block_restart_interval: 1 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: metadata_block_size: 4096 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: partition_filters: 0 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: use_delta_encoding: 1 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: whole_key_filtering: 1 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: verify_compression: 0 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:51:33.555 INFO:journalctl@ceph.mon.b.smithi145.stdout: format_version: 4 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout: enable_index_compression: 1 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_align: 0 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression: NoCompression 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.num_levels: 7 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:51:33.556 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:51:33.557 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:51:33.558 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:51:33.559 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.ttl: 2592000 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:51:33.560 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912293411020, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912293411321, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726912293, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "79669804-cb43-4794-a416-e99ba43af622", "db_session_id": "0F8FJK38X1G5HYHEJX91"}} 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912293411637, "job": 1, "event": "recovery_finished"} 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55689802e700 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: DB pointer 0x556897ffe000 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** DB Stats ** 2024-09-21T09:51:33.561 INFO:journalctl@ceph.mon.b.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:33.562 INFO:journalctl@ceph.mon.b.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.563 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:51:33.564 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: using public_addr v2:172.21.15.145:0/0 -> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: starting mon.b rank -1 at public addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] at bind addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(???) e0 preinit fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: e1 2024-09-21T09:51:33.565 INFO:journalctl@ceph.mon.b.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout: legacy client fscid: -1 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout: No filesystems configured 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:51:33.566 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2123906216' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b;count:3 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:33.567 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: Deploying daemon mon.b on smithi145 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/489275773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 ceph-mon[33206]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 bash[33147]: f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 2024-09-21T09:51:33.568 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:33 smithi145 systemd[1]: Started Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:34.281 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:51:34.281 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:51:34.532 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: mon.a calling monitor election 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: mon.b calling monitor election 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: fsmap 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: mgrmap e13: y(active, since 39s) 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: overall HEALTH_OK 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:38.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: mon.a calling monitor election 2024-09-21T09:51:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: mon.b calling monitor election 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: fsmap 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: mgrmap e13: y(active, since 39s) 2024-09-21T09:51:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: overall HEALTH_OK 2024-09-21T09:51:38.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:38.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:38.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:38.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:38.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:39.426 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:51:39.427 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":2,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:51:33.436479Z","created":"2024-09-21T09:50:05.190645Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T09:51:39.427 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 2 2024-09-21T09:51:39.728 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:39 smithi145 ceph-mon[33206]: Deploying daemon mon.c on smithi119 2024-09-21T09:51:39.728 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:39 smithi145 ceph-mon[33206]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:39.728 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:39 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/3258565336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:39.728 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:39 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:39.848 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:39 smithi119 ceph-mon[29367]: Deploying daemon mon.c on smithi119 2024-09-21T09:51:39.849 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:39 smithi119 ceph-mon[29367]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:39.849 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:39 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/3258565336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:39.849 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:39 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.169 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T09:51:41.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T09:51:41.416 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:41.522 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 podman[40746]: 2024-09-21 09:51:41.413514676 +0000 UTC m=+0.114420959 container create d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T09:51:41.780 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:41 smithi145 ceph-mon[33206]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[29367]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 podman[40746]: 2024-09-21 09:51:41.599225017 +0000 UTC m=+0.300131313 container init d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T09:51:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 podman[40746]: 2024-09-21 09:51:41.603891906 +0000 UTC m=+0.304798184 container start d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T09:51:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T09:51:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable), process ceph-mon, pid 7 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pidfile_write: ignore empty --pid-file 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: load: jerasure load: lrc 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: RocksDB version: 6.15.5 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Compile date Sep 20 2024 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: DB SUMMARY 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: DB Session ID: FA0QZRWK1E6MO81EYFOT 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: CURRENT file: CURRENT 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T09:51:41.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.error_if_exists: 0 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.create_if_missing: 0 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.paranoid_checks: 1 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.env: 0x55b62c4af440 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.fs: Posix File System 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.info_log: 0x55b62e6ecec0 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.statistics: (nil) 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.use_fsync: 0 2024-09-21T09:51:41.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_log_file_size: 0 2024-09-21T09:51:41.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.allow_fallocate: 1 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.use_direct_reads: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.db_log_dir: 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T09:51:41.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.write_buffer_manager: 0x55b62e7aa7b0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T09:51:41.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.unordered_write: 0 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.row_cache: None 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.wal_filter: None 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.preserve_deletes: 0 2024-09-21T09:51:41.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.two_write_queues: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.atomic_flush: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.log_readahead_size: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_background_jobs: 2 2024-09-21T09:51:41.874 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_background_compactions: -1 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_subcompactions: 1 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_open_files: -1 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T09:51:41.875 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_background_flushes: -1 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Compression algorithms supported: 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kZSTD supported: 0 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kXpressCompression supported: 0 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kBZip2Compression supported: 0 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kLZ4Compression supported: 1 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kZlibCompression supported: 1 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: kSnappyCompression supported: 1 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-21T09:51:41.876 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.merge_operator: 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_filter: None 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b62e693a38) 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_type: 0 2024-09-21T09:51:41.877 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_block_index_type: 0 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_shortening: 1 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: hash_index_allow_collision: 1 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: checksum: 1 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: no_block_cache: 0 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache: 0x55b62e6c91f0 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_options: 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: capacity : 536870912 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: num_shard_bits : 4 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: strict_capacity_limit : 0 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_compressed: (nil) 2024-09-21T09:51:41.878 INFO:journalctl@ceph.mon.c.smithi119.stdout: persistent_cache: (nil) 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_size: 4096 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_size_deviation: 10 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_restart_interval: 16 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_block_restart_interval: 1 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: metadata_block_size: 4096 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: partition_filters: 0 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: use_delta_encoding: 1 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: whole_key_filtering: 1 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: verify_compression: 0 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-09-21T09:51:41.879 INFO:journalctl@ceph.mon.c.smithi119.stdout: format_version: 4 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout: enable_index_compression: 1 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_align: 0 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression: NoCompression 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.num_levels: 7 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T09:51:41.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T09:51:41.881 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T09:51:41.883 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T09:51:41.883 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T09:51:41.883 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T09:51:41.883 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T09:51:41.884 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.arena_block_size: 4194304 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T09:51:41.885 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.inplace_update_support: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.bloom_locality: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.max_successive_merges: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T09:51:41.886 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.ttl: 2592000 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.enable_blob_files: false 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.min_blob_size: 0 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-21T09:51:41.887 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912301627587, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912301627890, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726912301, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "2b353165-0970-4537-af19-91d6471d4537", "db_session_id": "FA0QZRWK1E6MO81EYFOT"}} 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912301628200, "job": 1, "event": "recovery_finished"} 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55b62e6fc700 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: DB pointer 0x55b62e6cc000 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** DB Stats ** 2024-09-21T09:51:41.888 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:41.891 INFO:journalctl@ceph.mon.c.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:51:41.892 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-21T09:51:41.893 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T09:51:41.894 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: using public_addrv [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: starting mon.c rank -1 at public addrs [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] at bind addrs [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(???) e0 preinit fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: e1 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: legacy client fscid: -1 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout: No filesystems configured 2024-09-21T09:51:41.895 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2123906216' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:41.896 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi145:172.21.15.145=b;count:3 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: Deploying daemon mon.b on smithi145 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/489275773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.a calling monitor election 2024-09-21T09:51:41.897 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.b calling monitor election 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.898 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: fsmap 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mgrmap e13: y(active, since 39s) 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: overall HEALTH_OK 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: Deploying daemon mon.c on smithi119 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.899 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/3258565336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:41.900 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:41.900 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:41.900 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 ceph-mon[40785]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T09:51:41.900 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 bash[40746]: d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 2024-09-21T09:51:41.900 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:41 smithi119 systemd[1]: Started Ceph mon.c for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: mon.a calling monitor election 2024-09-21T09:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: mon.b calling monitor election 2024-09-21T09:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: fsmap 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: mgrmap e13: y(active, since 48s) 2024-09-21T09:51:47.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: mon.c (rank 2) addr [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] is down (out of quorum) 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: mon.a calling monitor election 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: mon.b calling monitor election 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: fsmap 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: mgrmap e13: y(active, since 48s) 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: mon.c (rank 2) addr [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] is down (out of quorum) 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:47.497 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:51:47.497 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":3,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","modified":"2024-09-21T09:51:41.651239Z","created":"2024-09-21T09:50:05.190645Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3301","nonce":0},{"type":"v1","addr":"172.21.15.119:6790","nonce":0}]},"addr":"172.21.15.119:6790/0","public_addr":"172.21.15.119:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T09:51:47.497 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 3 2024-09-21T09:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:47 smithi145 ceph-mon[33206]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/935116515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:47 smithi119 ceph-mon[29367]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/935116515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.559 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T09:51:48.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T09:51:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:51:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:51:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: mon.a calling monitor election 2024-09-21T09:51:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: mon.b calling monitor election 2024-09-21T09:51:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: fsmap 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: mgrmap e13: y(active, since 48s) 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T09:51:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: mon.c (rank 2) addr [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] is down (out of quorum) 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/935116515' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T09:51:48.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:48 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:49.439 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: mon.c calling monitor election 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: Reconfiguring daemon mon.a on smithi119 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: mon.c calling monitor election 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: mon.b calling monitor election 2024-09-21T09:51:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: mon.a calling monitor election 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: fsmap 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: mgrmap e13: y(active, since 50s) 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:51:49.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:49 smithi145 ceph-mon[33206]: overall HEALTH_OK 2024-09-21T09:51:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: mon.c calling monitor election 2024-09-21T09:51:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:51:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: Reconfiguring daemon mon.a on smithi119 2024-09-21T09:51:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: mon.c calling monitor election 2024-09-21T09:51:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: mon.b calling monitor election 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: mon.a calling monitor election 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: fsmap 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: mgrmap e13: y(active, since 50s) 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:51:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[29367]: overall HEALTH_OK 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: mon.c calling monitor election 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: Reconfiguring daemon mon.a on smithi119 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: mon.c calling monitor election 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: mon.b calling monitor election 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: mon.a calling monitor election 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T09:51:49.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: fsmap 2024-09-21T09:51:49.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T09:51:49.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: mgrmap e13: y(active, since 50s) 2024-09-21T09:51:49.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T09:51:49.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:51:49.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:49 smithi119 ceph-mon[40785]: overall HEALTH_OK 2024-09-21T09:51:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:51:50.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:50.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:51:50.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:50.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[29367]: Reconfiguring daemon mon.c on smithi119 2024-09-21T09:51:50.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:51:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:51:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:50.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:50 smithi119 ceph-mon[40785]: Reconfiguring daemon mon.c on smithi119 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: Reconfiguring mon.c (monmap changed)... 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:50 smithi145 ceph-mon[33206]: Reconfiguring daemon mon.c on smithi119 2024-09-21T09:51:52.247 INFO:teuthology.orchestra.run.smithi119.stdout:# minimal ceph.conf for c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:52.247 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-21T09:51:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T09:51:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: mon_host = [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] 2024-09-21T09:51:52.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:52 smithi119 ceph-mon[29367]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:52 smithi119 ceph-mon[40785]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:52 smithi145 ceph-mon[33206]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:53.095 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T09:51:53.096 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:53.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:51:53.123 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:53.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:53.187 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:51:53.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T09:51:53.197 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:53 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/4093535543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:53.197 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:53.197 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:53.197 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:53.197 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:51:53.197 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:53.215 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:51:53.215 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:51:53.281 INFO:tasks.cephadm:Adding mgr.y on smithi119 2024-09-21T09:51:53.282 INFO:tasks.cephadm:Adding mgr.x on smithi145 2024-09-21T09:51:53.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi119=y;smithi145=x' 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/4093535543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/4093535543' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:53.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T09:51:53.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T09:51:53.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:53.987 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:51:54.353 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: Reconfiguring daemon mon.b on smithi145 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:54.354 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: Reconfiguring daemon mon.b on smithi145 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: Reconfiguring mon.b (monmap changed)... 2024-09-21T09:51:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: Reconfiguring daemon mon.b on smithi145 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:55.687 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled mgr update... 2024-09-21T09:51:56.236 DEBUG:teuthology.orchestra.run.smithi145:mgr.x> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T09:51:56.238 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T09:51:56.238 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T09:51:56.238 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:51:56.239 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:51:56.256 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:51:56.256 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:51:56.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.311 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:51:56.311 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:56.311 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-21T09:51:56.311 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:56.311 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:56.311 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-21 09:51:41.141162991 +0000 2024-09-21T09:51:56.312 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.312 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.312 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:51:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:51:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T09:51:56.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:51:56.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:51:56.375 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-21T09:51:56.375 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-21T09:51:56.375 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000213167 s, 2.4 MB/s 2024-09-21T09:51:56.377 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:51:56.434 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-21 09:51:41.141162991 +0000 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.489 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:51:56.552 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-21T09:51:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-21T09:51:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000181709 s, 2.8 MB/s 2024-09-21T09:51:56.554 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:51:56.610 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-09-21T09:51:56.666 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:51:56.666 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:56.666 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-21T09:51:56.666 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:56.667 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:56.667 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-21 09:51:41.141162991 +0000 2024-09-21T09:51:56.667 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.667 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.667 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-21 09:47:10.018114234 +0000 2024-09-21T09:51:56.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:51:56.731 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-21T09:51:56.731 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-21T09:51:56.731 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000216662 s, 2.4 MB/s 2024-09-21T09:51:56.732 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:51:56.789 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-09-21T09:51:56.845 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:51:56.845 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:56.845 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T09:51:56.845 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:56.846 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:56.846 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-21 09:51:41.142162965 +0000 2024-09-21T09:51:56.846 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-21 09:47:10.019114209 +0000 2024-09-21T09:51:56.846 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-21 09:47:10.019114209 +0000 2024-09-21T09:51:56.846 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-21 09:47:10.019114209 +0000 2024-09-21T09:51:56.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:51:56.909 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-21T09:51:56.909 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-21T09:51:56.909 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000272083 s, 1.9 MB/s 2024-09-21T09:51:56.910 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:51:56.966 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:51:56.966 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T09:51:56.983 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T09:51:56.983 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-21 09:51:31.544378019 +0000 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.039 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T09:51:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-21T09:51:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-21T09:51:57.103 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000246645 s, 2.1 MB/s 2024-09-21T09:51:57.104 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T09:51:57.160 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-09-21T09:51:57.216 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T09:51:57.216 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 976 Links: 1 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-21 09:51:31.544378019 +0000 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.217 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T09:51:57.280 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-21T09:51:57.280 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-21T09:51:57.280 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000213485 s, 2.4 MB/s 2024-09-21T09:51:57.281 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T09:51:57.337 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-09-21T09:51:57.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:57 smithi119 ceph-mon[29367]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi145=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:57 smithi119 ceph-mon[29367]: Saving service mgr spec with placement smithi119=y;smithi145=x;count:2 2024-09-21T09:51:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:57 smithi119 ceph-mon[29367]: Deploying daemon mgr.x on smithi145 2024-09-21T09:51:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:57 smithi119 ceph-mon[40785]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi145=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:57 smithi119 ceph-mon[40785]: Saving service mgr spec with placement smithi119=y;smithi145=x;count:2 2024-09-21T09:51:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:57 smithi119 ceph-mon[40785]: Deploying daemon mgr.x on smithi145 2024-09-21T09:51:57.392 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-21 09:51:31.545377998 +0000 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-21 09:47:09.012011833 +0000 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-21 09:47:09.012011833 +0000 2024-09-21T09:51:57.393 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-09-21 09:47:09.012011833 +0000 2024-09-21T09:51:57.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T09:51:57.411 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:57 smithi145 ceph-mon[33206]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi145=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:51:57.411 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:57 smithi145 ceph-mon[33206]: Saving service mgr spec with placement smithi119=y;smithi145=x;count:2 2024-09-21T09:51:57.411 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:57 smithi145 ceph-mon[33206]: Deploying daemon mgr.x on smithi145 2024-09-21T09:51:57.572 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-21T09:51:57.572 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-21T09:51:57.573 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000199392 s, 2.6 MB/s 2024-09-21T09:51:57.574 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T09:51:57.589 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 980 Links: 1 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-21 09:51:57.525822434 +0000 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.647 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.648 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-09-21 09:47:09.013011793 +0000 2024-09-21T09:51:57.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T09:51:57.746 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-21T09:51:57.746 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-21T09:51:57.746 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000218754 s, 2.3 MB/s 2024-09-21T09:51:57.747 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T09:51:57.764 INFO:tasks.cephadm:Deploying osd.0 on smithi119 with /dev/vg_nvme/lv_4... 2024-09-21T09:51:57.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:51:57.969 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:57 smithi145 systemd[1]: Starting Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:51:57.999 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:51:58.219 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-mon[33206]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:58.220 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:57 smithi145 podman[35270]: 2024-09-21 09:51:57.870204946 +0000 UTC m=+0.015378400 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T09:51:58.220 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:57 smithi145 podman[35270]: 2024-09-21 09:51:57.969969766 +0000 UTC m=+0.115143252 container create cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2) 2024-09-21T09:51:58.220 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 podman[35270]: 2024-09-21 09:51:58.218719585 +0000 UTC m=+0.363893031 container init cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T09:51:58.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:58 smithi119 ceph-mon[29367]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:58 smithi119 ceph-mon[40785]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:58.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 podman[35270]: 2024-09-21 09:51:58.223560788 +0000 UTC m=+0.368734229 container start cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T09:51:58.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-mgr[35308]: -- 172.21.15.145:0/1672042758 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bd56088340 con 0x55bd56094800 2024-09-21T09:51:58.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 bash[35270]: cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a 2024-09-21T09:51:58.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 systemd[1]: Started Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:51:58.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:58.337+0000 7ff198959ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:51:58.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:58.395+0000 7ff198959ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:51:58.545 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:58.462+0000 7ff198959ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:51:59.016 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:58.592+0000 7ff198959ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:51:59.016 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:58.740+0000 7ff198959ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:51:59.016 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:59.014+0000 7ff198959ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:51:59.277 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:59.077+0000 7ff198959ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:51:59.277 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:51:59.246+0000 7ff198959ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:51:59 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:51:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:51:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:51:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:51:59 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:00.099+0000 7ff198959ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:52:00.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:00.162+0000 7ff198959ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:52:00.559 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:52:00.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-09-21T09:52:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[29367]: Reconfiguring daemon mgr.y on smithi119 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:52:00.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:52:00.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:52:00.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:00.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:00 smithi119 ceph-mon[40785]: Reconfiguring daemon mgr.y on smithi119 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T09:52:00.868 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:52:00.869 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:52:00.869 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:00.869 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-mon[33206]: Reconfiguring daemon mgr.y on smithi119 2024-09-21T09:52:01.157 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:00.866+0000 7ff198959ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:52:01.157 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:00.976+0000 7ff198959ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:52:01.157 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.096+0000 7ff198959ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:52:01.255 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:01.537 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.155+0000 7ff198959ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:52:01.537 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.280+0000 7ff198959ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:52:01.537 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.346+0000 7ff198959ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:52:01.537 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.403+0000 7ff198959ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:52:01.538 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.535+0000 7ff198959ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:52:01.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:01 smithi119 ceph-mon[29367]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:01.711 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:01 smithi119 ceph-mon[40785]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-mon[33206]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:01.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.598+0000 7ff198959ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:52:01.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.663+0000 7ff198959ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:52:01.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.744+0000 7ff198959ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:52:02.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.801+0000 7ff198959ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:52:02.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:01.857+0000 7ff198959ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:52:03.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:52:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:52:02.599+0000 7ff198959ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: Standby manager daemon x started 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: Standby manager daemon x started 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:52:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:03.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:03.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:03 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: Standby manager daemon x started 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3421424038' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:03 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[29367]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[29367]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[40785]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-21T09:52:04.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:52:04.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[40785]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:04.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:04.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:04 smithi145 ceph-mon[33206]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-21T09:52:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:52:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:04 smithi145 ceph-mon[33206]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:04.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:05.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:05 smithi119 ceph-mon[29367]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:05.515 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:05 smithi119 ceph-mon[40785]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:05 smithi145 ceph-mon[33206]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3748930273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]: dispatch 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]: dispatch 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]': finished 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[29367]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3554769204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3748930273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]: dispatch 2024-09-21T09:52:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]: dispatch 2024-09-21T09:52:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]': finished 2024-09-21T09:52:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[40785]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:52:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:06 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3554769204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:06 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3748930273' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]: dispatch 2024-09-21T09:52:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:06 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]: dispatch 2024-09-21T09:52:06.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:06 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cac46a83-240b-41e9-9cbe-6b685ebbb92b"}]': finished 2024-09-21T09:52:06.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:06 smithi145 ceph-mon[33206]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T09:52:06.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:06 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:06.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:06 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3554769204' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:07.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:07 smithi119 ceph-mon[29367]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:07 smithi119 ceph-mon[40785]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:07 smithi145 ceph-mon[33206]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:09.787 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:09 smithi119 ceph-mon[29367]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:09.787 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:09 smithi119 ceph-mon[40785]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:09 smithi145 ceph-mon[33206]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:11.764 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:11 smithi119 ceph-mon[29367]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:11.765 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:11 smithi119 ceph-mon[40785]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:11 smithi145 ceph-mon[33206]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:12 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:52:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:12 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:12 smithi119 ceph-mon[40785]: Deploying daemon osd.0 on smithi119 2024-09-21T09:52:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:12 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:52:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:12 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:12 smithi119 ceph-mon[29367]: Deploying daemon osd.0 on smithi119 2024-09-21T09:52:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:12 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T09:52:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:12 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:12 smithi145 ceph-mon[33206]: Deploying daemon osd.0 on smithi119 2024-09-21T09:52:13.734 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:13 smithi119 ceph-mon[29367]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:13.735 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:13 smithi119 ceph-mon[40785]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:13 smithi145 ceph-mon[33206]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:15 smithi145 ceph-mon[33206]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:16.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:15 smithi119 ceph-mon[29367]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:16.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:15 smithi119 ceph-mon[40785]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:16 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:16 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:16 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:16 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:16 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:16 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:16 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:16 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:16 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:18.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:17 smithi119 ceph-mon[29367]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:18.010 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:17 smithi119 ceph-mon[40785]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:18.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:17 smithi145 ceph-mon[33206]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:19.637 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 0 on host 'smithi119' 2024-09-21T09:52:19.949 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:19 smithi119 ceph-mon[29367]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:19.950 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:19 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:19.950 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:19 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:19.950 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:19 smithi119 ceph-mon[40785]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:19.950 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:19 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:19.950 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:19 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:19 smithi145 ceph-mon[33206]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:19 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:19 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:20.274 DEBUG:teuthology.orchestra.run.smithi119:osd.0> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T09:52:20.277 INFO:tasks.cephadm:Deploying osd.1 on smithi119 with /dev/vg_nvme/lv_3... 2024-09-21T09:52:20.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:52:20.569 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:21.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:20 smithi119 ceph-mon[29367]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:52:21.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:21.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:21.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:20 smithi119 ceph-mon[40785]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:52:21.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:21.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:20 smithi145 ceph-mon[33206]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T09:52:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[29367]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[29367]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:52:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[29367]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:52:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[29367]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:52:21.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[40785]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:21.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[40785]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:52:21.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[40785]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:52:21.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:21.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-mon[40785]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:52:21.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[45163]: 2024-09-21T09:52:21.665+0000 7fd84af16640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T09:52:21.869 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:52:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[45163]: 2024-09-21T09:52:21.667+0000 7fd843d49640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:52:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:21 smithi145 ceph-mon[33206]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:21 smithi145 ceph-mon[33206]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T09:52:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:21 smithi145 ceph-mon[33206]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T09:52:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:21 smithi145 ceph-mon[33206]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[29367]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[29367]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[40785]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[40785]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:22.856 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:22 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:22 smithi145 ceph-mon[33206]: from='osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:52:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:22 smithi145 ceph-mon[33206]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T09:52:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:22 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:22 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218] boot 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:52:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218] boot 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:52:23.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:24.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:52:24.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:52:24.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T09:52:24.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:24.047 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: osd.0 [v2:172.21.15.119:6802/2992214218,v1:172.21.15.119:6803/2992214218] boot 2024-09-21T09:52:24.047 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T09:52:24.047 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:23 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:52:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:52:24.211 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-09-21T09:52:24.755 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:24.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:24 smithi119 ceph-mon[29367]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:52:24.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:24 smithi119 ceph-mon[40785]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:52:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:24 smithi145 ceph-mon[33206]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T09:52:26.015 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:25 smithi119 ceph-mon[40785]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:26.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:25 smithi119 ceph-mon[29367]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:26.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:25 smithi145 ceph-mon[33206]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:27.048 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi119 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi119 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi119 to 16064M 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi119 to 16064M 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:27 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi119 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi119 to 16064M 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:27 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:28.294 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[40785]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:28.294 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[40785]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[29367]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[29367]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:28.295 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:28 smithi145 ceph-mon[33206]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:28 smithi145 ceph-mon[33206]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[29367]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1304462974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe034449-410d-4164-9895-411181da04f2"}]: dispatch 2024-09-21T09:52:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1304462974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe034449-410d-4164-9895-411181da04f2"}]': finished 2024-09-21T09:52:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[29367]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2520917536' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[40785]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1304462974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe034449-410d-4164-9895-411181da04f2"}]: dispatch 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1304462974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe034449-410d-4164-9895-411181da04f2"}]': finished 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[40785]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:30 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2520917536' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:30 smithi145 ceph-mon[33206]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:30 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1304462974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe034449-410d-4164-9895-411181da04f2"}]: dispatch 2024-09-21T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:30 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1304462974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe034449-410d-4164-9895-411181da04f2"}]': finished 2024-09-21T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:30 smithi145 ceph-mon[33206]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:30 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:30 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2520917536' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:32.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:32 smithi119 ceph-mon[29367]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:32.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:32 smithi119 ceph-mon[40785]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:32 smithi145 ceph-mon[33206]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:34.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:34 smithi119 ceph-mon[29367]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:34.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:34 smithi119 ceph-mon[40785]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:34 smithi145 ceph-mon[33206]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:36.195 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:36 smithi119 ceph-mon[29367]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:36.195 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:36 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:52:36.195 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:36 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:36.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:36 smithi119 ceph-mon[40785]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:36.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:36 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:52:36.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:36 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:36 smithi145 ceph-mon[33206]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:36 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T09:52:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:36 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:37 smithi119 ceph-mon[29367]: Deploying daemon osd.1 on smithi119 2024-09-21T09:52:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:37 smithi119 ceph-mon[40785]: Deploying daemon osd.1 on smithi119 2024-09-21T09:52:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:37 smithi145 ceph-mon[33206]: Deploying daemon osd.1 on smithi119 2024-09-21T09:52:38.255 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:38 smithi119 ceph-mon[29367]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:38.256 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:38 smithi119 ceph-mon[40785]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:38 smithi145 ceph-mon[33206]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:40.346 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:40 smithi145 ceph-mon[33206]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:40 smithi119 ceph-mon[29367]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:40 smithi119 ceph-mon[40785]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:42.351 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:42 smithi119 ceph-mon[40785]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:42.351 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:42 smithi119 ceph-mon[29367]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:42 smithi145 ceph-mon[33206]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:44.145 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:44.145 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.145 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.145 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[40785]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.146 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:44 smithi119 ceph-mon[29367]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:52:44.182 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 1 on host 'smithi119' 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:44 smithi145 ceph-mon[33206]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T09:52:44.823 DEBUG:teuthology.orchestra.run.smithi119:osd.1> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T09:52:44.825 INFO:tasks.cephadm:Deploying osd.2 on smithi119 with /dev/vg_nvme/lv_2... 2024-09-21T09:52:44.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:52:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:52:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:52:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[40785]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-mon[29367]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:45 smithi145 ceph-mon[33206]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:52:45.626 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:45.746 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[51253]: 2024-09-21T09:52:45.552+0000 7f4fadd48640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T09:52:45.746 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:52:45 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[51253]: 2024-09-21T09:52:45.555+0000 7f4faab83640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:52:46.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[29367]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:46.636 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[29367]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[29367]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[40785]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[40785]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[40785]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:46.637 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:46 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:46.638 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:46 smithi145 ceph-mon[33206]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:46.638 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:46 smithi145 ceph-mon[33206]: from='osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:52:46.638 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:46 smithi145 ceph-mon[33206]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T09:52:46.638 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:46.639 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[40785]: osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975] boot 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[40785]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[29367]: osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975] boot 2024-09-21T09:52:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[29367]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:52:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:47 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:52:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:47 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:52:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:47 smithi145 ceph-mon[33206]: osd.1 [v2:172.21.15.119:6810/2164564975,v1:172.21.15.119:6811/2164564975] boot 2024-09-21T09:52:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:47 smithi145 ceph-mon[33206]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T09:52:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:52:48.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:48 smithi119 ceph-mon[29367]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:48.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:48 smithi119 ceph-mon[29367]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:52:48.361 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:48 smithi119 ceph-mon[40785]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:48.361 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:48 smithi119 ceph-mon[40785]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:52:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:48 smithi145 ceph-mon[33206]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-21T09:52:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:48 smithi145 ceph-mon[33206]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T09:52:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi119 2024-09-21T09:52:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:50.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:50.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi119 to 8032M 2024-09-21T09:52:50.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:50.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:50.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:49 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.052 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:50.052 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi119 2024-09-21T09:52:50.052 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.052 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.052 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi119 to 8032M 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi119 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:50.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:52:50.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi119 to 8032M 2024-09-21T09:52:50.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:50.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:52:50.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:49 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:52:50.057 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:52:50.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-09-21T09:52:50.317 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:52:52.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:51 smithi145 ceph-mon[33206]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:51 smithi119 ceph-mon[29367]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:51 smithi119 ceph-mon[40785]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:52.831 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[40785]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:52.831 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:52.832 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:52.832 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:52.832 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[29367]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:52.832 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:52.832 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:52.832 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:52 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:52 smithi145 ceph-mon[33206]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:52:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:52 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:52:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:52 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:52:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:52 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:52:53.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:53 smithi119 ceph-mon[29367]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:53.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:53 smithi119 ceph-mon[40785]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:53 smithi145 ceph-mon[33206]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:54 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3718222006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db36e41e-f53c-4033-85fe-11dd420f633b"}]: dispatch 2024-09-21T09:52:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:54 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3718222006' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db36e41e-f53c-4033-85fe-11dd420f633b"}]': finished 2024-09-21T09:52:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:54 smithi145 ceph-mon[33206]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:52:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:52:55.088 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3718222006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db36e41e-f53c-4033-85fe-11dd420f633b"}]: dispatch 2024-09-21T09:52:55.088 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3718222006' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db36e41e-f53c-4033-85fe-11dd420f633b"}]': finished 2024-09-21T09:52:55.089 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[29367]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:52:55.089 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:52:55.089 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3718222006' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "db36e41e-f53c-4033-85fe-11dd420f633b"}]: dispatch 2024-09-21T09:52:55.089 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3718222006' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db36e41e-f53c-4033-85fe-11dd420f633b"}]': finished 2024-09-21T09:52:55.089 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[40785]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T09:52:55.089 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:52:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:55 smithi145 ceph-mon[33206]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:55 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3620000495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:55 smithi119 ceph-mon[29367]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:55 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3620000495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:55 smithi119 ceph-mon[40785]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:55 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3620000495' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:52:57.911 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:57 smithi119 ceph-mon[40785]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:57.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:57 smithi119 ceph-mon[29367]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:57 smithi145 ceph-mon[33206]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:52:59 smithi119 ceph-mon[29367]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:52:59.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:52:59 smithi119 ceph-mon[40785]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:52:59 smithi145 ceph-mon[33206]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:01.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:53:01.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:01.035 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:53:01.035 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T09:53:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:01.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:01 smithi119 ceph-mon[29367]: Deploying daemon osd.2 on smithi119 2024-09-21T09:53:01.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:01 smithi119 ceph-mon[29367]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:01 smithi119 ceph-mon[40785]: Deploying daemon osd.2 on smithi119 2024-09-21T09:53:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:01 smithi119 ceph-mon[40785]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:01 smithi145 ceph-mon[33206]: Deploying daemon osd.2 on smithi119 2024-09-21T09:53:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:01 smithi145 ceph-mon[33206]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:03.975 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:03 smithi119 ceph-mon[29367]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:03.976 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:03 smithi119 ceph-mon[40785]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:03 smithi145 ceph-mon[33206]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:05.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[29367]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:05.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:05.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:05.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:05.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[40785]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:05.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:05.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:05.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:05 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:05 smithi145 ceph-mon[33206]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:05 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:05 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:05 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:07 smithi145 ceph-mon[33206]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:08.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:07 smithi119 ceph-mon[29367]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:07 smithi119 ceph-mon[40785]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:08.792 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 2 on host 'smithi119' 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.436 DEBUG:teuthology.orchestra.run.smithi119:osd.2> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T09:53:09.439 INFO:tasks.cephadm:Deploying osd.3 on smithi119 with /dev/vg_nvme/lv_1... 2024-09-21T09:53:09.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:10.225 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:10.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[40785]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:10.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[40785]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:53:10.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[40785]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:53:10.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:10.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[40785]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:10.346 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[29367]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:10.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[29367]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:53:10.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[29367]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:53:10.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:10.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-mon[29367]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:10.347 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[57200]: 2024-09-21T09:53:10.080+0000 7f077696d640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T09:53:10.347 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:10 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[57200]: 2024-09-21T09:53:10.084+0000 7f076f7a0640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:53:10.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:10 smithi145 ceph-mon[33206]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:10.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:10 smithi145 ceph-mon[33206]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T09:53:10.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:10 smithi145 ceph-mon[33206]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T09:53:10.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:10 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:10.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:10 smithi145 ceph-mon[33206]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[29367]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:53:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[29367]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:53:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:11.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[40785]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:53:11.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[40785]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:53:11.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:11.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:11 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:11 smithi145 ceph-mon[33206]: from='osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:53:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:11 smithi145 ceph-mon[33206]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T09:53:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:11 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:11 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063] boot 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:53:12.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063] boot 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:53:12.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:12 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: pgmap v73: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: osd.2 [v2:172.21.15.119:6818/3612940063,v1:172.21.15.119:6819/3612940063] boot 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T09:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:12 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:53:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:13 smithi119 ceph-mon[40785]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:53:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:53:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:13 smithi119 ceph-mon[29367]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:53:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:53:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:13 smithi145 ceph-mon[33206]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T09:53:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-21T09:53:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: pgmap v76: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:53:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:53:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:53:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: pgmap v76: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:53:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:53:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:53:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:14.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:14.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:14 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: pgmap v76: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:14 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:14.789 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:53:14.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-09-21T09:53:15.059 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:53:15.116 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:14 smithi119 sudo[61097]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:53:15.117 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:14 smithi119 sudo[61097]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:15.117 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:14 smithi119 sudo[61097]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:15.117 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:14 smithi119 sudo[61097]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:15.417 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi119 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi119 to 5354M 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[40785]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi119 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi119 to 5354M 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:53:15.418 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:15 smithi119 ceph-mon[29367]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:53:15.419 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61155]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:53:15.419 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61155]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:15.419 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61155]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:15.419 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61155]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:15 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi119 2024-09-21T09:53:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:15 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi119 to 5354M 2024-09-21T09:53:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:15 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T09:53:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:15 smithi145 ceph-mon[33206]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T09:53:15.674 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61177]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:53:15.674 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61177]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:15.674 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61177]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:15.674 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61177]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61197]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61197]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61197]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61209]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61209]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61209]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:15.936 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:15 smithi119 sudo[61209]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:16.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 ceph-mon[29367]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:16.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 ceph-mon[29367]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T09:53:16.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:16 smithi119 ceph-mon[40785]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:16 smithi119 ceph-mon[40785]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T09:53:16.224 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:16 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:16.225 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:53:16.225 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:16.225 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61212]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:16.225 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61212]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:16.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61235]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:53:16.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61235]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:16.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61235]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:16.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61235]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:16.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 ceph-mon[33206]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 ceph-mon[33206]: osdmap e22: 3 total, 3 up, 3 in 2024-09-21T09:53:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:16.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61260]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:53:16.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61260]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:16.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61260]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:16.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:16 smithi119 sudo[61260]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:17.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 sudo[35961]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:53:17.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 sudo[35961]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:17.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 sudo[35961]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:17.189 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:16 smithi145 sudo[35961]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:17.466 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:17.466 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:17.466 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:17.466 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:17.467 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:53:17.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 sudo[35964]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:53:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 sudo[35964]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 sudo[35964]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 sudo[35964]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:53:17.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:17.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61409]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T09:53:17.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61409]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:17.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61409]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:17.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61409]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61449]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T09:53:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61449]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T09:53:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61449]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T09:53:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:17 smithi119 sudo[61449]: pam_unix(sudo:session): session closed for user root 2024-09-21T09:53:18.393 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:18.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:18.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:18.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:18.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:53:18.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:18.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:18.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:18.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:18.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:18.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[29367]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:18.396 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:18.396 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:18.396 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:18.396 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:18.397 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:53:18.397 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:18.397 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:18.397 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:18.397 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:18.398 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:18.398 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:18 smithi119 ceph-mon[40785]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T09:53:18.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:18.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T09:53:18.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:53:18.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:53:18.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:53:18.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:18 smithi145 ceph-mon[33206]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T09:53:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:19 smithi119 ceph-mon[40785]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:19 smithi119 ceph-mon[29367]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:19 smithi145 ceph-mon[33206]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:20 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2279212486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]: dispatch 2024-09-21T09:53:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:20 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]: dispatch 2024-09-21T09:53:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:20 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]': finished 2024-09-21T09:53:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:20 smithi145 ceph-mon[33206]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:53:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:20 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1182638900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:53:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2279212486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]: dispatch 2024-09-21T09:53:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]: dispatch 2024-09-21T09:53:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]': finished 2024-09-21T09:53:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[29367]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:53:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1182638900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:53:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2279212486' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]: dispatch 2024-09-21T09:53:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]: dispatch 2024-09-21T09:53:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2f44491e-2550-4886-88cc-5d1c790f8783"}]': finished 2024-09-21T09:53:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[40785]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T09:53:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:20 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1182638900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:53:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:21 smithi145 ceph-mon[33206]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:21.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:21 smithi119 ceph-mon[29367]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:21 smithi119 ceph-mon[40785]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:23 smithi119 ceph-mon[29367]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-09-21T09:53:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:23 smithi119 ceph-mon[40785]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-09-21T09:53:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:23 smithi145 ceph-mon[33206]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-09-21T09:53:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:25 smithi119 ceph-mon[29367]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-21T09:53:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:53:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:25 smithi119 ceph-mon[40785]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-21T09:53:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:53:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:25 smithi145 ceph-mon[33206]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-09-21T09:53:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T09:53:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:26.930 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:26 smithi119 ceph-mon[40785]: Deploying daemon osd.3 on smithi119 2024-09-21T09:53:26.931 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:26 smithi119 ceph-mon[29367]: Deploying daemon osd.3 on smithi119 2024-09-21T09:53:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:26 smithi145 ceph-mon[33206]: Deploying daemon osd.3 on smithi119 2024-09-21T09:53:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:27 smithi145 ceph-mon[33206]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-21T09:53:28.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:27 smithi119 ceph-mon[40785]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-21T09:53:28.088 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:27 smithi119 ceph-mon[29367]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-09-21T09:53:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:29 smithi119 ceph-mon[29367]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-21T09:53:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:29 smithi119 ceph-mon[40785]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-21T09:53:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:29 smithi145 ceph-mon[33206]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-09-21T09:53:30.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:30 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:30.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:30 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:30.952 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:30 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:30.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:30 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:30.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:30 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:30.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:30 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:30 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:30 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:30 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:31.990 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:31 smithi119 ceph-mon[29367]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 819 B/s rd, 23 KiB/s wr, 2 op/s 2024-09-21T09:53:31.990 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:31 smithi119 ceph-mon[40785]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 819 B/s rd, 23 KiB/s wr, 2 op/s 2024-09-21T09:53:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:31 smithi145 ceph-mon[33206]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 819 B/s rd, 23 KiB/s wr, 2 op/s 2024-09-21T09:53:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:33 smithi119 ceph-mon[29367]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-21T09:53:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:33 smithi119 ceph-mon[29367]: from='osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:53:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:33 smithi119 ceph-mon[29367]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:53:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:33 smithi119 ceph-mon[40785]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-21T09:53:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:33 smithi119 ceph-mon[40785]: from='osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:53:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:33 smithi119 ceph-mon[40785]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:53:34.026 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 3 on host 'smithi119' 2024-09-21T09:53:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:33 smithi145 ceph-mon[33206]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-09-21T09:53:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:33 smithi145 ceph-mon[33206]: from='osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:53:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:33 smithi145 ceph-mon[33206]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T09:53:34.666 DEBUG:teuthology.orchestra.run.smithi119:osd.3> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T09:53:34.669 INFO:tasks.cephadm:Deploying osd.4 on smithi145 with /dev/vg_nvme/lv_4... 2024-09-21T09:53:34.669 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T09:53:34.916 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:35.036 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:53:35.037 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:53:35.037 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.037 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: from='osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:35.037 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:35.037 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.037 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:34 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: from='osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T09:53:35.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T09:53:35.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: from='osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:35.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-21T09:53:35.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:34 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:53:35.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T09:53:35.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:35 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:35.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-21T09:53:35.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: osdmap e25: 4 total, 3 up, 4 in 2024-09-21T09:53:35.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:35.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:35.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:35 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:36.812 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:36 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:53:36.812 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:36 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:53:36.812 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:36 smithi145 ceph-mon[33206]: osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818] boot 2024-09-21T09:53:36.812 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:36 smithi145 ceph-mon[33206]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:53:36.812 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:36 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:53:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:53:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[40785]: osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818] boot 2024-09-21T09:53:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[40785]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:53:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:53:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:53:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[29367]: osd.3 [v2:172.21.15.119:6826/3712858818,v1:172.21.15.119:6827/3712858818] boot 2024-09-21T09:53:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[29367]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T09:53:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:36 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:53:37.321 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:53:37.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi145:vg_nvme/lv_4 2024-09-21T09:53:37.575 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:37.852 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:37 smithi119 ceph-mon[29367]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:37.852 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:37 smithi119 ceph-mon[29367]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T09:53:37.852 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:37 smithi119 ceph-mon[40785]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:37.853 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:37 smithi119 ceph-mon[40785]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T09:53:37.969 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:37 smithi145 ceph-mon[33206]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-21T09:53:37.969 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:37 smithi145 ceph-mon[33206]: osdmap e27: 4 total, 4 up, 4 in 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi119 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi119 to 4016M 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:38.950 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:38.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:38.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:38.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:38.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:38.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:38.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:38 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi119 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi119 to 4016M 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi119 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi119 to 4016M 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:39.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:38 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:39 smithi145 ceph-mon[33206]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:39 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:53:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:39 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:53:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:39 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[29367]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[40785]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:53:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:39 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:41.203 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:40 smithi145 ceph-mon[33206]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:40 smithi119 ceph-mon[29367]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:40 smithi119 ceph-mon[40785]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:53:42.146 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:41 smithi145 ceph-mon[33206]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:42.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:41 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/761830686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]: dispatch 2024-09-21T09:53:42.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:41 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]: dispatch 2024-09-21T09:53:42.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:41 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]': finished 2024-09-21T09:53:42.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:41 smithi145 ceph-mon[33206]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:53:42.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[29367]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/761830686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]: dispatch 2024-09-21T09:53:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]: dispatch 2024-09-21T09:53:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]': finished 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[29367]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[40785]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/761830686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]: dispatch 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]: dispatch 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2315f8d6-fbe6-4590-8842-d8aba506a0b8"}]': finished 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[40785]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T09:53:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:42 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/1513648201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:53:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:42 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/1513648201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:53:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:42 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/1513648201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:53:44.280 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:43 smithi145 ceph-mon[33206]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:43 smithi119 ceph-mon[40785]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:43 smithi119 ceph-mon[29367]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:46.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:45 smithi145 ceph-mon[33206]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:45 smithi119 ceph-mon[40785]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:45 smithi119 ceph-mon[29367]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:48.235 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:47 smithi145 ceph-mon[33206]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:48.235 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:53:48.235 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:47 smithi119 ceph-mon[40785]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:47 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:53:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:47 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:47 smithi119 ceph-mon[29367]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T09:53:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:49.281 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:49 smithi145 ceph-mon[33206]: Deploying daemon osd.4 on smithi145 2024-09-21T09:53:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:49 smithi119 ceph-mon[29367]: Deploying daemon osd.4 on smithi145 2024-09-21T09:53:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:49 smithi119 ceph-mon[40785]: Deploying daemon osd.4 on smithi145 2024-09-21T09:53:50.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:50 smithi145 ceph-mon[33206]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:50 smithi119 ceph-mon[40785]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:50.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:50 smithi119 ceph-mon[29367]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:52.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:52 smithi145 ceph-mon[33206]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:52 smithi119 ceph-mon[40785]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:52 smithi119 ceph-mon[29367]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:53 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:53 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:53 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:54.028 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:54 smithi145 ceph-mon[33206]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:54 smithi119 ceph-mon[29367]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:54 smithi119 ceph-mon[40785]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:55.589 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 4 on host 'smithi145' 2024-09-21T09:53:56.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:56.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:56.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:56.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:53:56.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:53:56.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:56.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.188 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:56 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.216 DEBUG:teuthology.orchestra.run.smithi145:osd.4> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T09:53:56.218 INFO:tasks.cephadm:Deploying osd.5 on smithi145 with /dev/vg_nvme/lv_3... 2024-09-21T09:53:56.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:53:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:53:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:53:56.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:53:56.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T09:53:56.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:53:56.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:56 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:53:56.920 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:53:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-mon[33206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:53:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-mon[33206]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:53:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-mon[33206]: from='osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:53:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-mon[33206]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:53:57.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-mon[33206]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:57.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[38603]: 2024-09-21T09:53:57.228+0000 7f50f0f4e640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T09:53:57.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 09:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[38603]: 2024-09-21T09:53:57.232+0000 7f50ebd85640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[29367]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[29367]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[29367]: from='osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[29367]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[29367]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[40785]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T09:53:57.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[40785]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T09:53:57.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:57.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[40785]: from='osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:53:57.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[40785]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:53:57.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:57 smithi119 ceph-mon[40785]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-21T09:53:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:53:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T09:53:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610] boot 2024-09-21T09:53:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:53:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:58 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:53:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T09:53:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610] boot 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: osd.4 [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610] boot 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T09:53:58.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:58 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:53:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:59 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:53:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:59 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:53:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:59 smithi145 ceph-mon[33206]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:53:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:53:59 smithi145 ceph-mon[33206]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:53:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:53:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:53:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[29367]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:53:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[29367]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:53:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:53:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:53:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[40785]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:53:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:53:59 smithi119 ceph-mon[40785]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T09:54:01.263 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi145 2024-09-21T09:54:01.263 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.263 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.263 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:01.264 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi145 to 17088M 2024-09-21T09:54:01.264 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.264 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:01.264 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:01.264 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.264 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:00 smithi145 ceph-mon[33206]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:54:01.268 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:54:01.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi145:vg_nvme/lv_3 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi145 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi145 to 17088M 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[29367]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi145 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi145 to 17088M 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:00 smithi119 ceph-mon[40785]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T09:54:01.526 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:54:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:02 smithi145 ceph-mon[33206]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:02 smithi119 ceph-mon[40785]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:02.408 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:02 smithi119 ceph-mon[29367]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:04 smithi145 ceph-mon[33206]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-21T09:54:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:04 smithi145 ceph-mon[33206]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:04 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[29367]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[29367]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[40785]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-21T09:54:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[40785]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:04 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-21T09:54:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/3993454885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]: dispatch 2024-09-21T09:54:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]: dispatch 2024-09-21T09:54:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]': finished 2024-09-21T09:54:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:54:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/3355323414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/3993454885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]: dispatch 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]: dispatch 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]': finished 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:06.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:06 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/3355323414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/3993454885' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]: dispatch 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]: dispatch 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e02fdc71-b568-4361-ab0f-1569f55f6034"}]': finished 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: osdmap e34: 6 total, 5 up, 6 in 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:06 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/3355323414' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:08 smithi119 ceph-mon[29367]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-21T09:54:08.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:08 smithi119 ceph-mon[40785]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-21T09:54:08.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:08 smithi145 ceph-mon[33206]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-21T09:54:10.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:10 smithi145 ceph-mon[33206]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-21T09:54:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:10 smithi119 ceph-mon[40785]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-21T09:54:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:10 smithi119 ceph-mon[29367]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-21T09:54:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:12 smithi119 ceph-mon[40785]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:54:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:12 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:54:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:12 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:12.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:12 smithi119 ceph-mon[29367]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:54:12.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:12 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:54:12.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:12 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:12.369 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:12 smithi145 ceph-mon[33206]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T09:54:12.369 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:12 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T09:54:12.369 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:12 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:13.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:13 smithi119 ceph-mon[29367]: Deploying daemon osd.5 on smithi145 2024-09-21T09:54:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:13 smithi119 ceph-mon[40785]: Deploying daemon osd.5 on smithi145 2024-09-21T09:54:13.409 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:13 smithi145 ceph-mon[33206]: Deploying daemon osd.5 on smithi145 2024-09-21T09:54:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:14 smithi145 ceph-mon[33206]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:14 smithi119 ceph-mon[29367]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:14 smithi119 ceph-mon[40785]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:16.372 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:16 smithi145 ceph-mon[33206]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:16 smithi119 ceph-mon[29367]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:16 smithi119 ceph-mon[40785]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:17.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:17.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:17.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:17 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:17 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:17 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:18.522 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:18 smithi145 ceph-mon[33206]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:18 smithi119 ceph-mon[29367]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:18 smithi119 ceph-mon[40785]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:20.032 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 5 on host 'smithi145' 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:54:20.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:54:20.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:20.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:20 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.588 DEBUG:teuthology.orchestra.run.smithi145:osd.5> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T09:54:20.590 INFO:tasks.cephadm:Deploying osd.6 on smithi145 with /dev/vg_nvme/lv_2... 2024-09-21T09:54:20.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:20.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:20 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:21.268 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:54:21.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-mon[33206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:54:21.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-mon[33206]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:54:21.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:21.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-mon[33206]: from='osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:21.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-mon[33206]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:21.406 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T09:54:21.160+0000 7f70e8c56640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T09:54:21.406 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 09:54:21 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T09:54:21.165+0000 7f70e0264640 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[29367]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[29367]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[29367]: from='osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[29367]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[40785]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[40785]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T09:54:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[40785]: from='osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:21 smithi119 ceph-mon[40785]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:22.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:22 smithi145 ceph-mon[33206]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:22.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:22 smithi145 ceph-mon[33206]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:54:22.185 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:22 smithi145 ceph-mon[33206]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:54:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:22 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:22 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[29367]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[29367]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[29367]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[40785]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-21T09:54:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[40785]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:54:22.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[40785]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T09:54:22.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:22.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:22 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:23 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:54:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:23 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:54:23.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:23 smithi145 ceph-mon[33206]: osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320] boot 2024-09-21T09:54:23.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:23 smithi145 ceph-mon[33206]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:54:23.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:23 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:23.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:23 smithi145 ceph-mon[33206]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[29367]: osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320] boot 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[29367]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[29367]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:54:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[40785]: osd.5 [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320] boot 2024-09-21T09:54:23.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[40785]: osdmap e37: 6 total, 6 up, 6 in 2024-09-21T09:54:23.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:54:23.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:23 smithi119 ceph-mon[40785]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T09:54:24.447 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:24 smithi145 ceph-mon[33206]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:24.447 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:24 smithi145 ceph-mon[33206]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:54:24.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:24 smithi119 ceph-mon[29367]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:24 smithi119 ceph-mon[29367]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:54:24.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:24 smithi119 ceph-mon[40785]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:24.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:24 smithi119 ceph-mon[40785]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi145 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi145 to 8544M 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:25 smithi145 ceph-mon[33206]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi145 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi145 to 8544M 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[29367]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi145 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi145 to 8544M 2024-09-21T09:54:25.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:25.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:25.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:25.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:25 smithi119 ceph-mon[40785]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:25.901 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:54:25.920 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi145:vg_nvme/lv_2 2024-09-21T09:54:26.158 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:54:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:27 smithi145 ceph-mon[33206]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:27 smithi119 ceph-mon[29367]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:27 smithi119 ceph-mon[40785]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:28.934 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[40785]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:28.934 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:28.934 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:28.935 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:28.935 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[29367]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:28.935 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:28.935 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:28.935 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:28 smithi145 ceph-mon[33206]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:29.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:29 smithi119 ceph-mon[40785]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:29 smithi119 ceph-mon[29367]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:29.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:29 smithi145 ceph-mon[33206]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:30 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/3546531940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]: dispatch 2024-09-21T09:54:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:30 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]: dispatch 2024-09-21T09:54:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:30 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]': finished 2024-09-21T09:54:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:30 smithi145 ceph-mon[33206]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:54:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:30 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/3546531940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]: dispatch 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]: dispatch 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]': finished 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[29367]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/3546531940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]: dispatch 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]: dispatch 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5bc8e69-00be-456e-b6fd-d6e4e43029a9"}]': finished 2024-09-21T09:54:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[40785]: osdmap e40: 7 total, 6 up, 7 in 2024-09-21T09:54:31.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:30 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:31 smithi145 ceph-mon[33206]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:31 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/2471152281' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:32.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:31 smithi119 ceph-mon[40785]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:32.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:31 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/2471152281' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:31 smithi119 ceph-mon[29367]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:31 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/2471152281' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:33 smithi145 ceph-mon[33206]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:33 smithi119 ceph-mon[40785]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:33 smithi119 ceph-mon[29367]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:35 smithi145 ceph-mon[33206]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:36.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:35 smithi119 ceph-mon[29367]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:35 smithi119 ceph-mon[40785]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:36 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:54:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:36 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:37.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:36 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:54:37.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:36 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:37.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:36 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T09:54:37.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:36 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:37.896 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:37 smithi145 ceph-mon[33206]: Deploying daemon osd.6 on smithi145 2024-09-21T09:54:37.896 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:37 smithi145 ceph-mon[33206]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:37 smithi119 ceph-mon[29367]: Deploying daemon osd.6 on smithi145 2024-09-21T09:54:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:37 smithi119 ceph-mon[29367]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:37 smithi119 ceph-mon[40785]: Deploying daemon osd.6 on smithi145 2024-09-21T09:54:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:37 smithi119 ceph-mon[40785]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:39.955 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:39 smithi119 ceph-mon[40785]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:39.956 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:39 smithi119 ceph-mon[29367]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:39.983 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:39 smithi145 ceph-mon[33206]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:41 smithi145 ceph-mon[33206]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:41 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[29367]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[40785]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:41 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:43 smithi145 ceph-mon[33206]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:43 smithi119 ceph-mon[29367]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:43 smithi119 ceph-mon[40785]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:44.790 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 6 on host 'smithi145' 2024-09-21T09:54:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:44 smithi145 ceph-mon[33206]: from='osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:54:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:44 smithi145 ceph-mon[33206]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:54:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:44 smithi119 ceph-mon[29367]: from='osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:54:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:44 smithi119 ceph-mon[29367]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:54:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:44 smithi119 ceph-mon[40785]: from='osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:54:45.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:44 smithi119 ceph-mon[40785]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T09:54:45.451 DEBUG:teuthology.orchestra.run.smithi145:osd.6> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T09:54:45.454 INFO:tasks.cephadm:Deploying osd.7 on smithi145 with /dev/vg_nvme/lv_1... 2024-09-21T09:54:45.454 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T09:54:45.752 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:54:45.993 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[50814]: 2024-09-21T09:54:45.710+0000 7f90f00e1640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T09:54:45.993 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[50814]: 2024-09-21T09:54:45.717+0000 7f90e76ef640 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:45.994 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:45.995 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:45.995 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:45 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:54:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:54:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: osdmap e41: 7 total, 6 up, 7 in 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:45 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:46.892 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:46 smithi145 ceph-mon[33206]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:54:46.892 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:46 smithi145 ceph-mon[33206]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T09:54:46.892 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:46.892 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:46 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[29367]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:54:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[29367]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T09:54:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[40785]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:54:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[40785]: osdmap e42: 7 total, 6 up, 7 in 2024-09-21T09:54:47.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:47.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:46 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:47.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:47 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:54:47.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:47 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:54:47.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:47 smithi145 ceph-mon[33206]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:47.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:47 smithi145 ceph-mon[33206]: osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366] boot 2024-09-21T09:54:47.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:47 smithi145 ceph-mon[33206]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:54:47.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:47 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[29367]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[29367]: osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366] boot 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[29367]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:54:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:54:48.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[40785]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-21T09:54:48.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[40785]: osd.6 [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366] boot 2024-09-21T09:54:48.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[40785]: osdmap e43: 7 total, 7 up, 7 in 2024-09-21T09:54:48.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:47 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:54:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-21T09:54:48.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi145:vg_nvme/lv_1 2024-09-21T09:54:49.022 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:48 smithi145 ceph-mon[33206]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:54:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:48 smithi119 ceph-mon[40785]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:54:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:48 smithi119 ceph-mon[29367]: osdmap e44: 7 total, 7 up, 7 in 2024-09-21T09:54:49.530 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:54:49.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:49 smithi145 ceph-mon[33206]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:49.937 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:49 smithi145 ceph-mon[33206]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T09:54:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:49 smithi119 ceph-mon[40785]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:49 smithi119 ceph-mon[40785]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T09:54:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:49 smithi119 ceph-mon[29367]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:49 smithi119 ceph-mon[29367]: osdmap e45: 7 total, 7 up, 7 in 2024-09-21T09:54:51.972 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi145 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi145 to 5696M 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:51.973 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:51.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:51.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:51.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:51 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi145 2024-09-21T09:54:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi145 to 5696M 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi145 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi145 to 5696M 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:52.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:54:52.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:52.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:54:52.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:51 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:54:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:52 smithi145 ceph-mon[33206]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:52 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:52 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:52 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[29367]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[40785]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:54:53.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T09:54:53.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T09:54:53.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:52 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:54:54.277 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:53 smithi145 ceph-mon[33206]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:53 smithi119 ceph-mon[29367]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:53 smithi119 ceph-mon[40785]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:55.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:54 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/3433749667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]: dispatch 2024-09-21T09:54:55.147 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:54 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]: dispatch 2024-09-21T09:54:55.148 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:54 smithi145 ceph-mon[33206]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]': finished 2024-09-21T09:54:55.148 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:54 smithi145 ceph-mon[33206]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:54:55.148 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:54 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/3433749667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]: dispatch 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]: dispatch 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[40785]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]': finished 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[40785]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/3433749667' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]: dispatch 2024-09-21T09:54:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]: dispatch 2024-09-21T09:54:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[29367]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfc1495b-a36e-4f6f-8491-781460970f42"}]': finished 2024-09-21T09:54:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[29367]: osdmap e46: 8 total, 7 up, 8 in 2024-09-21T09:54:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:54 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:54:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:55 smithi145 ceph-mon[33206]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:55 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/858703155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:55 smithi119 ceph-mon[29367]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:55 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/858703155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:55 smithi119 ceph-mon[40785]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:55 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/858703155' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T09:54:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:57 smithi145 ceph-mon[33206]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:57 smithi119 ceph-mon[29367]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:54:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:57 smithi119 ceph-mon[40785]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:00.192 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:54:59 smithi145 ceph-mon[33206]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:54:59 smithi119 ceph-mon[29367]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:54:59 smithi119 ceph-mon[40785]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:01.266 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:55:01.266 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:00 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:55:01.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:00 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T09:55:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:00 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:02.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:01 smithi145 ceph-mon[33206]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:02.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:01 smithi145 ceph-mon[33206]: Deploying daemon osd.7 on smithi145 2024-09-21T09:55:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:01 smithi119 ceph-mon[40785]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:01 smithi119 ceph-mon[40785]: Deploying daemon osd.7 on smithi145 2024-09-21T09:55:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:01 smithi119 ceph-mon[29367]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:01 smithi119 ceph-mon[29367]: Deploying daemon osd.7 on smithi145 2024-09-21T09:55:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:03 smithi145 ceph-mon[33206]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:03 smithi119 ceph-mon[40785]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:03 smithi119 ceph-mon[29367]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:06.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:05 smithi145 ceph-mon[33206]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:06.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:05 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:06.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:05 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:06.271 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:05 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[40785]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:06.390 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:06.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:06.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:06.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[29367]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:06.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:06.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:06.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:05 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:08.233 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:07 smithi145 ceph-mon[33206]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:07 smithi119 ceph-mon[40785]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:07 smithi119 ceph-mon[29367]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:08.566 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 7 on host 'smithi145' 2024-09-21T09:55:09.593 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:09.594 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:09 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:09.594 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:09 smithi145 ceph-mon[33206]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:09.594 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:09 smithi145 ceph-mon[33206]: from='osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:55:09.594 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:09 smithi145 ceph-mon[33206]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:55:09.618 DEBUG:teuthology.orchestra.run.smithi145:osd.7> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T09:55:09.622 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T09:55:09.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T09:55:09.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[29367]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[29367]: from='osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[29367]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[40785]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[40785]: from='osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:55:09.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:09 smithi119 ceph-mon[40785]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T09:55:09.876 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: from='osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: from='osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:10 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:10.883 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T09:55:10.883 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: osdmap e47: 8 total, 7 up, 8 in 2024-09-21T09:55:10.883 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:10.883 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: from='osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:55:10.884 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-09-21T09:55:10.884 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:10.884 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:10.884 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T09:55:10.576+0000 7f14f145b640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T09:55:10.884 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 09:55:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T09:55:10.582+0000 7f14e8a69640 -1 osd.7 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T09:55:11.742 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741] boot 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:55:11.743 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T09:55:11.744 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.744 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.744 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:11.744 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741] boot 2024-09-21T09:55:11.744 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:55:11.744 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:11 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: osdmap e48: 8 total, 7 up, 8 in 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: osd.7 [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741] boot 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: osdmap e49: 8 total, 8 up, 8 in 2024-09-21T09:55:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:11 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:12.453 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":49,"num_osds":8,"num_up_osds":8,"osd_up_since":1726912511,"num_in_osds":8,"osd_in_since":1726912494,"num_remapped_pgs":1} 2024-09-21T09:55:12.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:55:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:12.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:12 smithi145 ceph-mon[33206]: purged_snaps scrub starts 2024-09-21T09:55:12.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:12 smithi145 ceph-mon[33206]: purged_snaps scrub ok 2024-09-21T09:55:12.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:12 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/660478117' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:55:12.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:12 smithi145 ceph-mon[33206]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:55:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[40785]: purged_snaps scrub starts 2024-09-21T09:55:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[40785]: purged_snaps scrub ok 2024-09-21T09:55:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/660478117' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:55:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[40785]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:55:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[29367]: purged_snaps scrub starts 2024-09-21T09:55:12.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[29367]: purged_snaps scrub ok 2024-09-21T09:55:12.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/660478117' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T09:55:12.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:12 smithi119 ceph-mon[29367]: osdmap e50: 8 total, 8 up, 8 in 2024-09-21T09:55:14.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:14.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: Detected new or changed devices on smithi145 2024-09-21T09:55:14.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi145 to 4272M 2024-09-21T09:55:14.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:14.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.109 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[29367]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:55:14.109 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:14.109 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: Detected new or changed devices on smithi145 2024-09-21T09:55:14.109 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.109 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi145 to 4272M 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:14.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:14.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:14.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:14.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:14.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:13 smithi119 ceph-mon[40785]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:55:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: Detected new or changed devices on smithi145 2024-09-21T09:55:14.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:14.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi145 to 4272M 2024-09-21T09:55:14.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:14.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:14.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:14.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:14.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:14.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:14.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:13 smithi145 ceph-mon[33206]: osdmap e51: 8 total, 8 up, 8 in 2024-09-21T09:55:14.591 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:55:14.591 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":51,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T09:55:13.573725+0000","last_up_change":"2024-09-21T09:55:11.571016+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6803","nonce":2992214218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6805","nonce":2992214218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6809","nonce":2992214218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6807","nonce":2992214218}]},"public_addr":"172.21.15.119:6803/2992214218","cluster_addr":"172.21.15.119:6805/2992214218","heartbeat_back_addr":"172.21.15.119:6809/2992214218","heartbeat_front_addr":"172.21.15.119:6807/2992214218","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6811","nonce":2164564975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6813","nonce":2164564975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6817","nonce":2164564975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6815","nonce":2164564975}]},"public_addr":"172.21.15.119:6811/2164564975","cluster_addr":"172.21.15.119:6813/2164564975","heartbeat_back_addr":"172.21.15.119:6817/2164564975","heartbeat_front_addr":"172.21.15.119:6815/2164564975","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6819","nonce":3612940063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6821","nonce":3612940063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6825","nonce":3612940063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6823","nonce":3612940063}]},"public_addr":"172.21.15.119:6819/3612940063","cluster_addr":"172.21.15.119:6821/3612940063","heartbeat_back_addr":"172.21.15.119:6825/3612940063","heartbeat_front_addr":"172.21.15.119:6823/3612940063","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6827","nonce":3712858818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6829","nonce":3712858818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6833","nonce":3712858818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6831","nonce":3712858818}]},"public_addr":"172.21.15.119:6827/3712858818","cluster_addr":"172.21.15.119:6829/3712858818","heartbeat_back_addr":"172.21.15.119:6833/3712858818","heartbeat_front_addr":"172.21.15.119:6831/3712858818","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:55:15.116 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:14 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/609341249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:15.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:14 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/609341249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:14 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/609341249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:55:15.430 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T09:53:12.866105+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-21T09:55:15.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T09:55:15.671 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:16.081 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:15 smithi119 ceph-mon[40785]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:16.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:15 smithi119 ceph-mon[29367]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:15 smithi145 ceph-mon[33206]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:17.489 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-21T09:55:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:17 smithi119 ceph-mon[29367]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:17 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3032829264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:55:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:17 smithi119 ceph-mon[40785]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:17 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3032829264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:55:18.092 INFO:tasks.cephadm:Adding prometheus.a on smithi145 2024-09-21T09:55:18.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi145=a' 2024-09-21T09:55:18.120 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:17 smithi145 ceph-mon[33206]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:18.121 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:17 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3032829264' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T09:55:18.348 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:55:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:19 smithi119 ceph-mon[29367]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:19 smithi119 ceph-mon[40785]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:20.199 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled prometheus update... 2024-09-21T09:55:20.199 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:19 smithi145 ceph-mon[33206]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:20.833 DEBUG:teuthology.orchestra.run.smithi145:prometheus.a> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T09:55:20.896 INFO:tasks.cephadm:Adding node-exporter.a on smithi119 2024-09-21T09:55:20.896 INFO:tasks.cephadm:Adding node-exporter.b on smithi145 2024-09-21T09:55:20.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi119=a;smithi145=b' 2024-09-21T09:55:21.186 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: Saving service prometheus spec with placement smithi145=a;count:1 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:21 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: Saving service prometheus spec with placement smithi145=a;count:1 2024-09-21T09:55:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: Saving service prometheus spec with placement smithi145=a;count:1 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:21.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:21 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:22 smithi145 ceph-mon[33206]: Deploying daemon prometheus.a on smithi145 2024-09-21T09:55:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:22 smithi145 ceph-mon[33206]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:22.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:22 smithi119 ceph-mon[40785]: Deploying daemon prometheus.a on smithi145 2024-09-21T09:55:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:22 smithi119 ceph-mon[40785]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:22 smithi119 ceph-mon[29367]: Deploying daemon prometheus.a on smithi145 2024-09-21T09:55:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:22 smithi119 ceph-mon[29367]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:23.576 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled node-exporter update... 2024-09-21T09:55:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:24 smithi145 ceph-mon[33206]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:24 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:24.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:24 smithi119 ceph-mon[40785]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:24.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:24 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:24 smithi119 ceph-mon[29367]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:24 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:24.742 DEBUG:teuthology.orchestra.run.smithi119:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T09:55:24.745 DEBUG:teuthology.orchestra.run.smithi145:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T09:55:24.747 INFO:tasks.cephadm:Adding alertmanager.a on smithi119 2024-09-21T09:55:24.748 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi119=a' 2024-09-21T09:55:25.129 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:55:25.464 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:25 smithi145 ceph-mon[33206]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi145=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.464 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:25 smithi145 ceph-mon[33206]: Saving service node-exporter spec with placement smithi119=a;smithi145=b;count:2 2024-09-21T09:55:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:25 smithi119 ceph-mon[40785]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi145=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:25 smithi119 ceph-mon[40785]: Saving service node-exporter spec with placement smithi119=a;smithi145=b;count:2 2024-09-21T09:55:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:25 smithi119 ceph-mon[29367]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi145=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:25 smithi119 ceph-mon[29367]: Saving service node-exporter spec with placement smithi119=a;smithi145=b;count:2 2024-09-21T09:55:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:26 smithi145 ceph-mon[33206]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:26 smithi119 ceph-mon[40785]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:26 smithi119 ceph-mon[29367]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:27.235 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled alertmanager update... 2024-09-21T09:55:27.847 DEBUG:teuthology.orchestra.run.smithi119:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T09:55:27.850 INFO:tasks.cephadm:Adding grafana.a on smithi145 2024-09-21T09:55:27.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi145=a' 2024-09-21T09:55:28.088 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:55:28.467 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:28 smithi145 ceph-mon[33206]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:28.467 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:28 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:28 smithi119 ceph-mon[29367]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:28 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:28 smithi119 ceph-mon[40785]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:28 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:29.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:29 smithi119 ceph-mon[29367]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:29 smithi119 ceph-mon[29367]: Saving service alertmanager spec with placement smithi119=a;count:1 2024-09-21T09:55:29.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:29 smithi119 ceph-mon[40785]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:29 smithi119 ceph-mon[40785]: Saving service alertmanager spec with placement smithi119=a;count:1 2024-09-21T09:55:29.480 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:29 smithi145 ceph-mon[33206]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:29.480 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:29 smithi145 ceph-mon[33206]: Saving service alertmanager spec with placement smithi119=a;count:1 2024-09-21T09:55:30.407 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled grafana update... 2024-09-21T09:55:30.407 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:30 smithi145 ceph-mon[33206]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:30 smithi119 ceph-mon[40785]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:30 smithi119 ceph-mon[29367]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:31.589 DEBUG:teuthology.orchestra.run.smithi145:grafana.a> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T09:55:31.594 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T09:55:31.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:55:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:31 smithi145 ceph-mon[33206]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:31 smithi145 ceph-mon[33206]: Saving service grafana spec with placement smithi145=a;count:1 2024-09-21T09:55:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:31 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:31 smithi145 ceph-mon[33206]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:31.849 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[29367]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[29367]: Saving service grafana spec with placement smithi145=a;count:1 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[29367]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[40785]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi145=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[40785]: Saving service grafana spec with placement smithi145=a;count:1 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:31 smithi119 ceph-mon[40785]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:33.855 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-09-21T09:55:33.856 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQAVmO5mYeRzMhAAg3xgxGzyNy02/e2v9UDTbA== 2024-09-21T09:55:33.942 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:33 smithi145 ceph-mon[33206]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:33 smithi119 ceph-mon[29367]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:33 smithi119 ceph-mon[40785]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:34.537 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:55:34.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T09:55:34.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T09:55:34.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T09:55:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:34 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/258649327' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:34 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:34 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:55:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:34 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/258649327' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:34 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:34 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:55:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:34 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/258649327' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:34 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:34 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:55:35.264 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T09:55:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:35 smithi145 ceph-mon[33206]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:36.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:35 smithi119 ceph-mon[29367]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:35 smithi119 ceph-mon[40785]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:37.155 INFO:teuthology.orchestra.run.smithi145.stdout:[client.1] 2024-09-21T09:55:37.243 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQAZmO5m+VjSCBAAb3aR8Vv0RMWgQKAm3HsLUw== 2024-09-21T09:55:38.018 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-21T09:55:38.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T09:55:38.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T09:55:38.056 INFO:tasks.cephadm:Creating RBD pool 2024-09-21T09:55:38.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create rbd 8 2024-09-21T09:55:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='client.? 172.21.15.145:0/1819591022' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:55:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='client.? 172.21.15.145:0/1819591022' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:37 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:55:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='client.? 172.21.15.145:0/1819591022' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T09:55:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T09:55:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:38.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:37 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T09:55:38.311 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:39.040 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:38 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ignoring --setuser ceph since I am not root 2024-09-21T09:55:39.040 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:38 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ignoring --setgroup ceph since I am not root 2024-09-21T09:55:39.040 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:38 smithi145 ceph-mgr[35308]: -- 172.21.15.145:0/14813993 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a7a9824340 con 0x55a7a9830c00 2024-09-21T09:55:39.040 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:38 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:38.799+0000 7f7336b5fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:55:39.041 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:38 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:38.853+0000 7f7336b5fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:55:39.041 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:38 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:38.917+0000 7f7336b5fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:55:39.043 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setuser ceph since I am not root 2024-09-21T09:55:39.043 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setgroup ceph since I am not root 2024-09-21T09:55:39.043 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:38 smithi119 ceph-mgr[29640]: -- 172.21.15.119:0/3944702826 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ebe6db8340 con 0x55ebe6dc5000 2024-09-21T09:55:39.044 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:38.797+0000 7f4e0cdd5ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:55:39.044 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:38.851+0000 7f4e0cdd5ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:55:39.044 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:38 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:38.916+0000 7f4e0cdd5ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:55:39.044 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:39.041+0000 7f4e0cdd5ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:55:39.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:39.038+0000 7f7336b5fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:55:39.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:39.172+0000 7f7336b5fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:55:39.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:39.178+0000 7f4e0cdd5ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:55:39.764 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:39.764 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-mon[29367]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-mon[29367]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-mon[40785]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-mon[40785]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:39.431+0000 7f4e0cdd5ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:39.492+0000 7f4e0cdd5ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:55:39.765 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:39.661+0000 7f4e0cdd5ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:55:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' 2024-09-21T09:55:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-mon[33206]: from='mgr.14148 172.21.15.119:0/1405878547' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T09:55:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-mon[33206]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T09:55:39.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:39.424+0000 7f7336b5fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:55:39.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:39.483+0000 7f7336b5fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:55:39.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:39.641+0000 7f7336b5fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:55:40.640 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'rbd' created 2024-09-21T09:55:40.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:40 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1758888321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:55:40.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:40 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:55:40.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:40.478+0000 7f7336b5fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:55:40.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:40.543+0000 7f7336b5fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:55:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:40 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1758888321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:55:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:40 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:55:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:40 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1758888321' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:55:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:40 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]: dispatch 2024-09-21T09:55:40.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:40.529+0000 7f4e0cdd5ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:55:40.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:40.594+0000 7f4e0cdd5ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:55:41.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- rbd pool init rbd 2024-09-21T09:55:41.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.275+0000 7f7336b5fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:55:41.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.391+0000 7f7336b5fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:55:41.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.516+0000 7f7336b5fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:55:41.608 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.305+0000 7f4e0cdd5ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:55:41.608 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.416+0000 7f4e0cdd5ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:55:41.608 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.543+0000 7f4e0cdd5ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:55:41.608 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.605+0000 7f4e0cdd5ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:55:41.705 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:41.841 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:55:41.841 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-mon[33206]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:55:41.841 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.577+0000 7f7336b5fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:55:41.841 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.711+0000 7f7336b5fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:55:41.842 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.779+0000 7f7336b5fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:55:41.842 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.839+0000 7f7336b5fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:55:41.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:55:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-mon[29367]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:55:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]': finished 2024-09-21T09:55:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-mon[40785]: osdmap e52: 8 total, 8 up, 8 in 2024-09-21T09:55:41.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.747+0000 7f4e0cdd5ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:55:41.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.814+0000 7f4e0cdd5ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:55:42.109 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:41.978+0000 7f7336b5fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:55:42.109 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:42.041+0000 7f7336b5fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:55:42.110 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:42.106+0000 7f7336b5fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:55:42.128 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:41.872+0000 7f4e0cdd5ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:55:42.128 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:42.012+0000 7f4e0cdd5ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:55:42.128 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:42.077+0000 7f4e0cdd5ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:55:42.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:42.187+0000 7f7336b5fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:55:42.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:42.244+0000 7f7336b5fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:55:42.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:42.301+0000 7f7336b5fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:55:42.585 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:42.152+0000 7f4e0cdd5ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:55:42.585 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:42.233+0000 7f4e0cdd5ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:55:42.585 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:42.289+0000 7f4e0cdd5ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:55:42.585 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:42.345+0000 7f4e0cdd5ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:55:42.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-mon[40785]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:55:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:42 smithi119 ceph-mon[29367]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:55:43.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:42 smithi145 ceph-mon[33206]: osdmap e53: 8 total, 8 up, 8 in 2024-09-21T09:55:43.282 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:43.101+0000 7f4e0cdd5ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:55:43.282 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:55:43] ENGINE Bus STARTING 2024-09-21T09:55:43.282 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: CherryPy Checker: 2024-09-21T09:55:43.282 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: The Application mounted at '' has an empty config. 2024-09-21T09:55:43.282 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:43.447 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:43.050+0000 7f7336b5fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:55:43.447 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: [21/Sep/2024:09:55:43] ENGINE Bus STARTING 2024-09-21T09:55:43.448 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: CherryPy Checker: 2024-09-21T09:55:43.448 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: The Application mounted at '' has an empty config. 2024-09-21T09:55:43.448 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: 2024-09-21T09:55:43.448 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: [21/Sep/2024:09:55:43] ENGINE Serving on http://:::9283 2024-09-21T09:55:43.448 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: [21/Sep/2024:09:55:43] ENGINE Bus STARTED 2024-09-21T09:55:43.556 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:55:43] ENGINE Serving on http://:::9283 2024-09-21T09:55:43.556 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:55:43] ENGINE Bus STARTED 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: Standby manager daemon x restarted 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: Standby manager daemon x started 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: Active manager daemon y restarted 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: Activating manager daemon y 2024-09-21T09:55:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: mgrmap e17: y(active, starting, since 0.00694614s), standbys: x 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:55:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: Manager daemon y is now available 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1976018254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:55:43.797 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:43 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:55:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:55:43.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: Standby manager daemon x restarted 2024-09-21T09:55:43.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: Standby manager daemon x started 2024-09-21T09:55:43.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:55:43.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:55:43.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:55:43.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:55:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: Active manager daemon y restarted 2024-09-21T09:55:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: Activating manager daemon y 2024-09-21T09:55:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:55:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: mgrmap e17: y(active, starting, since 0.00694614s), standbys: x 2024-09-21T09:55:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:55:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:55:43.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: Manager daemon y is now available 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:43.872 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:43.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1976018254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:55:43.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:55:43.873 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T09:55:43.873 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: Standby manager daemon x restarted 2024-09-21T09:55:43.873 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: Standby manager daemon x started 2024-09-21T09:55:43.873 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2953403037' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: Active manager daemon y restarted 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: Activating manager daemon y 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: mgrmap e17: y(active, starting, since 0.00694614s), standbys: x 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:55:43.874 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: Manager daemon y is now available 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:43.875 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:43.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:43.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:55:43.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:43.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T09:55:43.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1976018254' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:55:43.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:43 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]: dispatch 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: mgrmap e18: y(active, since 1.01278s), standbys: x 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: pgmap v3: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: mgrmap e18: y(active, since 1.01278s), standbys: x 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: pgmap v3: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:45 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:44.909+0000 7f4dafb98640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:55:44] ENGINE Error in HTTPServer.serve 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: Traceback (most recent call last): 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._connections.run(self.expiration_interval) 2024-09-21T09:55:45.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._run(expiration_interval) 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: s = self.context.wrap_socket( 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: return self.sslsocket_class._create( 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self.do_handshake() 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:55:45.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: self._sslobj.do_handshake() 2024-09-21T09:55:45.370 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:55:45.370 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:55:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: mgrmap e18: y(active, since 1.01278s), standbys: x 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: pgmap v3: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]': finished 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:45 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.540 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: [21/Sep/2024:09:55:44] ENGINE Bus STARTING 2024-09-21T09:55:46.540 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: [21/Sep/2024:09:55:44] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T09:55:46.540 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: [21/Sep/2024:09:55:44] ENGINE Bus STARTED 2024-09-21T09:55:46.540 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: [21/Sep/2024:09:55:44] ENGINE Error in HTTPServer.serve 2024-09-21T09:55:46.540 INFO:journalctl@ceph.mon.c.smithi119.stdout: Traceback (most recent call last): 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._run(expiration_interval) 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:55:46.541 INFO:journalctl@ceph.mon.c.smithi119.stdout: self.do_handshake() 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: pgmap v5: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:55:46.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: [21/Sep/2024:09:55:44] ENGINE Bus STARTING 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: [21/Sep/2024:09:55:44] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: [21/Sep/2024:09:55:44] ENGINE Bus STARTED 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: [21/Sep/2024:09:55:44] ENGINE Error in HTTPServer.serve 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:55:46.543 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: pgmap v5: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:55:46.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:46 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:55:46.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: [21/Sep/2024:09:55:44] ENGINE Bus STARTING 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: [21/Sep/2024:09:55:44] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: [21/Sep/2024:09:55:44] ENGINE Bus STARTED 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: [21/Sep/2024:09:55:44] ENGINE Error in HTTPServer.serve 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: Traceback (most recent call last): 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._run(expiration_interval) 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:55:46.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: s = self.context.wrap_socket( 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: return self.sslsocket_class._create( 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: self.do_handshake() 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._sslobj.do_handshake() 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: pgmap v5: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:46.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T09:55:46.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:55:46.548 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:46 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:55:47.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T09:55:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:55:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi145 to 4016M 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:55:47.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi145 to 4016M 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:47.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T09:55:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi145 to 4016M 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:55:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:47.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:55:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[40785]: Deploying daemon alertmanager.a on smithi119 2024-09-21T09:55:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[40785]: pgmap v8: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[40785]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:55:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[29367]: Deploying daemon alertmanager.a on smithi119 2024-09-21T09:55:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[29367]: pgmap v8: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:48 smithi119 ceph-mon[29367]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:55:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:48 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:48 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:55:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:48 smithi145 ceph-mon[33206]: Deploying daemon alertmanager.a on smithi119 2024-09-21T09:55:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:48 smithi145 ceph-mon[33206]: pgmap v8: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:48 smithi145 ceph-mon[33206]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T09:55:49.245 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:55:50.414 INFO:teuthology.orchestra.run.smithi119.stdout:[{"version":18,"timestamp":"2024-09-21T09:55:46.176161+0000","name":"","changes":[{"name":"osd/host:smithi145/osd_memory_target","previous_value":"4479530393","new_value":"4211094937"}]}] 2024-09-21T09:55:50.414 INFO:tasks.ceph_manager:config epoch is 18 2024-09-21T09:55:50.414 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:55:50.415 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:55:50.415 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:55:50.437 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:50 smithi119 ceph-mon[29367]: pgmap v9: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:50.438 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:50 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2377223306' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:55:50.438 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:50 smithi119 ceph-mon[40785]: pgmap v9: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:50.438 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:50 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2377223306' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:55:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:50 smithi145 ceph-mon[33206]: pgmap v9: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:55:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:50 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2377223306' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T09:55:50.689 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:52 smithi145 ceph-mon[33206]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 294 B/s wr, 10 op/s 2024-09-21T09:55:52.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:52 smithi119 ceph-mon[29367]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 294 B/s wr, 10 op/s 2024-09-21T09:55:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:52 smithi119 ceph-mon[40785]: pgmap v10: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 294 B/s wr, 10 op/s 2024-09-21T09:55:53.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:55:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:55:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:55:53.638 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:55:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:54 smithi119 ceph-mon[29367]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:55:54.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:54 smithi119 ceph-mon[40785]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:55:54.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:54 smithi119 ceph-mon[40785]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 257 B/s wr, 9 op/s 2024-09-21T09:55:54.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:54 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1023126941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:55:54.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:54 smithi119 ceph-mon[29367]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 257 B/s wr, 9 op/s 2024-09-21T09:55:54.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:54 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1023126941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:55:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:54 smithi145 ceph-mon[33206]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T09:55:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:54 smithi145 ceph-mon[33206]: pgmap v12: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 257 B/s wr, 9 op/s 2024-09-21T09:55:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:54 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1023126941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:55:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:56 smithi119 ceph-mon[40785]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 228 B/s wr, 8 op/s 2024-09-21T09:55:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:56 smithi119 ceph-mon[29367]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 228 B/s wr, 8 op/s 2024-09-21T09:55:56.409 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":20,"flags":0,"active_gid":24347,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1780938388},{"type":"v1","addr":"172.21.15.119:6801","nonce":1780938388}]},"active_addr":"172.21.15.119:6801/1780938388","active_change":"2024-09-21T09:55:43.105595+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24340,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.119:8443/","prometheus":"http://172.21.15.119:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":55,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2566533967}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":49791021}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":547636249}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2781454362}]}]} 2024-09-21T09:55:56.414 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:55:56.414 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:55:56.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:55:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:56 smithi145 ceph-mon[33206]: pgmap v13: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 228 B/s wr, 8 op/s 2024-09-21T09:55:57.535 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:55:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:58 smithi119 ceph-mon[40785]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T09:55:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:55:58 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:58 smithi119 ceph-mon[29367]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T09:55:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:55:58 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:58 smithi145 ceph-mon[33206]: pgmap v14: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T09:55:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:55:58 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:55:59.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:55:59.388 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":59,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T09:55:53.104637+0000","last_up_change":"2024-09-21T09:55:11.571016+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6803","nonce":2992214218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6805","nonce":2992214218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6809","nonce":2992214218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6807","nonce":2992214218}]},"public_addr":"172.21.15.119:6803/2992214218","cluster_addr":"172.21.15.119:6805/2992214218","heartbeat_back_addr":"172.21.15.119:6809/2992214218","heartbeat_front_addr":"172.21.15.119:6807/2992214218","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6811","nonce":2164564975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6813","nonce":2164564975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6817","nonce":2164564975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6815","nonce":2164564975}]},"public_addr":"172.21.15.119:6811/2164564975","cluster_addr":"172.21.15.119:6813/2164564975","heartbeat_back_addr":"172.21.15.119:6817/2164564975","heartbeat_front_addr":"172.21.15.119:6815/2164564975","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6819","nonce":3612940063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6821","nonce":3612940063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6825","nonce":3612940063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6823","nonce":3612940063}]},"public_addr":"172.21.15.119:6819/3612940063","cluster_addr":"172.21.15.119:6821/3612940063","heartbeat_back_addr":"172.21.15.119:6825/3612940063","heartbeat_front_addr":"172.21.15.119:6823/3612940063","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6827","nonce":3712858818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6829","nonce":3712858818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6833","nonce":3712858818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6831","nonce":3712858818}]},"public_addr":"172.21.15.119:6827/3712858818","cluster_addr":"172.21.15.119:6829/3712858818","heartbeat_back_addr":"172.21.15.119:6833/3712858818","heartbeat_front_addr":"172.21.15.119:6831/3712858818","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:56:00.413 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:56:00.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:56:00.436 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:00 smithi119 ceph-mon[29367]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T09:56:00.436 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:00 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2933257337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:56:00.437 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:00 smithi119 ceph-mon[40785]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T09:56:00.437 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:00 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2933257337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:56:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:00 smithi145 ceph-mon[33206]: pgmap v15: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 18 KiB/s rd, 204 B/s wr, 7 op/s 2024-09-21T09:56:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:00 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2933257337' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:56:00.920 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:01.042 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:00 smithi119 systemd[1]: Started Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:01.315 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:56:01] ENGINE Bus STOPPING 2024-09-21T09:56:01.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:56:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:56:01.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:56:01] ENGINE Bus STOPPED 2024-09-21T09:56:01.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:56:01] ENGINE Bus STARTING 2024-09-21T09:56:01.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:56:01] ENGINE Serving on http://:::9283 2024-09-21T09:56:01.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:56:01] ENGINE Bus STARTED 2024-09-21T09:56:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: Deploying daemon grafana.a on smithi145 2024-09-21T09:56:02.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[29367]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:56:02.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:56:02.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: Deploying daemon grafana.a on smithi145 2024-09-21T09:56:02.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:01 smithi119 ceph-mon[40785]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:56:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T09:56:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: Deploying daemon grafana.a on smithi145 2024-09-21T09:56:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:01 smithi145 ceph-mon[33206]: pgmap v16: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:02.879 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:56:02.880 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":59,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T09:55:53.104637+0000","last_up_change":"2024-09-21T09:55:11.571016+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6803","nonce":2992214218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6805","nonce":2992214218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6809","nonce":2992214218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6807","nonce":2992214218}]},"public_addr":"172.21.15.119:6803/2992214218","cluster_addr":"172.21.15.119:6805/2992214218","heartbeat_back_addr":"172.21.15.119:6809/2992214218","heartbeat_front_addr":"172.21.15.119:6807/2992214218","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6811","nonce":2164564975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6813","nonce":2164564975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6817","nonce":2164564975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6815","nonce":2164564975}]},"public_addr":"172.21.15.119:6811/2164564975","cluster_addr":"172.21.15.119:6813/2164564975","heartbeat_back_addr":"172.21.15.119:6817/2164564975","heartbeat_front_addr":"172.21.15.119:6815/2164564975","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6819","nonce":3612940063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6821","nonce":3612940063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6825","nonce":3612940063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6823","nonce":3612940063}]},"public_addr":"172.21.15.119:6819/3612940063","cluster_addr":"172.21.15.119:6821/3612940063","heartbeat_back_addr":"172.21.15.119:6825/3612940063","heartbeat_front_addr":"172.21.15.119:6823/3612940063","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6827","nonce":3712858818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6829","nonce":3712858818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6833","nonce":3712858818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6831","nonce":3712858818}]},"public_addr":"172.21.15.119:6827/3712858818","cluster_addr":"172.21.15.119:6829/3712858818","heartbeat_back_addr":"172.21.15.119:6833/3712858818","heartbeat_front_addr":"172.21.15.119:6831/3712858818","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:56:03.043 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:56:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:56:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:56:03.065 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[71454]: ts=2024-09-21T09:56:02.793Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000573398s 2024-09-21T09:56:03.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:03 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2187870802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:56:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:03 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2187870802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:56:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:03 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2187870802' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:56:03.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:56:03.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:56:03.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:56:03.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:56:03.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:56:03.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:56:03.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:56:03.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:56:04.169 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.224 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.225 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.229 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.230 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.235 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.236 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:04.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:04 smithi119 ceph-mon[29367]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:04.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:04 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:04.425 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:04 smithi119 ceph-mon[40785]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:04.425 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:04 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:04.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:04 smithi145 ceph-mon[33206]: pgmap v17: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:04 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:06.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:06 smithi119 ceph-mon[29367]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:06 smithi119 ceph-mon[40785]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:06 smithi145 ceph-mon[33206]: pgmap v18: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:08.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:08 smithi119 ceph-mon[29367]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:08.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:08 smithi119 ceph-mon[40785]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:08 smithi145 ceph-mon[33206]: pgmap v19: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:10.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:10 smithi119 ceph-mon[29367]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:10.462 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:10 smithi119 ceph-mon[40785]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:10 smithi145 ceph-mon[33206]: pgmap v20: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:11.116 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:10 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[71454]: ts=2024-09-21T09:56:10.796Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003584162s 2024-09-21T09:56:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:12 smithi145 ceph-mon[33206]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:12 smithi119 ceph-mon[40785]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:12 smithi119 ceph-mon[29367]: pgmap v21: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:12.580 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 systemd[1]: Starting Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:12.848 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:56:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:56:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:56:12.848 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 podman[64245]: 2024-09-21 09:56:12.578144347 +0000 UTC m=+0.107435082 container create 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, architecture=x86_64, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897) 2024-09-21T09:56:12.849 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 podman[64245]: 2024-09-21 09:56:12.485465087 +0000 UTC m=+0.014755833 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-21T09:56:12.849 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 podman[64245]: 2024-09-21 09:56:12.847266641 +0000 UTC m=+0.376557387 container init 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, name=ubi8, description=Ceph Grafana Container, architecture=x86_64, vcs-type=git, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.buildah.version=1.32.0, io.openshift.expose-services=) 2024-09-21T09:56:13.100 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 podman[64245]: 2024-09-21 09:56:12.852338588 +0000 UTC m=+0.381629324 container start 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, version=8.5, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., vcs-type=git, maintainer=Paul Cuzner , release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, description=Ceph Grafana Container, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-09-21T09:56:13.100 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 bash[64245]: 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc 2024-09-21T09:56:13.100 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:12 smithi145 systemd[1]: Started Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:13.100 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.003707309Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T09:56:13.100 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.003963146Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.003989794Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004003462Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004014724Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004026885Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004038055Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004053249Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004071642Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T09:56:13.101 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004082952Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T09:56:13.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004093431Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T09:56:13.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004104049Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T09:56:13.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004114603Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T09:56:13.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=settings t=2024-09-21T09:56:13.004125413Z level=info msg="App mode production" 2024-09-21T09:56:13.102 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore t=2024-09-21T09:56:13.004179422Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore t=2024-09-21T09:56:13.004202291Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.005109041Z level=info msg="Starting DB migrations" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.00519696Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.005774144Z level=info msg="Executing migration" id="create user table" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.006267818Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.006749124Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.007220665Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.007863423Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.008311651Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.009042317Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T09:56:13.103 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.009456832Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.009838873Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.010348479Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.010691666Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.011068231Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.011506177Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.011803943Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.012233765Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.012570665Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.012997008Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.013419112Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.013826088Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.01501033Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T09:56:13.104 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.01546649Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.015917047Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.016322081Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.016718084Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.017126133Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.017422719Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.017781193Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.01817391Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.018534742Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.018893364Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.019542685Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.019913666Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T09:56:13.105 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.020781945Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.021189476Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.021549943Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.021952046Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.022301838Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.022619561Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.022966156Z level=info msg="Executing migration" id="create star table" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.023395397Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.023801939Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T09:56:13.106 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.02419606Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.024598439Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.024985433Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.025390494Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.025789302Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.02618188Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.026456999Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.026723197Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.027043617Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.027411958Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.027793684Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.028205598Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.028587087Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T09:56:13.107 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.02905084Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.029893953Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.030736767Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.031139333Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.031928906Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.032777084Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.033117003Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.033585035Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.033866801Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.034360605Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.034824866Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.035316847Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.03572861Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T09:56:13.108 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.036296481Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.0367268Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.037145875Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.037962154Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.038256755Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.038763121Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.039295744Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.040204269Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.040944756Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.041273068Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.042139117Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.042564702Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.043392181Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.043830362Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T09:56:13.109 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.044304337Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.045192195Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.045996251Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.046840981Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.047682924Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.048620626Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.049169801Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.051931097Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T09:56:13.110 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.05491041Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.057661281Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.058034273Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.060741225Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.063610291Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.066388953Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.06683907Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.067281603Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.069902207Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.070323331Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.072549846Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.073016428Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.075892668Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T09:56:13.111 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.078488541Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.081435881Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.082004647Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.084893271Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.087422432Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.090164772Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.090723259Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.093617781Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.096190842Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.098852191Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.099384521Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T09:56:13.112 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.101859933Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.104422569Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.107026947Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.107611555Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.110144209Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.112892097Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.115404142Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.118068298Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T09:56:13.351 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.120962967Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.123640607Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.124366949Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.125194749Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.12616434Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.126899505Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.127214715Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.127671355Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.127964193Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.128544504Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.129503938Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T09:56:13.352 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.131915848Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.134713179Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.137272806Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.140047588Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.142574828Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.145259851Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.147831431Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.150461464Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T09:56:13.353 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.153339975Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.155822194Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.158474974Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.161268895Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.163844698Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.166556432Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.169229312Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.171832664Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.172125347Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.174828162Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.177481479Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.179950516Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.182664749Z level=info msg="Executing migration" id="create session table" 2024-09-21T09:56:13.354 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.183409483Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.184195704Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.184708918Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.185094093Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.18604868Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.18668241Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.187513291Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.188316284Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.189182385Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.190143446Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.190842165Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.191638289Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.192520922Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T09:56:13.355 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.193331115Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.1941329Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.19494176Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.195732988Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.196751216Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.197456485Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.198264563Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.199069168Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.199940798Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.200872879Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.201609671Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.202577253Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.204063613Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T09:56:13.356 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.204950031Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.205954181Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.206613899Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.207412102Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.208252328Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.209235954Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.210067372Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.210850347Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.211628796Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.214313149Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.217099722Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T09:56:13.357 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.219810846Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T09:56:13.358 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.222487724Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T09:56:13.358 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.225217822Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T09:56:13.358 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.22791854Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T09:56:13.358 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.230668365Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T09:56:13.358 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.233437068Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.236128006Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.238694198Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.24121074Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.243927537Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.246679747Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.249548692Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.252251069Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.254855116Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.257695781Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.260229825Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.262694146Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.265276722Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T09:56:13.359 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.26797522Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.270509277Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.273204653Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.275877038Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.278392236Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.281050361Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.283606968Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.286420488Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.288934468Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.291447782Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.294283653Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.29701983Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.299689789Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T09:56:13.360 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.302207421Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.304864759Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.305746544Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.306423543Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.307361441Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.308273749Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.309128798Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.309916578Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.310858012Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.311690915Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.312470569Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.313317531Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.314294662Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.314963087Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T09:56:13.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.315883796Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.316605591Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.317419656Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.318369122Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.319030433Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.319890966Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.320719126Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.32149832Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.322307627Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.323236918Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.324001313Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.32484871Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.325662182Z level=info msg="Executing migration" id="create team table" 2024-09-21T09:56:13.362 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.326629778Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.327381277Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.328314021Z level=info msg="Executing migration" id="create team member table" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.328970695Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.329926951Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.330648106Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.331521326Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.332402659Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.333168491Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.334035072Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.336879055Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.3395531Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.342117038Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T09:56:13.363 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.342558164Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.34341774Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.344253246Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.344718806Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.345673587Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.34638933Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.347223564Z level=info msg="Executing migration" id="create tag table" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.347632061Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.348640783Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.349311041Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.350126264Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.350694601Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.353826434Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T09:56:13.364 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.354660961Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T09:56:13.365 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.35554884Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.355923515Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.356775853Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.357632875Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.36028404Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.360634853Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.363563774Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T09:56:13.601 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.366403284Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.369155603Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.369920354Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.372795284Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.375627321Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.378383465Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.378849595Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.38154987Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.384433767Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.38714388Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.387583115Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.390490826Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T09:56:13.602 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.393279073Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T09:56:13.603 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.396250648Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T09:56:13.603 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.396925214Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T09:56:13.603 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.399617909Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T09:56:13.603 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.402193629Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.402557239Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.403029896Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.405739451Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.406199938Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.406523965Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.406919797Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.407326175Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.40771609Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.408149199Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.408919834Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.409575339Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.409851306Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.410249794Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T09:56:13.604 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.41064543Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.411062001Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.411347955Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.411746045Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.412196485Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.412585418Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.413005068Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.413771742Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.414177966Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.414552278Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.417841673Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.421038932Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.42146035Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.421845428Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.422677383Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T09:56:13.605 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.423120546Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.423540771Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.423956939Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.424371314Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.42462899Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.425458751Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.426272741Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.427108752Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.427509024Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.427892055Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.428679907Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.429502242Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.429928639Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.43072574Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T09:56:13.606 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.431531221Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.431846171Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.432276437Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.435140938Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.435564558Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.435833485Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.436677944Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.438528564Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.441241114Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.444024034Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.444915742Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.447479537Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.45019889Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.452682185Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T09:56:13.607 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.453008511Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.455871846Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.458237144Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.461053275Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.463673812Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.464492629Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.465396045Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.466258099Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.466709545Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.467532271Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.468547311Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.469132848Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.469551379Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.469978323Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:56:13.608 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.470068316Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.470930588Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.471634999Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.47207983Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.472205366Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.47232698Z level=info msg="alerts found to migrate" alerts=0 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.472939908Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.473923723Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.474670046Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.475085826Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.475935074Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.476912765Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.477711065Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.478302338Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T09:56:13.609 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.479201606Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.48002508Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.484370044Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.485288475Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.486151788Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.490436667Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.49472088Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T09:56:13.610 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.495148443Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.496019354Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.497012525Z level=info msg="Executing migration" id="create permission table" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.497752881Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.498201784Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.499104453Z level=info msg="Executing migration" id="create role table" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.499873495Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.500798988Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.5017352Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.504363316Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.507027091Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.51006181Z level=info msg="Executing migration" id="create team role table" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.510514604Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.513384152Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T09:56:13.611 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.516153809Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.518946795Z level=info msg="Executing migration" id="create user role table" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.521503171Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.524214324Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.526961699Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.529792071Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.532343869Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.534931053Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.537784727Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.540572387Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.543371261Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.546142776Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.548841643Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.551473476Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T09:56:13.612 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.554028554Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.556462938Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.559163638Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.561782882Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.564597025Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.567103038Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.569722292Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.572319425Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.574929085Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.577495588Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.580345333Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.582842672Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.585519884Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.588094183Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.590941592Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T09:56:13.613 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.593635892Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.59603737Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.59870659Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.599780974Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.600621133Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.601438818Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.601525167Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.602266682Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.602349102Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:56:13.614 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.603108256Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.603903166Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.604716859Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.605542635Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.60639258Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.607173194Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.607960726Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.608796578Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.609609548Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.610446976Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T09:56:14.017 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.611266755Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.61205866Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.615093577Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.6160868Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.617071989Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.61801189Z level=info msg="Executing migration" id="add share column" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.619139638Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.619994438Z level=info msg="Executing migration" id="create file table" 2024-09-21T09:56:14.018 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.620952486Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.621637961Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.622606208Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.623316519Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.624170051Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.625032192Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.625734393Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.626571041Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.627403485Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.628485072Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.629357502Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.630156665Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T09:56:14.019 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.630980114Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.633809428Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.636701287Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.639000222Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.641734489Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.644608192Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.654411021Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.657228505Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.660060139Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.663521218Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.666500299Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.668989189Z level=info msg="Executing migration" id="create folder table" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.671773522Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T09:56:14.020 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.674332921Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.677239716Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.679964605Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=migrator t=2024-09-21T09:56:13.682740611Z level=info msg="migrations completed" performed=485 skipped=0 duration=677.563858ms 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore t=2024-09-21T09:56:13.683145308Z level=info msg="Created default organization" 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=plugin.loader t=2024-09-21T09:56:13.74108381Z level=info msg="Plugin registered" pluginID=input 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=plugin.loader t=2024-09-21T09:56:13.769906668Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=plugin.loader t=2024-09-21T09:56:13.769936045Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=secrets t=2024-09-21T09:56:13.769971507Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=query_data t=2024-09-21T09:56:13.771649144Z level=info msg="Query Service initialization" 2024-09-21T09:56:14.021 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=live.push_http t=2024-09-21T09:56:13.773934963Z level=info msg="Live Push Gateway initialization" 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: Deploying daemon node-exporter.a on smithi119 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: Deploying daemon node-exporter.a on smithi119 2024-09-21T09:56:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: Deploying daemon node-exporter.a on smithi119 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: pgmap v22: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:14.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=infra.usagestats.collector t=2024-09-21T09:56:14.481028763Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T09:56:14.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=provisioning.datasources t=2024-09-21T09:56:14.481337026Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-21T09:56:14.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=provisioning.datasources t=2024-09-21T09:56:14.51516748Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T09:56:14.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=provisioning.datasources t=2024-09-21T09:56:14.54692413Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-21T09:56:14.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=provisioning.alerting t=2024-09-21T09:56:14.568297881Z level=info msg="starting to provision alerting" 2024-09-21T09:56:14.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=provisioning.alerting t=2024-09-21T09:56:14.56831765Z level=info msg="finished to provision alerting" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=ngalert.state.manager t=2024-09-21T09:56:14.568459685Z level=info msg="Warming state cache for startup" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=grafanaStorageLogger t=2024-09-21T09:56:14.568504247Z level=info msg="storage starting" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=http.server t=2024-09-21T09:56:14.570240235Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore.transactions t=2024-09-21T09:56:14.579245947Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=ngalert.state.manager t=2024-09-21T09:56:14.579332272Z level=info msg="State cache has been initialized" states=0 duration=10.871507ms 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=ticker t=2024-09-21T09:56:14.579410779Z level=info msg=starting first_tick=2024-09-21T09:56:20Z 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=ngalert.multiorg.alertmanager t=2024-09-21T09:56:14.579404311Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore.transactions t=2024-09-21T09:56:14.629480188Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore.transactions t=2024-09-21T09:56:14.640695658Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T09:56:14.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 09:56:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[64259]: logger=sqlstore.transactions t=2024-09-21T09:56:14.678077871Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T09:56:14.796 INFO:teuthology.orchestra.run.smithi119.stdout:158913789976 2024-09-21T09:56:14.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:56:16.013 INFO:teuthology.orchestra.run.smithi119.stdout:34359738418 2024-09-21T09:56:16.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:56:16.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:16 smithi119 ceph-mon[40785]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:16.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:16 smithi119 ceph-mon[29367]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:16 smithi145 ceph-mon[33206]: pgmap v23: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:17.178 INFO:teuthology.orchestra.run.smithi119.stdout:111669149730 2024-09-21T09:56:17.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:56:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:18 smithi145 ceph-mon[33206]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:18.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:18 smithi119 ceph-mon[40785]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:18 smithi119 ceph-mon[29367]: pgmap v24: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:18.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:18 smithi119 bash[73377]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:56:18.668 INFO:teuthology.orchestra.run.smithi119.stdout:184683593748 2024-09-21T09:56:18.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:56:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:20 smithi119 ceph-mon[29367]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:20 smithi119 ceph-mon[40785]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:20.367 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:20 smithi119 bash[73377]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:56:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:20 smithi145 ceph-mon[33206]: pgmap v25: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:20.866 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:20 smithi119 bash[73377]: Writing manifest to image destination 2024-09-21T09:56:21.367 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 podman[73377]: 2024-09-21 09:56:21.002747065 +0000 UTC m=+3.622153016 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:56:21.367 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 podman[73377]: 2024-09-21 09:56:21.13922979 +0000 UTC m=+3.758635732 container create 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:21.392 INFO:teuthology.orchestra.run.smithi119.stdout:77309411368 2024-09-21T09:56:21.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 podman[73377]: 2024-09-21 09:56:21.476512981 +0000 UTC m=+4.095918931 container init 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 podman[73377]: 2024-09-21 09:56:21.481966714 +0000 UTC m=+4.101372657 container start 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.486Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.486Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.486Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.486Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:56:21.848 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:56:21.849 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:56:21.850 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:56:21.851 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:56:21.852 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[73713]: ts=2024-09-21T09:56:21.487Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:56:22.263 INFO:teuthology.orchestra.run.smithi119.stdout:210453397520 2024-09-21T09:56:22.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:56:22.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:22 smithi119 ceph-mon[29367]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:22.507 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:22 smithi119 ceph-mon[40785]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:22.520 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:22 smithi145 ceph-mon[33206]: pgmap v26: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:22.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:56:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:56:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:56:23.389 INFO:teuthology.orchestra.run.smithi119.stdout:133143986206 2024-09-21T09:56:23.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:56:23.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:23 smithi119 bash[73377]: 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 2024-09-21T09:56:23.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 09:56:23 smithi119 systemd[1]: Started Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:23.783 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:23.793 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:23.801 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:23.801 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:24 smithi145 ceph-mon[33206]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:24 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:24 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:24 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:24 smithi145 ceph-mon[33206]: Deploying daemon node-exporter.b on smithi145 2024-09-21T09:56:25.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:24 smithi145 bash[64527]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[40785]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[40785]: Deploying daemon node-exporter.b on smithi145 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[29367]: pgmap v27: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:25.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:25.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:24 smithi119 ceph-mon[29367]: Deploying daemon node-exporter.b on smithi145 2024-09-21T09:56:25.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:25 smithi145 bash[64527]: Getting image source signatures 2024-09-21T09:56:25.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:25 smithi145 bash[64527]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T09:56:25.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:25 smithi145 bash[64527]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T09:56:25.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:25 smithi145 bash[64527]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T09:56:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:26 smithi119 ceph-mon[29367]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:26.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:26 smithi119 ceph-mon[40785]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:26 smithi145 ceph-mon[33206]: pgmap v28: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:28.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:27 smithi145 bash[64527]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T09:56:28.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:27 smithi145 bash[64527]: Writing manifest to image destination 2024-09-21T09:56:28.388 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 podman[64527]: 2024-09-21 09:56:28.035462784 +0000 UTC m=+3.228810899 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T09:56:28.388 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 podman[64527]: 2024-09-21 09:56:28.367133344 +0000 UTC m=+3.560481512 container create 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 podman[64527]: 2024-09-21 09:56:28.619305843 +0000 UTC m=+3.812653975 container init 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 podman[64527]: 2024-09-21 09:56:28.625193465 +0000 UTC m=+3.818541581 container start 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.629Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.629Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.629Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.629Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T09:56:28.684 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T09:56:28.685 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T09:56:28.686 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T09:56:28.687 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T09:56:28.688 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T09:56:28.689 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T09:56:28.689 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.630Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T09:56:28.689 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.631Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T09:56:28.689 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[64602]: ts=2024-09-21T09:56:28.631Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T09:56:28.713 INFO:teuthology.orchestra.run.smithi119.stdout:55834574894 2024-09-21T09:56:28.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:56:28.961 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:28 smithi119 ceph-mon[29367]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:28.961 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:28 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:28.962 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:28 smithi119 ceph-mon[40785]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:28.962 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:28 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-mon[33206]: pgmap v29: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:28 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:29.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 bash[64527]: 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e 2024-09-21T09:56:29.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 09:56:28 smithi145 systemd[1]: Started Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:29.423 INFO:teuthology.orchestra.run.smithi119.stdout:34359738420 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:29 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3523625187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:56:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3523625187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:56:30.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: pgmap v30: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:30.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:29 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3523625187' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:56:30.516 INFO:teuthology.orchestra.run.smithi119.stdout:111669149733 2024-09-21T09:56:30.989 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:30 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1599524704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:56:30.989 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:30 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1599524704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:56:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:30 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1599524704' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:56:31.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149733 for osd.3 2024-09-21T09:56:31.306 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:31.957 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:31 smithi119 ceph-mon[29367]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:31.958 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:31 smithi119 ceph-mon[40785]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:32.103 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:31 smithi145 ceph-mon[33206]: pgmap v31: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:32.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738420 for osd.0 2024-09-21T09:56:32.354 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:33.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:56:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:56:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:56:33.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:33 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:33 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:33 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:33 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:33 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:33 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:33 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:33 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:33 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:34.469 INFO:teuthology.orchestra.run.smithi119.stdout:210453397522 2024-09-21T09:56:34.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:34 smithi119 ceph-mon[40785]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:34.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:34 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1676059865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:56:34.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:34 smithi119 ceph-mon[29367]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:34.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:34 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1676059865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:56:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:34 smithi145 ceph-mon[33206]: pgmap v32: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:34 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1676059865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:56:35.197 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397520 got 210453397522 for osd.7 2024-09-21T09:56:35.197 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:35.838 INFO:teuthology.orchestra.run.smithi119.stdout:158913789981 2024-09-21T09:56:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:36 smithi145 ceph-mon[33206]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:36 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3701824602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:56:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:36 smithi119 ceph-mon[29367]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:36 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3701824602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:56:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:36 smithi119 ceph-mon[40785]: pgmap v33: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:36 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3701824602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:56:37.197 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789976 got 158913789981 for osd.5 2024-09-21T09:56:37.198 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:37.553 INFO:teuthology.orchestra.run.smithi119.stdout:133143986209 2024-09-21T09:56:37.756 INFO:teuthology.orchestra.run.smithi119.stdout:77309411372 2024-09-21T09:56:38.097 INFO:teuthology.orchestra.run.smithi119.stdout:184683593752 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[29367]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/58216342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2288162240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2985549418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[40785]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/58216342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2288162240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:56:38.656 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:38 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2985549418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:56:38.679 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986206 got 133143986209 for osd.4 2024-09-21T09:56:38.679 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:38 smithi145 ceph-mon[33206]: pgmap v34: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:38 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/58216342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:56:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:38 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2288162240' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:56:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:38 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2985549418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:56:39.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411372 for osd.2 2024-09-21T09:56:39.444 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:39.631 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:40.344 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593748 got 184683593752 for osd.6 2024-09-21T09:56:40.344 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:40.743 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:40 smithi119 ceph-mon[29367]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:40.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:40 smithi119 ceph-mon[40785]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:40.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:40 smithi145 ceph-mon[33206]: pgmap v35: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:41.852 INFO:teuthology.orchestra.run.smithi119.stdout:55834574898 2024-09-21T09:56:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:42 smithi119 ceph-mon[29367]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:42 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1324482464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:56:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:42 smithi119 ceph-mon[40785]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:42 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1324482464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:56:42.670 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574898 for osd.1 2024-09-21T09:56:42.670 DEBUG:teuthology.parallel:result is None 2024-09-21T09:56:42.670 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:56:42.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:56:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:42 smithi145 ceph-mon[33206]: pgmap v36: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:42 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1324482464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:56:42.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:56:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:56:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:56:43.263 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:43.652 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:43.653 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:44.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:44 smithi119 ceph-mon[29367]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:44.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:44 smithi119 ceph-mon[40785]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:44 smithi145 ceph-mon[33206]: pgmap v37: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:45.106 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:56:45.106 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T09:56:45.942 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-21T09:56:43.128529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385160,"kb_used_data":6312,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576056,"statfs":{"total":767960285184,"available":765517881344,"internally_reserved":0,"allocated":6463488,"data_stored":3735149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002059"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.233800+0000","last_change":"2024-09-21T09:55:46.174897+0000","last_active":"2024-09-21T09:55:53.233800+0000","last_peered":"2024-09-21T09:55:53.233800+0000","last_clean":"2024-09-21T09:55:53.233800+0000","last_became_active":"2024-09-21T09:55:41.640876+0000","last_became_peered":"2024-09-21T09:55:41.640876+0000","last_unstale":"2024-09-21T09:55:53.233800+0000","last_undegraded":"2024-09-21T09:55:53.233800+0000","last_fullsized":"2024-09-21T09:55:53.233800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:47.920765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.112252+0000","last_change":"2024-09-21T09:55:46.170602+0000","last_active":"2024-09-21T09:55:53.112252+0000","last_peered":"2024-09-21T09:55:53.112252+0000","last_clean":"2024-09-21T09:55:53.112252+0000","last_became_active":"2024-09-21T09:55:41.639796+0000","last_became_peered":"2024-09-21T09:55:41.639796+0000","last_unstale":"2024-09-21T09:55:53.112252+0000","last_undegraded":"2024-09-21T09:55:53.112252+0000","last_fullsized":"2024-09-21T09:55:53.112252+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:13.050021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000210194,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.250604+0000","last_change":"2024-09-21T09:55:46.321183+0000","last_active":"2024-09-21T09:55:53.250604+0000","last_peered":"2024-09-21T09:55:53.250604+0000","last_clean":"2024-09-21T09:55:53.250604+0000","last_became_active":"2024-09-21T09:55:41.914202+0000","last_became_peered":"2024-09-21T09:55:41.914202+0000","last_unstale":"2024-09-21T09:55:53.250604+0000","last_undegraded":"2024-09-21T09:55:53.250604+0000","last_fullsized":"2024-09-21T09:55:53.250604+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:39.713100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.112127+0000","last_change":"2024-09-21T09:55:46.170662+0000","last_active":"2024-09-21T09:55:53.112127+0000","last_peered":"2024-09-21T09:55:53.112127+0000","last_clean":"2024-09-21T09:55:53.112127+0000","last_became_active":"2024-09-21T09:55:41.914180+0000","last_became_peered":"2024-09-21T09:55:41.914180+0000","last_unstale":"2024-09-21T09:55:53.112127+0000","last_undegraded":"2024-09-21T09:55:53.112127+0000","last_fullsized":"2024-09-21T09:55:53.112127+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:50.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":21,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.367684+0000","last_change":"2024-09-21T09:55:46.169743+0000","last_active":"2024-09-21T09:55:53.367684+0000","last_peered":"2024-09-21T09:55:53.367684+0000","last_clean":"2024-09-21T09:55:53.367684+0000","last_became_active":"2024-09-21T09:55:41.641386+0000","last_became_peered":"2024-09-21T09:55:41.641386+0000","last_unstale":"2024-09-21T09:55:53.367684+0000","last_undegraded":"2024-09-21T09:55:53.367684+0000","last_fullsized":"2024-09-21T09:55:53.367684+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:03.000891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.367582+0000","last_change":"2024-09-21T09:55:46.165011+0000","last_active":"2024-09-21T09:55:53.367582+0000","last_peered":"2024-09-21T09:55:53.367582+0000","last_clean":"2024-09-21T09:55:53.367582+0000","last_became_active":"2024-09-21T09:55:41.914164+0000","last_became_peered":"2024-09-21T09:55:41.914164+0000","last_unstale":"2024-09-21T09:55:53.367582+0000","last_undegraded":"2024-09-21T09:55:53.367582+0000","last_fullsized":"2024-09-21T09:55:53.367582+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:55.377328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.250575+0000","last_change":"2024-09-21T09:55:46.321119+0000","last_active":"2024-09-21T09:55:53.250575+0000","last_peered":"2024-09-21T09:55:53.250575+0000","last_clean":"2024-09-21T09:55:53.250575+0000","last_became_active":"2024-09-21T09:55:41.914064+0000","last_became_peered":"2024-09-21T09:55:41.914064+0000","last_unstale":"2024-09-21T09:55:53.250575+0000","last_undegraded":"2024-09-21T09:55:53.250575+0000","last_fullsized":"2024-09-21T09:55:53.250575+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:38.386562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.367690+0000","last_change":"2024-09-21T09:55:46.169262+0000","last_active":"2024-09-21T09:55:53.367690+0000","last_peered":"2024-09-21T09:55:53.367690+0000","last_clean":"2024-09-21T09:55:53.367690+0000","last_became_active":"2024-09-21T09:55:41.641506+0000","last_became_peered":"2024-09-21T09:55:41.641506+0000","last_unstale":"2024-09-21T09:55:53.367690+0000","last_undegraded":"2024-09-21T09:55:53.367690+0000","last_fullsized":"2024-09-21T09:55:53.367690+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:21.952844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:55:46.320944+0000","last_change":"2024-09-21T09:55:13.590990+0000","last_active":"2024-09-21T09:55:46.320944+0000","last_peered":"2024-09-21T09:55:46.320944+0000","last_clean":"2024-09-21T09:55:46.320944+0000","last_became_active":"2024-09-21T09:55:13.585775+0000","last_became_peered":"2024-09-21T09:55:13.585775+0000","last_unstale":"2024-09-21T09:55:46.320944+0000","last_undegraded":"2024-09-21T09:55:46.320944+0000","last_fullsized":"2024-09-21T09:55:46.320944+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:07.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397524,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.442},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.317}]},{"osd":1,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.45800000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.42099999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.35699999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.379},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48499999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.498}]},{"osd":6,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.51300000000000001}]}]},{"osd":6,"up_from":43,"seq":184683593753,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":835728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.59699999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.629}]},{"osd":3,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.64100000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.48299999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.60999999999999999}]}]},{"osd":5,"up_from":37,"seq":158913789982,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49399999999999999,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65900000000000003,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.503},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.61199999999999999,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.40899999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56100000000000005,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61399999999999999,"5min":1.327,"15min":1.327},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.64100000000000001,"5min":1.26,"15min":1.26},"last":0.56499999999999995}]},{"osd":3,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63400000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63300000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.28399999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69299999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.497}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.48699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68300000000000005,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.375},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69199999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69299999999999995,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.65100000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.443}]},{"osd":3,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64200000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.39100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71699999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.42299999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.38300000000000001,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.501}]},{"osd":6,"last update":"Sat Sep 21 09:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Sep 21 09:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56899999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53100000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.39000000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.505,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.379,"15min":0.38100000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.50800000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.312,"5min":0.254,"15min":0.254},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.435}]},{"osd":2,"last update":"Sat Sep 21 09:55:39 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.30399999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.53000000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.54500000000000004,"5min":0.625,"15min":0.625},"last":0.36099999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58799999999999997,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.5},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59899999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86299999999999999,"5min":0.876,"15min":0.876},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.48199999999999998}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.52800000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.505},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.35399999999999998},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.28100000000000003}]},{"osd":1,"last update":"Sat Sep 21 09:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35799999999999998,"15min":0.34899999999999998},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.317}]},{"osd":3,"last update":"Sat Sep 21 09:55:37 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.41199999999999998,"15min":0.40699999999999997},"min":{"1min":0.32700000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.307},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.435,"15min":0.433},"min":{"1min":0.33000000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34899999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.66700000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.497}]},{"osd":5,"last update":"Sat Sep 21 09:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.52600000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.47299999999999998}]},{"osd":7,"last update":"Sat Sep 21 09:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447228,"statfs":{"total":95995035648,"available":95689961472,"internally_reserved":0,"allocated":581632,"data_stored":245045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34000000000000002,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.312},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.315},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.497,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.33200000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38,"15min":0.38},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.58599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.372},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.379,"15min":0.379},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.26000000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59299999999999997,"5min":0.624,"15min":0.624},"last":0.438},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.436},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.35499999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.70099999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59199999999999997}]},{"osd":5,"last update":"Sat Sep 21 09:56:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.57999999999999996}]},{"osd":6,"last update":"Sat Sep 21 09:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67900000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56999999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.32100000000000001},"min":{"1min":0.22700000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.249},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34300000000000003,"15min":0.32200000000000001},"min":{"1min":0.26200000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60699999999999998,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.312}]},{"osd":2,"last update":"Sat Sep 21 09:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.37,"15min":0.36399999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.47899999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.28199999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.32700000000000001}]},{"osd":3,"last update":"Sat Sep 21 09:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.437,"15min":0.438},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.45500000000000002,"15min":0.45300000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.35799999999999998}]},{"osd":4,"last update":"Sat Sep 21 09:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84799999999999998,"5min":0.88,"15min":0.88},"last":0.438}]},{"osd":5,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.625,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.51300000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.55900000000000005}]},{"osd":7,"last update":"Sat Sep 21 09:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:56:45.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:56:46.324 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:46.738 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:46 smithi119 ceph-mon[29367]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:56:46.738 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:46 smithi119 ceph-mon[29367]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:46.738 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:46 smithi119 ceph-mon[40785]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:56:46.738 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:46 smithi119 ceph-mon[40785]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:46 smithi145 ceph-mon[33206]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:56:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:46 smithi145 ceph-mon[33206]: pgmap v38: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:48.925 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:56:48.925 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[40785]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[29367]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:48.936 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:48.937 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:48.937 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:48.937 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:48 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:48 smithi145 ceph-mon[33206]: pgmap v39: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:48 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:48 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:48 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:56:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:48 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:56:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:48 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:49 smithi119 ceph-mon[29367]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:56:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:49 smithi119 ceph-mon[29367]: Reconfiguring daemon alertmanager.a on smithi119 2024-09-21T09:56:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:49 smithi119 ceph-mon[40785]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:56:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:49 smithi119 ceph-mon[40785]: Reconfiguring daemon alertmanager.a on smithi119 2024-09-21T09:56:49.909 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-21T09:56:47.129162+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":115,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":27,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385160,"kb_used_data":6312,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747576056,"statfs":{"total":767960285184,"available":765517881344,"internally_reserved":0,"allocated":6463488,"data_stored":3735149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001914"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.233800+0000","last_change":"2024-09-21T09:55:46.174897+0000","last_active":"2024-09-21T09:55:53.233800+0000","last_peered":"2024-09-21T09:55:53.233800+0000","last_clean":"2024-09-21T09:55:53.233800+0000","last_became_active":"2024-09-21T09:55:41.640876+0000","last_became_peered":"2024-09-21T09:55:41.640876+0000","last_unstale":"2024-09-21T09:55:53.233800+0000","last_undegraded":"2024-09-21T09:55:53.233800+0000","last_fullsized":"2024-09-21T09:55:53.233800+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:47.920765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.112252+0000","last_change":"2024-09-21T09:55:46.170602+0000","last_active":"2024-09-21T09:55:53.112252+0000","last_peered":"2024-09-21T09:55:53.112252+0000","last_clean":"2024-09-21T09:55:53.112252+0000","last_became_active":"2024-09-21T09:55:41.639796+0000","last_became_peered":"2024-09-21T09:55:41.639796+0000","last_unstale":"2024-09-21T09:55:53.112252+0000","last_undegraded":"2024-09-21T09:55:53.112252+0000","last_fullsized":"2024-09-21T09:55:53.112252+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:13.050021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000210194,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.250604+0000","last_change":"2024-09-21T09:55:46.321183+0000","last_active":"2024-09-21T09:55:53.250604+0000","last_peered":"2024-09-21T09:55:53.250604+0000","last_clean":"2024-09-21T09:55:53.250604+0000","last_became_active":"2024-09-21T09:55:41.914202+0000","last_became_peered":"2024-09-21T09:55:41.914202+0000","last_unstale":"2024-09-21T09:55:53.250604+0000","last_undegraded":"2024-09-21T09:55:53.250604+0000","last_fullsized":"2024-09-21T09:55:53.250604+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:39.713100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.112127+0000","last_change":"2024-09-21T09:55:46.170662+0000","last_active":"2024-09-21T09:55:53.112127+0000","last_peered":"2024-09-21T09:55:53.112127+0000","last_clean":"2024-09-21T09:55:53.112127+0000","last_became_active":"2024-09-21T09:55:41.914180+0000","last_became_peered":"2024-09-21T09:55:41.914180+0000","last_unstale":"2024-09-21T09:55:53.112127+0000","last_undegraded":"2024-09-21T09:55:53.112127+0000","last_fullsized":"2024-09-21T09:55:53.112127+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:50.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":21,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.367684+0000","last_change":"2024-09-21T09:55:46.169743+0000","last_active":"2024-09-21T09:55:53.367684+0000","last_peered":"2024-09-21T09:55:53.367684+0000","last_clean":"2024-09-21T09:55:53.367684+0000","last_became_active":"2024-09-21T09:55:41.641386+0000","last_became_peered":"2024-09-21T09:55:41.641386+0000","last_unstale":"2024-09-21T09:55:53.367684+0000","last_undegraded":"2024-09-21T09:55:53.367684+0000","last_fullsized":"2024-09-21T09:55:53.367684+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:03.000891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.367582+0000","last_change":"2024-09-21T09:55:46.165011+0000","last_active":"2024-09-21T09:55:53.367582+0000","last_peered":"2024-09-21T09:55:53.367582+0000","last_clean":"2024-09-21T09:55:53.367582+0000","last_became_active":"2024-09-21T09:55:41.914164+0000","last_became_peered":"2024-09-21T09:55:41.914164+0000","last_unstale":"2024-09-21T09:55:53.367582+0000","last_undegraded":"2024-09-21T09:55:53.367582+0000","last_fullsized":"2024-09-21T09:55:53.367582+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:55.377328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.250575+0000","last_change":"2024-09-21T09:55:46.321119+0000","last_active":"2024-09-21T09:55:53.250575+0000","last_peered":"2024-09-21T09:55:53.250575+0000","last_clean":"2024-09-21T09:55:53.250575+0000","last_became_active":"2024-09-21T09:55:41.914064+0000","last_became_peered":"2024-09-21T09:55:41.914064+0000","last_unstale":"2024-09-21T09:55:53.250575+0000","last_undegraded":"2024-09-21T09:55:53.250575+0000","last_fullsized":"2024-09-21T09:55:53.250575+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:38.386562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":20,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T09:55:53.367690+0000","last_change":"2024-09-21T09:55:46.169262+0000","last_active":"2024-09-21T09:55:53.367690+0000","last_peered":"2024-09-21T09:55:53.367690+0000","last_clean":"2024-09-21T09:55:53.367690+0000","last_became_active":"2024-09-21T09:55:41.641506+0000","last_became_peered":"2024-09-21T09:55:41.641506+0000","last_unstale":"2024-09-21T09:55:53.367690+0000","last_undegraded":"2024-09-21T09:55:53.367690+0000","last_fullsized":"2024-09-21T09:55:53.367690+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:21.952844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":48,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-09-21T09:55:46.320944+0000","last_change":"2024-09-21T09:55:13.590990+0000","last_active":"2024-09-21T09:55:46.320944+0000","last_peered":"2024-09-21T09:55:46.320944+0000","last_clean":"2024-09-21T09:55:46.320944+0000","last_became_active":"2024-09-21T09:55:13.585775+0000","last_became_peered":"2024-09-21T09:55:13.585775+0000","last_unstale":"2024-09-21T09:55:46.320944+0000","last_undegraded":"2024-09-21T09:55:46.320944+0000","last_fullsized":"2024-09-21T09:55:46.320944+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:07.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.37}]},{"osd":2,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.441},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.432}]},{"osd":3,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.38900000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.35599999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:56:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.41899999999999998}]}]},{"osd":6,"up_from":43,"seq":184683593754,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":835728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.57599999999999996}]},{"osd":1,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.60599999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.629},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.72899999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.498}]},{"osd":5,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.443},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.45200000000000001}]},{"osd":7,"last update":"Sat Sep 21 09:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.52200000000000002}]}]},{"osd":5,"up_from":37,"seq":158913789983,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.438},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.49399999999999999,"15min":0.495},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.65900000000000003,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.42499999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.502,"15min":0.503},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.69999999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.55800000000000005,"15min":0.56599999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.61199999999999999,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56100000000000005,"15min":0.57399999999999995},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.61399999999999999,"5min":1.327,"15min":1.327},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.64100000000000001,"5min":1.26,"15min":1.26},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.50700000000000001},"min":{"1min":0.34399999999999997,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.63400000000000001,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.63300000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.39600000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.28399999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.44,"15min":0.442},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69299999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.54400000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.503}]},{"osd":7,"last update":"Sat Sep 21 09:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.38600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.68300000000000005,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.375},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69199999999999995,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.69299999999999995,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55500000000000005}]},{"osd":2,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.52800000000000002,"15min":0.53600000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.65100000000000002,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.443}]},{"osd":3,"last update":"Sat Sep 21 09:56:00 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.64200000000000002,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.39100000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71699999999999997,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.42299999999999999}]},{"osd":5,"last update":"Sat Sep 21 09:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.38300000000000001,"15min":0.37},"min":{"1min":0.30099999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.39800000000000002,"15min":0.38600000000000001},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.501}]},{"osd":6,"last update":"Sat Sep 21 09:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.623}]},{"osd":7,"last update":"Sat Sep 21 09:56:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56899999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38400000000000001,"15min":0.40000000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.57599999999999996,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.36199999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.36599999999999999,"15min":0.377},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.42199999999999999,"5min":0.51100000000000001,"15min":0.51100000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.26000000000000001,"5min":0.254,"15min":0.254},"max":{"1min":0.502,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.38100000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.44600000000000001,"15min":0.45800000000000002},"min":{"1min":0.28599999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61199999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.39700000000000002,"15min":0.40100000000000002},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.318}]},{"osd":4,"last update":"Sat Sep 21 09:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.58799999999999997,"15min":0.58499999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57499999999999996,"15min":0.57099999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.47299999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58399999999999996,"15min":0.57999999999999996},"min":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.86299999999999999,"5min":0.876,"15min":0.876},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56299999999999994,"15min":0.55900000000000005},"min":{"1min":0.45300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.55200000000000005}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56399999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.58799999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297928,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447224,"statfs":{"total":95995035648,"available":95689957376,"internally_reserved":0,"allocated":585728,"data_stored":245656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36099999999999999,"15min":0.35399999999999998},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.54400000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.35799999999999998,"15min":0.34899999999999998},"min":{"1min":0.26100000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.38200000000000001,"15min":0.373},"min":{"1min":0.27000000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.40799999999999997,"15min":0.40600000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.54400000000000004,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42599999999999999,"15min":0.42999999999999999},"min":{"1min":0.29799999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:56:02 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.45200000000000001,"5min":0.442,"15min":0.442},"max":{"1min":0.66700000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55200000000000005,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.65400000000000003}]},{"osd":5,"last update":"Sat Sep 21 09:56:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.67400000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:55:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.622},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.68700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297924,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447228,"statfs":{"total":95995035648,"available":95689961472,"internally_reserved":0,"allocated":581632,"data_stored":245045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:55:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34000000000000002,"15min":0.32400000000000001},"min":{"1min":0.20399999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.315},"min":{"1min":0.223,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.497,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.40799999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:56:18 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.38,"15min":0.38},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.58599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.379,"15min":0.379},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.55000000000000004,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.33800000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:55:40 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.32600000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59299999999999997,"5min":0.624,"15min":0.624},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.435,"15min":0.436},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:56:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.70099999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.55100000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:56:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56299999999999994,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.495},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54300000000000004,"15min":0.54900000000000004},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.54300000000000004}]},{"osd":6,"last update":"Sat Sep 21 09:55:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.56599999999999995}]},{"osd":7,"last update":"Sat Sep 21 09:56:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.64700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298508,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446644,"statfs":{"total":95995035648,"available":95689363456,"internally_reserved":0,"allocated":1179648,"data_stored":836024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.34300000000000003,"15min":0.32100000000000001},"min":{"1min":0.22700000000000001,"5min":0.158,"15min":0.158},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34300000000000003,"15min":0.32200000000000001},"min":{"1min":0.26200000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.60699999999999998,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.29499999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.37,"15min":0.36399999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.47899999999999998,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.25},{"interface":"front","average":{"1min":0.437,"5min":0.40999999999999998,"15min":0.39900000000000002},"min":{"1min":0.28199999999999997,"5min":0.224,"15min":0.224},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35599999999999998}]},{"osd":3,"last update":"Sat Sep 21 09:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44800000000000001,"15min":0.45100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40500000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78300000000000003,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84799999999999998,"5min":0.88,"15min":0.88},"last":0.54700000000000004}]},{"osd":5,"last update":"Sat Sep 21 09:56:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61699999999999999,"15min":0.624},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.627},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.625,"15min":0.64000000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76100000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Sat Sep 21 09:55:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.52200000000000002}]},{"osd":7,"last update":"Sat Sep 21 09:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:56:49.911 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:56:49.911 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:56:49.911 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:56:49.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:56:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:49 smithi145 ceph-mon[33206]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:56:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:49 smithi145 ceph-mon[33206]: Reconfiguring daemon alertmanager.a on smithi119 2024-09-21T09:56:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:50 smithi119 ceph-mon[29367]: from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:56:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:50 smithi119 ceph-mon[29367]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:50 smithi119 ceph-mon[40785]: from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:56:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:50 smithi119 ceph-mon[40785]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:50 smithi145 ceph-mon[33206]: from='client.24515 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:56:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:50 smithi145 ceph-mon[33206]: pgmap v40: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:52.455 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:52.616 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:52 smithi119 systemd[1]: Stopping Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:53.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:52 smithi119 ceph-mon[29367]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:53.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:52 smithi119 ceph-mon[40785]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:52 smithi145 ceph-mon[33206]: pgmap v41: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:53.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:56:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:56:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:56:53.867 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:53 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[71454]: ts=2024-09-21T09:56:53.776Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:56:54.471 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:54 smithi119 podman[76637]: 2024-09-21 09:56:54.126282131 +0000 UTC m=+0.484647312 container died 82b6fcd6eeb121fff156d21a268f46af8ec7e6e4373175cdc0f1d9f5c13d26bc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:54.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:54 smithi119 ceph-mon[29367]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:54.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:54 smithi119 ceph-mon[40785]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:54.821 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:54 smithi119 podman[76637]: 2024-09-21 09:56:54.568006699 +0000 UTC m=+0.926371868 container cleanup 82b6fcd6eeb121fff156d21a268f46af8ec7e6e4373175cdc0f1d9f5c13d26bc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:54.821 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:54 smithi119 bash[76637]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T09:56:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:54 smithi145 ceph-mon[33206]: pgmap v42: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:55.116 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:54 smithi119 podman[76673]: 2024-09-21 09:56:54.819378315 +0000 UTC m=+1.024463083 container remove 82b6fcd6eeb121fff156d21a268f46af8ec7e6e4373175cdc0f1d9f5c13d26bc (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:55.117 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:54 smithi119 podman[76673]: 2024-09-21 09:56:54.928409645 +0000 UTC m=+1.133494407 volume remove f451e56aac638483bca0ad4416b3161f7a6a3cf238dedd9ffaaf4da7d3624408 2024-09-21T09:56:55.385 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:56:55.385 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:56:55.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:55 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T09:56:55.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:55 smithi119 systemd[1]: Stopped Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:55.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:55 smithi119 systemd[1]: Starting Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:56:55.985 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:55 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3638868769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:56:55.985 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:55 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3638868769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:56:56.009 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:56:56.009 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T09:56:56.009 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T09:56:56.018 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-09-21T09:56:56.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph fs volume create foo' 2024-09-21T09:56:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:55 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3638868769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:56:56.244 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 podman[76855]: 2024-09-21 09:56:56.130475997 +0000 UTC m=+0.016316223 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T09:56:56.554 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 podman[76855]: 2024-09-21 09:56:56.243210072 +0000 UTC m=+0.129050286 volume create f5eccdf0653cca84c61ebaffedab4cfcd9ee88ed9f103d0b023c4f41041680b8 2024-09-21T09:56:56.554 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 podman[76855]: 2024-09-21 09:56:56.335211839 +0000 UTC m=+0.221052052 container create b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:56:56.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-mon[29367]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:56.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-mon[40785]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:56.970 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 podman[76855]: 2024-09-21 09:56:56.663277843 +0000 UTC m=+0.549118060 container init b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 podman[76855]: 2024-09-21 09:56:56.66803293 +0000 UTC m=+0.553873144 container start b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.680Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.680Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.680Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.119 port=9094 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.681Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.704Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.705Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.707Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:56.707Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 bash[76855]: b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb 2024-09-21T09:56:56.971 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:56 smithi119 systemd[1]: Started Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:56:57.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:56 smithi145 ceph-mon[33206]: pgmap v43: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:57 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:57 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:57 smithi145 ceph-mon[33206]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:56:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:57 smithi145 ceph-mon[33206]: Reconfiguring daemon prometheus.a on smithi145 2024-09-21T09:56:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:57 smithi145 ceph-mon[33206]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[40785]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[40785]: Reconfiguring daemon prometheus.a on smithi145 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[40785]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[29367]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[29367]: Reconfiguring daemon prometheus.a on smithi145 2024-09-21T09:56:58.047 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:57 smithi119 ceph-mon[29367]: pgmap v44: 9 pgs: 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:56:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:58 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:58 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:58 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:56:59.117 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:56:58 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:56:58.682Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000134128s 2024-09-21T09:56:59.970 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:56:59.970 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:56:59.970 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:56:59.971 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:56:59.971 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:56:59.971 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:56:59.971 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:56:59.971 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:56:59 smithi145 ceph-mon[33206]: pgmap v46: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:00.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[40785]: pgmap v46: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: from='client.24527 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foo", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]: dispatch 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foo.meta"}]': finished 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:00.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-mon[29367]: pgmap v46: 41 pgs: 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:00.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:56:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[29343]: 2024-09-21T09:56:59.993+0000 7fb536571640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:57:00.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 systemd[1]: Stopping Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T09:57:00.709 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T09:57:00.710 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T09:57:00.710 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T09:57:00.710 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T09:57:00.710 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T09:57:00.710 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[62350]: ts=2024-09-21T09:57:00.425Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T09:57:00.710 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 podman[65484]: 2024-09-21 09:57:00.435707677 +0000 UTC m=+0.142168913 container died 7f5ace8445883a7d5713a93b100f7d26436ab0aefa59305222c87e8881163b0c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:00.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-21T09:57:01.042 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 podman[65484]: 2024-09-21 09:57:00.814648777 +0000 UTC m=+0.521110012 container cleanup 7f5ace8445883a7d5713a93b100f7d26436ab0aefa59305222c87e8881163b0c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:01.042 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:00 smithi145 bash[65484]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:57:01.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: fsmap foo:0 2024-09-21T09:57:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: Saving service mds.foo spec with placement count:2 2024-09-21T09:57:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:00 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:01.104 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:57:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: fsmap foo:0 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: Saving service mds.foo spec with placement count:2 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.foo.data"}]': finished 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]: dispatch 2024-09-21T09:57:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-21T09:57:01.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-21T09:57:01.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T09:57:01.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "fs new", "fs_name": "foo", "metadata": "cephfs.foo.meta", "data": "cephfs.foo.data"}]': finished 2024-09-21T09:57:01.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: fsmap foo:0 2024-09-21T09:57:01.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: Saving service mds.foo spec with placement count:2 2024-09-21T09:57:01.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:00 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:01.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 podman[65501]: 2024-09-21 09:57:01.041215283 +0000 UTC m=+0.604627568 container remove 7f5ace8445883a7d5713a93b100f7d26436ab0aefa59305222c87e8881163b0c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:01.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T09:57:01.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 systemd[1]: Stopped Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:57:01.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.091s CPU time. 2024-09-21T09:57:01.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 systemd[1]: Starting Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:57:01.657 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 podman[65597]: 2024-09-21 09:57:01.293459755 +0000 UTC m=+0.016838586 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T09:57:01.657 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 podman[65597]: 2024-09-21 09:57:01.392085343 +0000 UTC m=+0.115464217 container create 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:01.657 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 podman[65597]: 2024-09-21 09:57:01.627370764 +0000 UTC m=+0.350749596 container init 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:01.657 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 podman[65597]: 2024-09-21 09:57:01.631940907 +0000 UTC m=+0.355319738 container start 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.655Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.655Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.655Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi145 (none))" 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.655Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.655Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.657Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T09:57:02.010 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.657Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.658Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.658Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.660Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.660Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.68µs 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.660Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.661Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.661Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.661Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.256µs wal_replay_duration=402.776µs wbl_replay_duration=113ns total_replay_duration=438.413µs 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.661Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.661Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.661Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.677Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.269428ms db_storage=785ns remote_storage=944ns web_handler=300ns query_engine=509ns scrape=438.739µs scrape_sd=24.528µs notify=28.801µs notify_sd=6.605µs rules=14.550757ms tracing=7.125µs 2024-09-21T09:57:02.011 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.677Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T09:57:02.012 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T09:57:01.677Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T09:57:02.012 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 bash[65597]: 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 2024-09-21T09:57:02.012 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 09:57:01 smithi145 systemd[1]: Started Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:57:02.093 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:01] ENGINE Bus STOPPING 2024-09-21T09:57:02.093 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:57:02.093 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:01] ENGINE Bus STOPPED 2024-09-21T09:57:02.093 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:01 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:01] ENGINE Bus STARTING 2024-09-21T09:57:02.093 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Serving on http://:::9283 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Bus STARTED 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: pgmap v50: 73 pgs: 32 unknown, 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:02.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: pgmap v50: 73 pgs: 32 unknown, 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:57:02.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: pgmap v50: 73 pgs: 32 unknown, 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:57:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:02.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:02.358 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Bus STOPPING 2024-09-21T09:57:02.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:57:02.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Bus STOPPED 2024-09-21T09:57:02.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Bus STARTING 2024-09-21T09:57:02.961 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Serving on http://:::9283 2024-09-21T09:57:02.962 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Bus STARTED 2024-09-21T09:57:02.962 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:02] ENGINE Bus STOPPING 2024-09-21T09:57:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:57:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:57:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:57:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:57:03.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:57:03.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:57:03.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:03 smithi145 ceph-mon[33206]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:03] ENGINE Bus STOPPED 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:03] ENGINE Bus STARTING 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:57:03.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[40785]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi145.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi145.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:57:03.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-mon[29367]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T09:57:03.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:03] ENGINE Serving on http://:::9283 2024-09-21T09:57:03.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:57:03] ENGINE Bus STARTED 2024-09-21T09:57:03.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-21T09:57:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:04 smithi145 ceph-mon[33206]: pgmap v52: 73 pgs: 32 unknown, 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:04 smithi119 ceph-mon[40785]: pgmap v52: 73 pgs: 32 unknown, 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:04 smithi119 ceph-mon[29367]: pgmap v52: 73 pgs: 32 unknown, 32 creating+peering, 9 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:04.512 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:06.686 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-mon[29367]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:06.686 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:06.686 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:06.686 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-mon[40785]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:06.686 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:06.686 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:06 smithi145 ceph-mon[33206]: pgmap v53: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:06 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:06 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:07.117 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 09:57:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T09:57:06.683Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00181673s 2024-09-21T09:57:07.155 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T09:57:07.166 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T09:57:07.166 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T09:57:07.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T09:57:07.551 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:08 smithi119 ceph-mon[29367]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:08 smithi119 ceph-mon[40785]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:08 smithi145 ceph-mon[33206]: pgmap v54: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:10.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: Deploying daemon mds.foo.smithi119.oikglq on smithi119 2024-09-21T09:57:10.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[40785]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: Deploying daemon mds.foo.smithi119.oikglq on smithi119 2024-09-21T09:57:10.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:09 smithi119 ceph-mon[29367]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:10.179 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:57:10.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:10.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:10.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:10.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:10.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:10.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:57:10.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:10.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: Deploying daemon mds.foo.smithi119.oikglq on smithi119 2024-09-21T09:57:10.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:09 smithi145 ceph-mon[33206]: pgmap v55: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:11.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:10 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1995994816' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:57:11.359 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:10 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1995994816' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:57:11.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:10 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1995994816' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T09:57:11.384 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":20,"flags":0,"active_gid":24347,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1780938388},{"type":"v1","addr":"172.21.15.119:6801","nonce":1780938388}]},"active_addr":"172.21.15.119:6801/1780938388","active_change":"2024-09-21T09:55:43.105595+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24340,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.119:8443/","prometheus":"http://172.21.15.119:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":55,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2566533967}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":49791021}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":547636249}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":2781454362}]}]} 2024-09-21T09:57:11.391 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T09:57:11.391 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T09:57:11.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:57:11.838 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:12.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:11 smithi145 ceph-mon[33206]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:11 smithi119 ceph-mon[29367]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:11 smithi119 ceph-mon[40785]: pgmap v56: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:12.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:57:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:13.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:12 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:12] "GET /metrics HTTP/1.1" 200 203815 "" "Prometheus/2.43.0" 2024-09-21T09:57:14.024 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:57:14.024 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":64,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T09:57:02.000662+0000","last_up_change":"2024-09-21T09:55:11.571016+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:56:58.923308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:56:59.224014+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6803","nonce":2992214218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6805","nonce":2992214218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6809","nonce":2992214218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6807","nonce":2992214218}]},"public_addr":"172.21.15.119:6803/2992214218","cluster_addr":"172.21.15.119:6805/2992214218","heartbeat_back_addr":"172.21.15.119:6809/2992214218","heartbeat_front_addr":"172.21.15.119:6807/2992214218","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6811","nonce":2164564975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6813","nonce":2164564975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6817","nonce":2164564975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6815","nonce":2164564975}]},"public_addr":"172.21.15.119:6811/2164564975","cluster_addr":"172.21.15.119:6813/2164564975","heartbeat_back_addr":"172.21.15.119:6817/2164564975","heartbeat_front_addr":"172.21.15.119:6815/2164564975","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6819","nonce":3612940063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6821","nonce":3612940063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6825","nonce":3612940063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6823","nonce":3612940063}]},"public_addr":"172.21.15.119:6819/3612940063","cluster_addr":"172.21.15.119:6821/3612940063","heartbeat_back_addr":"172.21.15.119:6825/3612940063","heartbeat_front_addr":"172.21.15.119:6823/3612940063","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6827","nonce":3712858818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6829","nonce":3712858818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6833","nonce":3712858818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6831","nonce":3712858818}]},"public_addr":"172.21.15.119:6827/3712858818","cluster_addr":"172.21.15.119:6829/3712858818","heartbeat_back_addr":"172.21.15.119:6833/3712858818","heartbeat_front_addr":"172.21.15.119:6831/3712858818","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:57:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2578595651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:57:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2578595651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:57:14.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:14 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: pgmap v57: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2578595651' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-21T09:57:14.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:14 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:14.839 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T09:57:14.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T09:57:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:15.509 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: Deploying daemon mds.foo.smithi145.tjomfr on smithi145 2024-09-21T09:57:15.509 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:boot 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: daemon mds.foo.smithi119.oikglq assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: fsmap foo:0 1 up:standby 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: fsmap foo:1 {0=foo.smithi119.oikglq=up:creating} 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[29367]: daemon mds.foo.smithi119.oikglq is now active in filesystem foo as rank 0 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: Deploying daemon mds.foo.smithi145.tjomfr on smithi145 2024-09-21T09:57:15.510 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:boot 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: daemon mds.foo.smithi119.oikglq assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: fsmap foo:0 1 up:standby 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:creating} 2024-09-21T09:57:15.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:15 smithi119 ceph-mon[40785]: daemon mds.foo.smithi119.oikglq is now active in filesystem foo as rank 0 2024-09-21T09:57:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: Deploying daemon mds.foo.smithi145.tjomfr on smithi145 2024-09-21T09:57:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:boot 2024-09-21T09:57:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: daemon mds.foo.smithi119.oikglq assigned to filesystem foo as rank 0 (now has 1 ranks) 2024-09-21T09:57:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-21T09:57:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-21T09:57:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:57:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T09:57:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: fsmap foo:0 1 up:standby 2024-09-21T09:57:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:creating} 2024-09-21T09:57:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:15 smithi145 ceph-mon[33206]: daemon mds.foo.smithi119.oikglq is now active in filesystem foo as rank 0 2024-09-21T09:57:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:16 smithi145 ceph-mon[33206]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:16 smithi145 ceph-mon[33206]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:active 2024-09-21T09:57:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:16 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 2024-09-21T09:57:16.305 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:16 smithi119 ceph-mon[40785]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:16.305 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:16 smithi119 ceph-mon[40785]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:active 2024-09-21T09:57:16.305 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:16 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 2024-09-21T09:57:16.305 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:16 smithi119 ceph-mon[29367]: pgmap v58: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:16.306 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:16 smithi119 ceph-mon[29367]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:active 2024-09-21T09:57:16.306 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:16 smithi119 ceph-mon[29367]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 2024-09-21T09:57:17.002 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:57:17.003 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":64,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T09:57:02.000662+0000","last_up_change":"2024-09-21T09:55:11.571016+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:56:58.923308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:56:59.224014+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6803","nonce":2992214218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6805","nonce":2992214218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6809","nonce":2992214218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2992214218},{"type":"v1","addr":"172.21.15.119:6807","nonce":2992214218}]},"public_addr":"172.21.15.119:6803/2992214218","cluster_addr":"172.21.15.119:6805/2992214218","heartbeat_back_addr":"172.21.15.119:6809/2992214218","heartbeat_front_addr":"172.21.15.119:6807/2992214218","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6811","nonce":2164564975}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6813","nonce":2164564975}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6817","nonce":2164564975}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2164564975},{"type":"v1","addr":"172.21.15.119:6815","nonce":2164564975}]},"public_addr":"172.21.15.119:6811/2164564975","cluster_addr":"172.21.15.119:6813/2164564975","heartbeat_back_addr":"172.21.15.119:6817/2164564975","heartbeat_front_addr":"172.21.15.119:6815/2164564975","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6819","nonce":3612940063}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6821","nonce":3612940063}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6825","nonce":3612940063}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3612940063},{"type":"v1","addr":"172.21.15.119:6823","nonce":3612940063}]},"public_addr":"172.21.15.119:6819/3612940063","cluster_addr":"172.21.15.119:6821/3612940063","heartbeat_back_addr":"172.21.15.119:6825/3612940063","heartbeat_front_addr":"172.21.15.119:6823/3612940063","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6827","nonce":3712858818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6829","nonce":3712858818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6833","nonce":3712858818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3712858818},{"type":"v1","addr":"172.21.15.119:6831","nonce":3712858818}]},"public_addr":"172.21.15.119:6827/3712858818","cluster_addr":"172.21.15.119:6829/3712858818","heartbeat_back_addr":"172.21.15.119:6833/3712858818","heartbeat_front_addr":"172.21.15.119:6831/3712858818","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T09:57:17.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T09:57:17.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T09:57:17.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T09:57:17.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T09:57:17.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T09:57:17.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T09:57:17.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T09:57:17.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T09:57:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1030604550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: mds.? [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] up:boot 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:18.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:17 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:18.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1030604550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: mds.? [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] up:boot 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[29367]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.063 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1030604550' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: mds.? [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] up:boot 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:18.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:17 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:18.622 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.624 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.626 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.634 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.634 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.635 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.635 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.667 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:18.974 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:18 smithi119 ceph-mon[29367]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:18.974 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:18 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:18.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:18 smithi119 ceph-mon[40785]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:18.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:18 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:18 smithi145 ceph-mon[33206]: pgmap v59: 73 pgs: 73 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:18 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:19.953 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:19 smithi145 ceph-mon[33206]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:active 2024-09-21T09:57:19.954 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:19 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:19.989 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:19 smithi119 ceph-mon[29367]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:active 2024-09-21T09:57:19.989 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:19 smithi119 ceph-mon[29367]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:19.989 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:19 smithi119 ceph-mon[40785]: mds.? [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] up:active 2024-09-21T09:57:19.989 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:19 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:20.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:20 smithi119 ceph-mon[29367]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:20.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:20 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:20.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:20 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:20 smithi119 ceph-mon[40785]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:20 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:20 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:20 smithi145 ceph-mon[33206]: pgmap v60: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:20 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:20 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-mon[29367]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-mon[29367]: mds.? [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] up:standby 2024-09-21T09:57:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-mon[29367]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-mon[40785]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-mon[40785]: mds.? [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] up:standby 2024-09-21T09:57:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:22.585 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:22 smithi145 ceph-mon[33206]: pgmap v61: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:22.585 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:22 smithi145 ceph-mon[33206]: mds.? [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] up:standby 2024-09-21T09:57:22.585 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:22 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T09:57:22.977 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:22] "GET /metrics HTTP/1.1" 200 208802 "" "Prometheus/2.43.0" 2024-09-21T09:57:23.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:57:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:24.495 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:24 smithi119 ceph-mon[29367]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:24.496 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:24 smithi119 ceph-mon[40785]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:24 smithi145 ceph-mon[33206]: pgmap v62: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:25.857 INFO:teuthology.orchestra.run.smithi119.stdout:55834574907 2024-09-21T09:57:25.857 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T09:57:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:26 smithi119 ceph-mon[40785]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:26 smithi119 ceph-mon[29367]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:26 smithi145 ceph-mon[33206]: pgmap v63: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:28 smithi119 ceph-mon[40785]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:28 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:28 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:28 smithi119 ceph-mon[29367]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:28 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:28 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:28 smithi145 ceph-mon[33206]: pgmap v64: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:28 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:28 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:30 smithi119 ceph-mon[29367]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:30 smithi119 ceph-mon[40785]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:30 smithi145 ceph-mon[33206]: pgmap v65: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-21T09:57:30.841 INFO:teuthology.orchestra.run.smithi119.stdout:133143986220 2024-09-21T09:57:30.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T09:57:32.418 INFO:teuthology.orchestra.run.smithi119.stdout:184683593764 2024-09-21T09:57:32.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T09:57:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:32 smithi119 ceph-mon[29367]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:32 smithi119 ceph-mon[40785]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:32.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:32 smithi145 ceph-mon[33206]: pgmap v66: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:33.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:57:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:33.054 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:32] "GET /metrics HTTP/1.1" 200 239777 "" "Prometheus/2.43.0" 2024-09-21T09:57:33.080 INFO:teuthology.orchestra.run.smithi119.stdout:210453397535 2024-09-21T09:57:33.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T09:57:33.903 INFO:teuthology.orchestra.run.smithi119.stdout:158913789993 2024-09-21T09:57:33.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T09:57:34.573 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:34 smithi119 ceph-mon[29367]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:34.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:34 smithi119 ceph-mon[40785]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:34 smithi145 ceph-mon[33206]: pgmap v67: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:34.974 INFO:teuthology.orchestra.run.smithi119.stdout:34359738435 2024-09-21T09:57:34.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T09:57:35.347 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:35.379 INFO:teuthology.orchestra.run.smithi119.stdout:77309411384 2024-09-21T09:57:35.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T09:57:35.399 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:35.399 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:35.632 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:36.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:36 smithi119 ceph-mon[40785]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:36 smithi119 ceph-mon[29367]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:36 smithi145 ceph-mon[33206]: pgmap v68: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:37.445 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:37.501 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:37.983 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:38.556 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:38 smithi119 ceph-mon[40785]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:38.556 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:38 smithi119 ceph-mon[29367]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:38 smithi145 ceph-mon[33206]: pgmap v69: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:40.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:40 smithi119 ceph-mon[29367]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:40.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:40 smithi119 ceph-mon[40785]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:40.596 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:40 smithi145 ceph-mon[33206]: pgmap v70: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:42.543 INFO:teuthology.orchestra.run.smithi119.stdout:210453397537 2024-09-21T09:57:42.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:42 smithi119 ceph-mon[29367]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:42.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:42 smithi119 ceph-mon[40785]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:42.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:42 smithi145 ceph-mon[33206]: pgmap v71: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:43.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:57:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:43.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:42] "GET /metrics HTTP/1.1" 200 239777 "" "Prometheus/2.43.0" 2024-09-21T09:57:43.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2478529376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:57:43.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-21T09:57:43.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.584 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.584 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-21T09:57:43.584 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.584 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:43.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2478529376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:57:43.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-21T09:57:43.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-21T09:57:43.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2478529376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T09:57:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-21T09:57:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-21T09:57:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T09:57:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[40785]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[40785]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[29367]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T09:57:44.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T09:57:44.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T09:57:44.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:44 smithi119 ceph-mon[29367]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:57:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:44 smithi145 ceph-mon[33206]: pgmap v72: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-09-21T09:57:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-21T09:57:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T09:57:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:44 smithi145 ceph-mon[33206]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T09:57:45.011 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397535 got 210453397537 for osd.7 2024-09-21T09:57:45.012 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:45.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:45 smithi119 ceph-mon[40785]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:57:45.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:45 smithi119 ceph-mon[29367]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:57:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:45 smithi145 ceph-mon[33206]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T09:57:46.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:46 smithi119 ceph-mon[29367]: pgmap v75: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:46.783 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:46 smithi119 ceph-mon[40785]: pgmap v75: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:46.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:46 smithi145 ceph-mon[33206]: pgmap v75: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:47.160 INFO:teuthology.orchestra.run.smithi119.stdout:158913789996 2024-09-21T09:57:47.236 INFO:teuthology.orchestra.run.smithi119.stdout:34359738437 2024-09-21T09:57:47.725 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:47 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/314615241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:57:47.725 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:47 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/64501373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:57:47.725 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/314615241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:57:47.726 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/64501373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:57:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/314615241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T09:57:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/64501373' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T09:57:48.148 INFO:teuthology.orchestra.run.smithi119.stdout:111669149748 2024-09-21T09:57:48.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T09:57:48.654 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:48 smithi119 ceph-mon[29367]: pgmap v76: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:48.654 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:48 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2945033569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:57:48.655 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:48 smithi119 ceph-mon[40785]: pgmap v76: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:48.655 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:48 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2945033569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:57:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:48 smithi145 ceph-mon[33206]: pgmap v76: 73 pgs: 3 peering, 70 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:48 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2945033569' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T09:57:49.053 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738435 got 34359738437 for osd.0 2024-09-21T09:57:49.053 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:49.455 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789993 got 158913789996 for osd.5 2024-09-21T09:57:49.455 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:49.659 INFO:teuthology.orchestra.run.smithi119.stdout:133143986224 2024-09-21T09:57:49.659 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:49 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1893060553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:57:49.659 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:49 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1893060553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:57:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:49 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1893060553' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T09:57:50.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986220 got 133143986224 for osd.4 2024-09-21T09:57:50.552 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:50.640 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:50 smithi119 ceph-mon[29367]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:50.640 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:50 smithi119 ceph-mon[40785]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:50.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:50 smithi145 ceph-mon[33206]: pgmap v77: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:50.893 INFO:teuthology.orchestra.run.smithi119.stdout:77309411388 2024-09-21T09:57:51.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:51 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/4211295988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:57:51.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:51 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/4211295988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:57:51.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:51 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/4211295988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T09:57:52.252 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411384 got 77309411388 for osd.2 2024-09-21T09:57:52.253 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:52.372 INFO:teuthology.orchestra.run.smithi119.stdout:184683593768 2024-09-21T09:57:52.689 INFO:teuthology.orchestra.run.smithi119.stdout:55834574912 2024-09-21T09:57:52.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:52 smithi119 ceph-mon[29367]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:52.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:52 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1950028850' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:57:52.690 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:52 smithi119 ceph-mon[40785]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:52.690 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:52 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1950028850' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:57:52.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:52 smithi145 ceph-mon[33206]: pgmap v78: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:52.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:52 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1950028850' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T09:57:53.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:57:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:57:53.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:57:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:57:52] "GET /metrics HTTP/1.1" 200 239760 "" "Prometheus/2.43.0" 2024-09-21T09:57:53.483 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574907 got 55834574912 for osd.1 2024-09-21T09:57:53.483 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:54.306 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:54.325 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593764 got 184683593768 for osd.6 2024-09-21T09:57:54.325 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:54.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:54 smithi119 ceph-mon[29367]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:54.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:54 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:54.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:54 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:54.549 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:54 smithi119 ceph-mon[40785]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:54.549 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:54 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:54.550 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:54 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:54 smithi145 ceph-mon[33206]: pgmap v79: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:54 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:54 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:55.714 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:55.714 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:55.715 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:55.715 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:55.715 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:55.715 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:55.715 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:55.715 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:55 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:55.799 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:55 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:57:55.800 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:55 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:57:55.800 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:55 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:55.800 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:55 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:57:56.125 INFO:teuthology.orchestra.run.smithi119.stdout:111669149751 2024-09-21T09:57:56.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:56 smithi119 ceph-mon[40785]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:56.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:56 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3155825791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:57:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:56 smithi119 ceph-mon[29367]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:56 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3155825791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:57:56.923 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149748 got 111669149751 for osd.3 2024-09-21T09:57:56.923 DEBUG:teuthology.parallel:result is None 2024-09-21T09:57:56.923 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T09:57:56.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:57:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:56 smithi145 ceph-mon[33206]: pgmap v80: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:56 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3155825791' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T09:57:57.506 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:57:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:58 smithi145 ceph-mon[33206]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:58 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:58 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:57:58 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[40785]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[29367]: pgmap v81: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:57:58 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:57:59.667 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:57:59.668 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T09:58:00.629 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-21T09:57:59.143674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385952,"kb_used_data":7080,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575264,"statfs":{"total":767960285184,"available":765517070336,"internally_reserved":0,"allocated":7249920,"data_stored":4145579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002597"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353393+0000","last_change":"2024-09-21T09:57:00.027187+0000","last_active":"2024-09-21T09:57:43.353393+0000","last_peered":"2024-09-21T09:57:43.353393+0000","last_clean":"2024-09-21T09:57:43.353393+0000","last_became_active":"2024-09-21T09:57:00.026930+0000","last_became_peered":"2024-09-21T09:57:00.026930+0000","last_unstale":"2024-09-21T09:57:43.353393+0000","last_undegraded":"2024-09-21T09:57:43.353393+0000","last_fullsized":"2024-09-21T09:57:43.353393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:38.389724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357283+0000","last_change":"2024-09-21T09:57:00.005080+0000","last_active":"2024-09-21T09:57:43.357283+0000","last_peered":"2024-09-21T09:57:43.357283+0000","last_clean":"2024-09-21T09:57:43.357283+0000","last_became_active":"2024-09-21T09:57:00.004738+0000","last_became_peered":"2024-09-21T09:57:00.004738+0000","last_unstale":"2024-09-21T09:57:43.357283+0000","last_undegraded":"2024-09-21T09:57:43.357283+0000","last_fullsized":"2024-09-21T09:57:43.357283+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:04.953587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356045+0000","last_change":"2024-09-21T09:57:01.007638+0000","last_active":"2024-09-21T09:57:43.356045+0000","last_peered":"2024-09-21T09:57:43.356045+0000","last_clean":"2024-09-21T09:57:43.356045+0000","last_became_active":"2024-09-21T09:57:01.007262+0000","last_became_peered":"2024-09-21T09:57:01.007262+0000","last_unstale":"2024-09-21T09:57:43.356045+0000","last_undegraded":"2024-09-21T09:57:43.356045+0000","last_fullsized":"2024-09-21T09:57:43.356045+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:41.701342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632726+0000","last_change":"2024-09-21T09:57:00.006492+0000","last_active":"2024-09-21T09:57:43.632726+0000","last_peered":"2024-09-21T09:57:43.632726+0000","last_clean":"2024-09-21T09:57:43.632726+0000","last_became_active":"2024-09-21T09:57:00.006208+0000","last_became_peered":"2024-09-21T09:57:00.006208+0000","last_unstale":"2024-09-21T09:57:43.632726+0000","last_undegraded":"2024-09-21T09:57:43.632726+0000","last_fullsized":"2024-09-21T09:57:43.632726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:32.022273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632930+0000","last_change":"2024-09-21T09:57:01.005438+0000","last_active":"2024-09-21T09:57:43.632930+0000","last_peered":"2024-09-21T09:57:43.632930+0000","last_clean":"2024-09-21T09:57:43.632930+0000","last_became_active":"2024-09-21T09:57:01.005127+0000","last_became_peered":"2024-09-21T09:57:01.005127+0000","last_unstale":"2024-09-21T09:57:43.632930+0000","last_undegraded":"2024-09-21T09:57:43.632930+0000","last_fullsized":"2024-09-21T09:57:43.632930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:06.271723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357001+0000","last_change":"2024-09-21T09:57:00.004995+0000","last_active":"2024-09-21T09:57:43.357001+0000","last_peered":"2024-09-21T09:57:43.357001+0000","last_clean":"2024-09-21T09:57:43.357001+0000","last_became_active":"2024-09-21T09:57:00.004651+0000","last_became_peered":"2024-09-21T09:57:00.004651+0000","last_unstale":"2024-09-21T09:57:43.357001+0000","last_undegraded":"2024-09-21T09:57:43.357001+0000","last_fullsized":"2024-09-21T09:57:43.357001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:04.502797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:57:44.733976+0000","last_change":"2024-09-21T09:57:44.733976+0000","last_active":"2024-09-21T09:57:44.733976+0000","last_peered":"2024-09-21T09:57:44.733976+0000","last_clean":"2024-09-21T09:57:44.733976+0000","last_became_active":"2024-09-21T09:57:44.733644+0000","last_became_peered":"2024-09-21T09:57:44.733644+0000","last_unstale":"2024-09-21T09:57:44.733976+0000","last_undegraded":"2024-09-21T09:57:44.733976+0000","last_fullsized":"2024-09-21T09:57:44.733976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:26.737299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633075+0000","last_change":"2024-09-21T09:57:00.002705+0000","last_active":"2024-09-21T09:57:43.633075+0000","last_peered":"2024-09-21T09:57:43.633075+0000","last_clean":"2024-09-21T09:57:43.633075+0000","last_became_active":"2024-09-21T09:57:00.002438+0000","last_became_peered":"2024-09-21T09:57:00.002438+0000","last_unstale":"2024-09-21T09:57:43.633075+0000","last_undegraded":"2024-09-21T09:57:43.633075+0000","last_fullsized":"2024-09-21T09:57:43.633075+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:26.972249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357964+0000","last_change":"2024-09-21T09:57:00.009174+0000","last_active":"2024-09-21T09:57:43.357964+0000","last_peered":"2024-09-21T09:57:43.357964+0000","last_clean":"2024-09-21T09:57:43.357964+0000","last_became_active":"2024-09-21T09:57:00.008890+0000","last_became_peered":"2024-09-21T09:57:00.008890+0000","last_unstale":"2024-09-21T09:57:43.357964+0000","last_undegraded":"2024-09-21T09:57:43.357964+0000","last_fullsized":"2024-09-21T09:57:43.357964+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:14.607174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355770+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T09:57:43.355770+0000","last_peered":"2024-09-21T09:57:43.355770+0000","last_clean":"2024-09-21T09:57:43.355770+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T09:57:43.355770+0000","last_undegraded":"2024-09-21T09:57:43.355770+0000","last_fullsized":"2024-09-21T09:57:43.355770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357811+0000","last_change":"2024-09-21T09:57:00.010363+0000","last_active":"2024-09-21T09:57:43.357811+0000","last_peered":"2024-09-21T09:57:43.357811+0000","last_clean":"2024-09-21T09:57:43.357811+0000","last_became_active":"2024-09-21T09:57:00.010065+0000","last_became_peered":"2024-09-21T09:57:00.010065+0000","last_unstale":"2024-09-21T09:57:43.357811+0000","last_undegraded":"2024-09-21T09:57:43.357811+0000","last_fullsized":"2024-09-21T09:57:43.357811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:10.964008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355517+0000","last_change":"2024-09-21T09:57:00.019748+0000","last_active":"2024-09-21T09:57:43.355517+0000","last_peered":"2024-09-21T09:57:43.355517+0000","last_clean":"2024-09-21T09:57:43.355517+0000","last_became_active":"2024-09-21T09:57:00.019267+0000","last_became_peered":"2024-09-21T09:57:00.019267+0000","last_unstale":"2024-09-21T09:57:43.355517+0000","last_undegraded":"2024-09-21T09:57:43.355517+0000","last_fullsized":"2024-09-21T09:57:43.355517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:10.423332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357610+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T09:57:43.357610+0000","last_peered":"2024-09-21T09:57:43.357610+0000","last_clean":"2024-09-21T09:57:43.357610+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T09:57:43.357610+0000","last_undegraded":"2024-09-21T09:57:43.357610+0000","last_fullsized":"2024-09-21T09:57:43.357610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633106+0000","last_change":"2024-09-21T09:57:01.044324+0000","last_active":"2024-09-21T09:57:43.633106+0000","last_peered":"2024-09-21T09:57:43.633106+0000","last_clean":"2024-09-21T09:57:43.633106+0000","last_became_active":"2024-09-21T09:57:01.044036+0000","last_became_peered":"2024-09-21T09:57:01.044036+0000","last_unstale":"2024-09-21T09:57:43.633106+0000","last_undegraded":"2024-09-21T09:57:43.633106+0000","last_fullsized":"2024-09-21T09:57:43.633106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:07.712026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356550+0000","last_change":"2024-09-21T09:57:00.000710+0000","last_active":"2024-09-21T09:57:43.356550+0000","last_peered":"2024-09-21T09:57:43.356550+0000","last_clean":"2024-09-21T09:57:43.356550+0000","last_became_active":"2024-09-21T09:57:00.000348+0000","last_became_peered":"2024-09-21T09:57:00.000348+0000","last_unstale":"2024-09-21T09:57:43.356550+0000","last_undegraded":"2024-09-21T09:57:43.356550+0000","last_fullsized":"2024-09-21T09:57:43.356550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:07.844682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632729+0000","last_change":"2024-09-21T09:57:01.044295+0000","last_active":"2024-09-21T09:57:43.632729+0000","last_peered":"2024-09-21T09:57:43.632729+0000","last_clean":"2024-09-21T09:57:43.632729+0000","last_became_active":"2024-09-21T09:57:01.043955+0000","last_became_peered":"2024-09-21T09:57:01.043955+0000","last_unstale":"2024-09-21T09:57:43.632729+0000","last_undegraded":"2024-09-21T09:57:43.632729+0000","last_fullsized":"2024-09-21T09:57:43.632729+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:04.236619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353007+0000","last_change":"2024-09-21T09:57:00.003942+0000","last_active":"2024-09-21T09:57:43.353007+0000","last_peered":"2024-09-21T09:57:43.353007+0000","last_clean":"2024-09-21T09:57:43.353007+0000","last_became_active":"2024-09-21T09:57:00.003251+0000","last_became_peered":"2024-09-21T09:57:00.003251+0000","last_unstale":"2024-09-21T09:57:43.353007+0000","last_undegraded":"2024-09-21T09:57:43.353007+0000","last_fullsized":"2024-09-21T09:57:43.353007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:31.031132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355653+0000","last_change":"2024-09-21T09:57:00.093132+0000","last_active":"2024-09-21T09:57:43.355653+0000","last_peered":"2024-09-21T09:57:43.355653+0000","last_clean":"2024-09-21T09:57:43.355653+0000","last_became_active":"2024-09-21T09:57:00.092674+0000","last_became_peered":"2024-09-21T09:57:00.092674+0000","last_unstale":"2024-09-21T09:57:43.355653+0000","last_undegraded":"2024-09-21T09:57:43.355653+0000","last_fullsized":"2024-09-21T09:57:43.355653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:17:10.177034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633241+0000","last_change":"2024-09-21T09:57:00.010794+0000","last_active":"2024-09-21T09:57:43.633241+0000","last_peered":"2024-09-21T09:57:43.633241+0000","last_clean":"2024-09-21T09:57:43.633241+0000","last_became_active":"2024-09-21T09:57:00.010522+0000","last_became_peered":"2024-09-21T09:57:00.010522+0000","last_unstale":"2024-09-21T09:57:43.633241+0000","last_undegraded":"2024-09-21T09:57:43.633241+0000","last_fullsized":"2024-09-21T09:57:43.633241+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:43:24.880300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:57:44.734642+0000","last_change":"2024-09-21T09:57:44.734642+0000","last_active":"2024-09-21T09:57:44.734642+0000","last_peered":"2024-09-21T09:57:44.734642+0000","last_clean":"2024-09-21T09:57:44.734642+0000","last_became_active":"2024-09-21T09:57:44.734289+0000","last_became_peered":"2024-09-21T09:57:44.734289+0000","last_unstale":"2024-09-21T09:57:44.734642+0000","last_undegraded":"2024-09-21T09:57:44.734642+0000","last_fullsized":"2024-09-21T09:57:44.734642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:42.522827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355868+0000","last_change":"2024-09-21T09:57:00.011268+0000","last_active":"2024-09-21T09:57:43.355868+0000","last_peered":"2024-09-21T09:57:43.355868+0000","last_clean":"2024-09-21T09:57:43.355868+0000","last_became_active":"2024-09-21T09:57:00.010320+0000","last_became_peered":"2024-09-21T09:57:00.010320+0000","last_unstale":"2024-09-21T09:57:43.355868+0000","last_undegraded":"2024-09-21T09:57:43.355868+0000","last_fullsized":"2024-09-21T09:57:43.355868+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:45:28.233874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355688+0000","last_change":"2024-09-21T09:57:00.093759+0000","last_active":"2024-09-21T09:57:43.355688+0000","last_peered":"2024-09-21T09:57:43.355688+0000","last_clean":"2024-09-21T09:57:43.355688+0000","last_became_active":"2024-09-21T09:57:00.093139+0000","last_became_peered":"2024-09-21T09:57:00.093139+0000","last_unstale":"2024-09-21T09:57:43.355688+0000","last_undegraded":"2024-09-21T09:57:43.355688+0000","last_fullsized":"2024-09-21T09:57:43.355688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:01.810647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352993+0000","last_change":"2024-09-21T09:57:00.001863+0000","last_active":"2024-09-21T09:57:43.352993+0000","last_peered":"2024-09-21T09:57:43.352993+0000","last_clean":"2024-09-21T09:57:43.352993+0000","last_became_active":"2024-09-21T09:57:00.001607+0000","last_became_peered":"2024-09-21T09:57:00.001607+0000","last_unstale":"2024-09-21T09:57:43.352993+0000","last_undegraded":"2024-09-21T09:57:43.352993+0000","last_fullsized":"2024-09-21T09:57:43.352993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:39.440239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355214+0000","last_change":"2024-09-21T09:57:00.019042+0000","last_active":"2024-09-21T09:57:43.355214+0000","last_peered":"2024-09-21T09:57:43.355214+0000","last_clean":"2024-09-21T09:57:43.355214+0000","last_became_active":"2024-09-21T09:57:00.018783+0000","last_became_peered":"2024-09-21T09:57:00.018783+0000","last_unstale":"2024-09-21T09:57:43.355214+0000","last_undegraded":"2024-09-21T09:57:43.355214+0000","last_fullsized":"2024-09-21T09:57:43.355214+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:54.550378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":55,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356934+0000","last_change":"2024-09-21T09:55:13.590990+0000","last_active":"2024-09-21T09:57:43.356934+0000","last_peered":"2024-09-21T09:57:43.356934+0000","last_clean":"2024-09-21T09:57:43.356934+0000","last_became_active":"2024-09-21T09:55:13.585775+0000","last_became_peered":"2024-09-21T09:55:13.585775+0000","last_unstale":"2024-09-21T09:57:43.356934+0000","last_undegraded":"2024-09-21T09:57:43.356934+0000","last_fullsized":"2024-09-21T09:57:43.356934+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:07.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355673+0000","last_change":"2024-09-21T09:55:46.169262+0000","last_active":"2024-09-21T09:57:43.355673+0000","last_peered":"2024-09-21T09:57:43.355673+0000","last_clean":"2024-09-21T09:57:43.355673+0000","last_became_active":"2024-09-21T09:55:41.641506+0000","last_became_peered":"2024-09-21T09:55:41.641506+0000","last_unstale":"2024-09-21T09:57:43.355673+0000","last_undegraded":"2024-09-21T09:57:43.355673+0000","last_fullsized":"2024-09-21T09:57:43.355673+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:21.952844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353247+0000","last_change":"2024-09-21T09:57:00.002507+0000","last_active":"2024-09-21T09:57:43.353247+0000","last_peered":"2024-09-21T09:57:43.353247+0000","last_clean":"2024-09-21T09:57:43.353247+0000","last_became_active":"2024-09-21T09:57:00.002269+0000","last_became_peered":"2024-09-21T09:57:00.002269+0000","last_unstale":"2024-09-21T09:57:43.353247+0000","last_undegraded":"2024-09-21T09:57:43.353247+0000","last_fullsized":"2024-09-21T09:57:43.353247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:46.331435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633382+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T09:57:43.633382+0000","last_peered":"2024-09-21T09:57:43.633382+0000","last_clean":"2024-09-21T09:57:43.633382+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T09:57:43.633382+0000","last_undegraded":"2024-09-21T09:57:43.633382+0000","last_fullsized":"2024-09-21T09:57:43.633382+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356872+0000","last_change":"2024-09-21T09:55:46.321119+0000","last_active":"2024-09-21T09:57:43.356872+0000","last_peered":"2024-09-21T09:57:43.356872+0000","last_clean":"2024-09-21T09:57:43.356872+0000","last_became_active":"2024-09-21T09:55:41.914064+0000","last_became_peered":"2024-09-21T09:55:41.914064+0000","last_unstale":"2024-09-21T09:57:43.356872+0000","last_undegraded":"2024-09-21T09:57:43.356872+0000","last_fullsized":"2024-09-21T09:57:43.356872+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:38.386562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632648+0000","last_change":"2024-09-21T09:57:00.009407+0000","last_active":"2024-09-21T09:57:43.632648+0000","last_peered":"2024-09-21T09:57:43.632648+0000","last_clean":"2024-09-21T09:57:43.632648+0000","last_became_active":"2024-09-21T09:57:00.009142+0000","last_became_peered":"2024-09-21T09:57:00.009142+0000","last_unstale":"2024-09-21T09:57:43.632648+0000","last_undegraded":"2024-09-21T09:57:43.632648+0000","last_fullsized":"2024-09-21T09:57:43.632648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:24.452913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632897+0000","last_change":"2024-09-21T09:57:01.044330+0000","last_active":"2024-09-21T09:57:43.632897+0000","last_peered":"2024-09-21T09:57:43.632897+0000","last_clean":"2024-09-21T09:57:43.632897+0000","last_became_active":"2024-09-21T09:57:01.043955+0000","last_became_peered":"2024-09-21T09:57:01.043955+0000","last_unstale":"2024-09-21T09:57:43.632897+0000","last_undegraded":"2024-09-21T09:57:43.632897+0000","last_fullsized":"2024-09-21T09:57:43.632897+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:38.206368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.351492+0000","last_change":"2024-09-21T09:55:46.165011+0000","last_active":"2024-09-21T09:57:43.351492+0000","last_peered":"2024-09-21T09:57:43.351492+0000","last_clean":"2024-09-21T09:57:43.351492+0000","last_became_active":"2024-09-21T09:55:41.914164+0000","last_became_peered":"2024-09-21T09:55:41.914164+0000","last_unstale":"2024-09-21T09:57:43.351492+0000","last_undegraded":"2024-09-21T09:57:43.351492+0000","last_fullsized":"2024-09-21T09:57:43.351492+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:55.377328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354210+0000","last_change":"2024-09-21T09:57:00.007879+0000","last_active":"2024-09-21T09:57:43.354210+0000","last_peered":"2024-09-21T09:57:43.354210+0000","last_clean":"2024-09-21T09:57:43.354210+0000","last_became_active":"2024-09-21T09:57:00.007622+0000","last_became_peered":"2024-09-21T09:57:00.007622+0000","last_unstale":"2024-09-21T09:57:43.354210+0000","last_undegraded":"2024-09-21T09:57:43.354210+0000","last_fullsized":"2024-09-21T09:57:43.354210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:14.570442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355288+0000","last_change":"2024-09-21T09:57:00.233199+0000","last_active":"2024-09-21T09:57:43.355288+0000","last_peered":"2024-09-21T09:57:43.355288+0000","last_clean":"2024-09-21T09:57:43.355288+0000","last_became_active":"2024-09-21T09:57:00.232832+0000","last_became_peered":"2024-09-21T09:57:00.232832+0000","last_unstale":"2024-09-21T09:57:43.355288+0000","last_undegraded":"2024-09-21T09:57:43.355288+0000","last_fullsized":"2024-09-21T09:57:43.355288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:52.446033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":28,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356569+0000","last_change":"2024-09-21T09:55:46.169743+0000","last_active":"2024-09-21T09:57:43.356569+0000","last_peered":"2024-09-21T09:57:43.356569+0000","last_clean":"2024-09-21T09:57:43.356569+0000","last_became_active":"2024-09-21T09:55:41.641386+0000","last_became_peered":"2024-09-21T09:55:41.641386+0000","last_unstale":"2024-09-21T09:57:43.356569+0000","last_undegraded":"2024-09-21T09:57:43.356569+0000","last_fullsized":"2024-09-21T09:57:43.356569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:03.000891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355921+0000","last_change":"2024-09-21T09:57:00.008676+0000","last_active":"2024-09-21T09:57:43.355921+0000","last_peered":"2024-09-21T09:57:43.355921+0000","last_clean":"2024-09-21T09:57:43.355921+0000","last_became_active":"2024-09-21T09:57:00.008406+0000","last_became_peered":"2024-09-21T09:57:00.008406+0000","last_unstale":"2024-09-21T09:57:43.355921+0000","last_undegraded":"2024-09-21T09:57:43.355921+0000","last_fullsized":"2024-09-21T09:57:43.355921+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:19:12.487639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354459+0000","last_change":"2024-09-21T09:57:00.022701+0000","last_active":"2024-09-21T09:57:43.354459+0000","last_peered":"2024-09-21T09:57:43.354459+0000","last_clean":"2024-09-21T09:57:43.354459+0000","last_became_active":"2024-09-21T09:57:00.022388+0000","last_became_peered":"2024-09-21T09:57:00.022388+0000","last_unstale":"2024-09-21T09:57:43.354459+0000","last_undegraded":"2024-09-21T09:57:43.354459+0000","last_fullsized":"2024-09-21T09:57:43.354459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:21.317748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356231+0000","last_change":"2024-09-21T09:57:00.002223+0000","last_active":"2024-09-21T09:57:43.356231+0000","last_peered":"2024-09-21T09:57:43.356231+0000","last_clean":"2024-09-21T09:57:43.356231+0000","last_became_active":"2024-09-21T09:57:00.001901+0000","last_became_peered":"2024-09-21T09:57:00.001901+0000","last_unstale":"2024-09-21T09:57:43.356231+0000","last_undegraded":"2024-09-21T09:57:43.356231+0000","last_fullsized":"2024-09-21T09:57:43.356231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:37.206528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354647+0000","last_change":"2024-09-21T09:55:46.170662+0000","last_active":"2024-09-21T09:57:43.354647+0000","last_peered":"2024-09-21T09:57:43.354647+0000","last_clean":"2024-09-21T09:57:43.354647+0000","last_became_active":"2024-09-21T09:55:41.914180+0000","last_became_peered":"2024-09-21T09:55:41.914180+0000","last_unstale":"2024-09-21T09:57:43.354647+0000","last_undegraded":"2024-09-21T09:57:43.354647+0000","last_fullsized":"2024-09-21T09:57:43.354647+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:50.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354259+0000","last_change":"2024-09-21T09:57:00.018283+0000","last_active":"2024-09-21T09:57:43.354259+0000","last_peered":"2024-09-21T09:57:43.354259+0000","last_clean":"2024-09-21T09:57:43.354259+0000","last_became_active":"2024-09-21T09:57:00.018034+0000","last_became_peered":"2024-09-21T09:57:00.018034+0000","last_unstale":"2024-09-21T09:57:43.354259+0000","last_undegraded":"2024-09-21T09:57:43.354259+0000","last_fullsized":"2024-09-21T09:57:43.354259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:23:44.388036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354708+0000","last_change":"2024-09-21T09:57:00.001823+0000","last_active":"2024-09-21T09:57:43.354708+0000","last_peered":"2024-09-21T09:57:43.354708+0000","last_clean":"2024-09-21T09:57:43.354708+0000","last_became_active":"2024-09-21T09:57:00.001553+0000","last_became_peered":"2024-09-21T09:57:00.001553+0000","last_unstale":"2024-09-21T09:57:43.354708+0000","last_undegraded":"2024-09-21T09:57:43.354708+0000","last_fullsized":"2024-09-21T09:57:43.354708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:44.299510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357701+0000","last_change":"2024-09-21T09:55:46.321183+0000","last_active":"2024-09-21T09:57:43.357701+0000","last_peered":"2024-09-21T09:57:43.357701+0000","last_clean":"2024-09-21T09:57:43.357701+0000","last_became_active":"2024-09-21T09:55:41.914202+0000","last_became_peered":"2024-09-21T09:55:41.914202+0000","last_unstale":"2024-09-21T09:57:43.357701+0000","last_undegraded":"2024-09-21T09:57:43.357701+0000","last_fullsized":"2024-09-21T09:57:43.357701+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:39.713100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632772+0000","last_change":"2024-09-21T09:57:01.007568+0000","last_active":"2024-09-21T09:57:43.632772+0000","last_peered":"2024-09-21T09:57:43.632772+0000","last_clean":"2024-09-21T09:57:43.632772+0000","last_became_active":"2024-09-21T09:57:01.007283+0000","last_became_peered":"2024-09-21T09:57:01.007283+0000","last_unstale":"2024-09-21T09:57:43.632772+0000","last_undegraded":"2024-09-21T09:57:43.632772+0000","last_fullsized":"2024-09-21T09:57:43.632772+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:50.108853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353344+0000","last_change":"2024-09-21T09:57:00.003035+0000","last_active":"2024-09-21T09:57:43.353344+0000","last_peered":"2024-09-21T09:57:43.353344+0000","last_clean":"2024-09-21T09:57:43.353344+0000","last_became_active":"2024-09-21T09:57:00.002761+0000","last_became_peered":"2024-09-21T09:57:00.002761+0000","last_unstale":"2024-09-21T09:57:43.353344+0000","last_undegraded":"2024-09-21T09:57:43.353344+0000","last_fullsized":"2024-09-21T09:57:43.353344+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:56.828478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354678+0000","last_change":"2024-09-21T09:55:46.170602+0000","last_active":"2024-09-21T09:57:43.354678+0000","last_peered":"2024-09-21T09:57:43.354678+0000","last_clean":"2024-09-21T09:57:43.354678+0000","last_became_active":"2024-09-21T09:55:41.639796+0000","last_became_peered":"2024-09-21T09:55:41.639796+0000","last_unstale":"2024-09-21T09:57:43.354678+0000","last_undegraded":"2024-09-21T09:57:43.354678+0000","last_fullsized":"2024-09-21T09:57:43.354678+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:13.050021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000210194,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352823+0000","last_change":"2024-09-21T09:57:00.232708+0000","last_active":"2024-09-21T09:57:43.352823+0000","last_peered":"2024-09-21T09:57:43.352823+0000","last_clean":"2024-09-21T09:57:43.352823+0000","last_became_active":"2024-09-21T09:57:00.232129+0000","last_became_peered":"2024-09-21T09:57:00.232129+0000","last_unstale":"2024-09-21T09:57:43.352823+0000","last_undegraded":"2024-09-21T09:57:43.352823+0000","last_fullsized":"2024-09-21T09:57:43.352823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:09.877466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632990+0000","last_change":"2024-09-21T09:57:00.015519+0000","last_active":"2024-09-21T09:57:43.632990+0000","last_peered":"2024-09-21T09:57:43.632990+0000","last_clean":"2024-09-21T09:57:43.632990+0000","last_became_active":"2024-09-21T09:57:00.015224+0000","last_became_peered":"2024-09-21T09:57:00.015224+0000","last_unstale":"2024-09-21T09:57:43.632990+0000","last_undegraded":"2024-09-21T09:57:43.632990+0000","last_fullsized":"2024-09-21T09:57:43.632990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:12.170419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633431+0000","last_change":"2024-09-21T09:55:46.174897+0000","last_active":"2024-09-21T09:57:43.633431+0000","last_peered":"2024-09-21T09:57:43.633431+0000","last_clean":"2024-09-21T09:57:43.633431+0000","last_became_active":"2024-09-21T09:55:41.640876+0000","last_became_peered":"2024-09-21T09:55:41.640876+0000","last_unstale":"2024-09-21T09:57:43.633431+0000","last_undegraded":"2024-09-21T09:57:43.633431+0000","last_fullsized":"2024-09-21T09:57:43.633431+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:47.920765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355829+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T09:57:43.355829+0000","last_peered":"2024-09-21T09:57:43.355829+0000","last_clean":"2024-09-21T09:57:43.355829+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T09:57:43.355829+0000","last_undegraded":"2024-09-21T09:57:43.355829+0000","last_fullsized":"2024-09-21T09:57:43.355829+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355788+0000","last_change":"2024-09-21T09:57:00.094313+0000","last_active":"2024-09-21T09:57:43.355788+0000","last_peered":"2024-09-21T09:57:43.355788+0000","last_clean":"2024-09-21T09:57:43.355788+0000","last_became_active":"2024-09-21T09:57:00.093496+0000","last_became_peered":"2024-09-21T09:57:00.093496+0000","last_unstale":"2024-09-21T09:57:43.355788+0000","last_undegraded":"2024-09-21T09:57:43.355788+0000","last_fullsized":"2024-09-21T09:57:43.355788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:07.957884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"64'6","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356103+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T09:57:43.356103+0000","last_peered":"2024-09-21T09:57:43.356103+0000","last_clean":"2024-09-21T09:57:43.356103+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T09:57:43.356103+0000","last_undegraded":"2024-09-21T09:57:43.356103+0000","last_fullsized":"2024-09-21T09:57:43.356103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354191+0000","last_change":"2024-09-21T09:57:00.093690+0000","last_active":"2024-09-21T09:57:43.354191+0000","last_peered":"2024-09-21T09:57:43.354191+0000","last_clean":"2024-09-21T09:57:43.354191+0000","last_became_active":"2024-09-21T09:57:00.093425+0000","last_became_peered":"2024-09-21T09:57:00.093425+0000","last_unstale":"2024-09-21T09:57:43.354191+0000","last_undegraded":"2024-09-21T09:57:43.354191+0000","last_fullsized":"2024-09-21T09:57:43.354191+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:34.762490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357439+0000","last_change":"2024-09-21T09:57:00.008480+0000","last_active":"2024-09-21T09:57:43.357439+0000","last_peered":"2024-09-21T09:57:43.357439+0000","last_clean":"2024-09-21T09:57:43.357439+0000","last_became_active":"2024-09-21T09:57:00.008184+0000","last_became_peered":"2024-09-21T09:57:00.008184+0000","last_unstale":"2024-09-21T09:57:43.357439+0000","last_undegraded":"2024-09-21T09:57:43.357439+0000","last_fullsized":"2024-09-21T09:57:43.357439+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:57.720039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356331+0000","last_change":"2024-09-21T09:57:00.003035+0000","last_active":"2024-09-21T09:57:43.356331+0000","last_peered":"2024-09-21T09:57:43.356331+0000","last_clean":"2024-09-21T09:57:43.356331+0000","last_became_active":"2024-09-21T09:57:00.002395+0000","last_became_peered":"2024-09-21T09:57:00.002395+0000","last_unstale":"2024-09-21T09:57:43.356331+0000","last_undegraded":"2024-09-21T09:57:43.356331+0000","last_fullsized":"2024-09-21T09:57:43.356331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:28:00.234060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353002+0000","last_change":"2024-09-21T09:57:00.093428+0000","last_active":"2024-09-21T09:57:43.353002+0000","last_peered":"2024-09-21T09:57:43.353002+0000","last_clean":"2024-09-21T09:57:43.353002+0000","last_became_active":"2024-09-21T09:57:00.093153+0000","last_became_peered":"2024-09-21T09:57:00.093153+0000","last_unstale":"2024-09-21T09:57:43.353002+0000","last_undegraded":"2024-09-21T09:57:43.353002+0000","last_fullsized":"2024-09-21T09:57:43.353002+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:30.018563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633013+0000","last_change":"2024-09-21T09:57:00.007135+0000","last_active":"2024-09-21T09:57:43.633013+0000","last_peered":"2024-09-21T09:57:43.633013+0000","last_clean":"2024-09-21T09:57:43.633013+0000","last_became_active":"2024-09-21T09:57:00.006210+0000","last_became_peered":"2024-09-21T09:57:00.006210+0000","last_unstale":"2024-09-21T09:57:43.633013+0000","last_undegraded":"2024-09-21T09:57:43.633013+0000","last_fullsized":"2024-09-21T09:57:43.633013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:40.426691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352822+0000","last_change":"2024-09-21T09:57:00.093696+0000","last_active":"2024-09-21T09:57:43.352822+0000","last_peered":"2024-09-21T09:57:43.352822+0000","last_clean":"2024-09-21T09:57:43.352822+0000","last_became_active":"2024-09-21T09:57:00.093406+0000","last_became_peered":"2024-09-21T09:57:00.093406+0000","last_unstale":"2024-09-21T09:57:43.352822+0000","last_undegraded":"2024-09-21T09:57:43.352822+0000","last_fullsized":"2024-09-21T09:57:43.352822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:28.966544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633419+0000","last_change":"2024-09-21T09:57:01.044215+0000","last_active":"2024-09-21T09:57:43.633419+0000","last_peered":"2024-09-21T09:57:43.633419+0000","last_clean":"2024-09-21T09:57:43.633419+0000","last_became_active":"2024-09-21T09:57:01.043958+0000","last_became_peered":"2024-09-21T09:57:01.043958+0000","last_unstale":"2024-09-21T09:57:43.633419+0000","last_undegraded":"2024-09-21T09:57:43.633419+0000","last_fullsized":"2024-09-21T09:57:43.633419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:27:09.128981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352917+0000","last_change":"2024-09-21T09:57:00.004259+0000","last_active":"2024-09-21T09:57:43.352917+0000","last_peered":"2024-09-21T09:57:43.352917+0000","last_clean":"2024-09-21T09:57:43.352917+0000","last_became_active":"2024-09-21T09:57:00.003232+0000","last_became_peered":"2024-09-21T09:57:00.003232+0000","last_unstale":"2024-09-21T09:57:43.352917+0000","last_undegraded":"2024-09-21T09:57:43.352917+0000","last_fullsized":"2024-09-21T09:57:43.352917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:48.945740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632868+0000","last_change":"2024-09-21T09:57:01.005264+0000","last_active":"2024-09-21T09:57:43.632868+0000","last_peered":"2024-09-21T09:57:43.632868+0000","last_clean":"2024-09-21T09:57:43.632868+0000","last_became_active":"2024-09-21T09:57:01.004994+0000","last_became_peered":"2024-09-21T09:57:01.004994+0000","last_unstale":"2024-09-21T09:57:43.632868+0000","last_undegraded":"2024-09-21T09:57:43.632868+0000","last_fullsized":"2024-09-21T09:57:43.632868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:05.086017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353952+0000","last_change":"2024-09-21T09:57:00.011765+0000","last_active":"2024-09-21T09:57:43.353952+0000","last_peered":"2024-09-21T09:57:43.353952+0000","last_clean":"2024-09-21T09:57:43.353952+0000","last_became_active":"2024-09-21T09:57:00.011345+0000","last_became_peered":"2024-09-21T09:57:00.011345+0000","last_unstale":"2024-09-21T09:57:43.353952+0000","last_undegraded":"2024-09-21T09:57:43.353952+0000","last_fullsized":"2024-09-21T09:57:43.353952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:29.973001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353355+0000","last_change":"2024-09-21T09:57:00.020868+0000","last_active":"2024-09-21T09:57:43.353355+0000","last_peered":"2024-09-21T09:57:43.353355+0000","last_clean":"2024-09-21T09:57:43.353355+0000","last_became_active":"2024-09-21T09:57:00.020447+0000","last_became_peered":"2024-09-21T09:57:00.020447+0000","last_unstale":"2024-09-21T09:57:43.353355+0000","last_undegraded":"2024-09-21T09:57:43.353355+0000","last_fullsized":"2024-09-21T09:57:43.353355+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.317211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355440+0000","last_change":"2024-09-21T09:57:00.012697+0000","last_active":"2024-09-21T09:57:43.355440+0000","last_peered":"2024-09-21T09:57:43.355440+0000","last_clean":"2024-09-21T09:57:43.355440+0000","last_became_active":"2024-09-21T09:57:00.012361+0000","last_became_peered":"2024-09-21T09:57:00.012361+0000","last_unstale":"2024-09-21T09:57:43.355440+0000","last_undegraded":"2024-09-21T09:57:43.355440+0000","last_fullsized":"2024-09-21T09:57:43.355440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:02.861671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:57:44.734702+0000","last_change":"2024-09-21T09:57:44.734702+0000","last_active":"2024-09-21T09:57:44.734702+0000","last_peered":"2024-09-21T09:57:44.734702+0000","last_clean":"2024-09-21T09:57:44.734702+0000","last_became_active":"2024-09-21T09:57:44.734375+0000","last_became_peered":"2024-09-21T09:57:44.734375+0000","last_unstale":"2024-09-21T09:57:44.734702+0000","last_undegraded":"2024-09-21T09:57:44.734702+0000","last_fullsized":"2024-09-21T09:57:44.734702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:34.972770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632862+0000","last_change":"2024-09-21T09:57:00.014650+0000","last_active":"2024-09-21T09:57:43.632862+0000","last_peered":"2024-09-21T09:57:43.632862+0000","last_clean":"2024-09-21T09:57:43.632862+0000","last_became_active":"2024-09-21T09:57:00.014390+0000","last_became_peered":"2024-09-21T09:57:00.014390+0000","last_unstale":"2024-09-21T09:57:43.632862+0000","last_undegraded":"2024-09-21T09:57:43.632862+0000","last_fullsized":"2024-09-21T09:57:43.632862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:57.999280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355740+0000","last_change":"2024-09-21T09:57:00.020059+0000","last_active":"2024-09-21T09:57:43.355740+0000","last_peered":"2024-09-21T09:57:43.355740+0000","last_clean":"2024-09-21T09:57:43.355740+0000","last_became_active":"2024-09-21T09:57:00.019751+0000","last_became_peered":"2024-09-21T09:57:00.019751+0000","last_unstale":"2024-09-21T09:57:43.355740+0000","last_undegraded":"2024-09-21T09:57:43.355740+0000","last_fullsized":"2024-09-21T09:57:43.355740+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:15.324825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356472+0000","last_change":"2024-09-21T09:57:00.009079+0000","last_active":"2024-09-21T09:57:43.356472+0000","last_peered":"2024-09-21T09:57:43.356472+0000","last_clean":"2024-09-21T09:57:43.356472+0000","last_became_active":"2024-09-21T09:57:00.008767+0000","last_became_peered":"2024-09-21T09:57:00.008767+0000","last_unstale":"2024-09-21T09:57:43.356472+0000","last_undegraded":"2024-09-21T09:57:43.356472+0000","last_fullsized":"2024-09-21T09:57:43.356472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:00.762208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355513+0000","last_change":"2024-09-21T09:57:00.232451+0000","last_active":"2024-09-21T09:57:43.355513+0000","last_peered":"2024-09-21T09:57:43.355513+0000","last_clean":"2024-09-21T09:57:43.355513+0000","last_became_active":"2024-09-21T09:57:00.232212+0000","last_became_peered":"2024-09-21T09:57:00.232212+0000","last_unstale":"2024-09-21T09:57:43.355513+0000","last_undegraded":"2024-09-21T09:57:43.355513+0000","last_fullsized":"2024-09-21T09:57:43.355513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:13.690154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356135+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T09:57:43.356135+0000","last_peered":"2024-09-21T09:57:43.356135+0000","last_clean":"2024-09-21T09:57:43.356135+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T09:57:43.356135+0000","last_undegraded":"2024-09-21T09:57:43.356135+0000","last_fullsized":"2024-09-21T09:57:43.356135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353083+0000","last_change":"2024-09-21T09:57:00.232407+0000","last_active":"2024-09-21T09:57:43.353083+0000","last_peered":"2024-09-21T09:57:43.353083+0000","last_clean":"2024-09-21T09:57:43.353083+0000","last_became_active":"2024-09-21T09:57:00.232121+0000","last_became_peered":"2024-09-21T09:57:00.232121+0000","last_unstale":"2024-09-21T09:57:43.353083+0000","last_undegraded":"2024-09-21T09:57:43.353083+0000","last_fullsized":"2024-09-21T09:57:43.353083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:11.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353223+0000","last_change":"2024-09-21T09:57:00.000040+0000","last_active":"2024-09-21T09:57:43.353223+0000","last_peered":"2024-09-21T09:57:43.353223+0000","last_clean":"2024-09-21T09:57:43.353223+0000","last_became_active":"2024-09-21T09:56:59.999759+0000","last_became_peered":"2024-09-21T09:56:59.999759+0000","last_unstale":"2024-09-21T09:57:43.353223+0000","last_undegraded":"2024-09-21T09:57:43.353223+0000","last_fullsized":"2024-09-21T09:57:43.353223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:13.332271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355250+0000","last_change":"2024-09-21T09:57:00.093953+0000","last_active":"2024-09-21T09:57:43.355250+0000","last_peered":"2024-09-21T09:57:43.355250+0000","last_clean":"2024-09-21T09:57:43.355250+0000","last_became_active":"2024-09-21T09:57:00.093674+0000","last_became_peered":"2024-09-21T09:57:00.093674+0000","last_unstale":"2024-09-21T09:57:43.355250+0000","last_undegraded":"2024-09-21T09:57:43.355250+0000","last_fullsized":"2024-09-21T09:57:43.355250+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:06.967283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632980+0000","last_change":"2024-09-21T09:57:00.001817+0000","last_active":"2024-09-21T09:57:43.632980+0000","last_peered":"2024-09-21T09:57:43.632980+0000","last_clean":"2024-09-21T09:57:43.632980+0000","last_became_active":"2024-09-21T09:57:00.001545+0000","last_became_peered":"2024-09-21T09:57:00.001545+0000","last_unstale":"2024-09-21T09:57:43.632980+0000","last_undegraded":"2024-09-21T09:57:43.632980+0000","last_fullsized":"2024-09-21T09:57:43.632980+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:48.501153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397541,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298624,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446528,"statfs":{"total":95995035648,"available":95689244672,"internally_reserved":0,"allocated":1294336,"data_stored":888219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.501,"15min":0.502},"min":{"1min":0.38100000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66100000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.435}]},{"osd":2,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.501},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.497},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.45700000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71499999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.372},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.32700000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.312}]},{"osd":6,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76900000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.36099999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593770,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298612,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446540,"statfs":{"total":95995035648,"available":95689256960,"internally_reserved":0,"allocated":1282048,"data_stored":887777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64200000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.627},{"interface":"front","average":{"1min":0.501,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.38600000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.67000000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.67000000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71699999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.59799999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.495},"min":{"1min":0.38100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68700000000000006,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.5},"min":{"1min":0.40200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70299999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.70299999999999996}]},{"osd":3,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.59499999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.51000000000000001},"min":{"1min":0.443,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73699999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.73699999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41799999999999998,"15min":0.39600000000000002},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.34000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.44500000000000001,"15min":0.42699999999999999},"min":{"1min":0.35699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72799999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.755}]},{"osd":7,"last update":"Sat Sep 21 09:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.78200000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789999,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":679936,"data_stored":296427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.503},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.66700000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67700000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.35299999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78400000000000003,"5min":1.327,"15min":1.327},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.434,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.26,"15min":1.26},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.433},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.39600000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.66800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.40500000000000003},"min":{"1min":0.312,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63900000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"min":{"1min":0.374,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63500000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.29799999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.35599999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64300000000000002,"5min":0.754,"15min":0.754},"last":0.315},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.39600000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66600000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 09:57:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53200000000000003,"15min":0.53300000000000003},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.5,"15min":0.499},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298036,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447116,"statfs":{"total":95995035648,"available":95689846784,"internally_reserved":0,"allocated":696320,"data_stored":297781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.441}]},{"osd":2,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.76800000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.56699999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.33500000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.65300000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.41399999999999998,"15min":0.39200000000000002},"min":{"1min":0.307,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.56699999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.59399999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63700000000000001,"5min":0.753,"15min":0.753},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.433,"15min":0.41099999999999998},"min":{"1min":0.29099999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.318,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70099999999999996,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":675840,"data_stored":296173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.37,"15min":0.39600000000000002},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.253},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.376,"15min":0.38600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53400000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.496}]},{"osd":1,"last update":"Sat Sep 21 09:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40999999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51800000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.43099999999999999,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.312},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39200000000000002,"15min":0.39900000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.69899999999999995,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83399999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.441}]},{"osd":5,"last update":"Sat Sep 21 09:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77600000000000002,"5min":0.876,"15min":0.876},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82699999999999996,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45300000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":0.998,"15min":0.998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.53600000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63500000000000001,"15min":0.64300000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.008},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.88400000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":679936,"data_stored":296687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.35099999999999998,"15min":0.35899999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.40699999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.35499999999999998,"15min":0.35199999999999998},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.35099999999999998,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56000000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.376,"15min":0.371},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.47799999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:57:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51600000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.40500000000000003,"15min":0.42299999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.45100000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.49299999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.502},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.377,"5min":0.375,"15min":0.375},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.623}]},{"osd":7,"last update":"Sat Sep 21 09:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70099999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.81499999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":667648,"data_stored":295472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34799999999999998,"15min":0.32700000000000001},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41399999999999998,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.34599999999999997,"15min":0.318},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.50900000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.433,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.315,"5min":0.36599999999999999,"15min":0.375},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.34499999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.41199999999999998,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.52300000000000002,"5min":0.624,"15min":0.624},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.54000000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.315}]},{"osd":4,"last update":"Sat Sep 21 09:57:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.78700000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.70399999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.65600000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.45600000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76500000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 21 09:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.66500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.621},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69899999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298604,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446548,"statfs":{"total":95995035648,"available":95689265152,"internally_reserved":0,"allocated":1273856,"data_stored":887043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.32600000000000001},"min":{"1min":0.191,"5min":0.158,"15min":0.158},"max":{"1min":0.44,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.313},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.47499999999999998,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.36399999999999999,"15min":0.36199999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.63900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60099999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.378}]},{"osd":4,"last update":"Sat Sep 21 09:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.92500000000000004,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58999999999999997},"min":{"1min":0.34999999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.873,"5min":0.88,"15min":0.88},"last":0.34899999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78300000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.36399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85899999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.39900000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.377,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70299999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.436},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79500000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 09:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.38,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84099999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:58:00.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T09:58:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:00 smithi119 ceph-mon[29367]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:00 smithi119 ceph-mon[29367]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:58:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:00 smithi119 ceph-mon[40785]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:00 smithi119 ceph-mon[40785]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:58:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:00 smithi145 ceph-mon[33206]: pgmap v82: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:00 smithi145 ceph-mon[33206]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:58:01.418 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:02.940 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[40785]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:02.940 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:02.940 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:02.940 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:02.940 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:02.940 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:02.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[29367]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:02.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:02.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:02.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:02.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:02.941 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:03.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-mon[33206]: pgmap v83: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:03.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:02 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:03.312 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:58:03.312 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:02] "GET /metrics HTTP/1.1" 200 239793 "" "Prometheus/2.43.0" 2024-09-21T09:58:03.315 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T09:58:03.960 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-21T09:58:03.144418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592725,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":160,"num_write_kb":1643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121,"ondisk_log_size":121,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385952,"kb_used_data":7080,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575264,"statfs":{"total":767960285184,"available":765517070336,"internally_reserved":0,"allocated":7249920,"data_stored":4145579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002591"},"pg_stats":[{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353393+0000","last_change":"2024-09-21T09:57:00.027187+0000","last_active":"2024-09-21T09:57:43.353393+0000","last_peered":"2024-09-21T09:57:43.353393+0000","last_clean":"2024-09-21T09:57:43.353393+0000","last_became_active":"2024-09-21T09:57:00.026930+0000","last_became_peered":"2024-09-21T09:57:00.026930+0000","last_unstale":"2024-09-21T09:57:43.353393+0000","last_undegraded":"2024-09-21T09:57:43.353393+0000","last_fullsized":"2024-09-21T09:57:43.353393+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:38.389724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357283+0000","last_change":"2024-09-21T09:57:00.005080+0000","last_active":"2024-09-21T09:57:43.357283+0000","last_peered":"2024-09-21T09:57:43.357283+0000","last_clean":"2024-09-21T09:57:43.357283+0000","last_became_active":"2024-09-21T09:57:00.004738+0000","last_became_peered":"2024-09-21T09:57:00.004738+0000","last_unstale":"2024-09-21T09:57:43.357283+0000","last_undegraded":"2024-09-21T09:57:43.357283+0000","last_fullsized":"2024-09-21T09:57:43.357283+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:04.953587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356045+0000","last_change":"2024-09-21T09:57:01.007638+0000","last_active":"2024-09-21T09:57:43.356045+0000","last_peered":"2024-09-21T09:57:43.356045+0000","last_clean":"2024-09-21T09:57:43.356045+0000","last_became_active":"2024-09-21T09:57:01.007262+0000","last_became_peered":"2024-09-21T09:57:01.007262+0000","last_unstale":"2024-09-21T09:57:43.356045+0000","last_undegraded":"2024-09-21T09:57:43.356045+0000","last_fullsized":"2024-09-21T09:57:43.356045+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:41:41.701342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632726+0000","last_change":"2024-09-21T09:57:00.006492+0000","last_active":"2024-09-21T09:57:43.632726+0000","last_peered":"2024-09-21T09:57:43.632726+0000","last_clean":"2024-09-21T09:57:43.632726+0000","last_became_active":"2024-09-21T09:57:00.006208+0000","last_became_peered":"2024-09-21T09:57:00.006208+0000","last_unstale":"2024-09-21T09:57:43.632726+0000","last_undegraded":"2024-09-21T09:57:43.632726+0000","last_fullsized":"2024-09-21T09:57:43.632726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:48:32.022273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632930+0000","last_change":"2024-09-21T09:57:01.005438+0000","last_active":"2024-09-21T09:57:43.632930+0000","last_peered":"2024-09-21T09:57:43.632930+0000","last_clean":"2024-09-21T09:57:43.632930+0000","last_became_active":"2024-09-21T09:57:01.005127+0000","last_became_peered":"2024-09-21T09:57:01.005127+0000","last_unstale":"2024-09-21T09:57:43.632930+0000","last_undegraded":"2024-09-21T09:57:43.632930+0000","last_fullsized":"2024-09-21T09:57:43.632930+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:06.271723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357001+0000","last_change":"2024-09-21T09:57:00.004995+0000","last_active":"2024-09-21T09:57:43.357001+0000","last_peered":"2024-09-21T09:57:43.357001+0000","last_clean":"2024-09-21T09:57:43.357001+0000","last_became_active":"2024-09-21T09:57:00.004651+0000","last_became_peered":"2024-09-21T09:57:00.004651+0000","last_unstale":"2024-09-21T09:57:43.357001+0000","last_undegraded":"2024-09-21T09:57:43.357001+0000","last_fullsized":"2024-09-21T09:57:43.357001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:26:04.502797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:57:44.733976+0000","last_change":"2024-09-21T09:57:44.733976+0000","last_active":"2024-09-21T09:57:44.733976+0000","last_peered":"2024-09-21T09:57:44.733976+0000","last_clean":"2024-09-21T09:57:44.733976+0000","last_became_active":"2024-09-21T09:57:44.733644+0000","last_became_peered":"2024-09-21T09:57:44.733644+0000","last_unstale":"2024-09-21T09:57:44.733976+0000","last_undegraded":"2024-09-21T09:57:44.733976+0000","last_fullsized":"2024-09-21T09:57:44.733976+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:51:26.737299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":22,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633075+0000","last_change":"2024-09-21T09:57:00.002705+0000","last_active":"2024-09-21T09:57:43.633075+0000","last_peered":"2024-09-21T09:57:43.633075+0000","last_clean":"2024-09-21T09:57:43.633075+0000","last_became_active":"2024-09-21T09:57:00.002438+0000","last_became_peered":"2024-09-21T09:57:00.002438+0000","last_unstale":"2024-09-21T09:57:43.633075+0000","last_undegraded":"2024-09-21T09:57:43.633075+0000","last_fullsized":"2024-09-21T09:57:43.633075+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:07:26.972249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357964+0000","last_change":"2024-09-21T09:57:00.009174+0000","last_active":"2024-09-21T09:57:43.357964+0000","last_peered":"2024-09-21T09:57:43.357964+0000","last_clean":"2024-09-21T09:57:43.357964+0000","last_became_active":"2024-09-21T09:57:00.008890+0000","last_became_peered":"2024-09-21T09:57:00.008890+0000","last_unstale":"2024-09-21T09:57:43.357964+0000","last_undegraded":"2024-09-21T09:57:43.357964+0000","last_fullsized":"2024-09-21T09:57:43.357964+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:14.607174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355770+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T09:57:43.355770+0000","last_peered":"2024-09-21T09:57:43.355770+0000","last_clean":"2024-09-21T09:57:43.355770+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T09:57:43.355770+0000","last_undegraded":"2024-09-21T09:57:43.355770+0000","last_fullsized":"2024-09-21T09:57:43.355770+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357811+0000","last_change":"2024-09-21T09:57:00.010363+0000","last_active":"2024-09-21T09:57:43.357811+0000","last_peered":"2024-09-21T09:57:43.357811+0000","last_clean":"2024-09-21T09:57:43.357811+0000","last_became_active":"2024-09-21T09:57:00.010065+0000","last_became_peered":"2024-09-21T09:57:00.010065+0000","last_unstale":"2024-09-21T09:57:43.357811+0000","last_undegraded":"2024-09-21T09:57:43.357811+0000","last_fullsized":"2024-09-21T09:57:43.357811+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:10.964008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355517+0000","last_change":"2024-09-21T09:57:00.019748+0000","last_active":"2024-09-21T09:57:43.355517+0000","last_peered":"2024-09-21T09:57:43.355517+0000","last_clean":"2024-09-21T09:57:43.355517+0000","last_became_active":"2024-09-21T09:57:00.019267+0000","last_became_peered":"2024-09-21T09:57:00.019267+0000","last_unstale":"2024-09-21T09:57:43.355517+0000","last_undegraded":"2024-09-21T09:57:43.355517+0000","last_fullsized":"2024-09-21T09:57:43.355517+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:10.423332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357610+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T09:57:43.357610+0000","last_peered":"2024-09-21T09:57:43.357610+0000","last_clean":"2024-09-21T09:57:43.357610+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T09:57:43.357610+0000","last_undegraded":"2024-09-21T09:57:43.357610+0000","last_fullsized":"2024-09-21T09:57:43.357610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633106+0000","last_change":"2024-09-21T09:57:01.044324+0000","last_active":"2024-09-21T09:57:43.633106+0000","last_peered":"2024-09-21T09:57:43.633106+0000","last_clean":"2024-09-21T09:57:43.633106+0000","last_became_active":"2024-09-21T09:57:01.044036+0000","last_became_peered":"2024-09-21T09:57:01.044036+0000","last_unstale":"2024-09-21T09:57:43.633106+0000","last_undegraded":"2024-09-21T09:57:43.633106+0000","last_fullsized":"2024-09-21T09:57:43.633106+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:07.712026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356550+0000","last_change":"2024-09-21T09:57:00.000710+0000","last_active":"2024-09-21T09:57:43.356550+0000","last_peered":"2024-09-21T09:57:43.356550+0000","last_clean":"2024-09-21T09:57:43.356550+0000","last_became_active":"2024-09-21T09:57:00.000348+0000","last_became_peered":"2024-09-21T09:57:00.000348+0000","last_unstale":"2024-09-21T09:57:43.356550+0000","last_undegraded":"2024-09-21T09:57:43.356550+0000","last_fullsized":"2024-09-21T09:57:43.356550+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:07.844682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632729+0000","last_change":"2024-09-21T09:57:01.044295+0000","last_active":"2024-09-21T09:57:43.632729+0000","last_peered":"2024-09-21T09:57:43.632729+0000","last_clean":"2024-09-21T09:57:43.632729+0000","last_became_active":"2024-09-21T09:57:01.043955+0000","last_became_peered":"2024-09-21T09:57:01.043955+0000","last_unstale":"2024-09-21T09:57:43.632729+0000","last_undegraded":"2024-09-21T09:57:43.632729+0000","last_fullsized":"2024-09-21T09:57:43.632729+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:04.236619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353007+0000","last_change":"2024-09-21T09:57:00.003942+0000","last_active":"2024-09-21T09:57:43.353007+0000","last_peered":"2024-09-21T09:57:43.353007+0000","last_clean":"2024-09-21T09:57:43.353007+0000","last_became_active":"2024-09-21T09:57:00.003251+0000","last_became_peered":"2024-09-21T09:57:00.003251+0000","last_unstale":"2024-09-21T09:57:43.353007+0000","last_undegraded":"2024-09-21T09:57:43.353007+0000","last_fullsized":"2024-09-21T09:57:43.353007+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:31.031132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355653+0000","last_change":"2024-09-21T09:57:00.093132+0000","last_active":"2024-09-21T09:57:43.355653+0000","last_peered":"2024-09-21T09:57:43.355653+0000","last_clean":"2024-09-21T09:57:43.355653+0000","last_became_active":"2024-09-21T09:57:00.092674+0000","last_became_peered":"2024-09-21T09:57:00.092674+0000","last_unstale":"2024-09-21T09:57:43.355653+0000","last_undegraded":"2024-09-21T09:57:43.355653+0000","last_fullsized":"2024-09-21T09:57:43.355653+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:17:10.177034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633241+0000","last_change":"2024-09-21T09:57:00.010794+0000","last_active":"2024-09-21T09:57:43.633241+0000","last_peered":"2024-09-21T09:57:43.633241+0000","last_clean":"2024-09-21T09:57:43.633241+0000","last_became_active":"2024-09-21T09:57:00.010522+0000","last_became_peered":"2024-09-21T09:57:00.010522+0000","last_unstale":"2024-09-21T09:57:43.633241+0000","last_undegraded":"2024-09-21T09:57:43.633241+0000","last_fullsized":"2024-09-21T09:57:43.633241+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:43:24.880300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:57:44.734642+0000","last_change":"2024-09-21T09:57:44.734642+0000","last_active":"2024-09-21T09:57:44.734642+0000","last_peered":"2024-09-21T09:57:44.734642+0000","last_clean":"2024-09-21T09:57:44.734642+0000","last_became_active":"2024-09-21T09:57:44.734289+0000","last_became_peered":"2024-09-21T09:57:44.734289+0000","last_unstale":"2024-09-21T09:57:44.734642+0000","last_undegraded":"2024-09-21T09:57:44.734642+0000","last_fullsized":"2024-09-21T09:57:44.734642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:42.522827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355868+0000","last_change":"2024-09-21T09:57:00.011268+0000","last_active":"2024-09-21T09:57:43.355868+0000","last_peered":"2024-09-21T09:57:43.355868+0000","last_clean":"2024-09-21T09:57:43.355868+0000","last_became_active":"2024-09-21T09:57:00.010320+0000","last_became_peered":"2024-09-21T09:57:00.010320+0000","last_unstale":"2024-09-21T09:57:43.355868+0000","last_undegraded":"2024-09-21T09:57:43.355868+0000","last_fullsized":"2024-09-21T09:57:43.355868+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:45:28.233874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355688+0000","last_change":"2024-09-21T09:57:00.093759+0000","last_active":"2024-09-21T09:57:43.355688+0000","last_peered":"2024-09-21T09:57:43.355688+0000","last_clean":"2024-09-21T09:57:43.355688+0000","last_became_active":"2024-09-21T09:57:00.093139+0000","last_became_peered":"2024-09-21T09:57:00.093139+0000","last_unstale":"2024-09-21T09:57:43.355688+0000","last_undegraded":"2024-09-21T09:57:43.355688+0000","last_fullsized":"2024-09-21T09:57:43.355688+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:01.810647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352993+0000","last_change":"2024-09-21T09:57:00.001863+0000","last_active":"2024-09-21T09:57:43.352993+0000","last_peered":"2024-09-21T09:57:43.352993+0000","last_clean":"2024-09-21T09:57:43.352993+0000","last_became_active":"2024-09-21T09:57:00.001607+0000","last_became_peered":"2024-09-21T09:57:00.001607+0000","last_unstale":"2024-09-21T09:57:43.352993+0000","last_undegraded":"2024-09-21T09:57:43.352993+0000","last_fullsized":"2024-09-21T09:57:43.352993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:39.440239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355214+0000","last_change":"2024-09-21T09:57:00.019042+0000","last_active":"2024-09-21T09:57:43.355214+0000","last_peered":"2024-09-21T09:57:43.355214+0000","last_clean":"2024-09-21T09:57:43.355214+0000","last_became_active":"2024-09-21T09:57:00.018783+0000","last_became_peered":"2024-09-21T09:57:00.018783+0000","last_unstale":"2024-09-21T09:57:43.355214+0000","last_undegraded":"2024-09-21T09:57:43.355214+0000","last_fullsized":"2024-09-21T09:57:43.355214+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:19:54.550378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"57'79","reported_seq":55,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356934+0000","last_change":"2024-09-21T09:55:13.590990+0000","last_active":"2024-09-21T09:57:43.356934+0000","last_peered":"2024-09-21T09:57:43.356934+0000","last_clean":"2024-09-21T09:57:43.356934+0000","last_became_active":"2024-09-21T09:55:13.585775+0000","last_became_peered":"2024-09-21T09:55:13.585775+0000","last_unstale":"2024-09-21T09:57:43.356934+0000","last_undegraded":"2024-09-21T09:57:43.356934+0000","last_fullsized":"2024-09-21T09:57:43.356934+0000","mapping_epoch":50,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:54:07.685495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355673+0000","last_change":"2024-09-21T09:55:46.169262+0000","last_active":"2024-09-21T09:57:43.355673+0000","last_peered":"2024-09-21T09:57:43.355673+0000","last_clean":"2024-09-21T09:57:43.355673+0000","last_became_active":"2024-09-21T09:55:41.641506+0000","last_became_peered":"2024-09-21T09:55:41.641506+0000","last_unstale":"2024-09-21T09:57:43.355673+0000","last_undegraded":"2024-09-21T09:57:43.355673+0000","last_fullsized":"2024-09-21T09:57:43.355673+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:21.952844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353247+0000","last_change":"2024-09-21T09:57:00.002507+0000","last_active":"2024-09-21T09:57:43.353247+0000","last_peered":"2024-09-21T09:57:43.353247+0000","last_clean":"2024-09-21T09:57:43.353247+0000","last_became_active":"2024-09-21T09:57:00.002269+0000","last_became_peered":"2024-09-21T09:57:00.002269+0000","last_unstale":"2024-09-21T09:57:43.353247+0000","last_undegraded":"2024-09-21T09:57:43.353247+0000","last_fullsized":"2024-09-21T09:57:43.353247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:46.331435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633382+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T09:57:43.633382+0000","last_peered":"2024-09-21T09:57:43.633382+0000","last_clean":"2024-09-21T09:57:43.633382+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T09:57:43.633382+0000","last_undegraded":"2024-09-21T09:57:43.633382+0000","last_fullsized":"2024-09-21T09:57:43.633382+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356872+0000","last_change":"2024-09-21T09:55:46.321119+0000","last_active":"2024-09-21T09:57:43.356872+0000","last_peered":"2024-09-21T09:57:43.356872+0000","last_clean":"2024-09-21T09:57:43.356872+0000","last_became_active":"2024-09-21T09:55:41.914064+0000","last_became_peered":"2024-09-21T09:55:41.914064+0000","last_unstale":"2024-09-21T09:57:43.356872+0000","last_undegraded":"2024-09-21T09:57:43.356872+0000","last_fullsized":"2024-09-21T09:57:43.356872+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:52:38.386562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632648+0000","last_change":"2024-09-21T09:57:00.009407+0000","last_active":"2024-09-21T09:57:43.632648+0000","last_peered":"2024-09-21T09:57:43.632648+0000","last_clean":"2024-09-21T09:57:43.632648+0000","last_became_active":"2024-09-21T09:57:00.009142+0000","last_became_peered":"2024-09-21T09:57:00.009142+0000","last_unstale":"2024-09-21T09:57:43.632648+0000","last_undegraded":"2024-09-21T09:57:43.632648+0000","last_fullsized":"2024-09-21T09:57:43.632648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:24.452913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632897+0000","last_change":"2024-09-21T09:57:01.044330+0000","last_active":"2024-09-21T09:57:43.632897+0000","last_peered":"2024-09-21T09:57:43.632897+0000","last_clean":"2024-09-21T09:57:43.632897+0000","last_became_active":"2024-09-21T09:57:01.043955+0000","last_became_peered":"2024-09-21T09:57:01.043955+0000","last_unstale":"2024-09-21T09:57:43.632897+0000","last_undegraded":"2024-09-21T09:57:43.632897+0000","last_fullsized":"2024-09-21T09:57:43.632897+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:25:38.206368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.351492+0000","last_change":"2024-09-21T09:55:46.165011+0000","last_active":"2024-09-21T09:57:43.351492+0000","last_peered":"2024-09-21T09:57:43.351492+0000","last_clean":"2024-09-21T09:57:43.351492+0000","last_became_active":"2024-09-21T09:55:41.914164+0000","last_became_peered":"2024-09-21T09:55:41.914164+0000","last_unstale":"2024-09-21T09:57:43.351492+0000","last_undegraded":"2024-09-21T09:57:43.351492+0000","last_fullsized":"2024-09-21T09:57:43.351492+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:41:55.377328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354210+0000","last_change":"2024-09-21T09:57:00.007879+0000","last_active":"2024-09-21T09:57:43.354210+0000","last_peered":"2024-09-21T09:57:43.354210+0000","last_clean":"2024-09-21T09:57:43.354210+0000","last_became_active":"2024-09-21T09:57:00.007622+0000","last_became_peered":"2024-09-21T09:57:00.007622+0000","last_unstale":"2024-09-21T09:57:43.354210+0000","last_undegraded":"2024-09-21T09:57:43.354210+0000","last_fullsized":"2024-09-21T09:57:43.354210+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:14.570442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355288+0000","last_change":"2024-09-21T09:57:00.233199+0000","last_active":"2024-09-21T09:57:43.355288+0000","last_peered":"2024-09-21T09:57:43.355288+0000","last_clean":"2024-09-21T09:57:43.355288+0000","last_became_active":"2024-09-21T09:57:00.232832+0000","last_became_peered":"2024-09-21T09:57:00.232832+0000","last_unstale":"2024-09-21T09:57:43.355288+0000","last_undegraded":"2024-09-21T09:57:43.355288+0000","last_fullsized":"2024-09-21T09:57:43.355288+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:52.446033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":28,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356569+0000","last_change":"2024-09-21T09:55:46.169743+0000","last_active":"2024-09-21T09:57:43.356569+0000","last_peered":"2024-09-21T09:57:43.356569+0000","last_clean":"2024-09-21T09:57:43.356569+0000","last_became_active":"2024-09-21T09:55:41.641386+0000","last_became_peered":"2024-09-21T09:55:41.641386+0000","last_unstale":"2024-09-21T09:57:43.356569+0000","last_undegraded":"2024-09-21T09:57:43.356569+0000","last_fullsized":"2024-09-21T09:57:43.356569+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:03.000891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355921+0000","last_change":"2024-09-21T09:57:00.008676+0000","last_active":"2024-09-21T09:57:43.355921+0000","last_peered":"2024-09-21T09:57:43.355921+0000","last_clean":"2024-09-21T09:57:43.355921+0000","last_became_active":"2024-09-21T09:57:00.008406+0000","last_became_peered":"2024-09-21T09:57:00.008406+0000","last_unstale":"2024-09-21T09:57:43.355921+0000","last_undegraded":"2024-09-21T09:57:43.355921+0000","last_fullsized":"2024-09-21T09:57:43.355921+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:19:12.487639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354459+0000","last_change":"2024-09-21T09:57:00.022701+0000","last_active":"2024-09-21T09:57:43.354459+0000","last_peered":"2024-09-21T09:57:43.354459+0000","last_clean":"2024-09-21T09:57:43.354459+0000","last_became_active":"2024-09-21T09:57:00.022388+0000","last_became_peered":"2024-09-21T09:57:00.022388+0000","last_unstale":"2024-09-21T09:57:43.354459+0000","last_undegraded":"2024-09-21T09:57:43.354459+0000","last_fullsized":"2024-09-21T09:57:43.354459+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:21.317748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356231+0000","last_change":"2024-09-21T09:57:00.002223+0000","last_active":"2024-09-21T09:57:43.356231+0000","last_peered":"2024-09-21T09:57:43.356231+0000","last_clean":"2024-09-21T09:57:43.356231+0000","last_became_active":"2024-09-21T09:57:00.001901+0000","last_became_peered":"2024-09-21T09:57:00.001901+0000","last_unstale":"2024-09-21T09:57:43.356231+0000","last_undegraded":"2024-09-21T09:57:43.356231+0000","last_fullsized":"2024-09-21T09:57:43.356231+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:43:37.206528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354647+0000","last_change":"2024-09-21T09:55:46.170662+0000","last_active":"2024-09-21T09:57:43.354647+0000","last_peered":"2024-09-21T09:57:43.354647+0000","last_clean":"2024-09-21T09:57:43.354647+0000","last_became_active":"2024-09-21T09:55:41.914180+0000","last_became_peered":"2024-09-21T09:55:41.914180+0000","last_unstale":"2024-09-21T09:57:43.354647+0000","last_undegraded":"2024-09-21T09:57:43.354647+0000","last_fullsized":"2024-09-21T09:57:43.354647+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:11:50.532273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354259+0000","last_change":"2024-09-21T09:57:00.018283+0000","last_active":"2024-09-21T09:57:43.354259+0000","last_peered":"2024-09-21T09:57:43.354259+0000","last_clean":"2024-09-21T09:57:43.354259+0000","last_became_active":"2024-09-21T09:57:00.018034+0000","last_became_peered":"2024-09-21T09:57:00.018034+0000","last_unstale":"2024-09-21T09:57:43.354259+0000","last_undegraded":"2024-09-21T09:57:43.354259+0000","last_fullsized":"2024-09-21T09:57:43.354259+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:23:44.388036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354708+0000","last_change":"2024-09-21T09:57:00.001823+0000","last_active":"2024-09-21T09:57:43.354708+0000","last_peered":"2024-09-21T09:57:43.354708+0000","last_clean":"2024-09-21T09:57:43.354708+0000","last_became_active":"2024-09-21T09:57:00.001553+0000","last_became_peered":"2024-09-21T09:57:00.001553+0000","last_unstale":"2024-09-21T09:57:43.354708+0000","last_undegraded":"2024-09-21T09:57:43.354708+0000","last_fullsized":"2024-09-21T09:57:43.354708+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:44.299510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357701+0000","last_change":"2024-09-21T09:55:46.321183+0000","last_active":"2024-09-21T09:57:43.357701+0000","last_peered":"2024-09-21T09:57:43.357701+0000","last_clean":"2024-09-21T09:57:43.357701+0000","last_became_active":"2024-09-21T09:55:41.914202+0000","last_became_peered":"2024-09-21T09:55:41.914202+0000","last_unstale":"2024-09-21T09:57:43.357701+0000","last_undegraded":"2024-09-21T09:57:43.357701+0000","last_fullsized":"2024-09-21T09:57:43.357701+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:39.713100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632772+0000","last_change":"2024-09-21T09:57:01.007568+0000","last_active":"2024-09-21T09:57:43.632772+0000","last_peered":"2024-09-21T09:57:43.632772+0000","last_clean":"2024-09-21T09:57:43.632772+0000","last_became_active":"2024-09-21T09:57:01.007283+0000","last_became_peered":"2024-09-21T09:57:01.007283+0000","last_unstale":"2024-09-21T09:57:43.632772+0000","last_undegraded":"2024-09-21T09:57:43.632772+0000","last_fullsized":"2024-09-21T09:57:43.632772+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:50.108853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353344+0000","last_change":"2024-09-21T09:57:00.003035+0000","last_active":"2024-09-21T09:57:43.353344+0000","last_peered":"2024-09-21T09:57:43.353344+0000","last_clean":"2024-09-21T09:57:43.353344+0000","last_became_active":"2024-09-21T09:57:00.002761+0000","last_became_peered":"2024-09-21T09:57:00.002761+0000","last_unstale":"2024-09-21T09:57:43.353344+0000","last_undegraded":"2024-09-21T09:57:43.353344+0000","last_fullsized":"2024-09-21T09:57:43.353344+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:56.828478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354678+0000","last_change":"2024-09-21T09:55:46.170602+0000","last_active":"2024-09-21T09:57:43.354678+0000","last_peered":"2024-09-21T09:57:43.354678+0000","last_clean":"2024-09-21T09:57:43.354678+0000","last_became_active":"2024-09-21T09:55:41.639796+0000","last_became_peered":"2024-09-21T09:55:41.639796+0000","last_unstale":"2024-09-21T09:57:43.354678+0000","last_undegraded":"2024-09-21T09:57:43.354678+0000","last_fullsized":"2024-09-21T09:57:43.354678+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:13.050021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000210194,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352823+0000","last_change":"2024-09-21T09:57:00.232708+0000","last_active":"2024-09-21T09:57:43.352823+0000","last_peered":"2024-09-21T09:57:43.352823+0000","last_clean":"2024-09-21T09:57:43.352823+0000","last_became_active":"2024-09-21T09:57:00.232129+0000","last_became_peered":"2024-09-21T09:57:00.232129+0000","last_unstale":"2024-09-21T09:57:43.352823+0000","last_undegraded":"2024-09-21T09:57:43.352823+0000","last_fullsized":"2024-09-21T09:57:43.352823+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:09.877466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632990+0000","last_change":"2024-09-21T09:57:00.015519+0000","last_active":"2024-09-21T09:57:43.632990+0000","last_peered":"2024-09-21T09:57:43.632990+0000","last_clean":"2024-09-21T09:57:43.632990+0000","last_became_active":"2024-09-21T09:57:00.015224+0000","last_became_peered":"2024-09-21T09:57:00.015224+0000","last_unstale":"2024-09-21T09:57:43.632990+0000","last_undegraded":"2024-09-21T09:57:43.632990+0000","last_fullsized":"2024-09-21T09:57:43.632990+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:12.170419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633431+0000","last_change":"2024-09-21T09:55:46.174897+0000","last_active":"2024-09-21T09:57:43.633431+0000","last_peered":"2024-09-21T09:57:43.633431+0000","last_clean":"2024-09-21T09:57:43.633431+0000","last_became_active":"2024-09-21T09:55:41.640876+0000","last_became_peered":"2024-09-21T09:55:41.640876+0000","last_unstale":"2024-09-21T09:57:43.633431+0000","last_undegraded":"2024-09-21T09:57:43.633431+0000","last_fullsized":"2024-09-21T09:57:43.633431+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:47.920765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355829+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T09:57:43.355829+0000","last_peered":"2024-09-21T09:57:43.355829+0000","last_clean":"2024-09-21T09:57:43.355829+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T09:57:43.355829+0000","last_undegraded":"2024-09-21T09:57:43.355829+0000","last_fullsized":"2024-09-21T09:57:43.355829+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355788+0000","last_change":"2024-09-21T09:57:00.094313+0000","last_active":"2024-09-21T09:57:43.355788+0000","last_peered":"2024-09-21T09:57:43.355788+0000","last_clean":"2024-09-21T09:57:43.355788+0000","last_became_active":"2024-09-21T09:57:00.093496+0000","last_became_peered":"2024-09-21T09:57:00.093496+0000","last_unstale":"2024-09-21T09:57:43.355788+0000","last_undegraded":"2024-09-21T09:57:43.355788+0000","last_fullsized":"2024-09-21T09:57:43.355788+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:07.957884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"64'6","reported_seq":20,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356103+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T09:57:43.356103+0000","last_peered":"2024-09-21T09:57:43.356103+0000","last_clean":"2024-09-21T09:57:43.356103+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T09:57:43.356103+0000","last_undegraded":"2024-09-21T09:57:43.356103+0000","last_fullsized":"2024-09-21T09:57:43.356103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1552,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.354191+0000","last_change":"2024-09-21T09:57:00.093690+0000","last_active":"2024-09-21T09:57:43.354191+0000","last_peered":"2024-09-21T09:57:43.354191+0000","last_clean":"2024-09-21T09:57:43.354191+0000","last_became_active":"2024-09-21T09:57:00.093425+0000","last_became_peered":"2024-09-21T09:57:00.093425+0000","last_unstale":"2024-09-21T09:57:43.354191+0000","last_undegraded":"2024-09-21T09:57:43.354191+0000","last_fullsized":"2024-09-21T09:57:43.354191+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:51:34.762490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":17,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.357439+0000","last_change":"2024-09-21T09:57:00.008480+0000","last_active":"2024-09-21T09:57:43.357439+0000","last_peered":"2024-09-21T09:57:43.357439+0000","last_clean":"2024-09-21T09:57:43.357439+0000","last_became_active":"2024-09-21T09:57:00.008184+0000","last_became_peered":"2024-09-21T09:57:00.008184+0000","last_unstale":"2024-09-21T09:57:43.357439+0000","last_undegraded":"2024-09-21T09:57:43.357439+0000","last_fullsized":"2024-09-21T09:57:43.357439+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:57.720039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356331+0000","last_change":"2024-09-21T09:57:00.003035+0000","last_active":"2024-09-21T09:57:43.356331+0000","last_peered":"2024-09-21T09:57:43.356331+0000","last_clean":"2024-09-21T09:57:43.356331+0000","last_became_active":"2024-09-21T09:57:00.002395+0000","last_became_peered":"2024-09-21T09:57:00.002395+0000","last_unstale":"2024-09-21T09:57:43.356331+0000","last_undegraded":"2024-09-21T09:57:43.356331+0000","last_fullsized":"2024-09-21T09:57:43.356331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:28:00.234060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353002+0000","last_change":"2024-09-21T09:57:00.093428+0000","last_active":"2024-09-21T09:57:43.353002+0000","last_peered":"2024-09-21T09:57:43.353002+0000","last_clean":"2024-09-21T09:57:43.353002+0000","last_became_active":"2024-09-21T09:57:00.093153+0000","last_became_peered":"2024-09-21T09:57:00.093153+0000","last_unstale":"2024-09-21T09:57:43.353002+0000","last_undegraded":"2024-09-21T09:57:43.353002+0000","last_fullsized":"2024-09-21T09:57:43.353002+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:57:30.018563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633013+0000","last_change":"2024-09-21T09:57:00.007135+0000","last_active":"2024-09-21T09:57:43.633013+0000","last_peered":"2024-09-21T09:57:43.633013+0000","last_clean":"2024-09-21T09:57:43.633013+0000","last_became_active":"2024-09-21T09:57:00.006210+0000","last_became_peered":"2024-09-21T09:57:00.006210+0000","last_unstale":"2024-09-21T09:57:43.633013+0000","last_undegraded":"2024-09-21T09:57:43.633013+0000","last_fullsized":"2024-09-21T09:57:43.633013+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:40.426691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352822+0000","last_change":"2024-09-21T09:57:00.093696+0000","last_active":"2024-09-21T09:57:43.352822+0000","last_peered":"2024-09-21T09:57:43.352822+0000","last_clean":"2024-09-21T09:57:43.352822+0000","last_became_active":"2024-09-21T09:57:00.093406+0000","last_became_peered":"2024-09-21T09:57:00.093406+0000","last_unstale":"2024-09-21T09:57:43.352822+0000","last_undegraded":"2024-09-21T09:57:43.352822+0000","last_fullsized":"2024-09-21T09:57:43.352822+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:05:28.966544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.633419+0000","last_change":"2024-09-21T09:57:01.044215+0000","last_active":"2024-09-21T09:57:43.633419+0000","last_peered":"2024-09-21T09:57:43.633419+0000","last_clean":"2024-09-21T09:57:43.633419+0000","last_became_active":"2024-09-21T09:57:01.043958+0000","last_became_peered":"2024-09-21T09:57:01.043958+0000","last_unstale":"2024-09-21T09:57:43.633419+0000","last_undegraded":"2024-09-21T09:57:43.633419+0000","last_fullsized":"2024-09-21T09:57:43.633419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:27:09.128981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.352917+0000","last_change":"2024-09-21T09:57:00.004259+0000","last_active":"2024-09-21T09:57:43.352917+0000","last_peered":"2024-09-21T09:57:43.352917+0000","last_clean":"2024-09-21T09:57:43.352917+0000","last_became_active":"2024-09-21T09:57:00.003232+0000","last_became_peered":"2024-09-21T09:57:00.003232+0000","last_unstale":"2024-09-21T09:57:43.352917+0000","last_undegraded":"2024-09-21T09:57:43.352917+0000","last_fullsized":"2024-09-21T09:57:43.352917+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:48.945740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632868+0000","last_change":"2024-09-21T09:57:01.005264+0000","last_active":"2024-09-21T09:57:43.632868+0000","last_peered":"2024-09-21T09:57:43.632868+0000","last_clean":"2024-09-21T09:57:43.632868+0000","last_became_active":"2024-09-21T09:57:01.004994+0000","last_became_peered":"2024-09-21T09:57:01.004994+0000","last_unstale":"2024-09-21T09:57:43.632868+0000","last_undegraded":"2024-09-21T09:57:43.632868+0000","last_fullsized":"2024-09-21T09:57:43.632868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:05.086017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353952+0000","last_change":"2024-09-21T09:57:00.011765+0000","last_active":"2024-09-21T09:57:43.353952+0000","last_peered":"2024-09-21T09:57:43.353952+0000","last_clean":"2024-09-21T09:57:43.353952+0000","last_became_active":"2024-09-21T09:57:00.011345+0000","last_became_peered":"2024-09-21T09:57:00.011345+0000","last_unstale":"2024-09-21T09:57:43.353952+0000","last_undegraded":"2024-09-21T09:57:43.353952+0000","last_fullsized":"2024-09-21T09:57:43.353952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:29.973001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353355+0000","last_change":"2024-09-21T09:57:00.020868+0000","last_active":"2024-09-21T09:57:43.353355+0000","last_peered":"2024-09-21T09:57:43.353355+0000","last_clean":"2024-09-21T09:57:43.353355+0000","last_became_active":"2024-09-21T09:57:00.020447+0000","last_became_peered":"2024-09-21T09:57:00.020447+0000","last_unstale":"2024-09-21T09:57:43.353355+0000","last_undegraded":"2024-09-21T09:57:43.353355+0000","last_fullsized":"2024-09-21T09:57:43.353355+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:22:26.317211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355440+0000","last_change":"2024-09-21T09:57:00.012697+0000","last_active":"2024-09-21T09:57:43.355440+0000","last_peered":"2024-09-21T09:57:43.355440+0000","last_clean":"2024-09-21T09:57:43.355440+0000","last_became_active":"2024-09-21T09:57:00.012361+0000","last_became_peered":"2024-09-21T09:57:00.012361+0000","last_unstale":"2024-09-21T09:57:43.355440+0000","last_undegraded":"2024-09-21T09:57:43.355440+0000","last_fullsized":"2024-09-21T09:57:43.355440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:02.861671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T09:57:44.734702+0000","last_change":"2024-09-21T09:57:44.734702+0000","last_active":"2024-09-21T09:57:44.734702+0000","last_peered":"2024-09-21T09:57:44.734702+0000","last_clean":"2024-09-21T09:57:44.734702+0000","last_became_active":"2024-09-21T09:57:44.734375+0000","last_became_peered":"2024-09-21T09:57:44.734375+0000","last_unstale":"2024-09-21T09:57:44.734702+0000","last_undegraded":"2024-09-21T09:57:44.734702+0000","last_fullsized":"2024-09-21T09:57:44.734702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:36:34.972770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632862+0000","last_change":"2024-09-21T09:57:00.014650+0000","last_active":"2024-09-21T09:57:43.632862+0000","last_peered":"2024-09-21T09:57:43.632862+0000","last_clean":"2024-09-21T09:57:43.632862+0000","last_became_active":"2024-09-21T09:57:00.014390+0000","last_became_peered":"2024-09-21T09:57:00.014390+0000","last_unstale":"2024-09-21T09:57:43.632862+0000","last_undegraded":"2024-09-21T09:57:43.632862+0000","last_fullsized":"2024-09-21T09:57:43.632862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:57.999280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355740+0000","last_change":"2024-09-21T09:57:00.020059+0000","last_active":"2024-09-21T09:57:43.355740+0000","last_peered":"2024-09-21T09:57:43.355740+0000","last_clean":"2024-09-21T09:57:43.355740+0000","last_became_active":"2024-09-21T09:57:00.019751+0000","last_became_peered":"2024-09-21T09:57:00.019751+0000","last_unstale":"2024-09-21T09:57:43.355740+0000","last_undegraded":"2024-09-21T09:57:43.355740+0000","last_fullsized":"2024-09-21T09:57:43.355740+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:15.324825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356472+0000","last_change":"2024-09-21T09:57:00.009079+0000","last_active":"2024-09-21T09:57:43.356472+0000","last_peered":"2024-09-21T09:57:43.356472+0000","last_clean":"2024-09-21T09:57:43.356472+0000","last_became_active":"2024-09-21T09:57:00.008767+0000","last_became_peered":"2024-09-21T09:57:00.008767+0000","last_unstale":"2024-09-21T09:57:43.356472+0000","last_undegraded":"2024-09-21T09:57:43.356472+0000","last_fullsized":"2024-09-21T09:57:43.356472+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:06:00.762208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355513+0000","last_change":"2024-09-21T09:57:00.232451+0000","last_active":"2024-09-21T09:57:43.355513+0000","last_peered":"2024-09-21T09:57:43.355513+0000","last_clean":"2024-09-21T09:57:43.355513+0000","last_became_active":"2024-09-21T09:57:00.232212+0000","last_became_peered":"2024-09-21T09:57:00.232212+0000","last_unstale":"2024-09-21T09:57:43.355513+0000","last_undegraded":"2024-09-21T09:57:43.355513+0000","last_fullsized":"2024-09-21T09:57:43.355513+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:13.690154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":16,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.356135+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T09:57:43.356135+0000","last_peered":"2024-09-21T09:57:43.356135+0000","last_clean":"2024-09-21T09:57:43.356135+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T09:57:43.356135+0000","last_undegraded":"2024-09-21T09:57:43.356135+0000","last_fullsized":"2024-09-21T09:57:43.356135+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353083+0000","last_change":"2024-09-21T09:57:00.232407+0000","last_active":"2024-09-21T09:57:43.353083+0000","last_peered":"2024-09-21T09:57:43.353083+0000","last_clean":"2024-09-21T09:57:43.353083+0000","last_became_active":"2024-09-21T09:57:00.232121+0000","last_became_peered":"2024-09-21T09:57:00.232121+0000","last_unstale":"2024-09-21T09:57:43.353083+0000","last_undegraded":"2024-09-21T09:57:43.353083+0000","last_fullsized":"2024-09-21T09:57:43.353083+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:11.087521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":15,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.353223+0000","last_change":"2024-09-21T09:57:00.000040+0000","last_active":"2024-09-21T09:57:43.353223+0000","last_peered":"2024-09-21T09:57:43.353223+0000","last_clean":"2024-09-21T09:57:43.353223+0000","last_became_active":"2024-09-21T09:56:59.999759+0000","last_became_peered":"2024-09-21T09:56:59.999759+0000","last_unstale":"2024-09-21T09:57:43.353223+0000","last_undegraded":"2024-09-21T09:57:43.353223+0000","last_fullsized":"2024-09-21T09:57:43.353223+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:34:13.332271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.355250+0000","last_change":"2024-09-21T09:57:00.093953+0000","last_active":"2024-09-21T09:57:43.355250+0000","last_peered":"2024-09-21T09:57:43.355250+0000","last_clean":"2024-09-21T09:57:43.355250+0000","last_became_active":"2024-09-21T09:57:00.093674+0000","last_became_peered":"2024-09-21T09:57:00.093674+0000","last_unstale":"2024-09-21T09:57:43.355250+0000","last_undegraded":"2024-09-21T09:57:43.355250+0000","last_fullsized":"2024-09-21T09:57:43.355250+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:07:06.967283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":18,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T09:57:43.632980+0000","last_change":"2024-09-21T09:57:00.001817+0000","last_active":"2024-09-21T09:57:43.632980+0000","last_peered":"2024-09-21T09:57:43.632980+0000","last_clean":"2024-09-21T09:57:43.632980+0000","last_became_active":"2024-09-21T09:57:00.001545+0000","last_became_peered":"2024-09-21T09:57:00.001545+0000","last_unstale":"2024-09-21T09:57:43.632980+0000","last_undegraded":"2024-09-21T09:57:43.632980+0000","last_fullsized":"2024-09-21T09:57:43.632980+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:48.501153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2338,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":7014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397541,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298624,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446528,"statfs":{"total":95995035648,"available":95689244672,"internally_reserved":0,"allocated":1294336,"data_stored":888219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.501,"15min":0.502},"min":{"1min":0.38100000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66100000000000003,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.50700000000000001,"15min":0.50900000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51300000000000001}]},{"osd":1,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49199999999999999,"15min":0.49099999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.37,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69699999999999995,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.435}]},{"osd":2,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.501},{"interface":"front","average":{"1min":0.496,"5min":0.497,"15min":0.497},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.45700000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.35699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.53400000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.34100000000000003,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71499999999999997,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.372},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.32700000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.34100000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.61499999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.312}]},{"osd":6,"last update":"Sat Sep 21 09:57:15 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55700000000000005,"15min":0.55900000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76900000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56100000000000005,"15min":0.56499999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.36099999999999999}]}]},{"osd":6,"up_from":43,"seq":184683593770,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298612,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446540,"statfs":{"total":95995035648,"available":95689256960,"internally_reserved":0,"allocated":1282048,"data_stored":887777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.501,"15min":0.48899999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64200000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.627},{"interface":"front","average":{"1min":0.501,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.38600000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.67000000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.67000000000000004}]},{"osd":1,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.71699999999999997,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.499,"15min":0.49199999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.59799999999999998,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.51500000000000001}]},{"osd":2,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.50700000000000001,"15min":0.495},"min":{"1min":0.38100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.68700000000000006,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51700000000000002,"15min":0.5},"min":{"1min":0.40200000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70299999999999996,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.70299999999999996}]},{"osd":3,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.377,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.59499999999999997,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53000000000000003,"15min":0.51000000000000001},"min":{"1min":0.443,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.73699999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.73699999999999999}]},{"osd":4,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41799999999999998,"15min":0.39600000000000002},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.441,"15min":0.42899999999999999},"min":{"1min":0.34000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.70899999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.44500000000000001,"15min":0.42699999999999999},"min":{"1min":0.35699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72799999999999998,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.46800000000000003,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.755}]},{"osd":7,"last update":"Sat Sep 21 09:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.48699999999999999,"15min":0.48499999999999999},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.749,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.36599999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.78200000000000003}]}]},{"osd":5,"up_from":37,"seq":158913789999,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":679936,"data_stored":296427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.32700000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.66300000000000003,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.49099999999999999,"15min":0.49399999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.41899999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.503},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.66700000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67700000000000005,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.40699999999999997}]},{"osd":2,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54900000000000004,"15min":0.56999999999999995},"min":{"1min":0.35299999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.78400000000000003,"5min":1.327,"15min":1.327},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56899999999999995,"15min":0.58599999999999997},"min":{"1min":0.434,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72999999999999998,"5min":1.26,"15min":1.26},"last":0.46800000000000003}]},{"osd":3,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.67900000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.433},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.39600000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.66800000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.46100000000000002}]},{"osd":4,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.41599999999999998,"15min":0.40500000000000003},"min":{"1min":0.312,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.63900000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.443,"15min":0.443},"min":{"1min":0.374,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63500000000000001,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.29799999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.443},"min":{"1min":0.35599999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64300000000000002,"5min":0.754,"15min":0.754},"last":0.315},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.49299999999999999,"15min":0.48699999999999999},"min":{"1min":0.39600000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66600000000000004,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 09:57:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.53200000000000003,"15min":0.53300000000000003},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70599999999999996,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.5,"15min":0.499},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.35599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298036,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447116,"statfs":{"total":95995035648,"available":95689846784,"internally_reserved":0,"allocated":696320,"data_stored":297781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53600000000000003,"15min":0.54400000000000004},"min":{"1min":0.372,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.55000000000000004,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.55000000000000004,"15min":0.56200000000000006},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.441}]},{"osd":2,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.40400000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.76800000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:57:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53100000000000003,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56000000000000005,"15min":0.56699999999999995},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63400000000000001}]},{"osd":5,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.33500000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.65300000000000002,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.41399999999999998,"15min":0.39200000000000002},"min":{"1min":0.307,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.56699999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.59399999999999997}]},{"osd":6,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.40600000000000003,"15min":0.38800000000000001},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.63700000000000001,"5min":0.753,"15min":0.753},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.433,"15min":0.41099999999999998},"min":{"1min":0.29099999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.67500000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57999999999999996}]},{"osd":7,"last update":"Sat Sep 21 09:57:20 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.318,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.495,"15min":0.497},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.70099999999999996,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60699999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298020,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447132,"statfs":{"total":95995035648,"available":95689863168,"internally_reserved":0,"allocated":675840,"data_stored":296173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.37,"15min":0.39600000000000002},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.253},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.376,"15min":0.38600000000000001},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":0.53400000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.496}]},{"osd":1,"last update":"Sat Sep 21 09:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35799999999999998,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.39500000000000002,"15min":0.40999999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.51800000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.28899999999999998}]},{"osd":2,"last update":"Sat Sep 21 09:57:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.43099999999999999,"15min":0.45300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.312},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.39200000000000002,"15min":0.39900000000000002},"min":{"1min":0.23000000000000001,"5min":0.218,"15min":0.218},"max":{"1min":0.69899999999999995,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46600000000000003}]},{"osd":4,"last update":"Sat Sep 21 09:57:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.83399999999999996,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.441}]},{"osd":5,"last update":"Sat Sep 21 09:57:27 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.41599999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.77600000000000002,"5min":0.876,"15min":0.876},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.41999999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82699999999999996,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45300000000000001}]},{"osd":6,"last update":"Sat Sep 21 09:57:51 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86399999999999999,"5min":0.998,"15min":0.998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.61499999999999999,"15min":0.60799999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.53600000000000003}]},{"osd":7,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.63500000000000001,"15min":0.64300000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94399999999999995,"5min":1.008,"15min":1.008},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.46999999999999997,"5min":0.443,"15min":0.443},"max":{"1min":0.88400000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.42199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298024,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447128,"statfs":{"total":95995035648,"available":95689859072,"internally_reserved":0,"allocated":679936,"data_stored":296687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.35099999999999998,"15min":0.35899999999999999},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.40699999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.35499999999999998,"15min":0.35199999999999998},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.54400000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.29099999999999998}]},{"osd":1,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.35099999999999998,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.56000000000000005,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.376,"15min":0.371},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.47799999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40000000000000002}]},{"osd":3,"last update":"Sat Sep 21 09:57:41 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.39300000000000002,"15min":0.40100000000000002},"min":{"1min":0.26700000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.51600000000000001,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.40500000000000003,"15min":0.42299999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.45100000000000001,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.44900000000000001}]},{"osd":4,"last update":"Sat Sep 21 09:57:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.55300000000000005}]},{"osd":5,"last update":"Sat Sep 21 09:57:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56000000000000005,"15min":0.56100000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.49299999999999999}]},{"osd":6,"last update":"Sat Sep 21 09:57:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.502},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.377,"5min":0.375,"15min":0.375},"max":{"1min":0.78500000000000003,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.623}]},{"osd":7,"last update":"Sat Sep 21 09:57:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70099999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.81499999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":667648,"data_stored":295472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 09:57:55 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34799999999999998,"15min":0.32700000000000001},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41399999999999998,"5min":0.56200000000000006,"15min":0.56200000000000006},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.34599999999999997,"15min":0.318},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.50900000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.23699999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.371,"15min":0.377},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.433,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.315,"5min":0.36599999999999999,"15min":0.375},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.42099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.34499999999999997}]},{"osd":3,"last update":"Sat Sep 21 09:57:45 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.41199999999999998,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.52300000000000002,"5min":0.624,"15min":0.624},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.40799999999999997,"15min":0.42699999999999999},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.54000000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.315}]},{"osd":4,"last update":"Sat Sep 21 09:57:10 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55400000000000005,"15min":0.56299999999999994},"min":{"1min":0.39100000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58699999999999997,"15min":0.60499999999999998},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.78700000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.57399999999999995}]},{"osd":5,"last update":"Sat Sep 21 09:57:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55700000000000005,"15min":0.56399999999999995},"min":{"1min":0.38200000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.70399999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.374,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.65600000000000003}]},{"osd":6,"last update":"Sat Sep 21 09:57:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.55900000000000005,"15min":0.54900000000000004},"min":{"1min":0.435,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.56200000000000006},"min":{"1min":0.45600000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76500000000000001,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 21 09:57:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.66500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.621},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.69899999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298604,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446548,"statfs":{"total":95995035648,"available":95689265152,"internally_reserved":0,"allocated":1273856,"data_stored":887043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.35899999999999999,"15min":0.32600000000000001},"min":{"1min":0.191,"5min":0.158,"15min":0.158},"max":{"1min":0.44,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.313},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.35499999999999998,"15min":0.32600000000000001},"min":{"1min":0.22700000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.47499999999999998,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.30099999999999999}]},{"osd":2,"last update":"Sat Sep 21 09:57:18 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.36399999999999999,"15min":0.36199999999999999},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40000000000000002,"15min":0.39600000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.63900000000000001,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35899999999999999}]},{"osd":3,"last update":"Sat Sep 21 09:57:46 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.40899999999999997,"15min":0.42899999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.435,"15min":0.44700000000000001},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60099999999999998,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.378}]},{"osd":4,"last update":"Sat Sep 21 09:57:07 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58599999999999997,"15min":0.60899999999999999},"min":{"1min":0.32200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.92500000000000004,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.58999999999999997},"min":{"1min":0.34999999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.873,"5min":0.88,"15min":0.88},"last":0.34899999999999998}]},{"osd":5,"last update":"Sat Sep 21 09:57:31 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.59999999999999998,"15min":0.61799999999999999},"min":{"1min":0.41799999999999998,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.78300000000000003,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.60399999999999998,"15min":0.63300000000000001},"min":{"1min":0.36399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.85899999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.39900000000000002}]},{"osd":6,"last update":"Sat Sep 21 09:57:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.60399999999999998,"15min":0.623},"min":{"1min":0.377,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.70299999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.436},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.59299999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.79500000000000004,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 09:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.38,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.61099999999999999,"15min":0.61699999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84099999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.45300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T09:58:03.964 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T09:58:03.964 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T09:58:03.964 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T09:58:03.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T09:58:04.210 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:04 smithi119 ceph-mon[40785]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:04 smithi119 ceph-mon[40785]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:58:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:04 smithi119 ceph-mon[29367]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:04 smithi119 ceph-mon[29367]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:58:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:04 smithi145 ceph-mon[33206]: pgmap v84: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:04 smithi145 ceph-mon[33206]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T09:58:06.109 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T09:58:06.116 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T09:58:06.851 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T09:58:06.851 INFO:teuthology.run_tasks:Running task print... 2024-09-21T09:58:06.867 INFO:teuthology.task.print:**** upgrading first half of cluster, with stress **** 2024-09-21T09:58:06.867 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T09:58:06.895 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T09:58:06.895 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:58:06.896 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T09:58:06.900 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-21T09:58:06.901 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T09:58:06.901 DEBUG:teuthology.orchestra.run.smithi119:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:58:06.902 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T09:58:06.902 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-21T09:58:06.902 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi119.front.sepia.ceph.com 2024-09-21T09:58:06.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set mgr mgr/cephadm/daemon_cache_timeout 60' 2024-09-21T09:58:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T09:58:06.918 INFO:teuthology.orchestra.run.smithi119.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T09:58:06.919 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T09:58:06.973 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T09:58:06.973 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T09:58:07.030 INFO:tasks.workunit:timeout=3h 2024-09-21T09:58:07.030 INFO:tasks.workunit:cleanup=True 2024-09-21T09:58:07.031 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T09:58:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:06 smithi145 ceph-mon[33206]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:06 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1662628843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:58:07.087 INFO:tasks.workunit.client.0.smithi119.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T09:58:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:06 smithi119 ceph-mon[29367]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:06 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1662628843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:58:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:06 smithi119 ceph-mon[40785]: pgmap v85: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:06 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1662628843' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T09:58:07.151 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:08.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:08 smithi119 ceph-mon[40785]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:08.971 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:08 smithi119 ceph-mon[29367]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:09.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:08 smithi145 ceph-mon[33206]: pgmap v86: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:09.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:58:09] ENGINE Bus STOPPING 2024-09-21T09:58:09.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:58:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:58:09.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:58:09] ENGINE Bus STOPPED 2024-09-21T09:58:09.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:58:09] ENGINE Bus STARTING 2024-09-21T09:58:09.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:58:09] ENGINE Serving on http://:::9283 2024-09-21T09:58:09.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:58:09] ENGINE Bus STARTED 2024-09-21T09:58:09.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-21T09:58:10.204 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/307440649' entity='client.admin' 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[29367]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/307440649' entity='client.admin' 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:10.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[40785]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:10.329 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:10.329 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:10.329 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:10 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:10 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/307440649' entity='client.admin' 2024-09-21T09:58:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:10 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:10 smithi145 ceph-mon[33206]: pgmap v87: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:10 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:10 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:10 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:12 smithi145 ceph-mon[33206]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:12 smithi119 ceph-mon[29367]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:12 smithi119 ceph-mon[40785]: pgmap v88: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:13.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:58:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:13.366 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:12 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:12] "GET /metrics HTTP/1.1" 200 239793 "" "Prometheus/2.43.0" 2024-09-21T09:58:13.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-21T09:58:13.444 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T09:58:13.519 INFO:tasks.workunit.client.0.smithi119.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T09:58:13.521 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T09:58:13.521 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T09:58:13.685 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:14 smithi119 ceph-mon[40785]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:14 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:14 smithi119 ceph-mon[29367]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:14 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:14 smithi145 ceph-mon[33206]: pgmap v89: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:14 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:15.325 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T09:58:15.434 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T09:58:15.481 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T09:58:15.482 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T09:58:15.482 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T09:58:15.521 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T09:58:15.524 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T09:58:15.525 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T09:58:15.581 INFO:tasks.workunit:Running workunits matching rbd/test_librbd.sh on client.0... 2024-09-21T09:58:15.582 INFO:tasks.workunit:Running workunit rbd/test_librbd.sh... 2024-09-21T09:58:15.582 DEBUG:teuthology.orchestra.run.smithi119:workunit test rbd/test_librbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh 2024-09-21T09:58:15.706 INFO:tasks.workunit.client.0.smithi119.stderr:warning: failed to enable lockdep 2024-09-21T09:58:15.707 INFO:tasks.workunit.client.0.smithi119.stdout:seed 84733 2024-09-21T09:58:15.707 INFO:tasks.workunit.client.0.smithi119.stdout:[==========] Running 425 tests from 16 test suites. 2024-09-21T09:58:15.707 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] Global test environment set-up. 2024-09-21T09:58:15.707 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 1 test from TestJournalStress 2024-09-21T09:58:16.173 INFO:teuthology.orchestra.run.smithi119.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:16 smithi145 ceph-mon[33206]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:16 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:16 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:16 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:16 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:16 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:16.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[40785]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:16.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:16.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:16.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[29367]: pgmap v90: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:58:16.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:16 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:16.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T09:58:17.204 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:17.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[29367]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:17.224 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalStress.DiscardWithPruneWriteOverlap 2024-09-21T09:58:17.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[29367]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:17.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[29367]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:58:17.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3181002387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:17.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:17.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[29367]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:17.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[40785]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:17.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[40785]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:17.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[40785]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:58:17.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3181002387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:17.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:17.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:17 smithi119 ceph-mon[40785]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:17 smithi145 ceph-mon[33206]: from='client.24716 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:17 smithi145 ceph-mon[33206]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:17 smithi145 ceph-mon[33206]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T09:58:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:17 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3181002387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:17 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:17 smithi145 ceph-mon[33206]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:62da54b4b83a96ca9976b7e1d0c8272a564a2208 2024-09-21T09:58:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:18 smithi145 ceph-mon[33206]: pgmap v92: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:18 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:18 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:18 smithi145 ceph-mon[33206]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:58:18.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[40785]: pgmap v92: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:18.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:18.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:18.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[40785]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:58:18.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[29367]: pgmap v92: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:18.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:18.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:18.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:18 smithi119 ceph-mon[29367]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T09:58:19.250 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:19.250 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalStress.DiscardWithPruneWriteOverlap (2026 ms) 2024-09-21T09:58:19.349 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T09:58:19.349 INFO:teuthology.orchestra.run.smithi119.stdout:alertmanager.a smithi119 *:9093,9094 running (83s) 17s ago 2m 26.3M - 0.25.0 c8568f914cd2 b73cbc93508c 2024-09-21T09:58:19.349 INFO:teuthology.orchestra.run.smithi119.stdout:grafana.a smithi145 *:3000 running (2m) 20s ago 2m 83.9M - 9.4.7 954c08fa6188 375d6b959d13 2024-09-21T09:58:19.349 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi119.oikglq smithi119 running (66s) 17s ago 65s 15.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4ff2238155bf 2024-09-21T09:58:19.349 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi145.tjomfr smithi145 running (63s) 20s ago 62s 14.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b1338cf49871 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi145 *:8443 running (6m) 20s ago 6m 399M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be cc664d51cc4b 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi119 *:9283,8765 running (8m) 17s ago 8m 445M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 34c990954610 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (8m) 17s ago 8m 44.5M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1467ec54a60b 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi145 running (6m) 20s ago 6m 39.1M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f45b53ab2476 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi119 running (6m) 17s ago 6m 36.4M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d64b219d7298 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.a smithi119 *:9100 running (118s) 17s ago 115s 16.2M - 1.5.0 0da6a335fe13 8272a9f7510c 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.b smithi145 *:9100 running (111s) 20s ago 110s 8715k - 1.5.0 0da6a335fe13 9040fb6ea6a6 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (6m) 17s ago 6m 42.1M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ce07d9950c3f 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (5m) 17s ago 5m 45.4M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b8f7b7d5ac64 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (5m) 17s ago 5m 43.5M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 87104f5801c3 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (4m) 17s ago 4m 44.1M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 700faab8401d 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi145 running (4m) 20s ago 4m 41.4M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 18086eb220ec 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi145 running (4m) 20s ago 4m 42.6M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 90ab6d4f8571 2024-09-21T09:58:19.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi145 running (3m) 20s ago 3m 41.5M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 772774e9be4a 2024-09-21T09:58:19.351 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi145 running (3m) 20s ago 3m 42.2M 4016M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1a18c8e91654 2024-09-21T09:58:19.351 INFO:teuthology.orchestra.run.smithi119.stdout:prometheus.a smithi145 *:9095 running (78s) 20s ago 2m 42.7M - 2.43.0 a07b618ecd1d 69f08aa0a165 2024-09-21T09:58:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:19 smithi145 ceph-mon[33206]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:58:19.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:19 smithi119 ceph-mon[29367]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:58:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:19 smithi119 ceph-mon[40785]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T09:58:20.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for minority of mons to upgrade' 2024-09-21T09:58:20.250 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 1 test from TestJournalStress (2026 ms total) 2024-09-21T09:58:20.250 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:20.250 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 16 tests from TestJournalReplay 2024-09-21T09:58:20.347 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[29367]: pgmap v95: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[29367]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[29367]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[29367]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[40785]: pgmap v95: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[40785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:58:20.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[40785]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:58:20.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:20 smithi119 ceph-mon[40785]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:20 smithi145 ceph-mon[33206]: pgmap v95: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-21T09:58:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:20 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:20 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:58:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:20 smithi145 ceph-mon[33206]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T09:58:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:20 smithi145 ceph-mon[33206]: from='client.24722 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:21 smithi145 ceph-mon[33206]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:58:21.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:21 smithi119 ceph-mon[40785]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:58:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:21 smithi119 ceph-mon[29367]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[29367]: pgmap v98: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[29367]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3619824773' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[40785]: pgmap v98: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[40785]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3619824773' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:22.257 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:22.259 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.AioDiscardEvent 2024-09-21T09:58:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:22 smithi145 ceph-mon[33206]: pgmap v98: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:22 smithi145 ceph-mon[33206]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T09:58:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:22 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3619824773' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:22 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:22.575 INFO:teuthology.orchestra.run.smithi119.stdout:wait for minority of mons to upgrade 2024-09-21T09:58:23.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:58:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:23.263 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:22] "GET /metrics HTTP/1.1" 200 239790 "" "Prometheus/2.43.0" 2024-09-21T09:58:23.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T09:58:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:23 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:23 smithi145 ceph-mon[33206]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:58:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:23 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:23 smithi119 ceph-mon[29367]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:58:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:23 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:23 smithi119 ceph-mon[40785]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T09:58:23.654 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T09:58:24.281 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.281 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.AioDiscardEvent (2022 ms) 2024-09-21T09:58:24.281 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.AioWriteEvent 2024-09-21T09:58:24.289 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.289 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.AioWriteEvent (8 ms) 2024-09-21T09:58:24.289 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.AioFlushEvent 2024-09-21T09:58:24.298 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.298 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.AioFlushEvent (9 ms) 2024-09-21T09:58:24.298 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.SnapCreate 2024-09-21T09:58:24.306 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.306 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.SnapCreate (8 ms) 2024-09-21T09:58:24.306 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.SnapProtect 2024-09-21T09:58:24.314 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.314 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.SnapProtect (8 ms) 2024-09-21T09:58:24.314 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.SnapUnprotect 2024-09-21T09:58:24.322 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.SnapUnprotect (8 ms) 2024-09-21T09:58:24.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.SnapRename 2024-09-21T09:58:24.329 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.329 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.SnapRename (7 ms) 2024-09-21T09:58:24.329 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.SnapRollback 2024-09-21T09:58:24.337 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.337 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.SnapRollback (8 ms) 2024-09-21T09:58:24.337 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.SnapRemove 2024-09-21T09:58:24.344 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.344 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.SnapRemove (7 ms) 2024-09-21T09:58:24.344 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.Rename 2024-09-21T09:58:24.352 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.352 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.Rename (8 ms) 2024-09-21T09:58:24.352 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.Resize 2024-09-21T09:58:24.360 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.360 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.Resize (8 ms) 2024-09-21T09:58:24.360 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.Flatten 2024-09-21T09:58:24.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:24 smithi119 ceph-mon[29367]: pgmap v101: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:24 smithi119 ceph-mon[29367]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:58:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:24 smithi119 ceph-mon[40785]: pgmap v101: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:24 smithi119 ceph-mon[40785]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:58:24.368 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.368 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.Flatten (8 ms) 2024-09-21T09:58:24.368 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.UpdateFeatures 2024-09-21T09:58:24.376 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.376 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.UpdateFeatures (8 ms) 2024-09-21T09:58:24.376 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.MetadataSet 2024-09-21T09:58:24.383 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.383 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.MetadataSet (7 ms) 2024-09-21T09:58:24.383 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.MetadataRemove 2024-09-21T09:58:24.391 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.391 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.MetadataRemove (8 ms) 2024-09-21T09:58:24.391 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalReplay.ObjectPosition 2024-09-21T09:58:24.398 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:24.398 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalReplay.ObjectPosition (7 ms) 2024-09-21T09:58:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:24 smithi145 ceph-mon[33206]: pgmap v101: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:24 smithi145 ceph-mon[33206]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T09:58:25.281 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 16 tests from TestJournalReplay (2139 ms total) 2024-09-21T09:58:25.281 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:25.282 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 4 tests from TestJournalEntries 2024-09-21T09:58:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:25 smithi145 ceph-mon[33206]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:58:25.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:25 smithi119 ceph-mon[29367]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:58:25.560 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:25 smithi119 ceph-mon[40785]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T09:58:26.108 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:26 smithi119 ceph-mon[40785]: pgmap v104: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:26 smithi119 ceph-mon[40785]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:58:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:26 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3780568447' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:26 smithi119 ceph-mon[29367]: pgmap v104: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:26 smithi119 ceph-mon[29367]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:58:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:26 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3780568447' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:26 smithi145 ceph-mon[33206]: pgmap v104: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:26 smithi145 ceph-mon[33206]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T09:58:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:26 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3780568447' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:27.296 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalEntries.AioWrite 2024-09-21T09:58:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:27 smithi145 ceph-mon[33206]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:27 smithi145 ceph-mon[33206]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:58:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:27 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/581557516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:27 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:27 smithi145 ceph-mon[33206]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:27.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[40785]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[40785]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/581557516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[40785]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[29367]: from='client.14841 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[29367]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/581557516' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:27 smithi119 ceph-mon[29367]: from='client.14847 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:28 smithi145 ceph-mon[33206]: pgmap v107: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:28 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:28 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:28 smithi145 ceph-mon[33206]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:58:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:28 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:28.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:28 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:28.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[40785]: pgmap v107: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:28.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:28.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:28.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[40785]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:58:28.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:28.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:28.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[29367]: pgmap v107: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:28.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:28.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:28.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[29367]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T09:58:28.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:28.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:28 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:29.070 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:29.318 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:29.319 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalEntries.AioWrite (2022 ms) 2024-09-21T09:58:29.319 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalEntries.AioDiscard 2024-09-21T09:58:29.326 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:29.326 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalEntries.AioDiscard (8 ms) 2024-09-21T09:58:29.326 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalEntries.AioDiscardWithPrune 2024-09-21T09:58:29.335 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:29.335 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalEntries.AioDiscardWithPrune (9 ms) 2024-09-21T09:58:29.335 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestJournalEntries.AioFlush 2024-09-21T09:58:29.343 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:29.343 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestJournalEntries.AioFlush (8 ms) 2024-09-21T09:58:29.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:29 smithi119 ceph-mon[40785]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:58:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:29 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2310418997' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:29 smithi119 ceph-mon[29367]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:58:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:29 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2310418997' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:29 smithi145 ceph-mon[33206]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T09:58:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:29 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2310418997' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:30.316 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 4 tests from TestJournalEntries (2047 ms total) 2024-09-21T09:58:30.316 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:30.316 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 2 tests from TestTrash 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[40785]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[40785]: pgmap v110: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[40785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[40785]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[40785]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[29367]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[29367]: pgmap v110: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[29367]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:58:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[29367]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:58:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:30 smithi119 ceph-mon[29367]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:30 smithi145 ceph-mon[33206]: from='client.24776 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:30 smithi145 ceph-mon[33206]: pgmap v110: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:30 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:30 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:58:30.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:30 smithi145 ceph-mon[33206]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T09:58:30.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:30 smithi145 ceph-mon[33206]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:31.629 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:31 smithi119 ceph-mon[29367]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:58:31.629 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:31 smithi119 ceph-mon[40785]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:58:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:31 smithi145 ceph-mon[33206]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T09:58:32.029 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:32.324 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestTrash.UserRemovingSource 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[29367]: pgmap v113: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[29367]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3547822755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3948915749' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[40785]: pgmap v113: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[40785]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3547822755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:32.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3948915749' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:32.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:32 smithi145 ceph-mon[33206]: pgmap v113: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:32.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:32 smithi145 ceph-mon[33206]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T09:58:32.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:32 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3547822755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:32.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:32 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3948915749' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:33.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:58:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:33.331 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:32] "GET /metrics HTTP/1.1" 200 242413 "" "Prometheus/2.43.0" 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[29367]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[29367]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3547822755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[29367]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[29367]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[40785]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[40785]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3547822755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:33.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[40785]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:58:33.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:33 smithi119 ceph-mon[40785]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:58:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:33 smithi145 ceph-mon[33206]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:33 smithi145 ceph-mon[33206]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:33 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3547822755' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:33 smithi145 ceph-mon[33206]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T09:58:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:33 smithi145 ceph-mon[33206]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T09:58:34.401 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestTrash.UserRemovingSource (2079 ms) 2024-09-21T09:58:34.402 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestTrash.RestoreMirroringSource 2024-09-21T09:58:34.440 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestTrash.RestoreMirroringSource (39 ms) 2024-09-21T09:58:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:34 smithi119 ceph-mon[29367]: pgmap v116: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:34 smithi119 ceph-mon[29367]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:58:34.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:34 smithi119 ceph-mon[40785]: pgmap v116: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:34.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:34 smithi119 ceph-mon[40785]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:58:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:34 smithi145 ceph-mon[33206]: pgmap v116: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:34 smithi145 ceph-mon[33206]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T09:58:34.973 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:35.337 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:35 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3345108458' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:35.346 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 2 tests from TestTrash (2118 ms total) 2024-09-21T09:58:35.346 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:35.346 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 1 test from TestOperations 2024-09-21T09:58:35.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:35 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3345108458' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:35 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3345108458' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[29367]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[29367]: pgmap v119: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[29367]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[29367]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[40785]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[40785]: pgmap v119: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[40785]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:36 smithi119 ceph-mon[40785]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:58:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:36 smithi145 ceph-mon[33206]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:36 smithi145 ceph-mon[33206]: pgmap v119: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:36 smithi145 ceph-mon[33206]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:36 smithi145 ceph-mon[33206]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T09:58:37.358 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestOperations.DisableJournalingCorrupt 2024-09-21T09:58:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:37 smithi119 ceph-mon[40785]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:58:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:37 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2035968886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:37 smithi119 ceph-mon[29367]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:58:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:37 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2035968886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:37 smithi145 ceph-mon[33206]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T09:58:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:37 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2035968886' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:37.938 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[40785]: pgmap v122: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2035968886' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[40785]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/989395507' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[29367]: pgmap v122: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:38.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2035968886' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:38.355 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[29367]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:58:38.355 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:38 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/989395507' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:38 smithi145 ceph-mon[33206]: pgmap v122: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:38 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:38 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2035968886' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:38 smithi145 ceph-mon[33206]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T09:58:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:38 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/989395507' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:39.371 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:39.371 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestOperations.DisableJournalingCorrupt (2013 ms) 2024-09-21T09:58:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:39 smithi119 ceph-mon[40785]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:39 smithi119 ceph-mon[40785]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:39 smithi119 ceph-mon[40785]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:58:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:39 smithi119 ceph-mon[29367]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:39 smithi119 ceph-mon[29367]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:39 smithi119 ceph-mon[29367]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:58:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:39 smithi145 ceph-mon[33206]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:39 smithi145 ceph-mon[33206]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:39 smithi145 ceph-mon[33206]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T09:58:40.373 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 1 test from TestOperations (2013 ms total) 2024-09-21T09:58:40.373 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:40.373 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 4 tests from TestObjectMap 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[29367]: pgmap v125: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[29367]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[29367]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[40785]: pgmap v125: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[40785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:58:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:40 smithi119 ceph-mon[40785]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:58:40.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:40 smithi145 ceph-mon[33206]: pgmap v125: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:40.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:40 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:40.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:40 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:58:40.696 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:40 smithi145 ceph-mon[33206]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T09:58:41.064 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:41 smithi145 ceph-mon[33206]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:58:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:41 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3831523927' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:41 smithi119 ceph-mon[40785]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:58:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3831523927' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:41 smithi119 ceph-mon[29367]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T09:58:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:41 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3831523927' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:42.378 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenCorrupt 2024-09-21T09:58:42.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:42 smithi145 ceph-mon[33206]: from='client.24679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:42.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:42 smithi145 ceph-mon[33206]: pgmap v128: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:42.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:42 smithi145 ceph-mon[33206]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T09:58:42.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:42 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/247518493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:42.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:42 smithi145 ceph-mon[33206]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[29367]: from='client.24679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[29367]: pgmap v128: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[29367]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T09:58:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/247518493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:42.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[29367]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[40785]: from='client.24679 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[40785]: pgmap v128: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[40785]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T09:58:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/247518493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-mon[40785]: from='client.24887 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:43.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:58:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:43.366 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:42] "GET /metrics HTTP/1.1" 200 242413 "" "Prometheus/2.43.0" 2024-09-21T09:58:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:43 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/247518493' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:43 smithi145 ceph-mon[33206]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T09:58:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:43 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/247518493' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[29367]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/247518493' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[40785]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:44.166 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:44.429 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenCorrupt (2051 ms) 2024-09-21T09:58:44.430 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestObjectMap.RefreshInvalidatesWhenTooSmall 2024-09-21T09:58:44.458 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestObjectMap.RefreshInvalidatesWhenTooSmall (29 ms) 2024-09-21T09:58:44.458 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestObjectMap.InvalidateFlagOnDisk 2024-09-21T09:58:44.495 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestObjectMap.InvalidateFlagOnDisk (36 ms) 2024-09-21T09:58:44.495 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall 2024-09-21T09:58:44.524 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestObjectMap.AcquireLockInvalidatesWhenTooSmall (30 ms) 2024-09-21T09:58:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:44 smithi119 ceph-mon[40785]: pgmap v131: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:44 smithi119 ceph-mon[40785]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T09:58:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:44 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/4286430396' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:44 smithi119 ceph-mon[29367]: pgmap v131: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:44 smithi119 ceph-mon[29367]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T09:58:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:44 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/4286430396' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:44 smithi145 ceph-mon[33206]: pgmap v131: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:44 smithi145 ceph-mon[33206]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T09:58:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:44 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/4286430396' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:45.405 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 4 tests from TestObjectMap (2146 ms total) 2024-09-21T09:58:45.406 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:45.406 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 2 tests from TestMirroringWatcher 2024-09-21T09:58:45.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:45 smithi119 ceph-mon[29367]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:45.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:45 smithi119 ceph-mon[29367]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T09:58:45.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:45 smithi119 ceph-mon[29367]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:45.695 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:45 smithi119 ceph-mon[40785]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:45.696 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:45 smithi119 ceph-mon[40785]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T09:58:45.696 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:45 smithi119 ceph-mon[40785]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:45 smithi145 ceph-mon[33206]: from='client.24899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:45 smithi145 ceph-mon[33206]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T09:58:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:45 smithi145 ceph-mon[33206]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:46.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:46 smithi119 ceph-mon[29367]: pgmap v134: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:46 smithi119 ceph-mon[29367]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:58:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:46 smithi119 ceph-mon[40785]: pgmap v134: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:46 smithi119 ceph-mon[40785]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:58:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:46 smithi145 ceph-mon[33206]: pgmap v134: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T09:58:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:46 smithi145 ceph-mon[33206]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T09:58:47.099 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:47.427 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroringWatcher.ModeUpdated 2024-09-21T09:58:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:58:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1320605089' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:58:47.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1896295024' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:58:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1320605089' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1896295024' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T09:58:47.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1320605089' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: Upgrade: Target is version 19.3.0-5092-g62da54b4 (unknown) 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, digests ['quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47'] 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T09:58:47.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:47 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1896295024' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi145 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: pgmap v137: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:58:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:48 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi145 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: pgmap v137: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:58:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 on smithi145 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: from='client.24929 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: pgmap v137: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T09:58:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:48 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:49.460 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroringWatcher.ModeUpdated (2033 ms) 2024-09-21T09:58:49.460 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroringWatcher.ImageStatusUpdated 2024-09-21T09:58:49.480 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroringWatcher.ImageStatusUpdated (20 ms) 2024-09-21T09:58:49.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:49 smithi145 ceph-mon[33206]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:58:49.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:49 smithi119 ceph-mon[40785]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:58:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:49 smithi119 ceph-mon[29367]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T09:58:49.944 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:50.450 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 2 tests from TestMirroringWatcher (2053 ms total) 2024-09-21T09:58:50.450 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T09:58:50.450 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 57 tests from TestMigration 2024-09-21T09:58:50.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:50 smithi145 ceph-mon[33206]: pgmap v140: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:50.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:50 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:50.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:50 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T09:58:50.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:50 smithi145 ceph-mon[33206]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:58:50.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:50 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1428234104' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[40785]: pgmap v140: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[40785]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[40785]: Cluster is now healthy 2024-09-21T09:58:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[40785]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:58:50.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1428234104' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:50.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[29367]: pgmap v140: 105 pgs: 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:50.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[29367]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T09:58:50.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[29367]: Cluster is now healthy 2024-09-21T09:58:50.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[29367]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T09:58:50.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:50 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1428234104' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:51.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:51 smithi145 ceph-mon[33206]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:51.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:51 smithi145 ceph-mon[33206]: from='client.24953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:51.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:51 smithi145 ceph-mon[33206]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:58:51.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:51 smithi119 ceph-mon[40785]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:51.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:51 smithi119 ceph-mon[40785]: from='client.24953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:51.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:51 smithi119 ceph-mon[40785]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:58:51.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:51 smithi119 ceph-mon[29367]: from='client.24947 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:51.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:51 smithi119 ceph-mon[29367]: from='client.24953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:51.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:51 smithi119 ceph-mon[29367]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T09:58:52.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:52 smithi145 ceph-mon[33206]: pgmap v143: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:52.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:52 smithi145 ceph-mon[33206]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T09:58:52.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:52 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/314305749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:52.724 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:52 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:52.823 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:52.823 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[40785]: pgmap v143: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:52.823 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[40785]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T09:58:52.824 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/314305749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:52.824 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:52.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[29367]: pgmap v143: 73 pgs: 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:52.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[29367]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T09:58:52.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/314305749' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:52.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-21T09:58:53.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:58:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:58:53.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:58:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:58:52] "GET /metrics HTTP/1.1" 200 242919 "" "Prometheus/2.43.0" 2024-09-21T09:58:53.466 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.Empty 2024-09-21T09:58:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:53 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:53 smithi145 ceph-mon[33206]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T09:58:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:53 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3223911379' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:53.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:53 smithi119 ceph-mon[29367]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:53.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:53 smithi119 ceph-mon[29367]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T09:58:53.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:53 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3223911379' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:53.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:53 smithi119 ceph-mon[40785]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T09:58:53.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:53 smithi119 ceph-mon[40785]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T09:58:53.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:53 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3223911379' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:54 smithi145 ceph-mon[33206]: from='client.24977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:54 smithi145 ceph-mon[33206]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:54 smithi145 ceph-mon[33206]: pgmap v146: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:54 smithi145 ceph-mon[33206]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T09:58:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[29367]: from='client.24977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[29367]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[29367]: pgmap v146: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[29367]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T09:58:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[40785]: from='client.24977 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[40785]: from='client.15066 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:54.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[40785]: pgmap v146: 105 pgs: 32 unknown, 73 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:54.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:54 smithi119 ceph-mon[40785]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T09:58:55.519 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:55.528 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:55.663 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:55.698 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:55.728 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:58:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:55 smithi145 ceph-mon[33206]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:58:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:55 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3631937978' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:55.810 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.Empty (2343 ms) 2024-09-21T09:58:55.810 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.OtherName 2024-09-21T09:58:55.837 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:55.846 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:55.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:55 smithi119 ceph-mon[40785]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:58:55.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:55 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3631937978' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:55.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:55 smithi119 ceph-mon[29367]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T09:58:55.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:55 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3631937978' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:55.971 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:56.030 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:58:56.103 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.OtherName (293 ms) 2024-09-21T09:58:56.103 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.OtherPool 2024-09-21T09:58:56.129 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:56.138 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:56 smithi145 ceph-mon[33206]: pgmap v149: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:56 smithi145 ceph-mon[33206]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:56 smithi145 ceph-mon[33206]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:58:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:56 smithi145 ceph-mon[33206]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[29367]: pgmap v149: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[29367]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[29367]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[29367]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[40785]: pgmap v149: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[40785]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T09:58:56.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[40785]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T09:58:56.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:56 smithi119 ceph-mon[40785]: from='client.24995 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:57.600 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:57.661 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:58:57.737 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.OtherPool (1634 ms) 2024-09-21T09:58:57.738 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.OtherNamespace 2024-09-21T09:58:57.765 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:57.777 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:57.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:57 smithi145 ceph-mon[33206]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:57.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:57 smithi145 ceph-mon[33206]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:58:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:57 smithi119 ceph-mon[29367]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:57 smithi119 ceph-mon[29367]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:58:57.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:57 smithi119 ceph-mon[40785]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:57.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:57 smithi119 ceph-mon[40785]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T09:58:57.899 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:57.958 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:58:58.036 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.OtherNamespace (299 ms) 2024-09-21T09:58:58.036 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.DataPool 2024-09-21T09:58:58.062 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:58.072 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:58.203 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:58.269 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:58:58.346 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.DataPool (310 ms) 2024-09-21T09:58:58.346 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.AbortAfterPrepare 2024-09-21T09:58:58.378 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:58.389 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:58.526 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T09:58:58.617 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.AbortAfterPrepare (271 ms) 2024-09-21T09:58:58.618 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.AbortAfterFailedPrepare 2024-09-21T09:58:58.646 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:58.649 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:58:58.657 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:58.756 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.AbortAfterFailedPrepare (139 ms) 2024-09-21T09:58:58.756 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.AbortAfterExecute 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[29367]: pgmap v152: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[29367]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/769224091' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[40785]: pgmap v152: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:58.768 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[40785]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:58:58.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:58.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:58.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:58 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/769224091' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:58.783 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:58.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:58 smithi145 ceph-mon[33206]: pgmap v152: 137 pgs: 32 creating+peering, 105 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T09:58:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:58 smithi145 ceph-mon[33206]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T09:58:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:58 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:58:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:58 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:58:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:58 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/769224091' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:58:58.794 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:58.928 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:58.993 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T09:58:59.088 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.AbortAfterExecute (331 ms) 2024-09-21T09:58:59.088 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.OtherPoolAbortAfterExecute 2024-09-21T09:58:59.117 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.126 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:59.255 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:59.320 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T09:58:59.416 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.OtherPoolAbortAfterExecute (329 ms) 2024-09-21T09:58:59.416 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.OtherNamespaceAbortAfterExecute 2024-09-21T09:58:59.444 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.455 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:58:59.581 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:58:59.639 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T09:58:59.741 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.OtherNamespaceAbortAfterExecute (325 ms) 2024-09-21T09:58:59.741 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.MirroringSamePool 2024-09-21T09:58:59.769 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.779 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:58:59 smithi119 ceph-mon[29367]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:59.779 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:58:59 smithi119 ceph-mon[40785]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:59.780 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:59.787 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.MirroringSamePool (46 ms) 2024-09-21T09:58:59.787 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.MirroringAbort 2024-09-21T09:58:59.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:58:59 smithi145 ceph-mon[33206]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:58:59.815 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.825 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:59.831 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.MirroringAbort (44 ms) 2024-09-21T09:58:59.831 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.MirroringOtherPoolDisabled 2024-09-21T09:58:59.859 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.869 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:59.875 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.MirroringOtherPoolDisabled (44 ms) 2024-09-21T09:58:59.876 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.MirroringOtherPoolEnabled 2024-09-21T09:58:59.903 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.913 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:59.919 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.MirroringOtherPoolEnabled (44 ms) 2024-09-21T09:58:59.919 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.MirroringPool 2024-09-21T09:58:59.946 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:58:59.956 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T09:58:59.963 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.MirroringPool (44 ms) 2024-09-21T09:58:59.963 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.Group 2024-09-21T09:58:59.990 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:00.009 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:00.144 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:00.206 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:00.296 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.Group (332 ms) 2024-09-21T09:59:00.296 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.GroupAbort 2024-09-21T09:59:00.327 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:00.344 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:00.482 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T09:59:00.598 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.GroupAbort (303 ms) 2024-09-21T09:59:00.599 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnaps 2024-09-21T09:59:00.629 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:00.639 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:00.698 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:00.749 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:00.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:00 smithi145 ceph-mon[33206]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:00.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:00 smithi145 ceph-mon[33206]: pgmap v154: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1017 KiB/s rd, 58 KiB/s wr, 1.29k op/s 2024-09-21T09:59:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:00 smithi119 ceph-mon[29367]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:00 smithi119 ceph-mon[29367]: pgmap v154: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1017 KiB/s rd, 58 KiB/s wr, 1.29k op/s 2024-09-21T09:59:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:00 smithi119 ceph-mon[40785]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:00 smithi119 ceph-mon[40785]: pgmap v154: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1017 KiB/s rd, 58 KiB/s wr, 1.29k op/s 2024-09-21T09:59:00.922 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:01.056 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:01.173 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnaps (575 ms) 2024-09-21T09:59:01.173 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsOtherPool 2024-09-21T09:59:01.200 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:01.210 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:01.274 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:01.322 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:01.328 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:01.380 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:01.546 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:01.645 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:01.656 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:01.764 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsOtherPool (591 ms) 2024-09-21T09:59:01.765 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsDataPool 2024-09-21T09:59:01.791 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:01.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:01 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/4034044906' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:01.801 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:01.855 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:01 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/4034044906' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:01.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:01 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/4034044906' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:01.911 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:02.090 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:02.193 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:02.313 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsDataPool (549 ms) 2024-09-21T09:59:02.313 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsShrinkAfterPrepare 2024-09-21T09:59:02.341 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:02.350 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:02.411 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:02.470 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:02.656 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T09:59:02.663 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:02.775 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:02 smithi145 ceph-mon[33206]: pgmap v155: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 795 KiB/s rd, 45 KiB/s wr, 1.01k op/s 2024-09-21T09:59:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:02 smithi145 ceph-mon[33206]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:02.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:59:02.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:02 smithi119 ceph-mon[40785]: pgmap v155: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 795 KiB/s rd, 45 KiB/s wr, 1.01k op/s 2024-09-21T09:59:02.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:02 smithi119 ceph-mon[40785]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:02.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:02 smithi119 ceph-mon[29367]: pgmap v155: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 795 KiB/s rd, 45 KiB/s wr, 1.01k op/s 2024-09-21T09:59:02.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:02 smithi119 ceph-mon[29367]: from='client.25031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:02.885 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsShrinkAfterPrepare (572 ms) 2024-09-21T09:59:02.885 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroBeforePrepare 2024-09-21T09:59:02.914 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:02.922 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:02.982 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:03.028 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 0 2024-09-21T09:59:03.040 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:03.145 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:03.214 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:03.256 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroBeforePrepare (371 ms) 2024-09-21T09:59:03.256 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsShrinkToZeroAfterPrepare 2024-09-21T09:59:03.282 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:03.291 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:03.351 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:03.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:02] "GET /metrics HTTP/1.1" 200 245487 "" "Prometheus/2.43.0" 2024-09-21T09:59:03.408 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:03.579 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 0 2024-09-21T09:59:03.590 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:03.651 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:03.702 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsShrinkToZeroAfterPrepare (446 ms) 2024-09-21T09:59:03.702 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsExpandAfterPrepare 2024-09-21T09:59:03.730 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:03.739 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:03 smithi145 ceph-mon[33206]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:03 smithi145 ceph-mon[33206]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:59:03.802 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:03.855 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:03.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:03 smithi119 ceph-mon[29367]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:03.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:03 smithi119 ceph-mon[29367]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:59:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:03 smithi119 ceph-mon[40785]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:03 smithi119 ceph-mon[40785]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T09:59:04.038 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T09:59:04.045 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:04.170 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:04.303 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsExpandAfterPrepare (601 ms) 2024-09-21T09:59:04.303 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.NoSnapsSnapAfterPrepare 2024-09-21T09:59:04.332 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:04.342 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:04.396 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:04.448 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:04.621 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: after_prepare_snap 2024-09-21T09:59:04.629 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:04 smithi145 ceph-mon[33206]: pgmap v157: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 715 KiB/s rd, 41 KiB/s wr, 904 op/s 2024-09-21T09:59:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:04 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/803238082' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:04.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:04 smithi119 ceph-mon[29367]: pgmap v157: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 715 KiB/s rd, 41 KiB/s wr, 904 op/s 2024-09-21T09:59:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:04 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/803238082' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:04 smithi119 ceph-mon[40785]: pgmap v157: 137 pgs: 137 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 715 KiB/s rd, 41 KiB/s wr, 904 op/s 2024-09-21T09:59:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:04 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/803238082' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:05 smithi145 ceph-mon[33206]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:05 smithi119 ceph-mon[29367]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:05 smithi119 ceph-mon[40785]: from='client.15132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:06.541 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T09:59:06.569 INFO:tasks.workunit.client.0.smithi119.stdout:write: * 0~1000 2024-09-21T09:59:06.574 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:06.686 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:06 smithi145 ceph-mon[33206]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:06 smithi145 ceph-mon[33206]: pgmap v158: 137 pgs: 137 active+clean; 169 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 52 MiB/s rd, 41 MiB/s wr, 2.15k op/s 2024-09-21T09:59:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:06 smithi145 ceph-mon[33206]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:59:06.828 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.NoSnapsSnapAfterPrepare (2525 ms) 2024-09-21T09:59:06.828 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.Snaps 2024-09-21T09:59:06.857 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:06.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:06 smithi119 ceph-mon[29367]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:06 smithi119 ceph-mon[29367]: pgmap v158: 137 pgs: 137 active+clean; 169 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 52 MiB/s rd, 41 MiB/s wr, 2.15k op/s 2024-09-21T09:59:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:06 smithi119 ceph-mon[29367]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:59:06.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:06 smithi119 ceph-mon[40785]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:06.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:06 smithi119 ceph-mon[40785]: pgmap v158: 137 pgs: 137 active+clean; 169 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 52 MiB/s rd, 41 MiB/s wr, 2.15k op/s 2024-09-21T09:59:06.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:06 smithi119 ceph-mon[40785]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T09:59:06.868 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:06.922 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T09:59:07.455 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:07 smithi145 ceph-mon[33206]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:59:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:07 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2503766628' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:07 smithi119 ceph-mon[29367]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:59:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:07 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2503766628' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:07 smithi119 ceph-mon[40785]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T09:59:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:07 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2503766628' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:08.552 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T09:59:08.562 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T09:59:08.562 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T09:59:08.562 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T09:59:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:08 smithi145 ceph-mon[33206]: pgmap v161: 137 pgs: 137 active+clean; 169 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 68 MiB/s rd, 55 MiB/s wr, 1.85k op/s 2024-09-21T09:59:08.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:08 smithi145 ceph-mon[33206]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:08.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:08 smithi145 ceph-mon[33206]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:59:08.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:08 smithi145 ceph-mon[33206]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:08.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:08 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[29367]: pgmap v161: 137 pgs: 137 active+clean; 169 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 68 MiB/s rd, 55 MiB/s wr, 1.85k op/s 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[29367]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[29367]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[29367]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[40785]: pgmap v161: 137 pgs: 137 active+clean; 169 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 68 MiB/s rd, 55 MiB/s wr, 1.85k op/s 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[40785]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[40785]: osdmap e112: 8 total, 8 up, 8 in 2024-09-21T09:59:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[40785]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:08.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:08 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:09 smithi119 ceph-mon[40785]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:59:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:09 smithi119 ceph-mon[29367]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:59:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:09 smithi145 ceph-mon[33206]: osdmap e113: 8 total, 8 up, 8 in 2024-09-21T09:59:10.302 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:10.567 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T09:59:10.572 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T09:59:10.572 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T09:59:10.580 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T09:59:10.599 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T09:59:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:10 smithi119 ceph-mon[40785]: pgmap v164: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 22 MiB/s wr, 522 op/s 2024-09-21T09:59:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:10 smithi119 ceph-mon[40785]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:59:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:10 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/576248552' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:10 smithi119 ceph-mon[29367]: pgmap v164: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 22 MiB/s wr, 522 op/s 2024-09-21T09:59:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:10 smithi119 ceph-mon[29367]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:59:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:10 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/576248552' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:10.945 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:10 smithi145 ceph-mon[33206]: pgmap v164: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 22 MiB/s wr, 522 op/s 2024-09-21T09:59:10.945 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:10 smithi145 ceph-mon[33206]: osdmap e114: 8 total, 8 up, 8 in 2024-09-21T09:59:10.945 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:10 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/576248552' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:11.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:11 smithi119 ceph-mon[40785]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:11.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:11 smithi119 ceph-mon[40785]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:11.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:11 smithi119 ceph-mon[40785]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:59:11.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:11 smithi119 ceph-mon[29367]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:11.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:11 smithi119 ceph-mon[29367]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:11.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:11 smithi119 ceph-mon[29367]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:59:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:11 smithi145 ceph-mon[33206]: from='client.25085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:11 smithi145 ceph-mon[33206]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:11 smithi145 ceph-mon[33206]: osdmap e115: 8 total, 8 up, 8 in 2024-09-21T09:59:12.583 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:12.603 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T09:59:12.610 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:12 smithi119 ceph-mon[40785]: pgmap v167: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 22 MiB/s wr, 522 op/s 2024-09-21T09:59:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:12 smithi119 ceph-mon[40785]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:59:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:12 smithi119 ceph-mon[29367]: pgmap v167: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 22 MiB/s wr, 522 op/s 2024-09-21T09:59:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:12 smithi119 ceph-mon[29367]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:59:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:12 smithi145 ceph-mon[33206]: pgmap v167: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 22 MiB/s wr, 522 op/s 2024-09-21T09:59:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:12 smithi145 ceph-mon[33206]: osdmap e116: 8 total, 8 up, 8 in 2024-09-21T09:59:13.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:59:13.148 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:13.366 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:12 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:12] "GET /metrics HTTP/1.1" 200 245487 "" "Prometheus/2.43.0" 2024-09-21T09:59:13.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:13 smithi119 ceph-mon[40785]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:59:13.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:13 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1757754708' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:13.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:13 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:13 smithi119 ceph-mon[29367]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:59:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:13 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1757754708' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:13 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:13 smithi145 ceph-mon[33206]: osdmap e117: 8 total, 8 up, 8 in 2024-09-21T09:59:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:13 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1757754708' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:13 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[40785]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[40785]: pgmap v170: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[40785]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[40785]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:59:14.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[29367]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:14.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[29367]: pgmap v170: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:14.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[29367]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:14.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:14 smithi119 ceph-mon[29367]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:59:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:14 smithi145 ceph-mon[33206]: from='client.15186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:14 smithi145 ceph-mon[33206]: pgmap v170: 137 pgs: 137 active+clean; 225 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:14 smithi145 ceph-mon[33206]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:14 smithi145 ceph-mon[33206]: osdmap e118: 8 total, 8 up, 8 in 2024-09-21T09:59:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:15 smithi119 ceph-mon[29367]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:59:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:15 smithi119 ceph-mon[40785]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:59:15.987 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:15 smithi145 ceph-mon[33206]: osdmap e119: 8 total, 8 up, 8 in 2024-09-21T09:59:16.077 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:16.139 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:16 smithi119 ceph-mon[40785]: pgmap v173: 137 pgs: 137 active+clean; 249 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:16 smithi119 ceph-mon[40785]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:59:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:16 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/771765623' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:16 smithi119 ceph-mon[29367]: pgmap v173: 137 pgs: 137 active+clean; 249 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:16 smithi119 ceph-mon[29367]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:59:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:16 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/771765623' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:16 smithi145 ceph-mon[33206]: pgmap v173: 137 pgs: 137 active+clean; 249 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:16 smithi145 ceph-mon[33206]: osdmap e120: 8 total, 8 up, 8 in 2024-09-21T09:59:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:16 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/771765623' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:17 smithi119 ceph-mon[40785]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:17 smithi119 ceph-mon[40785]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:17 smithi119 ceph-mon[40785]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:59:17.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:17 smithi119 ceph-mon[29367]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:17.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:17 smithi119 ceph-mon[29367]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:17.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:17 smithi119 ceph-mon[29367]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:59:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:17 smithi145 ceph-mon[33206]: from='client.25121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:17 smithi145 ceph-mon[33206]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:17 smithi145 ceph-mon[33206]: osdmap e121: 8 total, 8 up, 8 in 2024-09-21T09:59:18.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:18 smithi119 ceph-mon[40785]: pgmap v176: 137 pgs: 137 active+clean; 249 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:18.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:18 smithi119 ceph-mon[40785]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:59:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:18 smithi119 ceph-mon[29367]: pgmap v176: 137 pgs: 137 active+clean; 249 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:18 smithi119 ceph-mon[29367]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:59:18.898 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.Snaps (12070 ms) 2024-09-21T09:59:18.898 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsOtherPool 2024-09-21T09:59:18.927 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:18.937 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:18.981 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:19.000 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T09:59:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:18 smithi145 ceph-mon[33206]: pgmap v176: 137 pgs: 137 active+clean; 249 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:18 smithi145 ceph-mon[33206]: osdmap e122: 8 total, 8 up, 8 in 2024-09-21T09:59:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:19 smithi145 ceph-mon[33206]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T09:59:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:19 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1157694111' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:19 smithi119 ceph-mon[40785]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T09:59:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:19 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1157694111' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:19 smithi119 ceph-mon[29367]: osdmap e123: 8 total, 8 up, 8 in 2024-09-21T09:59:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:19 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1157694111' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:20.626 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T09:59:20.647 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T09:59:20.647 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T09:59:20.647 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T09:59:20.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:20 smithi145 ceph-mon[33206]: from='client.25139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:20.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:20 smithi145 ceph-mon[33206]: pgmap v179: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-21T09:59:20.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:20 smithi145 ceph-mon[33206]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:20.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:20 smithi145 ceph-mon[33206]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[29367]: from='client.25139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[29367]: pgmap v179: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[29367]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[29367]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[40785]: from='client.25139 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[40785]: pgmap v179: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[40785]: from='client.15228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:20 smithi119 ceph-mon[40785]: osdmap e124: 8 total, 8 up, 8 in 2024-09-21T09:59:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T09:59:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:59:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:59:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:59:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:21.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:21 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/694764852' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:21.815 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/694764852' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: osdmap e125: 8 total, 8 up, 8 in 2024-09-21T09:59:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:59:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T09:59:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T09:59:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:21 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/694764852' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:22.632 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:22 smithi145 systemd[1]: Stopping Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:22.641 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T09:59:22.648 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T09:59:22.648 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T09:59:22.656 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T09:59:22.681 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[29367]: Upgrade: Updating mgr.x 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[29367]: Deploying daemon mgr.x on smithi145 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[29367]: pgmap v182: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[29367]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[40785]: Upgrade: Updating mgr.x 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[40785]: Deploying daemon mgr.x on smithi145 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[40785]: pgmap v182: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-21T09:59:22.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-mon[40785]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:59:23.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[35285]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:59:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:22 smithi145 ceph-mon[33206]: Upgrade: Updating mgr.x 2024-09-21T09:59:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:22 smithi145 ceph-mon[33206]: Deploying daemon mgr.x on smithi145 2024-09-21T09:59:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:22 smithi145 ceph-mon[33206]: pgmap v182: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 70 MiB/s rd, 10 MiB/s wr, 491 op/s 2024-09-21T09:59:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:22 smithi145 ceph-mon[33206]: osdmap e126: 8 total, 8 up, 8 in 2024-09-21T09:59:23.360 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 podman[68485]: 2024-09-21 09:59:23.072872391 +0000 UTC m=+0.422169752 container died cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T09:59:23.366 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:22] "GET /metrics HTTP/1.1" 200 246257 "" "Prometheus/2.43.0" 2024-09-21T09:59:23.631 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 podman[68485]: 2024-09-21 09:59:23.51621747 +0000 UTC m=+0.865514830 container cleanup cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T09:59:23.631 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 bash[68485]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T09:59:23.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:23 smithi145 ceph-mon[33206]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:23.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:23 smithi145 ceph-mon[33206]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:23.974 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:23 smithi145 ceph-mon[33206]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:59:23.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:23 smithi145 ceph-mon[33206]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:59:23.975 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 podman[68503]: 2024-09-21 09:59:23.776513231 +0000 UTC m=+0.699061957 container remove cc664d51cc4b543ea8892aedfc4d905f696d9a0456b02d45130ac94e4f66dd3a (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T09:59:23.975 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[40785]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[40785]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[40785]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[40785]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[29367]: from='client.25157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[29367]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[29367]: osdmap e127: 8 total, 8 up, 8 in 2024-09-21T09:59:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:23 smithi119 ceph-mon[29367]: osdmap e128: 8 total, 8 up, 8 in 2024-09-21T09:59:24.158 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:24.179 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T09:59:24.187 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:24.195 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T09:59:24.258 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:24.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T09:59:24.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 systemd[1]: Stopped Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:24.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:23 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service: Consumed 10.097s CPU time. 2024-09-21T09:59:24.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 systemd[1]: Starting Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T09:59:24.679 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:24 smithi145 ceph-mon[33206]: pgmap v186: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:24.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:24 smithi145 ceph-mon[33206]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:59:24.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:24 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1977791616' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:24.796 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 podman[68634]: 2024-09-21 09:59:24.325812338 +0000 UTC m=+0.015740685 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T09:59:24.797 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 podman[68634]: 2024-09-21 09:59:24.427627964 +0000 UTC m=+0.117556311 container create 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T09:59:24.797 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 podman[68634]: 2024-09-21 09:59:24.672058168 +0000 UTC m=+0.361986528 container init 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T09:59:24.797 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 podman[68634]: 2024-09-21 09:59:24.677337522 +0000 UTC m=+0.367265870 container start 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T09:59:24.797 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:25.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 ceph-mgr[68671]: -- 172.21.15.145:0/883898070 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562257c190e0 con 0x562257bf6800 2024-09-21T09:59:25.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 bash[68634]: 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa 2024-09-21T09:59:25.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:24 smithi145 systemd[1]: Started Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T09:59:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:24 smithi119 ceph-mon[40785]: pgmap v186: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:24 smithi119 ceph-mon[40785]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:59:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:24 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1977791616' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:24 smithi119 ceph-mon[29367]: pgmap v186: 137 pgs: 137 active+clean; 265 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:24 smithi119 ceph-mon[29367]: osdmap e129: 8 total, 8 up, 8 in 2024-09-21T09:59:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:24 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1977791616' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[29367]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[29367]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:59:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[40785]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:25.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:25 smithi119 ceph-mon[40785]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:59:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:25 smithi145 ceph-mon[33206]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:25 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:25 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:25 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:25 smithi145 ceph-mon[33206]: osdmap e130: 8 total, 8 up, 8 in 2024-09-21T09:59:26.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:26 smithi119 ceph-mon[40785]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:26.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:26 smithi119 ceph-mon[40785]: pgmap v189: 137 pgs: 137 active+clean; 305 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 79 KiB/s rd, 9.1 MiB/s wr, 117 op/s 2024-09-21T09:59:26.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:26 smithi119 ceph-mon[40785]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:59:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:26 smithi119 ceph-mon[29367]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:26 smithi119 ceph-mon[29367]: pgmap v189: 137 pgs: 137 active+clean; 305 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 79 KiB/s rd, 9.1 MiB/s wr, 117 op/s 2024-09-21T09:59:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:26 smithi119 ceph-mon[29367]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:59:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:26 smithi145 ceph-mon[33206]: from='client.15270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:26 smithi145 ceph-mon[33206]: pgmap v189: 137 pgs: 137 active+clean; 305 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 79 KiB/s rd, 9.1 MiB/s wr, 117 op/s 2024-09-21T09:59:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:26 smithi145 ceph-mon[33206]: osdmap e131: 8 total, 8 up, 8 in 2024-09-21T09:59:27.582 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:27.625 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:27.784 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:28 smithi145 ceph-mon[33206]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:59:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:28 smithi145 ceph-mon[33206]: pgmap v192: 137 pgs: 137 active+clean; 305 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:59:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:28 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/552484275' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:28 smithi145 ceph-mon[33206]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:28 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:28 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[29367]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[29367]: pgmap v192: 137 pgs: 137 active+clean; 305 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/552484275' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[29367]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[40785]: osdmap e132: 8 total, 8 up, 8 in 2024-09-21T09:59:28.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[40785]: pgmap v192: 137 pgs: 137 active+clean; 305 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T09:59:28.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/552484275' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:28.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[40785]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:28.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:28.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:28 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-mon[33206]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-mon[33206]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:59:29.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:29 smithi119 ceph-mon[40785]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:29 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:29 smithi119 ceph-mon[40785]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:59:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:29 smithi119 ceph-mon[29367]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:29 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:29 smithi119 ceph-mon[29367]: osdmap e133: 8 total, 8 up, 8 in 2024-09-21T09:59:30.167 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:29] ENGINE Bus STARTING 2024-09-21T09:59:30.167 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: CherryPy Checker: 2024-09-21T09:59:30.167 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: The Application mounted at '' has an empty config. 2024-09-21T09:59:30.167 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:30.167 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:29] ENGINE Serving on http://:::9283 2024-09-21T09:59:30.168 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:29 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:29] ENGINE Bus STARTED 2024-09-21T09:59:30.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: pgmap v194: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 496 op/s 2024-09-21T09:59:30.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:59:30.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: Standby manager daemon x restarted 2024-09-21T09:59:30.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: Standby manager daemon x started 2024-09-21T09:59:30.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:59:30.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: pgmap v194: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 496 op/s 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: Standby manager daemon x restarted 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: Standby manager daemon x started 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:59:30.508 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:59:30.509 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:30 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:59:30.509 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: pgmap v194: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 496 op/s 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: osdmap e134: 8 total, 8 up, 8 in 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: Standby manager daemon x restarted 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: Standby manager daemon x started 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T09:59:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:30 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:59:31.118 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsOtherPool (12220 ms) 2024-09-21T09:59:31.118 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsDataPool 2024-09-21T09:59:31.155 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:31.158 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:31.217 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T09:59:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:31 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/182385427' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:31 smithi145 ceph-mon[33206]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:31 smithi145 ceph-mon[33206]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:59:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:31 smithi145 ceph-mon[33206]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/182385427' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[40785]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[40785]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[40785]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/182385427' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[29367]: from='client.25217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[29367]: osdmap e135: 8 total, 8 up, 8 in 2024-09-21T09:59:31.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:31 smithi119 ceph-mon[29367]: mgrmap e21: y(active, since 3m), standbys: x 2024-09-21T09:59:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:32 smithi145 ceph-mon[33206]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:32 smithi145 ceph-mon[33206]: pgmap v197: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 497 op/s 2024-09-21T09:59:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:32 smithi119 ceph-mon[29367]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:32 smithi119 ceph-mon[29367]: pgmap v197: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 497 op/s 2024-09-21T09:59:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:32 smithi119 ceph-mon[40785]: from='client.25223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:32 smithi119 ceph-mon[40785]: pgmap v197: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 46 MiB/s rd, 18 MiB/s wr, 497 op/s 2024-09-21T09:59:33.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:59:33.158 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T09:59:33.171 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T09:59:33.171 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T09:59:33.171 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T09:59:33.346 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:33.346 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:32] "GET /metrics HTTP/1.1" 200 246298 "" "Prometheus/2.43.0" 2024-09-21T09:59:33.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:33 smithi119 ceph-mon[29367]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:59:33.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:33 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1330768383' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:33.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:33 smithi119 ceph-mon[29367]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:59:33.347 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:33 smithi119 ceph-mon[40785]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:59:33.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:33 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1330768383' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:33.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:33 smithi119 ceph-mon[40785]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:59:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:33 smithi145 ceph-mon[33206]: osdmap e136: 8 total, 8 up, 8 in 2024-09-21T09:59:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:33 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1330768383' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:33 smithi145 ceph-mon[33206]: osdmap e137: 8 total, 8 up, 8 in 2024-09-21T09:59:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:34 smithi145 ceph-mon[33206]: pgmap v200: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:34 smithi145 ceph-mon[33206]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:34 smithi145 ceph-mon[33206]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:34 smithi119 ceph-mon[29367]: pgmap v200: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:34 smithi119 ceph-mon[29367]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:34 smithi119 ceph-mon[29367]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:34.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:34 smithi119 ceph-mon[40785]: pgmap v200: 137 pgs: 137 active+clean; 373 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:34.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:34 smithi119 ceph-mon[40785]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:34.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:34 smithi119 ceph-mon[40785]: from='client.15324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:35.209 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T09:59:35.214 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T09:59:35.214 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T09:59:35.222 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T09:59:35.246 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T09:59:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:35 smithi145 ceph-mon[33206]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:59:35.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:35 smithi119 ceph-mon[40785]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:59:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:35 smithi119 ceph-mon[29367]: osdmap e138: 8 total, 8 up, 8 in 2024-09-21T09:59:36.169 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:36 smithi145 ceph-mon[33206]: pgmap v202: 137 pgs: 137 active+clean; 369 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 33 MiB/s rd, 7.3 MiB/s wr, 121 op/s 2024-09-21T09:59:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:36 smithi145 ceph-mon[33206]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:59:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:36 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3551166160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:36 smithi119 ceph-mon[40785]: pgmap v202: 137 pgs: 137 active+clean; 369 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 33 MiB/s rd, 7.3 MiB/s wr, 121 op/s 2024-09-21T09:59:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:36 smithi119 ceph-mon[40785]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:59:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:36 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3551166160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:36 smithi119 ceph-mon[29367]: pgmap v202: 137 pgs: 137 active+clean; 369 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 33 MiB/s rd, 7.3 MiB/s wr, 121 op/s 2024-09-21T09:59:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:36 smithi119 ceph-mon[29367]: osdmap e139: 8 total, 8 up, 8 in 2024-09-21T09:59:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:36 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3551166160' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:37.219 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:37.241 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T09:59:37.248 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:37.473 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:37 smithi145 ceph-mon[33206]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.473 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:37 smithi145 ceph-mon[33206]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:59:37.473 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:37 smithi145 ceph-mon[33206]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:37 smithi119 ceph-mon[29367]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:37 smithi119 ceph-mon[29367]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:59:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:37 smithi119 ceph-mon[29367]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:37 smithi119 ceph-mon[40785]: from='client.25253 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:37 smithi119 ceph-mon[40785]: osdmap e140: 8 total, 8 up, 8 in 2024-09-21T09:59:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:37 smithi119 ceph-mon[40785]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:38 smithi145 ceph-mon[33206]: pgmap v205: 137 pgs: 137 active+clean; 369 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 8.0 MiB/s wr, 131 op/s 2024-09-21T09:59:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:38 smithi145 ceph-mon[33206]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T09:59:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:38 smithi145 ceph-mon[33206]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T09:59:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:38 smithi119 ceph-mon[29367]: pgmap v205: 137 pgs: 137 active+clean; 369 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 8.0 MiB/s wr, 131 op/s 2024-09-21T09:59:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:38 smithi119 ceph-mon[29367]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T09:59:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:38 smithi119 ceph-mon[29367]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T09:59:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:38 smithi119 ceph-mon[40785]: pgmap v205: 137 pgs: 137 active+clean; 369 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 36 MiB/s rd, 8.0 MiB/s wr, 131 op/s 2024-09-21T09:59:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:38 smithi119 ceph-mon[40785]: osdmap e141: 8 total, 8 up, 8 in 2024-09-21T09:59:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:38 smithi119 ceph-mon[40785]: osdmap e142: 8 total, 8 up, 8 in 2024-09-21T09:59:39.005 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:39.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:39 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/54853688' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:39.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:39 smithi119 ceph-mon[29367]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T09:59:39.336 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:39 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/54853688' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:39.336 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:39 smithi119 ceph-mon[40785]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T09:59:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:39 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/54853688' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:39 smithi145 ceph-mon[33206]: osdmap e143: 8 total, 8 up, 8 in 2024-09-21T09:59:40.537 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:40.683 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: pgmap v209: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 120 KiB/s rd, 4.0 MiB/s wr, 171 op/s 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: pgmap v209: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 120 KiB/s rd, 4.0 MiB/s wr, 171 op/s 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:40 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.914 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: from='client.15354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.914 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: pgmap v209: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 120 KiB/s rd, 4.0 MiB/s wr, 171 op/s 2024-09-21T09:59:40.914 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:40.914 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.914 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.914 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: osdmap e144: 8 total, 8 up, 8 in 2024-09-21T09:59:40.915 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:40.915 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:40 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:41 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/213568457' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:41.836 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/213568457' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:41 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/213568457' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:42.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:42 smithi145 ceph-mon[33206]: pgmap v211: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:42.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:42 smithi145 ceph-mon[33206]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T09:59:42.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:42 smithi119 ceph-mon[40785]: pgmap v211: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:42.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:42 smithi119 ceph-mon[40785]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T09:59:42.949 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:42 smithi119 ceph-mon[29367]: pgmap v211: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T09:59:42.949 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:42 smithi119 ceph-mon[29367]: osdmap e145: 8 total, 8 up, 8 in 2024-09-21T09:59:43.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:42 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:59:43.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:42] "GET /metrics HTTP/1.1" 200 246298 "" "Prometheus/2.43.0" 2024-09-21T09:59:43.965 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsDataPool (12847 ms) 2024-09-21T09:59:43.966 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsShrinkAfterPrepare 2024-09-21T09:59:43.984 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[40785]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:43.984 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[40785]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:43.984 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[40785]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T09:59:43.984 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:43.984 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setuser ceph since I am not root 2024-09-21T09:59:43.984 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ignoring --setgroup ceph since I am not root 2024-09-21T09:59:43.985 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mgr[29640]: -- 172.21.15.119:0/2545896366 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b418794340 con 0x55b4187a0c00 2024-09-21T09:59:43.985 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[29367]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:43.985 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[29367]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:43.985 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[29367]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T09:59:43.985 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:43.998 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:44.009 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:44.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:43 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:43] ENGINE Bus STOPPING 2024-09-21T09:59:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:43 smithi145 ceph-mon[33206]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:43 smithi145 ceph-mon[33206]: from='client.25295 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:43 smithi145 ceph-mon[33206]: osdmap e146: 8 total, 8 up, 8 in 2024-09-21T09:59:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:43 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:44.067 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T09:59:44.255 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:43 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:43.981+0000 7fb8a1d73ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T09:59:44.255 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.052+0000 7fb8a1d73ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T09:59:44.256 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.120+0000 7fb8a1d73ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T09:59:44.256 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.253+0000 7fb8a1d73ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:59:44.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T09:59:44.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:44] ENGINE Bus STOPPED 2024-09-21T09:59:44.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.392+0000 7fb8a1d73ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T09:59:44.852 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:44] ENGINE Bus STARTING 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.866+0000 7f025b89e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.866+0000 7f025b89e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.866+0000 7f025b89e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.866+0000 7f025b89e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.866+0000 7f025b89e640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.873+0000 7f0258097640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.873+0000 7f0258097640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.873+0000 7f0258097640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.903 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.873+0000 7f0258097640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:44.873+0000 7f0258097640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:44] ENGINE Serving on http://:::9283 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:09:59:44] ENGINE Bus STARTED 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: pgmap v214: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.904 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: Activating manager daemon x 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: mgrmap e22: x(active, starting, since 0.00955747s) 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:59:44.905 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:59:44.906 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: Manager daemon x is now available 2024-09-21T09:59:44.907 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:44 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1940510514' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:44.907 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.658+0000 7fb8a1d73ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T09:59:44.907 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.718+0000 7fb8a1d73ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T09:59:44.907 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: pgmap v214: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: Activating manager daemon x 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:59:44.908 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: mgrmap e22: x(active, starting, since 0.00955747s) 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:59:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: Manager daemon x is now available 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1940510514' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: pgmap v214: 137 pgs: 137 active+clean; 393 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: osdmap e147: 8 total, 8 up, 8 in 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.910 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 172.21.15.119:0/1104130524' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: Activating manager daemon x 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: osdmap e148: 8 total, 8 up, 8 in 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24347 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: mgrmap e22: x(active, starting, since 0.00955747s) 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T09:59:44.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: Manager daemon x is now available 2024-09-21T09:59:44.912 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1940510514' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:45.366 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:44.901+0000 7fb8a1d73ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T09:59:45.837 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T09:59:45.853 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T09:59:45.853 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T09:59:45.853 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T09:59:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: Checking for cert/key for grafana.a 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: Migrating grafana.a cert to cert store 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: Migrating grafana.a key to cert store 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: mgrmap e23: x(active, since 1.01486s) 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: pgmap v4: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: Deploying cephadm binary to smithi145 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: Deploying cephadm binary to smithi119 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: [21/Sep/2024:09:59:45] ENGINE Bus STARTING 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: [21/Sep/2024:09:59:45] ENGINE Serving on https://172.21.15.145:7150 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-mon[33206]: [21/Sep/2024:09:59:45] ENGINE Error in HTTPServer.serve 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout: Traceback (most recent call last): 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._run(expiration_interval) 2024-09-21T09:59:46.046 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: s = self.context.wrap_socket( 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: return self.sslsocket_class._create( 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: self.do_handshake() 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._sslobj.do_handshake() 2024-09-21T09:59:46.047 INFO:journalctl@ceph.mon.b.smithi145.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:45.666+0000 7f0243df2640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:09:59:45] ENGINE Error in HTTPServer.serve 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: Traceback (most recent call last): 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: self._connections.run(self.expiration_interval) 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: self._run(expiration_interval) 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: s = self.context.wrap_socket( 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:59:46.048 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: return self.sslsocket_class._create( 2024-09-21T09:59:46.049 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:59:46.049 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: self.do_handshake() 2024-09-21T09:59:46.049 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:59:46.049 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: self._sslobj.do_handshake() 2024-09-21T09:59:46.049 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:59:46.049 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T09:59:46.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:45.761+0000 7fb8a1d73ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T09:59:46.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:45.829+0000 7fb8a1d73ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T09:59:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: Checking for cert/key for grafana.a 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: Migrating grafana.a cert to cert store 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: Migrating grafana.a key to cert store 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:59:46.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: mgrmap e23: x(active, since 1.01486s) 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: pgmap v4: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: Deploying cephadm binary to smithi145 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: Deploying cephadm binary to smithi119 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: [21/Sep/2024:09:59:45] ENGINE Bus STARTING 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: [21/Sep/2024:09:59:45] ENGINE Serving on https://172.21.15.145:7150 2024-09-21T09:59:46.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[40785]: [21/Sep/2024:09:59:45] ENGINE Error in HTTPServer.serve 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: Traceback (most recent call last): 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._run(expiration_interval) 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:59:46.120 INFO:journalctl@ceph.mon.c.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.c.smithi119.stdout: self.do_handshake() 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.c.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: Checking for cert/key for grafana.a 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: Migrating grafana.a cert to cert store 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: Migrating grafana.a key to cert store 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: osdmap e149: 8 total, 8 up, 8 in 2024-09-21T09:59:46.122 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: mgrmap e23: x(active, since 1.01486s) 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='client.15402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: pgmap v4: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: Deploying cephadm binary to smithi145 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: Deploying cephadm binary to smithi119 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: [21/Sep/2024:09:59:45] ENGINE Bus STARTING 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: [21/Sep/2024:09:59:45] ENGINE Serving on https://172.21.15.145:7150 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:45 smithi119 ceph-mon[29367]: [21/Sep/2024:09:59:45] ENGINE Error in HTTPServer.serve 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-21T09:59:46.123 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T09:59:46.124 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T09:59:46.125 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-21T09:59:46.125 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T09:59:46.125 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T09:59:46.125 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T09:59:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:46.550+0000 7fb8a1d73ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T09:59:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:46.660+0000 7fb8a1d73ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T09:59:46.827 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:46.785+0000 7fb8a1d73ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T09:59:47.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[29367]: [21/Sep/2024:09:59:45] ENGINE Serving on http://172.21.15.145:8765 2024-09-21T09:59:47.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[29367]: [21/Sep/2024:09:59:45] ENGINE Bus STARTED 2024-09-21T09:59:47.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[29367]: pgmap v5: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:47.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[29367]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[40785]: [21/Sep/2024:09:59:45] ENGINE Serving on http://172.21.15.145:8765 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[40785]: [21/Sep/2024:09:59:45] ENGINE Bus STARTED 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[40785]: pgmap v5: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-mon[40785]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:46.854+0000 7fb8a1d73ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.002+0000 7fb8a1d73ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T09:59:47.083 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.080+0000 7fb8a1d73ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T09:59:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:46 smithi145 ceph-mon[33206]: [21/Sep/2024:09:59:45] ENGINE Serving on http://172.21.15.145:8765 2024-09-21T09:59:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:46 smithi145 ceph-mon[33206]: [21/Sep/2024:09:59:45] ENGINE Bus STARTED 2024-09-21T09:59:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:46 smithi145 ceph-mon[33206]: pgmap v5: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:46 smithi145 ceph-mon[33206]: osdmap e150: 8 total, 8 up, 8 in 2024-09-21T09:59:47.349 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.139+0000 7fb8a1d73ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T09:59:47.350 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.279+0000 7fb8a1d73ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T09:59:47.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.347+0000 7fb8a1d73ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T09:59:47.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.416+0000 7fb8a1d73ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T09:59:47.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.500+0000 7fb8a1d73ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T09:59:47.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.560+0000 7fb8a1d73ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T09:59:47.832 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:47.849 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T09:59:47.857 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T09:59:47.857 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T09:59:47.866 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T09:59:47.880 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:47.625+0000 7fb8a1d73ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-mon[29367]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-mon[29367]: mgrmap e24: x(active, since 3s) 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3296669547' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-mon[40785]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-mon[40785]: mgrmap e24: x(active, since 3s) 2024-09-21T09:59:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:47 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3296669547' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:48.201 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:47 smithi145 ceph-mon[33206]: osdmap e151: 8 total, 8 up, 8 in 2024-09-21T09:59:48.202 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:47 smithi145 ceph-mon[33206]: mgrmap e24: x(active, since 3s) 2024-09-21T09:59:48.202 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3296669547' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:48.842 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:48.483+0000 7fb8a1d73ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T09:59:48.842 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:59:48] ENGINE Bus STARTING 2024-09-21T09:59:48.842 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: CherryPy Checker: 2024-09-21T09:59:48.842 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: The Application mounted at '' has an empty config. 2024-09-21T09:59:48.842 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: 2024-09-21T09:59:48.843 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:59:48] ENGINE Serving on http://:::9283 2024-09-21T09:59:48.843 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: [21/Sep/2024:09:59:48] ENGINE Bus STARTED 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: pgmap v8: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: Standby manager daemon y started 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:59:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: pgmap v8: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: Standby manager daemon y started 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:59:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:59:49.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:48 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:59:49.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:49.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: pgmap v8: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:49.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: osdmap e152: 8 total, 8 up, 8 in 2024-09-21T09:59:49.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: osdmap e153: 8 total, 8 up, 8 in 2024-09-21T09:59:49.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:49.153 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: Standby manager daemon y started 2024-09-21T09:59:49.154 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T09:59:49.154 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T09:59:49.154 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T09:59:49.154 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:48 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/2852115395' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T09:59:49.165 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:49.184 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T09:59:49.192 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T09:59:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:50 smithi145 ceph-mon[33206]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T09:59:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:50 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:59:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:50 smithi145 ceph-mon[33206]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:59:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:50 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:50 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[29367]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[29367]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[40785]: osdmap e154: 8 total, 8 up, 8 in 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[40785]: mgrmap e25: x(active, since 5s), standbys: y 2024-09-21T09:59:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:50.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:50 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:50.809 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: pgmap v12: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi119 to 2480M 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:51 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3513105825' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: pgmap v12: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi119 to 2480M 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3513105825' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: pgmap v12: 137 pgs: 137 active+clean; 409 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: osdmap e155: 8 total, 8 up, 8 in 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi119 to 2480M 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:51.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:51 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3513105825' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: from='client.25388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.382 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.534 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T09:59:52.555 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: from='client.25388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: from='client.25388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: osdmap e156: 8 total, 8 up, 8 in 2024-09-21T09:59:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:52.689 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T09:59:53.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 09:59:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:52] "GET /metrics HTTP/1.1" 200 36294 "" "Prometheus/2.43.0" 2024-09-21T09:59:53.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 09:59:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:09:59:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T09:59:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:53 smithi119 ceph-mon[40785]: pgmap v15: 137 pgs: 137 active+clean; 449 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 185 KiB/s rd, 8.0 MiB/s wr, 220 op/s 2024-09-21T09:59:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:53 smithi119 ceph-mon[40785]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T09:59:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:53 smithi119 ceph-mon[29367]: pgmap v15: 137 pgs: 137 active+clean; 449 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 185 KiB/s rd, 8.0 MiB/s wr, 220 op/s 2024-09-21T09:59:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:53 smithi119 ceph-mon[29367]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T09:59:53.486 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:53 smithi145 ceph-mon[33206]: pgmap v15: 137 pgs: 137 active+clean; 449 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 185 KiB/s rd, 8.0 MiB/s wr, 220 op/s 2024-09-21T09:59:53.486 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:53 smithi145 ceph-mon[33206]: osdmap e157: 8 total, 8 up, 8 in 2024-09-21T09:59:53.715 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:54 smithi145 ceph-mon[33206]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T09:59:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:54 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/60061527' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:54 smithi145 ceph-mon[33206]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:54.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:54 smithi119 ceph-mon[29367]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T09:59:54.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:54 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/60061527' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:54.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:54 smithi119 ceph-mon[29367]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:54 smithi119 ceph-mon[40785]: osdmap e158: 8 total, 8 up, 8 in 2024-09-21T09:59:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:54 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/60061527' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:54 smithi119 ceph-mon[40785]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: pgmap v18: 137 pgs: 137 active+clean; 449 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 184 KiB/s rd, 8.0 MiB/s wr, 220 op/s 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:55 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:55.466 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsShrinkAfterPrepare (11501 ms) 2024-09-21T09:59:55.466 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroBeforePrepare 2024-09-21T09:59:55.497 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T09:59:55.508 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T09:59:55.567 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: pgmap v18: 137 pgs: 137 active+clean; 449 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 184 KiB/s rd, 8.0 MiB/s wr, 220 op/s 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: pgmap v18: 137 pgs: 137 active+clean; 449 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 184 KiB/s rd, 8.0 MiB/s wr, 220 op/s 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: osdmap e159: 8 total, 8 up, 8 in 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T09:59:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:55 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T09:59:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Adjusting osd_memory_target on smithi145 to 2480M 2024-09-21T09:59:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:59:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:59:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:59:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:59:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:59:56.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:56 smithi145 ceph-mon[33206]: Deploying daemon alertmanager.a on smithi119 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Adjusting osd_memory_target on smithi145 to 2480M 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T09:59:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:59:56.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[40785]: Deploying daemon alertmanager.a on smithi119 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Adjusting osd_memory_target on smithi145 to 2480M 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: osdmap e160: 8 total, 8 up, 8 in 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T09:59:56.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T09:59:56.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:56 smithi119 ceph-mon[29367]: Deploying daemon alertmanager.a on smithi119 2024-09-21T09:59:56.642 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:57 smithi145 ceph-mon[33206]: pgmap v21: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 434 op/s 2024-09-21T09:59:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:57 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3082389581' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:57 smithi145 ceph-mon[33206]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T09:59:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:57 smithi145 ceph-mon[33206]: from='client.25418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:57.566 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T09:59:57.583 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T09:59:57.584 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T09:59:57.584 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T09:59:57.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[40785]: pgmap v21: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 434 op/s 2024-09-21T09:59:57.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3082389581' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[40785]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T09:59:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[40785]: from='client.25418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[29367]: pgmap v21: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 434 op/s 2024-09-21T09:59:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3082389581' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[29367]: osdmap e161: 8 total, 8 up, 8 in 2024-09-21T09:59:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:57 smithi119 ceph-mon[29367]: from='client.25418 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:58.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:58 smithi119 ceph-mon[40785]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:58.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:58 smithi119 ceph-mon[40785]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T09:59:58.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:58 smithi119 ceph-mon[40785]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T09:59:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:58 smithi119 ceph-mon[29367]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:58 smithi119 ceph-mon[29367]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T09:59:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:58 smithi119 ceph-mon[29367]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T09:59:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:58 smithi145 ceph-mon[33206]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T09:59:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:58 smithi145 ceph-mon[33206]: osdmap e162: 8 total, 8 up, 8 in 2024-09-21T09:59:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:58 smithi145 ceph-mon[33206]: osdmap e163: 8 total, 8 up, 8 in 2024-09-21T09:59:59.173 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T09:59:59.179 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T09:59:59.179 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T09:59:59.189 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T09:59:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T09:59:59.622 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T09:59:59.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:59 smithi119 ceph-mon[29367]: pgmap v24: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 434 op/s 2024-09-21T09:59:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:59 smithi119 ceph-mon[29367]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T09:59:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 09:59:59 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/261883035' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T09:59:59.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:59 smithi119 ceph-mon[40785]: pgmap v24: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 434 op/s 2024-09-21T09:59:59.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:59 smithi119 ceph-mon[40785]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T09:59:59.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 09:59:59 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/261883035' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:59 smithi145 ceph-mon[33206]: pgmap v24: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 434 op/s 2024-09-21T10:00:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:59 smithi145 ceph-mon[33206]: osdmap e164: 8 total, 8 up, 8 in 2024-09-21T10:00:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 09:59:59 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/261883035' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[40785]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:00.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:00 smithi119 ceph-mon[29367]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:00:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:00 smithi145 ceph-mon[33206]: osdmap e165: 8 total, 8 up, 8 in 2024-09-21T10:00:01.172 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:01.193 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T10:00:01.201 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 0 2024-09-21T10:00:01.216 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:00:01.845 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:01 smithi119 ceph-mon[29367]: pgmap v27: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:01.846 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:01 smithi119 ceph-mon[29367]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:01.846 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:01 smithi119 ceph-mon[29367]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T10:00:01.846 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:01 smithi119 ceph-mon[40785]: pgmap v27: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:01.846 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:01 smithi119 ceph-mon[40785]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:01.846 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:01 smithi119 ceph-mon[40785]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T10:00:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:01 smithi145 ceph-mon[33206]: pgmap v27: 137 pgs: 137 active+clean; 469 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:00:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:01 smithi145 ceph-mon[33206]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:01 smithi145 ceph-mon[33206]: osdmap e166: 8 total, 8 up, 8 in 2024-09-21T10:00:02.573 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:03.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:00:02] "GET /metrics HTTP/1.1" 200 38917 "" "Prometheus/2.43.0" 2024-09-21T10:00:03.361 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:02 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:00:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[40785]: pgmap v30: 137 pgs: 137 active+clean; 505 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[40785]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1377064042' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[40785]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[40785]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[29367]: pgmap v30: 137 pgs: 137 active+clean; 505 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[29367]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T10:00:03.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1377064042' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:03.363 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[29367]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:03.363 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-mon[29367]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T10:00:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:03 smithi145 ceph-mon[33206]: pgmap v30: 137 pgs: 137 active+clean; 505 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-21T10:00:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:03 smithi145 ceph-mon[33206]: osdmap e167: 8 total, 8 up, 8 in 2024-09-21T10:00:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:03 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1377064042' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:03 smithi145 ceph-mon[33206]: from='client.15540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:03 smithi145 ceph-mon[33206]: osdmap e168: 8 total, 8 up, 8 in 2024-09-21T10:00:04.035 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:03 smithi119 systemd[1]: Stopping Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:04.035 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[76883]: ts=2024-09-21T10:00:03.992Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:00:04.353 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 podman[90513]: 2024-09-21 10:00:04.033405461 +0000 UTC m=+0.145722270 container died b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:04.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:04 smithi119 ceph-mon[40785]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:04.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:04 smithi119 ceph-mon[29367]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:04.481 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:00:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:04 smithi145 ceph-mon[33206]: from='client.15546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:04.616 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 podman[90513]: 2024-09-21 10:00:04.441367132 +0000 UTC m=+0.553683944 container cleanup b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:04.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 bash[90513]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T10:00:04.635 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:00:04.949 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 podman[90530]: 2024-09-21 10:00:04.693259489 +0000 UTC m=+0.656642276 container remove b73cbc93508c28f85199002e090e849dd3b28408661402854db436a73a58c4fb (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:04.949 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 podman[90530]: 2024-09-21 10:00:04.785903492 +0000 UTC m=+0.749286282 volume remove f5eccdf0653cca84c61ebaffedab4cfcd9ee88ed9f103d0b023c4f41041680b8 2024-09-21T10:00:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:05 smithi119 ceph-mon[29367]: pgmap v33: 137 pgs: 137 active+clean; 505 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-21T10:00:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:05 smithi119 ceph-mon[29367]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T10:00:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:05 smithi119 ceph-mon[40785]: pgmap v33: 137 pgs: 137 active+clean; 505 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-21T10:00:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:05 smithi119 ceph-mon[40785]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T10:00:05.367 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T10:00:05.367 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 systemd[1]: Stopped Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:05.368 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:04 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.181s CPU time. 2024-09-21T10:00:05.368 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:05 smithi119 systemd[1]: Starting Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:05.535 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:05 smithi145 ceph-mon[33206]: pgmap v33: 137 pgs: 137 active+clean; 505 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 32 MiB/s rd, 12 MiB/s wr, 193 op/s 2024-09-21T10:00:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:05 smithi145 ceph-mon[33206]: osdmap e169: 8 total, 8 up, 8 in 2024-09-21T10:00:05.831 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:05 smithi119 podman[90689]: 2024-09-21 10:00:05.28606128 +0000 UTC m=+0.016547532 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-09-21T10:00:05.831 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:05 smithi119 podman[90689]: 2024-09-21 10:00:05.487518089 +0000 UTC m=+0.218004328 volume create 3696cc076a32557327918e9d84820484bd812a46254b9134e990f1bf6e802be6 2024-09-21T10:00:05.831 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:05 smithi119 podman[90689]: 2024-09-21 10:00:05.579637796 +0000 UTC m=+0.310124037 container create e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:06.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-mon[29367]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T10:00:06.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3562054081' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:06.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-mon[29367]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:06.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-mon[40785]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T10:00:06.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3562054081' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:06.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-mon[40785]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 podman[90689]: 2024-09-21 10:00:06.366766464 +0000 UTC m=+1.097252703 container init e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 podman[90689]: 2024-09-21 10:00:06.372221 +0000 UTC m=+1.102707245 container start e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.382Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.382Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.383Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.119 port=9094 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.384Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.405Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.406Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T10:00:06.477 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.407Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-09-21T10:00:06.478 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:06.407Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T10:00:06.478 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 bash[90689]: e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 2024-09-21T10:00:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:06 smithi145 ceph-mon[33206]: osdmap e170: 8 total, 8 up, 8 in 2024-09-21T10:00:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:06 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3562054081' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:06 smithi145 ceph-mon[33206]: from='client.25472 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:06.825 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:06 smithi119 systemd[1]: Started Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:07.117 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:07 smithi119 systemd[1]: Stopping Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:07.424 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsShrinkToZeroBeforePrepare (11958 ms) 2024-09-21T10:00:07.424 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsShrinkToZeroAfterPrepare 2024-09-21T10:00:07.454 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:07.465 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:00:07.529 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: pgmap v36: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 500 op/s 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T10:00:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:07 smithi145 ceph-mon[33206]: Deploying daemon node-exporter.a on smithi119 2024-09-21T10:00:07.616 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:07 smithi119 podman[90899]: 2024-09-21 10:00:07.30103224 +0000 UTC m=+0.139502259 container died 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: pgmap v36: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 500 op/s 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[40785]: Deploying daemon node-exporter.a on smithi119 2024-09-21T10:00:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: from='client.25478 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: pgmap v36: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 500 op/s 2024-09-21T10:00:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: osdmap e171: 8 total, 8 up, 8 in 2024-09-21T10:00:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-21T10:00:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:07 smithi119 ceph-mon[29367]: Deploying daemon node-exporter.a on smithi119 2024-09-21T10:00:08.117 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:07 smithi119 podman[90899]: 2024-09-21 10:00:07.890038493 +0000 UTC m=+0.728508517 container cleanup 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:08.117 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:07 smithi119 bash[90899]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T10:00:08.497 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:08.497 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:08 smithi119 ceph-mon[40785]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T10:00:08.497 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:08 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/989404923' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:08.498 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:08.384Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000077705s 2024-09-21T10:00:08.498 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:08 smithi119 ceph-mon[29367]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T10:00:08.498 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:08 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/989404923' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:08.498 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 podman[90917]: 2024-09-21 10:00:08.147897118 +0000 UTC m=+0.847085615 container remove 8272a9f7510c4236c8d402628b7980ff0532b1012297a311cf3fbda67c5128a6 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:08.498 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:00:08.499 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T10:00:08.499 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 systemd[1]: Stopped Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:08.499 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 2.242s CPU time. 2024-09-21T10:00:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:08 smithi145 ceph-mon[33206]: osdmap e172: 8 total, 8 up, 8 in 2024-09-21T10:00:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:08 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/989404923' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:08.867 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 systemd[1]: Starting Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:08.867 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:08 smithi119 bash[91114]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T10:00:09.211 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T10:00:09.230 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T10:00:09.230 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T10:00:09.230 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:00:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:09 smithi145 ceph-mon[33206]: pgmap v39: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 500 op/s 2024-09-21T10:00:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:09 smithi145 ceph-mon[33206]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T10:00:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:09 smithi145 ceph-mon[33206]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:09 smithi119 ceph-mon[29367]: pgmap v39: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 500 op/s 2024-09-21T10:00:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:09 smithi119 ceph-mon[29367]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T10:00:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:09 smithi119 ceph-mon[29367]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:09.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:09 smithi119 bash[91114]: Getting image source signatures 2024-09-21T10:00:09.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:09 smithi119 bash[91114]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T10:00:09.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:09 smithi119 bash[91114]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T10:00:09.617 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:09 smithi119 bash[91114]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T10:00:09.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:09 smithi119 ceph-mon[40785]: pgmap v39: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 6.0 MiB/s wr, 500 op/s 2024-09-21T10:00:09.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:09 smithi119 ceph-mon[40785]: osdmap e173: 8 total, 8 up, 8 in 2024-09-21T10:00:09.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:09 smithi119 ceph-mon[40785]: from='client.25490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:10.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:10 smithi145 ceph-mon[33206]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:10.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:10 smithi145 ceph-mon[33206]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T10:00:10.580 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:10 smithi119 ceph-mon[29367]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:10.580 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:10 smithi119 ceph-mon[29367]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T10:00:10.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:10 smithi119 ceph-mon[40785]: from='client.25496 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:10.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:10 smithi119 ceph-mon[40785]: osdmap e174: 8 total, 8 up, 8 in 2024-09-21T10:00:10.866 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:10 smithi119 bash[91114]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T10:00:11.153 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:10 smithi119 bash[91114]: Writing manifest to image destination 2024-09-21T10:00:11.221 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T10:00:11.227 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T10:00:11.227 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:00:11.235 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T10:00:11.257 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:00:11.475 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:11.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-mon[40785]: pgmap v42: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:11.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-mon[40785]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T10:00:11.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/817090866' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:11.478 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 podman[91114]: 2024-09-21 10:00:11.158426622 +0000 UTC m=+2.544265607 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T10:00:11.478 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 podman[91114]: 2024-09-21 10:00:11.257028392 +0000 UTC m=+2.642867376 container create 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:11.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-mon[29367]: pgmap v42: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:11.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-mon[29367]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T10:00:11.478 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/817090866' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:11 smithi145 ceph-mon[33206]: pgmap v42: 137 pgs: 137 active+clean; 533 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:11 smithi145 ceph-mon[33206]: osdmap e175: 8 total, 8 up, 8 in 2024-09-21T10:00:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:11 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/817090866' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:11.736 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 podman[91114]: 2024-09-21 10:00:11.634557551 +0000 UTC m=+3.020396539 container init 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 podman[91114]: 2024-09-21 10:00:11.639410976 +0000 UTC m=+3.025249963 container start 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.643Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.643Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.643Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.643Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.644Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.644Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.644Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.644Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.644Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.644Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T10:00:11.737 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T10:00:11.738 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T10:00:11.739 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T10:00:11.740 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.645Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.646Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a[91237]: ts=2024-09-21T10:00:11.646Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T10:00:11.741 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 bash[91114]: 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 2024-09-21T10:00:12.117 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:00:11 smithi119 systemd[1]: Started Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:12 smithi145 ceph-mon[33206]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T10:00:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:12 smithi145 ceph-mon[33206]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:12 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:12 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[29367]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[29367]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[40785]: osdmap e176: 8 total, 8 up, 8 in 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[40785]: from='client.25508 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:13.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:00:12] "GET /metrics HTTP/1.1" 200 38917 "" "Prometheus/2.43.0" 2024-09-21T10:00:13.167 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:13.187 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T10:00:13.195 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:00:13.223 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:12 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[29619]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:00:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T10:00:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:13 smithi145 ceph-mon[33206]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:13 smithi145 ceph-mon[33206]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T10:00:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:13 smithi145 ceph-mon[33206]: pgmap v45: 137 pgs: 137 active+clean; 537 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 8.0 MiB/s wr, 135 op/s 2024-09-21T10:00:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:13 smithi145 ceph-mon[33206]: Deploying daemon prometheus.a on smithi145 2024-09-21T10:00:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:13 smithi145 ceph-mon[33206]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T10:00:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:13 smithi145 ceph-mon[33206]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T10:00:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[40785]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[40785]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T10:00:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[40785]: pgmap v45: 137 pgs: 137 active+clean; 537 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 8.0 MiB/s wr, 135 op/s 2024-09-21T10:00:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[40785]: Deploying daemon prometheus.a on smithi145 2024-09-21T10:00:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[40785]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T10:00:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[40785]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T10:00:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[29367]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[29367]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T10:00:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[29367]: pgmap v45: 137 pgs: 137 active+clean; 537 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 8.0 MiB/s wr, 135 op/s 2024-09-21T10:00:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[29367]: Deploying daemon prometheus.a on smithi145 2024-09-21T10:00:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[29367]: osdmap e177: 8 total, 8 up, 8 in 2024-09-21T10:00:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:13 smithi119 ceph-mon[29367]: osdmap e178: 8 total, 8 up, 8 in 2024-09-21T10:00:14.401 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:14 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2613896066' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:14 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2613896066' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:14 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2613896066' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:15 smithi145 ceph-mon[33206]: pgmap v48: 137 pgs: 137 active+clean; 537 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 8.0 MiB/s wr, 135 op/s 2024-09-21T10:00:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:15 smithi145 ceph-mon[33206]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T10:00:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:15 smithi145 ceph-mon[33206]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:15 smithi145 ceph-mon[33206]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:15 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[40785]: pgmap v48: 137 pgs: 137 active+clean; 537 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 8.0 MiB/s wr, 135 op/s 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[40785]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[40785]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[40785]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[29367]: pgmap v48: 137 pgs: 137 active+clean; 537 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 8.0 MiB/s wr, 135 op/s 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[29367]: osdmap e179: 8 total, 8 up, 8 in 2024-09-21T10:00:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[29367]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[29367]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:15 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:16 smithi145 ceph-mon[33206]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T10:00:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:16 smithi119 ceph-mon[29367]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T10:00:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:16 smithi119 ceph-mon[40785]: osdmap e180: 8 total, 8 up, 8 in 2024-09-21T10:00:16.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:00:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:00:16.387Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002173689s 2024-09-21T10:00:16.648 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 0 2024-09-21T10:00:16.731 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:00:16.861 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:00:17.350 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:17 smithi145 ceph-mon[33206]: pgmap v51: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T10:00:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:17 smithi145 ceph-mon[33206]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T10:00:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:17 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3307805795' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:17 smithi119 ceph-mon[29367]: pgmap v51: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T10:00:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:17 smithi119 ceph-mon[29367]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T10:00:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:17 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3307805795' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:17 smithi119 ceph-mon[40785]: pgmap v51: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T10:00:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:17 smithi119 ceph-mon[40785]: osdmap e181: 8 total, 8 up, 8 in 2024-09-21T10:00:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:17 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3307805795' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:18 smithi145 ceph-mon[33206]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T10:00:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:18 smithi145 ceph-mon[33206]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:18 smithi145 ceph-mon[33206]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:18 smithi119 ceph-mon[40785]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T10:00:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:18 smithi119 ceph-mon[40785]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:18 smithi119 ceph-mon[40785]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:18 smithi119 ceph-mon[29367]: osdmap e182: 8 total, 8 up, 8 in 2024-09-21T10:00:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:18 smithi119 ceph-mon[29367]: from='client.25544 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:18 smithi119 ceph-mon[29367]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:19.274 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:19 smithi145 ceph-mon[33206]: pgmap v54: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T10:00:19.275 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:19 smithi145 ceph-mon[33206]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T10:00:19.513 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsShrinkToZeroAfterPrepare (12089 ms) 2024-09-21T10:00:19.513 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare 2024-09-21T10:00:19.544 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:19.553 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:00:19.610 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:00:19.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:19 smithi119 ceph-mon[40785]: pgmap v54: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T10:00:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:19 smithi119 ceph-mon[40785]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T10:00:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:19 smithi119 ceph-mon[29367]: pgmap v54: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 119 KiB/s rd, 4.0 MiB/s wr, 170 op/s 2024-09-21T10:00:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:19 smithi119 ceph-mon[29367]: osdmap e183: 8 total, 8 up, 8 in 2024-09-21T10:00:20.256 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:20 smithi145 ceph-mon[33206]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T10:00:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:20 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1597280285' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:20 smithi119 ceph-mon[29367]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T10:00:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:20 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1597280285' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:20 smithi119 ceph-mon[40785]: osdmap e184: 8 total, 8 up, 8 in 2024-09-21T10:00:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:20 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1597280285' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:21.294 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T10:00:21.303 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T10:00:21.303 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T10:00:21.303 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:00:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:21 smithi145 ceph-mon[33206]: pgmap v57: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:21 smithi145 ceph-mon[33206]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:21 smithi145 ceph-mon[33206]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T10:00:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:21 smithi145 ceph-mon[33206]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[40785]: pgmap v57: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[40785]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[40785]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[40785]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[29367]: pgmap v57: 137 pgs: 137 active+clean; 561 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[29367]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[29367]: osdmap e185: 8 total, 8 up, 8 in 2024-09-21T10:00:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:21 smithi119 ceph-mon[29367]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:22.402 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-mon[33206]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T10:00:22.402 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 systemd[1]: Stopping Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:22 smithi119 ceph-mon[40785]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T10:00:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:22 smithi119 ceph-mon[29367]: osdmap e186: 8 total, 8 up, 8 in 2024-09-21T10:00:22.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:00:22.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T10:00:22.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T10:00:22.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T10:00:22.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T10:00:22.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.400Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.401Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.401Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[65612]: ts=2024-09-21T10:00:22.401Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T10:00:22.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 podman[76433]: 2024-09-21 10:00:22.410640192 +0000 UTC m=+0.139932568 container died 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:23.169 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T10:00:23.175 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T10:00:23.175 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:00:23.175 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:23.183 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T10:00:23.192 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 podman[76433]: 2024-09-21 10:00:22.90611266 +0000 UTC m=+0.635405038 container cleanup 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:23.192 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:22 smithi145 bash[76433]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T10:00:23.195 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:00:23.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:23 smithi145 ceph-mon[33206]: pgmap v60: 137 pgs: 137 active+clean; 577 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 66 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-09-21T10:00:23.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:23 smithi145 ceph-mon[33206]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T10:00:23.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:23 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3161307031' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:23.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:23 smithi145 ceph-mon[33206]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T10:00:23.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 podman[76449]: 2024-09-21 10:00:23.190611307 +0000 UTC m=+0.782334100 container remove 69f08aa0a165412f866c95582d4668928e02010bc45072dd3847e24ffb4fc288 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:23.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T10:00:23.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 systemd[1]: Stopped Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:23.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.740s CPU time. 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[40785]: pgmap v60: 137 pgs: 137 active+clean; 577 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 66 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[40785]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3161307031' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[40785]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[29367]: pgmap v60: 137 pgs: 137 active+clean; 577 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 66 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[29367]: osdmap e187: 8 total, 8 up, 8 in 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3161307031' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:23 smithi119 ceph-mon[29367]: osdmap e188: 8 total, 8 up, 8 in 2024-09-21T10:00:23.905 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 systemd[1]: Starting Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:23.905 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 podman[76581]: 2024-09-21 10:00:23.658611144 +0000 UTC m=+0.016192769 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T10:00:23.906 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:23 smithi145 podman[76581]: 2024-09-21 10:00:23.775743097 +0000 UTC m=+0.133324711 container create a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:24.287 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 podman[76581]: 2024-09-21 10:00:24.003022474 +0000 UTC m=+0.360604090 container init a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 podman[76581]: 2024-09-21 10:00:24.008105114 +0000 UTC m=+0.365686729 container start a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.028Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.028Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.028Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi145 (none))" 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.028Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.028Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.030Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.031Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.032Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.032Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.034Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T10:00:24.288 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.035Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=715.163µs 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.035Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.035Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.047Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.047Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.047Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.037µs wal_replay_duration=12.388119ms wbl_replay_duration=125ns total_replay_duration=13.155436ms 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.050Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.050Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.050Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.067Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.410864ms db_storage=874ns remote_storage=1.565µs web_handler=421ns query_engine=614ns scrape=620.358µs scrape_sd=110.239µs notify=32.61µs notify_sd=17.265µs rules=15.261654ms tracing=7.109µs 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.067Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:24.067Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 bash[76581]: a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 2024-09-21T10:00:24.289 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:24 smithi145 systemd[1]: Started Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-mon[33206]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-mon[33206]: from='client.25586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:24 smithi145 ceph-mon[33206]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T10:00:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[29367]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[29367]: from='client.25586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[29367]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T10:00:24.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[40785]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:24.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[40785]: from='client.25586 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:24.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:24.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:24.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:24 smithi119 ceph-mon[40785]: osdmap e189: 8 total, 8 up, 8 in 2024-09-21T10:00:25.179 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:25.200 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T10:00:25.208 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:00:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:25 smithi145 ceph-mon[33206]: pgmap v63: 137 pgs: 137 active+clean; 577 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 66 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-09-21T10:00:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:25 smithi145 ceph-mon[33206]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T10:00:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:25 smithi145 ceph-mon[33206]: Reconfiguring daemon grafana.a on smithi145 2024-09-21T10:00:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:25 smithi145 ceph-mon[33206]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[40785]: pgmap v63: 137 pgs: 137 active+clean; 577 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 66 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[40785]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[40785]: Reconfiguring daemon grafana.a on smithi145 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[40785]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[29367]: pgmap v63: 137 pgs: 137 active+clean; 577 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 66 MiB/s rd, 10 MiB/s wr, 540 op/s 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[29367]: Reconfiguring grafana.a (dependencies changed)... 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[29367]: Reconfiguring daemon grafana.a on smithi145 2024-09-21T10:00:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:25 smithi119 ceph-mon[29367]: osdmap e190: 8 total, 8 up, 8 in 2024-09-21T10:00:26.092 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:26 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/955842691' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:26 smithi119 ceph-mon[40785]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T10:00:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:26 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/955842691' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:26 smithi119 ceph-mon[29367]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T10:00:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:26 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/955842691' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:26 smithi145 ceph-mon[33206]: osdmap e191: 8 total, 8 up, 8 in 2024-09-21T10:00:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:27 smithi145 ceph-mon[33206]: pgmap v66: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T10:00:27.924 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:27 smithi145 ceph-mon[33206]: from='client.25598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:27.924 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:27 smithi145 ceph-mon[33206]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:27.924 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:27 smithi145 ceph-mon[33206]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T10:00:27.924 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[29367]: pgmap v66: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[29367]: from='client.25598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[29367]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[29367]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[40785]: pgmap v66: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[40785]: from='client.25598 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[40785]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:27.925 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:27 smithi119 ceph-mon[40785]: osdmap e192: 8 total, 8 up, 8 in 2024-09-21T10:00:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:00:28.588 INFO:tasks.workunit.client.0.smithi119.stdout:write: * 83886080~1000 2024-09-21T10:00:28.624 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:00:28.756 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:00:29.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:28 smithi145 systemd[1]: Stopping Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:29.057 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:29.361 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 podman[76843]: 2024-09-21 10:00:29.062617296 +0000 UTC m=+0.136122261 container died 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, vcs-type=git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, name=ubi8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public) 2024-09-21T10:00:29.361 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:29 smithi145 ceph-mon[33206]: pgmap v69: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T10:00:29.361 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:29 smithi145 ceph-mon[33206]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T10:00:29.361 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:29 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1082667133' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:29 smithi119 ceph-mon[29367]: pgmap v69: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T10:00:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:29 smithi119 ceph-mon[29367]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T10:00:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:29 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1082667133' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:29 smithi119 ceph-mon[40785]: pgmap v69: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 MiB/s wr, 103 op/s 2024-09-21T10:00:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:29 smithi119 ceph-mon[40785]: osdmap e193: 8 total, 8 up, 8 in 2024-09-21T10:00:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:29 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1082667133' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:29.644 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 podman[76843]: 2024-09-21 10:00:29.458542474 +0000 UTC m=+0.532047428 container cleanup 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, version=8.5, release=239.1651231664, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, description=Ceph Grafana Container, distribution-scope=public, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vcs-type=git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-09-21T10:00:29.644 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 bash[76843]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a 2024-09-21T10:00:30.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 podman[76862]: 2024-09-21 10:00:29.643068035 +0000 UTC m=+0.582875271 container remove 375d6b959d13053317b010e1f6d947f16f4d515a201f929ccd35f4772a1ba8fc (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, architecture=x86_64, vcs-type=git, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, name=ubi8, description=Ceph Grafana Container, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-21T10:00:30.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:00:30.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T10:00:30.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 systemd[1]: Stopped Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:30.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service: Consumed 5.387s CPU time. 2024-09-21T10:00:30.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:29 smithi145 systemd[1]: Starting Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:30.423 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-mon[33206]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:30.423 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-mon[33206]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T10:00:30.423 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-mon[33206]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:30.423 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:30.423 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 podman[76958]: 2024-09-21 10:00:29.976695488 +0000 UTC m=+0.015416488 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-21T10:00:30.423 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 podman[76958]: 2024-09-21 10:00:30.111095932 +0000 UTC m=+0.149816961 container create e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, release=239.1651231664, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, name=ubi8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner ) 2024-09-21T10:00:30.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[40785]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:30.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[40785]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T10:00:30.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[40785]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:30.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:30.591 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[29367]: from='client.25616 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:30.591 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[29367]: osdmap e194: 8 total, 8 up, 8 in 2024-09-21T10:00:30.591 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[29367]: from='client.15708 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:30.591 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:30 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 podman[76958]: 2024-09-21 10:00:30.421785392 +0000 UTC m=+0.460506377 container init e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=Ceph Grafana Container, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, vcs-type=git, io.openshift.expose-services=, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8) 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 podman[76958]: 2024-09-21 10:00:30.426997982 +0000 UTC m=+0.465718969 container start e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, architecture=x86_64, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, io.openshift.expose-services=, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, maintainer=Paul Cuzner , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public) 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 bash[76958]: e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 systemd[1]: Started Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583638322Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583823495Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583846197Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T10:00:30.690 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583853658Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583859655Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583865604Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583871276Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583877063Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583882966Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583888655Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583894155Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583899526Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T10:00:30.691 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.583905156Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=settings t=2024-09-21T10:00:30.58391075Z level=info msg="App mode production" 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=sqlstore t=2024-09-21T10:00:30.583962488Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=sqlstore t=2024-09-21T10:00:30.583979746Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=migrator t=2024-09-21T10:00:30.597973036Z level=info msg="Starting DB migrations" 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=migrator t=2024-09-21T10:00:30.602887945Z level=info msg="migrations completed" performed=0 skipped=481 duration=487.445µs 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=sqlstore t=2024-09-21T10:00:30.603270073Z level=info msg="Created default organization" 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=plugin.loader t=2024-09-21T10:00:30.659372649Z level=info msg="Plugin registered" pluginID=input 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=plugin.loader t=2024-09-21T10:00:30.688460627Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=plugin.loader t=2024-09-21T10:00:30.688498532Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T10:00:30.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=secrets t=2024-09-21T10:00:30.688534155Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T10:00:31.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=query_data t=2024-09-21T10:00:30.690336665Z level=info msg="Query Service initialization" 2024-09-21T10:00:31.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=live.push_http t=2024-09-21T10:00:30.692726629Z level=info msg="Live Push Gateway initialization" 2024-09-21T10:00:31.459 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:31 smithi145 systemd[1]: Stopping Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:31.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-mon[33206]: pgmap v72: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:31.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-mon[33206]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T10:00:31.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:31.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:31.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-mon[33206]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T10:00:31.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-mon[33206]: Deploying daemon node-exporter.b on smithi145 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[40785]: pgmap v72: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[40785]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[40785]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[40785]: Deploying daemon node-exporter.b on smithi145 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[29367]: pgmap v72: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[29367]: osdmap e195: 8 total, 8 up, 8 in 2024-09-21T10:00:31.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:31.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:31.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[29367]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-21T10:00:31.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:31 smithi119 ceph-mon[29367]: Deploying daemon node-exporter.b on smithi145 2024-09-21T10:00:31.627 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare (12114 ms) 2024-09-21T10:00:31.627 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsExpandAfterPrepare2 2024-09-21T10:00:31.663 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:31.673 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 41943040~10 2024-09-21T10:00:31.681 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:00:31.794 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:31 smithi145 podman[77119]: 2024-09-21 10:00:31.590393103 +0000 UTC m=+0.317542257 container died 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:00:31.794 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:00:31] "GET /metrics HTTP/1.1" 200 38915 "" "Prometheus/2.51.0" 2024-09-21T10:00:31.794 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=infra.usagestats.collector t=2024-09-21T10:00:31.457494312Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=provisioning.datasources t=2024-09-21T10:00:31.543260359Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=provisioning.datasources t=2024-09-21T10:00:31.543298545Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=provisioning.datasources t=2024-09-21T10:00:31.554732617Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=provisioning.alerting t=2024-09-21T10:00:31.637095001Z level=info msg="starting to provision alerting" 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=provisioning.alerting t=2024-09-21T10:00:31.637123289Z level=info msg="finished to provision alerting" 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=grafanaStorageLogger t=2024-09-21T10:00:31.637266973Z level=info msg="storage starting" 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=ngalert.state.manager t=2024-09-21T10:00:31.63736514Z level=info msg="Warming state cache for startup" 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=http.server t=2024-09-21T10:00:31.639578127Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=ngalert.state.manager t=2024-09-21T10:00:31.648054933Z level=info msg="State cache has been initialized" states=0 duration=10.688738ms 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=ngalert.multiorg.alertmanager t=2024-09-21T10:00:31.648108451Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T10:00:31.795 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:00:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=ticker t=2024-09-21T10:00:31.648139727Z level=info msg=starting first_tick=2024-09-21T10:00:40Z 2024-09-21T10:00:32.026 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:32.294 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 podman[77119]: 2024-09-21 10:00:32.070327685 +0000 UTC m=+0.797476837 container cleanup 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:00:32.294 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 bash[77119]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T10:00:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:32 smithi119 ceph-mon[40785]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T10:00:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:32 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1787777782' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:32 smithi119 ceph-mon[29367]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T10:00:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:32 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1787777782' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:32.702 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:32 smithi145 ceph-mon[33206]: osdmap e196: 8 total, 8 up, 8 in 2024-09-21T10:00:32.702 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:32 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1787777782' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:32.702 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 podman[77137]: 2024-09-21 10:00:32.33850665 +0000 UTC m=+0.752096248 container remove 9040fb6ea6a62e0bb3d46c9bd2f17838ab9c0b8d5d2a1936479087992edea25e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:00:32.702 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:00:32.702 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T10:00:32.702 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 systemd[1]: Stopped Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:32.703 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 2.214s CPU time. 2024-09-21T10:00:33.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 systemd[1]: Starting Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:33.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:32 smithi145 bash[77270]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-09-21T10:00:33.176 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T10:00:33.198 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:00:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:33 smithi145 ceph-mon[33206]: pgmap v75: 137 pgs: 137 active+clean; 637 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 525 op/s 2024-09-21T10:00:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:33 smithi145 ceph-mon[33206]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:33 smithi145 ceph-mon[33206]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:33 smithi145 ceph-mon[33206]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T10:00:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:33 smithi145 ceph-mon[33206]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T10:00:33.545 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:33 smithi145 bash[77270]: Getting image source signatures 2024-09-21T10:00:33.545 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:33 smithi145 bash[77270]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-09-21T10:00:33.545 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:33 smithi145 bash[77270]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-09-21T10:00:33.545 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:33 smithi145 bash[77270]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[40785]: pgmap v75: 137 pgs: 137 active+clean; 637 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 525 op/s 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[40785]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[40785]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[40785]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[40785]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[29367]: pgmap v75: 137 pgs: 137 active+clean; 637 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 525 op/s 2024-09-21T10:00:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[29367]: from='client.25634 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:33.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[29367]: from='client.15726 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:33.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[29367]: osdmap e197: 8 total, 8 up, 8 in 2024-09-21T10:00:33.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:33 smithi119 ceph-mon[29367]: osdmap e198: 8 total, 8 up, 8 in 2024-09-21T10:00:34.329 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:34.340 INFO:tasks.workunit.client.0.smithi119.stdout:write: Y 41943040~5 2024-09-21T10:00:34.438 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:00:34.510 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:00:34.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:34 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:34.966 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:35.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:34 smithi145 bash[77270]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-09-21T10:00:35.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:34 smithi145 bash[77270]: Writing manifest to image destination 2024-09-21T10:00:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:35 smithi119 ceph-mon[40785]: pgmap v78: 137 pgs: 137 active+clean; 637 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 525 op/s 2024-09-21T10:00:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:35 smithi119 ceph-mon[40785]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T10:00:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:35 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1834939031' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:35 smithi119 ceph-mon[29367]: pgmap v78: 137 pgs: 137 active+clean; 637 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 525 op/s 2024-09-21T10:00:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:35 smithi119 ceph-mon[29367]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T10:00:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:35 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1834939031' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:35.414 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsExpandAfterPrepare2 (3788 ms) 2024-09-21T10:00:35.415 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsSnapAfterPrepare 2024-09-21T10:00:35.447 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:35.457 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:00:35.494 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-mon[33206]: pgmap v78: 137 pgs: 137 active+clean; 637 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 38 MiB/s rd, 6.0 MiB/s wr, 525 op/s 2024-09-21T10:00:35.494 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-mon[33206]: osdmap e199: 8 total, 8 up, 8 in 2024-09-21T10:00:35.494 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1834939031' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:35.494 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 podman[77270]: 2024-09-21 10:00:35.226715936 +0000 UTC m=+2.434608714 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-09-21T10:00:35.494 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 podman[77270]: 2024-09-21 10:00:35.341860927 +0000 UTC m=+2.549753691 container create d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:00:35.511 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:00:36.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:36.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 podman[77270]: 2024-09-21 10:00:35.627953023 +0000 UTC m=+2.835845790 container init d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 podman[77270]: 2024-09-21 10:00:35.632986892 +0000 UTC m=+2.840879657 container start d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.636Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.636Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T10:00:36.045 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T10:00:36.046 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.637Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T10:00:36.047 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T10:00:36.048 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T10:00:36.049 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T10:00:36.049 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T10:00:36.049 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-09-21T10:00:36.049 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b[77345]: ts=2024-09-21T10:00:35.638Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T10:00:36.049 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 bash[77270]: d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 2024-09-21T10:00:36.049 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:00:35 smithi145 systemd[1]: Started Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:36.440 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:36.441 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:36 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='client.25652 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='client.15744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: osdmap e200: 8 total, 8 up, 8 in 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:00:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:36.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:37.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:36 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:37.341 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T10:00:37.360 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T10:00:37.360 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T10:00:37.360 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: pgmap v81: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 22 KiB/s wr, 418 op/s 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[40785]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T10:00:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T10:00:37.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:00:37.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:37.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T10:00:37.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T10:00:37.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: pgmap v81: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 22 KiB/s wr, 418 op/s 2024-09-21T10:00:37.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:37 smithi119 ceph-mon[29367]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: pgmap v81: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 22 KiB/s wr, 418 op/s 2024-09-21T10:00:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:37 smithi145 ceph-mon[33206]: osdmap e201: 8 total, 8 up, 8 in 2024-09-21T10:00:37.927 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:38.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T10:00:38.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: Upgrade: Updating mgr.y 2024-09-21T10:00:38.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:38.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: Deploying daemon mgr.y on smithi119 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3811627758' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[40785]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: Upgrade: Updating mgr.y 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:00:38.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:00:38.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T10:00:38.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:38.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: Deploying daemon mgr.y on smithi119 2024-09-21T10:00:38.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3811627758' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:38.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:38 smithi119 ceph-mon[29367]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: osdmap e202: 8 total, 8 up, 8 in 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: Upgrade: Updating mgr.y 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: Deploying daemon mgr.y on smithi119 2024-09-21T10:00:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3811627758' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:38 smithi145 ceph-mon[33206]: osdmap e203: 8 total, 8 up, 8 in 2024-09-21T10:00:39.181 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T10:00:39.187 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T10:00:39.187 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:00:39.195 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T10:00:39.216 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:00:39.615 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:39 smithi119 systemd[1]: Stopping Ceph mgr.y for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:39.615 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[40785]: pgmap v84: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 22 KiB/s wr, 418 op/s 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[40785]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[40785]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[40785]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[29367]: pgmap v84: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 22 KiB/s wr, 418 op/s 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[29367]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[29367]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:39 smithi119 ceph-mon[29367]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T10:00:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:39 smithi145 ceph-mon[33206]: pgmap v84: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 22 KiB/s wr, 418 op/s 2024-09-21T10:00:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:39 smithi145 ceph-mon[33206]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:39 smithi145 ceph-mon[33206]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:39 smithi145 ceph-mon[33206]: osdmap e204: 8 total, 8 up, 8 in 2024-09-21T10:00:39.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:39 smithi119 podman[92244]: 2024-09-21 10:00:39.61368277 +0000 UTC m=+0.141639732 container died 34c990954610e68d3a5d0f687cf3d0c1975cb0ad0f42790682f146d9b05840f5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:00:40.350 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 podman[92244]: 2024-09-21 10:00:40.122487999 +0000 UTC m=+0.650444957 container cleanup 34c990954610e68d3a5d0f687cf3d0c1975cb0ad0f42790682f146d9b05840f5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:00:40.351 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 bash[92244]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y 2024-09-21T10:00:40.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 podman[92262]: 2024-09-21 10:00:40.349401768 +0000 UTC m=+0.733367188 container remove 34c990954610e68d3a5d0f687cf3d0c1975cb0ad0f42790682f146d9b05840f5 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T10:00:40.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:00:40.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T10:00:40.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 systemd[1]: Stopped Ceph mgr.y for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:40.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service: Consumed 40.332s CPU time. 2024-09-21T10:00:40.892 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:41.062 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 systemd[1]: Starting Ceph mgr.y for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:00:41.062 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 podman[92439]: 2024-09-21 10:00:40.827775071 +0000 UTC m=+0.016255991 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:00:41.062 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:40 smithi119 podman[92439]: 2024-09-21 10:00:40.934205718 +0000 UTC m=+0.122686640 container create d7bcb26ddb6215674e498dd628dbd05731e2974e4761a9f7bb9804afec4307b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-21T10:00:41.180 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:41.198 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T10:00:41.206 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:00:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:41 smithi145 ceph-mon[33206]: pgmap v87: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:41 smithi145 ceph-mon[33206]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T10:00:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:41 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/807843383' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mon[29367]: pgmap v87: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mon[29367]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T10:00:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/807843383' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mon[40785]: pgmap v87: 137 pgs: 137 active+clean; 617 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mon[40785]: osdmap e205: 8 total, 8 up, 8 in 2024-09-21T10:00:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/807843383' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:41.618 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:41 smithi119 podman[92439]: 2024-09-21 10:00:41.195357067 +0000 UTC m=+0.383837991 container init d7bcb26ddb6215674e498dd628dbd05731e2974e4761a9f7bb9804afec4307b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, ceph=True) 2024-09-21T10:00:41.618 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:41 smithi119 podman[92439]: 2024-09-21 10:00:41.201048016 +0000 UTC m=+0.389528942 container start d7bcb26ddb6215674e498dd628dbd05731e2974e4761a9f7bb9804afec4307b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS) 2024-09-21T10:00:41.618 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:41 smithi119 ceph-mgr[92504]: -- 172.21.15.119:0/3357649641 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8bb3fb0e0 con 0x55f8bb3d8800 2024-09-21T10:00:41.618 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:41 smithi119 bash[92439]: d7bcb26ddb6215674e498dd628dbd05731e2974e4761a9f7bb9804afec4307b6 2024-09-21T10:00:41.618 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:41 smithi119 systemd[1]: Started Ceph mgr.y for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:00:41.817 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:41 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:00:41] "GET /metrics HTTP/1.1" 200 38915 "" "Prometheus/2.51.0" 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[29367]: from='client.25688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[29367]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[29367]: from='client.25694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[40785]: from='client.25688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[40785]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T10:00:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[40785]: from='client.25694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:42.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:42 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:42 smithi145 ceph-mon[33206]: from='client.25688 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:42 smithi145 ceph-mon[33206]: osdmap e206: 8 total, 8 up, 8 in 2024-09-21T10:00:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:42 smithi145 ceph-mon[33206]: from='client.25694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:42 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:42 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:42 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:43 smithi145 ceph-mon[33206]: pgmap v90: 137 pgs: 137 active+clean; 673 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 108 KiB/s rd, 12 MiB/s wr, 161 op/s 2024-09-21T10:00:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:43 smithi145 ceph-mon[33206]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T10:00:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:43 smithi145 ceph-mon[33206]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T10:00:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:43 smithi119 ceph-mon[29367]: pgmap v90: 137 pgs: 137 active+clean; 673 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 108 KiB/s rd, 12 MiB/s wr, 161 op/s 2024-09-21T10:00:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:43 smithi119 ceph-mon[29367]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T10:00:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:43 smithi119 ceph-mon[29367]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T10:00:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:43 smithi119 ceph-mon[40785]: pgmap v90: 137 pgs: 137 active+clean; 673 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 108 KiB/s rd, 12 MiB/s wr, 161 op/s 2024-09-21T10:00:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:43 smithi119 ceph-mon[40785]: osdmap e207: 8 total, 8 up, 8 in 2024-09-21T10:00:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:43 smithi119 ceph-mon[40785]: osdmap e208: 8 total, 8 up, 8 in 2024-09-21T10:00:43.859 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:44 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3274876508' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:44 smithi145 ceph-mon[33206]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T10:00:44.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:44.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:44 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3274876508' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:44 smithi119 ceph-mon[29367]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T10:00:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:44 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3274876508' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:44 smithi119 ceph-mon[40785]: osdmap e209: 8 total, 8 up, 8 in 2024-09-21T10:00:44.673 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: after_prepare_snap 2024-09-21T10:00:45.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:45 smithi145 ceph-mon[33206]: pgmap v93: 137 pgs: 137 active+clean; 673 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 108 KiB/s rd, 12 MiB/s wr, 161 op/s 2024-09-21T10:00:45.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:45 smithi145 ceph-mon[33206]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:45.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:45 smithi145 ceph-mon[33206]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:45.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:45 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[29367]: pgmap v93: 137 pgs: 137 active+clean; 673 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 108 KiB/s rd, 12 MiB/s wr, 161 op/s 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[29367]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[29367]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[40785]: pgmap v93: 137 pgs: 137 active+clean; 673 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 108 KiB/s rd, 12 MiB/s wr, 161 op/s 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[40785]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[40785]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:45 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:45.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:45.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:46.212 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T10:00:46.244 INFO:tasks.workunit.client.0.smithi119.stdout:write: * 0~1000 2024-09-21T10:00:46.253 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:00:46.432 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.528 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: osdmap e210: 8 total, 8 up, 8 in 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:46] ENGINE Bus STARTING 2024-09-21T10:00:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: CherryPy Checker: 2024-09-21T10:00:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: The Application mounted at '' has an empty config. 2024-09-21T10:00:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:46] ENGINE Serving on http://:::9283 2024-09-21T10:00:46.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:46] ENGINE Bus STARTED 2024-09-21T10:00:46.890 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:47.203 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:46 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: pgmap v96: 137 pgs: 137 active+clean; 673 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 44 MiB/s rd, 12 KiB/s wr, 322 op/s 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/284570925' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: Standby manager daemon y restarted 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: Standby manager daemon y started 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T10:00:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:47 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: pgmap v96: 137 pgs: 137 active+clean; 673 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 44 MiB/s rd, 12 KiB/s wr, 322 op/s 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/284570925' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: Standby manager daemon y restarted 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: Standby manager daemon y started 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T10:00:47.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: pgmap v96: 137 pgs: 137 active+clean; 673 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 44 MiB/s rd, 12 KiB/s wr, 322 op/s 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: osdmap e211: 8 total, 8 up, 8 in 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/284570925' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: Standby manager daemon y restarted 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: Standby manager daemon y started 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T10:00:47.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:47 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:48 smithi145 ceph-mon[33206]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:48 smithi145 ceph-mon[33206]: from='client.25736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:48 smithi145 ceph-mon[33206]: mgrmap e26: x(active, since 63s), standbys: y 2024-09-21T10:00:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:48 smithi145 ceph-mon[33206]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T10:00:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[40785]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[40785]: from='client.25736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[40785]: mgrmap e26: x(active, since 63s), standbys: y 2024-09-21T10:00:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[40785]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T10:00:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[29367]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[29367]: from='client.25736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[29367]: mgrmap e26: x(active, since 63s), standbys: y 2024-09-21T10:00:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:48 smithi119 ceph-mon[29367]: osdmap e212: 8 total, 8 up, 8 in 2024-09-21T10:00:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mon[33206]: pgmap v99: 137 pgs: 137 active+clean; 673 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 44 MiB/s rd, 12 KiB/s wr, 322 op/s 2024-09-21T10:00:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mon[33206]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T10:00:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:49.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:49.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:49.545 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T10:00:49.323+0000 7f02ed623640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T10:00:49.545 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ignoring --setuser ceph since I am not root 2024-09-21T10:00:49.545 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ignoring --setgroup ceph since I am not root 2024-09-21T10:00:49.545 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:49 smithi145 ceph-mgr[68671]: -- 172.21.15.145:0/1589249328 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555ef55570e0 con 0x555ef5534800 2024-09-21T10:00:49.546 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsSnapAfterPrepare (14131 ms) 2024-09-21T10:00:49.546 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapsSnapExpandAfterPrepare 2024-09-21T10:00:49.576 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:49.587 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:49] ENGINE Bus STOPPING 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[40785]: pgmap v99: 137 pgs: 137 active+clean; 673 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 44 MiB/s rd, 12 KiB/s wr, 322 op/s 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[40785]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[29367]: pgmap v99: 137 pgs: 137 active+clean; 673 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 44 MiB/s rd, 12 KiB/s wr, 322 op/s 2024-09-21T10:00:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[29367]: osdmap e213: 8 total, 8 up, 8 in 2024-09-21T10:00:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:49.645 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:00:50.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:00:50.117 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:49 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:49] ENGINE Bus STOPPED 2024-09-21T10:00:50.367 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:50.475 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:50.475 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T10:00:50.475 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.475 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T10:00:50.475 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: Activating manager daemon y 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.24964 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: mgrmap e27: y(active, starting, since 0.0128654s) 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T10:00:50.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3983940334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: Manager daemon y is now available 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:50.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.152+0000 7ff444b88640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.152+0000 7ff444b88640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.152+0000 7ff444b88640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.152+0000 7ff444b88640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.152+0000 7ff444b88640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.153+0000 7ff44838f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.478 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.153+0000 7ff44838f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.479 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.153+0000 7ff44838f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.479 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.153+0000 7ff44838f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.479 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:50.153+0000 7ff44838f640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:00:50.479 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:50] ENGINE Bus STARTING 2024-09-21T10:00:50.479 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:50] ENGINE Serving on http://:::9283 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: Activating manager daemon y 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.24964 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T10:00:50.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: mgrmap e27: y(active, starting, since 0.0128654s) 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T10:00:50.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T10:00:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3983940334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: Manager daemon y is now available 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T10:00:50.483 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T10:00:50.488 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:50.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' 2024-09-21T10:00:50.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: osdmap e214: 8 total, 8 up, 8 in 2024-09-21T10:00:50.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.24964 172.21.15.145:0/3084400927' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: Activating manager daemon y 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: osdmap e215: 8 total, 8 up, 8 in 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.24964 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: mgrmap e27: y(active, starting, since 0.0128654s) 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata", "who": "foo.smithi119.oikglq"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T10:00:50.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3983940334' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: Manager daemon y is now available 2024-09-21T10:00:50.490 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:00:50.491 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:00:50.491 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:00:50.491 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T10:00:50.491 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:50 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T10:00:50.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:50 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:00:50] ENGINE Bus STARTED 2024-09-21T10:00:51.366 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: mgrmap e28: y(active, since 1.02616s) 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: [21/Sep/2024:10:00:50] ENGINE Bus STARTING 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: [21/Sep/2024:10:00:50] ENGINE Serving on http://172.21.15.119:8765 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: [21/Sep/2024:10:00:51] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[29367]: [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-21T10:00:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: mgrmap e28: y(active, since 1.02616s) 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: [21/Sep/2024:10:00:50] ENGINE Bus STARTING 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: [21/Sep/2024:10:00:50] ENGINE Serving on http://172.21.15.119:8765 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: [21/Sep/2024:10:00:51] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:51.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-mon[40785]: [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: Traceback (most recent call last): 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._run(expiration_interval) 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T10:00:51.370 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mon.c.smithi119.stdout: self.do_handshake() 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mon.c.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:51.077+0000 7ff4260ce640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: Traceback (most recent call last): 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self._connections.run(self.expiration_interval) 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self._run(expiration_interval) 2024-09-21T10:00:51.371 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:51.372 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:51.372 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:51.372 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:51.372 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:51.372 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: s = self.context.wrap_socket( 2024-09-21T10:00:51.372 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:51.373 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: return self.sslsocket_class._create( 2024-09-21T10:00:51.373 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:51.373 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self.do_handshake() 2024-09-21T10:00:51.374 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:51.374 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self._sslobj.do_handshake() 2024-09-21T10:00:51.374 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:51.374 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:00:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:00:51.379 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T10:00:51.380 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T10:00:51.380 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:00:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: osdmap e216: 8 total, 8 up, 8 in 2024-09-21T10:00:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: mgrmap e28: y(active, since 1.02616s) 2024-09-21T10:00:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: [21/Sep/2024:10:00:50] ENGINE Bus STARTING 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: [21/Sep/2024:10:00:50] ENGINE Serving on http://172.21.15.119:8765 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: [21/Sep/2024:10:00:51] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: [21/Sep/2024:10:00:51] ENGINE Bus STARTED 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:51 smithi145 ceph-mon[33206]: [21/Sep/2024:10:00:51] ENGINE Error in HTTPServer.serve 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: Traceback (most recent call last): 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._run(expiration_interval) 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:00:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: s = self.context.wrap_socket( 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: return self.sslsocket_class._create( 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: self.do_handshake() 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._sslobj.do_handshake() 2024-09-21T10:00:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:00:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:52 smithi119 ceph-mon[29367]: pgmap v5: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:52 smithi119 ceph-mon[29367]: mgrmap e29: y(active, since 2s) 2024-09-21T10:00:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:52 smithi119 ceph-mon[29367]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T10:00:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:52 smithi119 ceph-mon[40785]: pgmap v5: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:52 smithi119 ceph-mon[40785]: mgrmap e29: y(active, since 2s) 2024-09-21T10:00:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:52 smithi119 ceph-mon[40785]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T10:00:52.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:52 smithi145 ceph-mon[33206]: pgmap v5: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:52.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:52 smithi145 ceph-mon[33206]: mgrmap e29: y(active, since 2s) 2024-09-21T10:00:52.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:52 smithi145 ceph-mon[33206]: osdmap e217: 8 total, 8 up, 8 in 2024-09-21T10:00:53.175 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T10:00:53.180 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T10:00:53.181 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:00:53.189 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T10:00:53.207 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:00:53.360 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:53.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:53 smithi119 ceph-mon[40785]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T10:00:53.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:53 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3696283793' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:53.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:53 smithi119 ceph-mon[40785]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T10:00:53.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:53 smithi119 ceph-mon[29367]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T10:00:53.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:53 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3696283793' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:53.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:53 smithi119 ceph-mon[29367]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T10:00:53.735 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:53 smithi145 ceph-mon[33206]: osdmap e218: 8 total, 8 up, 8 in 2024-09-21T10:00:53.735 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:53 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3696283793' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:53.735 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:53 smithi145 ceph-mon[33206]: osdmap e219: 8 total, 8 up, 8 in 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[40785]: pgmap v9: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[40785]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[40785]: mgrmap e30: y(active, since 4s) 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[40785]: from='client.25802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[40785]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[29367]: pgmap v9: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:54.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[29367]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[29367]: mgrmap e30: y(active, since 4s) 2024-09-21T10:00:54.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[29367]: from='client.25802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:54 smithi119 ceph-mon[29367]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T10:00:54.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-mon[33206]: pgmap v9: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:54.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-mon[33206]: from='client.25796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-mon[33206]: mgrmap e30: y(active, since 4s) 2024-09-21T10:00:54.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-mon[33206]: from='client.25802 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:54.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-mon[33206]: osdmap e220: 8 total, 8 up, 8 in 2024-09-21T10:00:55.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:10:00:54] ENGINE Bus STARTING 2024-09-21T10:00:55.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: CherryPy Checker: 2024-09-21T10:00:55.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: The Application mounted at '' has an empty config. 2024-09-21T10:00:55.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: 2024-09-21T10:00:55.045 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:10:00:54] ENGINE Serving on http://:::9283 2024-09-21T10:00:55.045 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:00:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: [21/Sep/2024:10:00:54] ENGINE Bus STARTED 2024-09-21T10:00:55.178 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:00:55.197 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T10:00:55.205 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: Standby manager daemon x started 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[40785]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: Standby manager daemon x started 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:55 smithi119 ceph-mon[29367]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: Standby manager daemon x started 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/2668895389' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:00:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:55.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:55 smithi145 ceph-mon[33206]: osdmap e221: 8 total, 8 up, 8 in 2024-09-21T10:00:56.283 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:56.742 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: pgmap v12: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:56.742 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T10:00:56.742 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T10:00:56.742 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:56.742 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:56.742 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/955833849' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[29367]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: pgmap v12: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/955833849' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:56.743 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:56 smithi119 ceph-mon[40785]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: pgmap v12: 137 pgs: 137 active+clean; 689 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/955833849' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:56.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-mon[33206]: osdmap e222: 8 total, 8 up, 8 in 2024-09-21T10:00:57.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:00:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:00:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:00:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:58 smithi119 ceph-mon[40785]: from='client.15903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:58 smithi119 ceph-mon[40785]: from='client.15909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:58 smithi119 ceph-mon[40785]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T10:00:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:58 smithi119 ceph-mon[29367]: from='client.15903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:58 smithi119 ceph-mon[29367]: from='client.15909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:58 smithi119 ceph-mon[29367]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T10:00:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:58 smithi145 ceph-mon[33206]: from='client.15903 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:58 smithi145 ceph-mon[33206]: from='client.15909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:00:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:58 smithi145 ceph-mon[33206]: osdmap e223: 8 total, 8 up, 8 in 2024-09-21T10:00:58.589 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: after_prepare_snap 2024-09-21T10:00:59.296 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: pgmap v15: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 184 KiB/s rd, 8.0 MiB/s wr, 219 op/s 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/862054362' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:00:59 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: pgmap v15: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 184 KiB/s rd, 8.0 MiB/s wr, 219 op/s 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/862054362' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: pgmap v15: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 184 KiB/s rd, 8.0 MiB/s wr, 219 op/s 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: osdmap e224: 8 total, 8 up, 8 in 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:00:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:00:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/862054362' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:00:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:00:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:00:59 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:00.509 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:01:00.522 INFO:tasks.workunit.client.0.smithi119.stdout:write: * 83886080~1000 2024-09-21T10:01:00.528 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T10:01:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T10:01:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T10:01:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:00 smithi145 ceph-mon[33206]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T10:01:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[29367]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T10:01:00.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:00 smithi119 ceph-mon[40785]: osdmap e225: 8 total, 8 up, 8 in 2024-09-21T10:01:00.689 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:01.397 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:01 smithi145 ceph-mon[33206]: from='client.25832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:01.397 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:01 smithi145 ceph-mon[33206]: Reconfiguring daemon prometheus.a on smithi145 2024-09-21T10:01:01.398 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:01 smithi145 ceph-mon[33206]: pgmap v17: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 177 KiB/s rd, 7.7 MiB/s wr, 210 op/s 2024-09-21T10:01:01.398 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:01 smithi145 ceph-mon[33206]: from='client.15927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:01.398 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:01 smithi145 ceph-mon[33206]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T10:01:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[40785]: from='client.25832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[40785]: Reconfiguring daemon prometheus.a on smithi145 2024-09-21T10:01:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[40785]: pgmap v17: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 177 KiB/s rd, 7.7 MiB/s wr, 210 op/s 2024-09-21T10:01:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[40785]: from='client.15927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[40785]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T10:01:01.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[29367]: from='client.25832 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:01.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[29367]: Reconfiguring daemon prometheus.a on smithi145 2024-09-21T10:01:01.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[29367]: pgmap v17: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 177 KiB/s rd, 7.7 MiB/s wr, 210 op/s 2024-09-21T10:01:01.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[29367]: from='client.15927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:01.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:01 smithi119 ceph-mon[29367]: osdmap e226: 8 total, 8 up, 8 in 2024-09-21T10:01:01.656 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[68648]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:01:01] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.51.0" 2024-09-21T10:01:02.214 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:02 smithi145 ceph-mon[33206]: pgmap v20: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 4.6 KiB/s wr, 171 op/s 2024-09-21T10:01:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:02 smithi145 ceph-mon[33206]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T10:01:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:02 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/923703890' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:02 smithi145 ceph-mon[33206]: from='client.15939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[29367]: pgmap v20: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 4.6 KiB/s wr, 171 op/s 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[29367]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/923703890' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[29367]: from='client.15939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[40785]: pgmap v20: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 31 MiB/s rd, 4.6 KiB/s wr, 171 op/s 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[40785]: osdmap e227: 8 total, 8 up, 8 in 2024-09-21T10:01:02.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/923703890' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:02.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:02 smithi119 ceph-mon[40785]: from='client.15939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.110 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:02 smithi145 systemd[1]: Stopping Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T10:01:03.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T10:01:03.419 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.108Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T10:01:03.419 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.110Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T10:01:03.419 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.110Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T10:01:03.419 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[76596]: ts=2024-09-21T10:01:03.110Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T10:01:03.419 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 podman[79401]: 2024-09-21 10:01:03.119000121 +0000 UTC m=+0.111785676 container died a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:01:03.751 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-mon[33206]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.751 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:03 smithi145 ceph-mon[33206]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T10:01:03.752 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 podman[79401]: 2024-09-21 10:01:03.523152168 +0000 UTC m=+0.515937709 container cleanup a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:01:03.752 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 bash[79401]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T10:01:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:03 smithi119 ceph-mon[40785]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:03 smithi119 ceph-mon[40785]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T10:01:03.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:03 smithi119 ceph-mon[29367]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:03.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:03 smithi119 ceph-mon[29367]: osdmap e228: 8 total, 8 up, 8 in 2024-09-21T10:01:03.892 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapsSnapExpandAfterPrepare (14346 ms) 2024-09-21T10:01:03.892 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.Clone 2024-09-21T10:01:03.921 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:03.931 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~8388609 2024-09-21T10:01:03.993 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 16777218~8388609 2024-09-21T10:01:04.004 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:03 smithi145 podman[79418]: 2024-09-21 10:01:03.749924582 +0000 UTC m=+0.631911435 container remove a941f74ce33e64dc0c12fa4c36dea02201c0f0eb2188b3d399114bd02068f7f2 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:01:04.048 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap 2024-09-21T10:01:04.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T10:01:04.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 systemd[1]: Stopped Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:04.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.154s CPU time. 2024-09-21T10:01:04.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 systemd[1]: Starting Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:04.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 podman[79517]: 2024-09-21 10:01:04.121428682 +0000 UTC m=+0.016793160 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-09-21T10:01:04.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 podman[79517]: 2024-09-21 10:01:04.23406742 +0000 UTC m=+0.129431944 container create d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:01:04.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:04 smithi119 ceph-mon[29367]: pgmap v23: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 4.7 KiB/s wr, 178 op/s 2024-09-21T10:01:04.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:04 smithi119 ceph-mon[29367]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T10:01:04.768 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:04 smithi119 ceph-mon[40785]: pgmap v23: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 4.7 KiB/s wr, 178 op/s 2024-09-21T10:01:04.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:04 smithi119 ceph-mon[40785]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T10:01:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-mon[33206]: pgmap v23: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 32 MiB/s rd, 4.7 KiB/s wr, 178 op/s 2024-09-21T10:01:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-mon[33206]: osdmap e229: 8 total, 8 up, 8 in 2024-09-21T10:01:04.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 podman[79517]: 2024-09-21 10:01:04.46980289 +0000 UTC m=+0.365167377 container init d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:01:04.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 podman[79517]: 2024-09-21 10:01:04.47470028 +0000 UTC m=+0.370064767 container start d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.494Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.494Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.494Z caller=main.go:623 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi145 (none))" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.494Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.494Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.496Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.497Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.498Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.498Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.500Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.501Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=732.91µs 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.501Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.501Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.514Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-21T10:01:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.521Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.521Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.521Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=45.892µs wal_replay_duration=19.872953ms wbl_replay_duration=410ns total_replay_duration=20.672979ms 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.525Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.525Z caller=main.go:1153 level=info msg="TSDB started" 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.525Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.542Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.246966ms db_storage=890ns remote_storage=1.058µs web_handler=550ns query_engine=722ns scrape=281.407µs scrape_sd=109.415µs notify=12.625µs notify_sd=13.421µs rules=15.445151ms tracing=6.127µs 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.542Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:04.542Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 bash[79517]: d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 2024-09-21T10:01:04.796 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:04 smithi145 systemd[1]: Started Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:05.135 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:05.541 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap 2024-09-21T10:01:05.551 INFO:tasks.workunit.client.0.smithi119.stdout:clone image105 -> image108 2024-09-21T10:01:05.642 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T10:01:05.704 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T10:01:05.706 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3903593959' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:05.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:05 smithi145 ceph-mon[33206]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3903593959' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[29367]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: osdmap e230: 8 total, 8 up, 8 in 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3903593959' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:05.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:05 smithi119 ceph-mon[40785]: from='client.15957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.765 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: pgmap v26: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: Upgrade: Updating mgr.x 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:06.766 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-mon[33206]: Deploying daemon mgr.x on smithi145 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: pgmap v26: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: Upgrade: Updating mgr.x 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[29367]: Deploying daemon mgr.x on smithi145 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: pgmap v26: 137 pgs: 137 active+clean; 729 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: from='client.25874 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: osdmap e231: 8 total, 8 up, 8 in 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: Upgrade: Updating mgr.x 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:06.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:06 smithi119 ceph-mon[40785]: Deploying daemon mgr.x on smithi145 2024-09-21T10:01:07.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:07.708 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap 2024-09-21T10:01:07.718 INFO:tasks.workunit.client.0.smithi119.stdout:clone image108 -> image110 2024-09-21T10:01:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:07 smithi145 ceph-mon[33206]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T10:01:07.828 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 1000~1000 2024-09-21T10:01:07.885 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 2010~1000 2024-09-21T10:01:07.887 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:08.107 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:08.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:07 smithi119 ceph-mon[29367]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T10:01:08.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:07 smithi119 ceph-mon[40785]: osdmap e232: 8 total, 8 up, 8 in 2024-09-21T10:01:08.168 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:07 smithi145 systemd[1]: Stopping Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:08.192 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:08.358 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:08.470 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 podman[79863]: 2024-09-21 10:01:08.165752262 +0000 UTC m=+0.141261214 container died 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:01:08.522 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.Clone (4630 ms) 2024-09-21T10:01:08.523 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneParent 2024-09-21T10:01:08.551 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:08.563 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap 2024-09-21T10:01:08.790 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:08 smithi145 ceph-mon[33206]: pgmap v29: 137 pgs: 137 active+clean; 761 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 14 MiB/s wr, 435 op/s 2024-09-21T10:01:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:08 smithi145 ceph-mon[33206]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T10:01:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:08 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1666907270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:08 smithi145 ceph-mon[33206]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:08.791 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:08 smithi145 ceph-mon[33206]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T10:01:08.791 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 podman[79863]: 2024-09-21 10:01:08.569731159 +0000 UTC m=+0.545240103 container cleanup 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-21T10:01:08.791 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 bash[79863]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T10:01:09.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 podman[79881]: 2024-09-21 10:01:08.789119491 +0000 UTC m=+0.624141901 container remove 068dda3fcdc9d6a84052b2ee706bf00001a1ace45d363b70b28c829f60cd9daa (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T10:01:09.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:01:09.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T10:01:09.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 systemd[1]: Stopped Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:09.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:08 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service: Consumed 17.553s CPU time. 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[40785]: pgmap v29: 137 pgs: 137 active+clean; 761 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 14 MiB/s wr, 435 op/s 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[40785]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1666907270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[40785]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[40785]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[29367]: pgmap v29: 137 pgs: 137 active+clean; 761 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 46 MiB/s rd, 14 MiB/s wr, 435 op/s 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[29367]: osdmap e233: 8 total, 8 up, 8 in 2024-09-21T10:01:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1666907270' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[29367]: from='client.25886 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:08 smithi119 ceph-mon[29367]: osdmap e234: 8 total, 8 up, 8 in 2024-09-21T10:01:09.489 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 systemd[1]: Starting Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:09.489 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 podman[80014]: 2024-09-21 10:01:09.232502921 +0000 UTC m=+0.015760907 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:09.490 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 podman[80014]: 2024-09-21 10:01:09.35724403 +0000 UTC m=+0.140502007 container create 1346375dd74716f801cfc6cb378cdd27c2af9787385fc074fc7afae957fcf1f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-62da54b, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T10:01:10.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 podman[80014]: 2024-09-21 10:01:09.612491502 +0000 UTC m=+0.395749478 container init 1346375dd74716f801cfc6cb378cdd27c2af9787385fc074fc7afae957fcf1f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T10:01:10.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 podman[80014]: 2024-09-21 10:01:09.617887179 +0000 UTC m=+0.401145157 container start 1346375dd74716f801cfc6cb378cdd27c2af9787385fc074fc7afae957fcf1f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=main-62da54b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T10:01:10.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 ceph-mgr[80052]: -- 172.21.15.145:0/2918912492 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c363d570e0 con 0x55c363d35400 2024-09-21T10:01:10.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 bash[80014]: 1346375dd74716f801cfc6cb378cdd27c2af9787385fc074fc7afae957fcf1f0 2024-09-21T10:01:10.044 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:09 smithi145 systemd[1]: Started Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:10.194 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap 2024-09-21T10:01:10.203 INFO:tasks.workunit.client.0.smithi119.stdout:clone image111 -> image114 2024-09-21T10:01:10.292 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:10.436 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:01:10.451 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:10 smithi145 ceph-mon[33206]: from='client.25892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:10.451 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:10 smithi145 ceph-mon[33206]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T10:01:10.451 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:10 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:10.451 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:10 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:10.451 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:10 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:10.452 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:10.563 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneParent (2041 ms) 2024-09-21T10:01:10.563 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneUpdateAfterPrepare 2024-09-21T10:01:10.591 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:10.601 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~10 2024-09-21T10:01:10.608 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap 2024-09-21T10:01:10.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[40785]: from='client.25892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[40785]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[29367]: from='client.25892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[29367]: osdmap e235: 8 total, 8 up, 8 in 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:10 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:11.084 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:11.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:11 smithi119 ceph-mon[29367]: pgmap v33: 137 pgs: 137 active+clean; 761 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:11.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:11 smithi119 ceph-mon[29367]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T10:01:11.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:11 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2453700770' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:11.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:11 smithi119 ceph-mon[40785]: pgmap v33: 137 pgs: 137 active+clean; 761 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:11.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:11 smithi119 ceph-mon[40785]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T10:01:11.359 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:11 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2453700770' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:11 smithi145 ceph-mon[33206]: pgmap v33: 137 pgs: 137 active+clean; 761 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:11 smithi145 ceph-mon[33206]: osdmap e236: 8 total, 8 up, 8 in 2024-09-21T10:01:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:11 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2453700770' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:12.204 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap 2024-09-21T10:01:12.214 INFO:tasks.workunit.client.0.smithi119.stdout:clone image115 -> image118 2024-09-21T10:01:12.301 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:12.443 INFO:tasks.workunit.client.0.smithi119.stdout:write: Y 0~1 2024-09-21T10:01:12.473 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:12.542 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:12.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:12 smithi145 ceph-mon[33206]: from='client.15996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:12 smithi145 ceph-mon[33206]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T10:01:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:12 smithi119 ceph-mon[29367]: from='client.15996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:12 smithi119 ceph-mon[29367]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T10:01:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:12 smithi119 ceph-mon[40785]: from='client.15996 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:12 smithi119 ceph-mon[40785]: osdmap e237: 8 total, 8 up, 8 in 2024-09-21T10:01:12.658 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneUpdateAfterPrepare (2094 ms) 2024-09-21T10:01:12.658 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.TriggerAssertSnapcSeq 2024-09-21T10:01:12.692 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:12.702 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 41943040~10 2024-09-21T10:01:12.710 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:13 smithi145 ceph-mon[33206]: pgmap v36: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 1.08k op/s 2024-09-21T10:01:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:13 smithi145 ceph-mon[33206]: from='client.16002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:13 smithi145 ceph-mon[33206]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T10:01:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:13 smithi119 ceph-mon[29367]: pgmap v36: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 1.08k op/s 2024-09-21T10:01:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:13 smithi119 ceph-mon[29367]: from='client.16002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:13 smithi119 ceph-mon[29367]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T10:01:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:13 smithi119 ceph-mon[40785]: pgmap v36: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 1.08k op/s 2024-09-21T10:01:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:13 smithi119 ceph-mon[40785]: from='client.16002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:13 smithi119 ceph-mon[40785]: osdmap e238: 8 total, 8 up, 8 in 2024-09-21T10:01:13.994 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:14.214 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 41943041~10 2024-09-21T10:01:14.218 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:14 smithi119 ceph-mon[29367]: osdmap e239: 8 total, 8 up, 8 in 2024-09-21T10:01:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:14 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/1444616164' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:14 smithi119 ceph-mon[40785]: osdmap e239: 8 total, 8 up, 8 in 2024-09-21T10:01:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:14 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1444616164' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:14 smithi145 ceph-mon[33206]: osdmap e239: 8 total, 8 up, 8 in 2024-09-21T10:01:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:14 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1444616164' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:01:14] "GET /metrics HTTP/1.1" 200 38912 "" "Prometheus/2.51.0" 2024-09-21T10:01:15.325 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 41943042~10 2024-09-21T10:01:15.339 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:01:15.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: pgmap v39: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 MiB/s rd, 5.0 MiB/s wr, 981 op/s 2024-09-21T10:01:15.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='client.16014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:15.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: osdmap e240: 8 total, 8 up, 8 in 2024-09-21T10:01:15.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.488 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: Standby manager daemon x restarted 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: Standby manager daemon x started 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: [21/Sep/2024:10:01:15] ENGINE Bus STARTING 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: CherryPy Checker: 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: The Application mounted at '' has an empty config. 2024-09-21T10:01:15.489 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: 2024-09-21T10:01:15.490 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: [21/Sep/2024:10:01:15] ENGINE Serving on http://:::9283 2024-09-21T10:01:15.490 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: [21/Sep/2024:10:01:15] ENGINE Bus STARTED 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: pgmap v39: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 MiB/s rd, 5.0 MiB/s wr, 981 op/s 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='client.16014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: osdmap e240: 8 total, 8 up, 8 in 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: Standby manager daemon x restarted 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: Standby manager daemon x started 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:01:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:01:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:01:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: pgmap v39: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 22 MiB/s rd, 5.0 MiB/s wr, 981 op/s 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='client.16014 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: osdmap e240: 8 total, 8 up, 8 in 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: Standby manager daemon x restarted 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: Standby manager daemon x started 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.? 172.21.15.145:0/1428310283' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:15.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:15 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:16.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:16.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:16 smithi145 ceph-mon[33206]: from='client.16020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:16 smithi145 ceph-mon[33206]: osdmap e241: 8 total, 8 up, 8 in 2024-09-21T10:01:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:16 smithi145 ceph-mon[33206]: mgrmap e32: y(active, since 26s), standbys: x 2024-09-21T10:01:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:16 smithi119 ceph-mon[40785]: from='client.16020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:16 smithi119 ceph-mon[40785]: osdmap e241: 8 total, 8 up, 8 in 2024-09-21T10:01:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:16 smithi119 ceph-mon[40785]: mgrmap e32: y(active, since 26s), standbys: x 2024-09-21T10:01:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:16 smithi119 ceph-mon[29367]: from='client.16020 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:16 smithi119 ceph-mon[29367]: osdmap e241: 8 total, 8 up, 8 in 2024-09-21T10:01:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:16 smithi119 ceph-mon[29367]: mgrmap e32: y(active, since 26s), standbys: x 2024-09-21T10:01:16.915 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:17.174 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:17.229 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T10:01:17.253 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:17.308 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:17 smithi145 ceph-mon[33206]: pgmap v42: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:17 smithi145 ceph-mon[33206]: osdmap e242: 8 total, 8 up, 8 in 2024-09-21T10:01:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:17 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3883518175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:17 smithi119 ceph-mon[40785]: pgmap v42: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:17 smithi119 ceph-mon[40785]: osdmap e242: 8 total, 8 up, 8 in 2024-09-21T10:01:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:17 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3883518175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:17 smithi119 ceph-mon[29367]: pgmap v42: 137 pgs: 137 active+clean; 777 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:17 smithi119 ceph-mon[29367]: osdmap e242: 8 total, 8 up, 8 in 2024-09-21T10:01:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:17 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/3883518175' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:18.342 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.TriggerAssertSnapcSeq (5685 ms) 2024-09-21T10:01:18.342 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.SnapTrimBeforePrepare 2024-09-21T10:01:18.370 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:18.380 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 41943040~10 2024-09-21T10:01:18.387 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:18.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:18 smithi145 ceph-mon[33206]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:18 smithi145 ceph-mon[33206]: from='client.25949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:18 smithi145 ceph-mon[33206]: osdmap e243: 8 total, 8 up, 8 in 2024-09-21T10:01:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:18 smithi119 ceph-mon[29367]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:18 smithi119 ceph-mon[29367]: from='client.25949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:18 smithi119 ceph-mon[29367]: osdmap e243: 8 total, 8 up, 8 in 2024-09-21T10:01:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:18 smithi119 ceph-mon[40785]: from='client.16032 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:18 smithi119 ceph-mon[40785]: from='client.25949 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:18 smithi119 ceph-mon[40785]: osdmap e243: 8 total, 8 up, 8 in 2024-09-21T10:01:19.346 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: pgmap v45: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 711 op/s 2024-09-21T10:01:19.346 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: osdmap e244: 8 total, 8 up, 8 in 2024-09-21T10:01:19.346 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:01:19.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: pgmap v45: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 711 op/s 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: osdmap e244: 8 total, 8 up, 8 in 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T10:01:19.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T10:01:19.349 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T10:01:19.349 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T10:01:19.349 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:01:19.349 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:19 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T10:01:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: pgmap v45: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 711 op/s 2024-09-21T10:01:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: osdmap e244: 8 total, 8 up, 8 in 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T10:01:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-21T10:01:19.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:19 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-21T10:01:19.835 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:20.255 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T10:01:20.276 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:20 smithi145 ceph-mon[33206]: Upgrade: Setting container_image for all mgr 2024-09-21T10:01:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:20 smithi145 ceph-mon[33206]: Upgrade: It appears safe to stop mon.a 2024-09-21T10:01:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:20 smithi145 ceph-mon[33206]: osdmap e245: 8 total, 8 up, 8 in 2024-09-21T10:01:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:20 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2872133465' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:20.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:20 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[29367]: Upgrade: Setting container_image for all mgr 2024-09-21T10:01:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[29367]: Upgrade: It appears safe to stop mon.a 2024-09-21T10:01:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[29367]: osdmap e245: 8 total, 8 up, 8 in 2024-09-21T10:01:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[29367]: from='client.? 172.21.15.119:0/2872133465' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[40785]: Upgrade: Setting container_image for all mgr 2024-09-21T10:01:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[40785]: Upgrade: It appears safe to stop mon.a 2024-09-21T10:01:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[40785]: osdmap e245: 8 total, 8 up, 8 in 2024-09-21T10:01:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/2872133465' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:20 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T10:01:21.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: pgmap v48: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 711 op/s 2024-09-21T10:01:21.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: from='client.25961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:21.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: from='client.25967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:21.368 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:21.378 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:01:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: pgmap v48: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 711 op/s 2024-09-21T10:01:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: from='client.25961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: from='client.25967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: osdmap e246: 8 total, 8 up, 8 in 2024-09-21T10:01:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: Upgrade: Updating mon.a 2024-09-21T10:01:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:21.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:01:21.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:01:21.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:21.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:21 smithi145 ceph-mon[33206]: Deploying daemon mon.a on smithi119 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: pgmap v48: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 711 op/s 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: from='client.25961 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: from='client.25967 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: osdmap e246: 8 total, 8 up, 8 in 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: Upgrade: Updating mon.a 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[40785]: Deploying daemon mon.a on smithi119 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: osdmap e246: 8 total, 8 up, 8 in 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: Upgrade: Updating mon.a 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T10:01:21.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:21 smithi119 ceph-mon[29367]: Deploying daemon mon.a on smithi119 2024-09-21T10:01:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 systemd[1]: Stopping Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:22 smithi145 ceph-mon[33206]: osdmap e247: 8 total, 8 up, 8 in 2024-09-21T10:01:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:22 smithi145 ceph-mon[33206]: pgmap v51: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 214 KiB/s rd, 23 KiB/s wr, 295 op/s 2024-09-21T10:01:22.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-mon[40785]: osdmap e247: 8 total, 8 up, 8 in 2024-09-21T10:01:22.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-mon[40785]: pgmap v51: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 214 KiB/s rd, 23 KiB/s wr, 295 op/s 2024-09-21T10:01:22.559 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:22] ENGINE Bus STOPPING 2024-09-21T10:01:22.559 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:01:22.559 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:22] ENGINE Bus STOPPED 2024-09-21T10:01:22.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-mon[29367]: osdmap e247: 8 total, 8 up, 8 in 2024-09-21T10:01:22.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-mon[29367]: pgmap v51: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 214 KiB/s rd, 23 KiB/s wr, 295 op/s 2024-09-21T10:01:22.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[29343]: 2024-09-21T10:01:22.306+0000 7fb53bd7c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:01:22.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[29343]: 2024-09-21T10:01:22.306+0000 7fb53bd7c640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T10:01:22.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 podman[101162]: 2024-09-21 10:01:22.378622145 +0000 UTC m=+0.200156316 container died 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:01:22.774 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:22.848 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:22] ENGINE Bus STARTING 2024-09-21T10:01:22.849 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:22] ENGINE Serving on http://:::9283 2024-09-21T10:01:22.849 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:22] ENGINE Bus STARTED 2024-09-21T10:01:23.109 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 podman[101162]: 2024-09-21 10:01:22.847450677 +0000 UTC m=+0.668984848 container cleanup 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux ) 2024-09-21T10:01:23.109 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:22 smithi119 bash[101162]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T10:01:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 podman[101197]: 2024-09-21 10:01:23.107513756 +0000 UTC m=+0.725016640 container remove 1467ec54a60b3e4abc0d06d24567765c497b782fb7d591b89a556e9324930fb0 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T10:01:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T10:01:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 systemd[1]: Stopped Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service: Consumed 15.480s CPU time. 2024-09-21T10:01:23.812 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 systemd[1]: Starting Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:01:23.813 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 podman[101394]: 2024-09-21 10:01:23.584574534 +0000 UTC m=+0.015524328 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:23.813 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 podman[101394]: 2024-09-21 10:01:23.692416398 +0000 UTC m=+0.123366222 container create 030d68226d6c3648ebdf56480c5b35f2ad9aad20c680ccecab0008a66565038e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T10:01:24.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 podman[101394]: 2024-09-21 10:01:23.894894989 +0000 UTC m=+0.325844771 container init 030d68226d6c3648ebdf56480c5b35f2ad9aad20c680ccecab0008a66565038e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, RELEASE=main-62da54b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T10:01:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 podman[101394]: 2024-09-21 10:01:23.89967605 +0000 UTC m=+0.330625834 container start 030d68226d6c3648ebdf56480c5b35f2ad9aad20c680ccecab0008a66565038e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:01:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T10:01:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: pidfile_write: ignore empty --pid-file 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: load: jerasure load: lrc 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: RocksDB version: 7.9.2 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Git sha 0 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: DB SUMMARY 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: DB Session ID: Z4K0WT838O1OTWU5OYBC 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: CURRENT file: CURRENT 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: MANIFEST file: MANIFEST-000015 size: 973 Bytes 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3664144 ; 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.error_if_exists: 0 2024-09-21T10:01:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.create_if_missing: 0 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.paranoid_checks: 1 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.env: 0x55e19d544120 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.info_log: 0x55e19e549700 2024-09-21T10:01:24.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.statistics: (nil) 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.use_fsync: 0 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_log_file_size: 0 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T10:01:24.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.allow_fallocate: 1 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.use_direct_reads: 0 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.db_log_dir: 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.wal_dir: 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T10:01:24.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.write_buffer_manager: 0x55e19e54c1e0 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T10:01:24.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.unordered_write: 0 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.row_cache: None 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.wal_filter: None 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T10:01:24.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.two_write_queues: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.wal_compression: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.atomic_flush: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.log_readahead_size: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T10:01:24.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_background_jobs: 2 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_background_compactions: -1 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_subcompactions: 1 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T10:01:24.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_open_files: -1 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_background_flushes: -1 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Compression algorithms supported: 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kZSTD supported: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kXpressCompression supported: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kBZip2Compression supported: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kLZ4Compression supported: 1 2024-09-21T10:01:24.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kZlibCompression supported: 1 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: kSnappyCompression supported: 1 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.merge_operator: 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_filter: None 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T10:01:24.378 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e19e549320) 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_type: 0 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_index_type: 0 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_shortening: 1 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: checksum: 4 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: no_block_cache: 0 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache: 0x55e19e53d610 2024-09-21T10:01:24.379 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_options: 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: capacity : 536870912 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: num_shard_bits : 4 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: strict_capacity_limit : 0 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_compressed: (nil) 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: persistent_cache: (nil) 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size: 4096 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size_deviation: 10 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_restart_interval: 16 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_block_restart_interval: 1 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: metadata_block_size: 4096 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: partition_filters: 0 2024-09-21T10:01:24.380 INFO:journalctl@ceph.mon.a.smithi119.stdout: use_delta_encoding: 1 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: filter_policy: bloomfilter 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: whole_key_filtering: 1 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: verify_compression: 0 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: format_version: 5 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_index_compression: 1 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_align: 0 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: max_auto_readahead_size: 262144 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: prepopulate_block_cache: 0 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: initial_auto_readahead_size: 8192 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T10:01:24.381 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression: NoCompression 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.num_levels: 7 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T10:01:24.382 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:01:24.383 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T10:01:24.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T10:01:24.385 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.inplace_update_support: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.bloom_locality: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.max_successive_merges: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T10:01:24.386 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.ttl: 2592000 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.enable_blob_files: false 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.min_blob_size: 0 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T10:01:24.387 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12806, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 34a153c9-dbdd-4107-a831-a2e275cf1a06 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912883926310, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912883936320, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3429496, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12807, "largest_seqno": 14319, "table_properties": {"data_size": 3420954, "index_size": 5769, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1733, "raw_key_size": 16331, "raw_average_key_size": 23, "raw_value_size": 3405244, "raw_average_value_size": 4956, "num_data_blocks": 254, "num_entries": 687, "num_filter_entries": 687, "num_deletions": 6, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912883, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "34a153c9-dbdd-4107-a831-a2e275cf1a06", "db_session_id": "Z4K0WT838O1OTWU5OYBC", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912883936439, "job": 1, "event": "recovery_finished"} 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-09-21T10:01:24.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e19e56ce00 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: DB pointer 0x55e19e66e000 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** DB Stats ** 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:01:24.389 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: L0 1/0 3.27 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 679.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: L6 1/0 7.80 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: Sum 2/0 11.07 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 679.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 679.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 679.7 0.00 0.00 1 0.005 0 0 0.0 0.0 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.390 INFO:journalctl@ceph.mon.a.smithi119.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative compaction: 0.00 GB write, 233.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval compaction: 0.00 GB write, 233.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Block cache BinnedLRUCache@0x55e19e53d610#7 capacity: 512.00 MB usage: 7.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,6.08 KB,0.00115931%) Misc(1,0.00 KB,0%) 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.391 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: starting mon.a rank 0 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???) e3 preinit fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).mds e9 new map 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).mds e9 print_map 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: e9 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: legacy client fscid: 1 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: Filesystem 'foo' (1) 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: fs_name foo 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: epoch 8 2024-09-21T10:01:24.392 INFO:journalctl@ceph.mon.a.smithi119.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: created 2024-09-21T09:56:59.994919+0000 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: modified 2024-09-21T09:57:18.635502+0000 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: tableserver 0 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: root 0 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: session_timeout 60 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: session_autoclose 300 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: max_file_size 1099511627776 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: max_xattr_size 65536 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: required_client_features {} 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: last_failure 0 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: last_failure_osd_epoch 0 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: max_mds 1 2024-09-21T10:01:24.393 INFO:journalctl@ceph.mon.a.smithi119.stdout: in 0 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: up {0=24557} 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: failed 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: damaged 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: stopped 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_pools [4] 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: metadata_pool 3 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: inline_data disabled 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: balancer 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: bal_rank_mask -1 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: standby_count_wanted 1 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: qdb_cluster leader: 0 members: 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: [mds.foo.smithi119.oikglq{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.394 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout: Standby daemons: 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout: [mds.foo.smithi145.tjomfr{-1:24457} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).osd e248 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).osd e248 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).osd e248 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).osd e248 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 ceph-mon[101432]: mon.a@-1(???).paxosservice(auth 1..18) refresh upgraded, format 0 -> 3 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 bash[101394]: 030d68226d6c3648ebdf56480c5b35f2ad9aad20c680ccecab0008a66565038e 2024-09-21T10:01:24.395 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:23 smithi119 systemd[1]: Started Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:01:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:01:24] "GET /metrics HTTP/1.1" 200 38913 "" "Prometheus/2.51.0" 2024-09-21T10:01:25.710 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:01:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:25.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:25.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:27.035 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 41943040~10 2024-09-21T10:01:27.040 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap4 2024-09-21T10:01:27.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: ignoring --setuser ceph since I am not root 2024-09-21T10:01:27.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: ignoring --setgroup ceph since I am not root 2024-09-21T10:01:27.294 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mgr[80052]: -- 172.21.15.145:0/1894016565 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557f9b5e90e0 con 0x557f9b5c7000 2024-09-21T10:01:27.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:26 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:27.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:01:27.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:01:27.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:01:27.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3565813891' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:27.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='client.25979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='client.25985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: pgmap v53: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 208 KiB/s rd, 23 KiB/s wr, 287 op/s 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: pgmap v54: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 142 KiB/s rd, 16 KiB/s wr, 196 op/s 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: mon.a calling monitor election 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/1815355035' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='client.26003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: monmap epoch 3 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: last_changed 2024-09-21T09:51:41.651239+0000 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:01:27.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: min_mon_release 17 (quincy) 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: election_strategy: 1 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: osdmap e248: 8 total, 8 up, 8 in 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: mgrmap e32: y(active, since 37s), standbys: x 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: from='mgr.15786 ' entity='' 2024-09-21T10:01:27.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:27 smithi145 ceph-mon[33206]: mgrmap e33: y(active, since 37s), standbys: x 2024-09-21T10:01:27.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ignoring --setuser ceph since I am not root 2024-09-21T10:01:27.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ignoring --setgroup ceph since I am not root 2024-09-21T10:01:27.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mgr[92504]: -- 172.21.15.119:0/2491545042 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a4c01e10e0 con 0x55a4c01bf400 2024-09-21T10:01:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:01:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3565813891' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='client.25979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='client.25985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: pgmap v53: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 208 KiB/s rd, 23 KiB/s wr, 287 op/s 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: pgmap v54: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 142 KiB/s rd, 16 KiB/s wr, 196 op/s 2024-09-21T10:01:27.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: mon.a calling monitor election 2024-09-21T10:01:27.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/1815355035' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:27.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='client.26003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:01:27.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: monmap epoch 3 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: last_changed 2024-09-21T09:51:41.651239+0000 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: min_mon_release 17 (quincy) 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: election_strategy: 1 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:01:27.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: osdmap e248: 8 total, 8 up, 8 in 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: mgrmap e32: y(active, since 37s), standbys: x 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:01:27.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:01:27.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:01:27.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: from='mgr.15786 ' entity='' 2024-09-21T10:01:27.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[40785]: mgrmap e33: y(active, since 37s), standbys: x 2024-09-21T10:01:27.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T10:01:27.373 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3565813891' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='mgr.15786 172.21.15.119:0/3305034640' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='client.25979 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='client.25985 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: pgmap v53: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 208 KiB/s rd, 23 KiB/s wr, 287 op/s 2024-09-21T10:01:27.374 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: pgmap v54: 137 pgs: 137 active+clean; 781 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 142 KiB/s rd, 16 KiB/s wr, 196 op/s 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: mon.a calling monitor election 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1815355035' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='client.25997 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='client.26003 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: monmap epoch 3 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: last_changed 2024-09-21T09:51:41.651239+0000 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: min_mon_release 17 (quincy) 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: election_strategy: 1 2024-09-21T10:01:27.375 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: osdmap e248: 8 total, 8 up, 8 in 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: mgrmap e32: y(active, since 37s), standbys: x 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: from='mgr.15786 ' entity='' 2024-09-21T10:01:27.376 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:27 smithi119 ceph-mon[101432]: mgrmap e33: y(active, since 37s), standbys: x 2024-09-21T10:01:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:28 smithi145 ceph-mon[33206]: osdmap e249: 8 total, 8 up, 8 in 2024-09-21T10:01:28.349 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 1 2024-09-21T10:01:28.349 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:28 smithi119 ceph-mon[101432]: osdmap e249: 8 total, 8 up, 8 in 2024-09-21T10:01:28.349 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:28 smithi119 ceph-mon[40785]: osdmap e249: 8 total, 8 up, 8 in 2024-09-21T10:01:28.959 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 41943040 2024-09-21T10:01:28.979 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:29.033 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:29.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:01:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:29 smithi145 ceph-mon[33206]: osdmap e250: 8 total, 8 up, 8 in 2024-09-21T10:01:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:29 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3626577181' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:01:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:29 smithi145 ceph-mon[33206]: osdmap e251: 8 total, 8 up, 8 in 2024-09-21T10:01:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:29 smithi119 ceph-mon[40785]: osdmap e250: 8 total, 8 up, 8 in 2024-09-21T10:01:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:29 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3626577181' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:01:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:29 smithi119 ceph-mon[40785]: osdmap e251: 8 total, 8 up, 8 in 2024-09-21T10:01:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:29 smithi119 ceph-mon[101432]: osdmap e250: 8 total, 8 up, 8 in 2024-09-21T10:01:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3626577181' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:01:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:29 smithi119 ceph-mon[101432]: osdmap e251: 8 total, 8 up, 8 in 2024-09-21T10:01:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:30.163 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.SnapTrimBeforePrepare (11821 ms) 2024-09-21T10:01:30.163 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.AbortInUseImage 2024-09-21T10:01:30.193 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:30.203 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:30.383 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.AbortInUseImage (220 ms) 2024-09-21T10:01:30.383 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.AbortWithoutSnapshots 2024-09-21T10:01:30.411 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:30.421 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:01:30.487 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T10:01:30.537 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:30.721 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:01:30.776 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T10:01:30.836 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:01:31.037 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.AbortWithoutSnapshots (654 ms) 2024-09-21T10:01:31.037 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.AbortWithSnapshots 2024-09-21T10:01:31.069 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:31.080 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:01:31.153 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:31 smithi145 ceph-mon[33206]: osdmap e252: 8 total, 8 up, 8 in 2024-09-21T10:01:31.364 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:31 smithi119 ceph-mon[101432]: osdmap e252: 8 total, 8 up, 8 in 2024-09-21T10:01:31.364 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:31 smithi119 ceph-mon[40785]: osdmap e252: 8 total, 8 up, 8 in 2024-09-21T10:01:32.390 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 8388609~8388609 2024-09-21T10:01:32.404 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 0~1000 2024-09-21T10:01:32.404 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1010~1000 2024-09-21T10:01:32.404 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:01:32.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: [21/Sep/2024:10:01:32] ENGINE Bus STARTING 2024-09-21T10:01:32.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: CherryPy Checker: 2024-09-21T10:01:32.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: The Application mounted at '' has an empty config. 2024-09-21T10:01:32.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: 2024-09-21T10:01:32.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: [21/Sep/2024:10:01:32] ENGINE Serving on http://:::9283 2024-09-21T10:01:32.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:01:32 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x[80029]: [21/Sep/2024:10:01:32] ENGINE Bus STARTED 2024-09-21T10:01:33.030 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.737+0000 7fc08e359640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.737+0000 7fc08e359640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.737+0000 7fc08e359640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.737+0000 7fc08e359640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.737+0000 7fc08e359640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.739+0000 7fc08bb54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.739+0000 7fc08bb54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.739+0000 7fc08bb54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.739+0000 7fc08bb54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.031 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:32.739+0000 7fc08bb54640 -1 client.0 error registering admin socket command: (17) File exists 2024-09-21T10:01:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: osdmap e253: 8 total, 8 up, 8 in 2024-09-21T10:01:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: Standby manager daemon x restarted 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: Standby manager daemon x started 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: Active manager daemon y restarted 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: Activating manager daemon y 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: osdmap e254: 8 total, 8 up, 8 in 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: mgrmap e34: y(active, starting, since 0.0123536s), standbys: x 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:01:33.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi119.oikglq"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T10:01:33.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: Manager daemon y is now available 2024-09-21T10:01:33.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:33.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:33.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:01:33.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T10:01:33.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T10:01:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: osdmap e253: 8 total, 8 up, 8 in 2024-09-21T10:01:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: Standby manager daemon x restarted 2024-09-21T10:01:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: Standby manager daemon x started 2024-09-21T10:01:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:01:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: Active manager daemon y restarted 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: Activating manager daemon y 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: osdmap e254: 8 total, 8 up, 8 in 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: mgrmap e34: y(active, starting, since 0.0123536s), standbys: x 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi119.oikglq"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T10:01:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: Manager daemon y is now available 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:33.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: osdmap e253: 8 total, 8 up, 8 in 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: Standby manager daemon x restarted 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: Standby manager daemon x started 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T10:01:33.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.? 172.21.15.145:0/187897577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: Active manager daemon y restarted 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: Activating manager daemon y 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: osdmap e254: 8 total, 8 up, 8 in 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: mgrmap e34: y(active, starting, since 0.0123536s), standbys: x 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi145.tjomfr"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata", "who": "foo.smithi119.oikglq"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:01:33.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: Manager daemon y is now available 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-21T10:01:33.372 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:alertmanager.a smithi119 *:9093,9094 running (88s) 38s ago 5m 17.5M - 0.27.0 11f11916f8cd e4cf1f6d8355 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:grafana.a smithi145 *:3000 running (63s) 18s ago 5m 76.2M - 9.4.7 954c08fa6188 e033bb16a7cd 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi119.oikglq smithi119 running (4m) 38s ago 4m 16.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4ff2238155bf 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi145.tjomfr smithi145 running (4m) 18s ago 4m 15.4M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b1338cf49871 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi145 *:8443,9283,8765 running (24s) 18s ago 9m 67.0M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 1346375dd747 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi119 *:8443,9283,8765 running (52s) 38s ago 11m 494M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d7bcb26ddb62 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (11m) 38s ago 11m 57.6M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1467ec54a60b 2024-09-21T10:01:33.405 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi145 running (10m) 18s ago 9m 50.6M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f45b53ab2476 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi119 running (9m) 38s ago 9m 46.1M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be d64b219d7298 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.a smithi119 *:9100 running (82s) 38s ago 5m 9.83M - 1.7.0 72c9c2088986 05912f78c070 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.b smithi145 *:9100 running (58s) 18s ago 5m 8284k - 1.7.0 72c9c2088986 d906bafb8d86 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (9m) 38s ago 9m 107M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ce07d9950c3f 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (8m) 38s ago 8m 115M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b8f7b7d5ac64 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (8m) 38s ago 8m 93.8M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 87104f5801c3 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (8m) 38s ago 8m 99.4M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 700faab8401d 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi145 running (7m) 18s ago 7m 112M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 18086eb220ec 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi145 running (7m) 18s ago 7m 80.9M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 90ab6d4f8571 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi145 running (6m) 18s ago 6m 120M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 772774e9be4a 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi145 running (6m) 18s ago 6m 101M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1a18c8e91654 2024-09-21T10:01:33.406 INFO:teuthology.orchestra.run.smithi119.stdout:prometheus.a smithi145 *:9095 running (29s) 18s ago 5m 41.5M - 2.51.0 1d3b7f56885b d2de6cb75d75 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:33.392+0000 7fc0be6c7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:33] ENGINE Bus STARTING 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: CherryPy Checker: 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: The Application mounted at '' has an empty config. 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:33] ENGINE Serving on http://:::9283 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:01:33] ENGINE Bus STARTED 2024-09-21T10:01:33.727 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:33.723+0000 7fc06c95c640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:10:01:33] ENGINE Error in HTTPServer.serve 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: Traceback (most recent call last): 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self._connections.run(self.expiration_interval) 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self._run(expiration_interval) 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: s = self.context.wrap_socket( 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: return self.sslsocket_class._create( 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:01:33.728 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self.do_handshake() 2024-09-21T10:01:33.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:01:33.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: self._sslobj.do_handshake() 2024-09-21T10:01:33.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:01:33.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:01:34.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:34.400 INFO:tasks.workunit.client.0.smithi119.stdout:write: X 8388609~1000 2024-09-21T10:01:34.409 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16777228~1000 2024-09-21T10:01:34.409 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 167772160 2024-09-21T10:01:34.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T10:01:34.418 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 83886079~8388609 2024-09-21T10:01:34.432 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:01:34] "GET /metrics HTTP/1.1" 200 36073 "" "Prometheus/2.51.0" 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: osdmap e255: 8 total, 8 up, 8 in 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: mgrmap e35: y(active, since 1.02031s), standbys: x 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: [21/Sep/2024:10:01:33] ENGINE Bus STARTING 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: [21/Sep/2024:10:01:33] ENGINE Serving on http://172.21.15.119:8765 2024-09-21T10:01:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: [21/Sep/2024:10:01:33] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: [21/Sep/2024:10:01:33] ENGINE Bus STARTED 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[101432]: [21/Sep/2024:10:01:33] ENGINE Error in HTTPServer.serve 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T10:01:34.618 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: osdmap e255: 8 total, 8 up, 8 in 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: mgrmap e35: y(active, since 1.02031s), standbys: x 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: [21/Sep/2024:10:01:33] ENGINE Bus STARTING 2024-09-21T10:01:34.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: [21/Sep/2024:10:01:33] ENGINE Serving on http://172.21.15.119:8765 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: [21/Sep/2024:10:01:33] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: [21/Sep/2024:10:01:33] ENGINE Bus STARTED 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:34 smithi119 ceph-mon[40785]: [21/Sep/2024:10:01:33] ENGINE Error in HTTPServer.serve 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: Traceback (most recent call last): 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._run(expiration_interval) 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:01:34.620 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: s = self.context.wrap_socket( 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: return self.sslsocket_class._create( 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: self.do_handshake() 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._sslobj.do_handshake() 2024-09-21T10:01:34.621 INFO:journalctl@ceph.mon.c.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:01:34.709 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: osdmap e255: 8 total, 8 up, 8 in 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: mgrmap e35: y(active, since 1.02031s), standbys: x 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: [21/Sep/2024:10:01:33] ENGINE Bus STARTING 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: [21/Sep/2024:10:01:33] ENGINE Serving on http://172.21.15.119:8765 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: [21/Sep/2024:10:01:33] ENGINE Serving on https://172.21.15.119:7150 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: [21/Sep/2024:10:01:33] ENGINE Bus STARTED 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:34 smithi145 ceph-mon[33206]: [21/Sep/2024:10:01:33] ENGINE Error in HTTPServer.serve 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout: Traceback (most recent call last): 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T10:01:34.710 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._connections.run(self.expiration_interval) 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._run(expiration_interval) 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: s = self.context.wrap_socket( 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: return self.sslsocket_class._create( 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: self.do_handshake() 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T10:01:34.711 INFO:journalctl@ceph.mon.b.smithi145.stdout: self._sslobj.do_handshake() 2024-09-21T10:01:34.712 INFO:journalctl@ceph.mon.b.smithi145.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T10:01:35.204 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:35.610 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:35 smithi119 ceph-mon[40785]: pgmap v5: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:35.610 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:35 smithi119 ceph-mon[40785]: osdmap e256: 8 total, 8 up, 8 in 2024-09-21T10:01:35.611 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:35 smithi119 ceph-mon[40785]: mgrmap e36: y(active, since 2s), standbys: x 2024-09-21T10:01:35.611 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:35 smithi119 ceph-mon[101432]: pgmap v5: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:35.611 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:35 smithi119 ceph-mon[101432]: osdmap e256: 8 total, 8 up, 8 in 2024-09-21T10:01:35.611 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:35 smithi119 ceph-mon[101432]: mgrmap e36: y(active, since 2s), standbys: x 2024-09-21T10:01:35.664 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:35 smithi145 ceph-mon[33206]: pgmap v5: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:35.664 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:35 smithi145 ceph-mon[33206]: osdmap e256: 8 total, 8 up, 8 in 2024-09-21T10:01:35.664 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:35 smithi145 ceph-mon[33206]: mgrmap e36: y(active, since 2s), standbys: x 2024-09-21T10:01:36.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:35.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:36.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:35.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:36.422 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:36.442 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83886079~1 2024-09-21T10:01:36.448 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:36.767 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:36 smithi119 ceph-mon[40785]: osdmap e257: 8 total, 8 up, 8 in 2024-09-21T10:01:36.767 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:36 smithi119 ceph-mon[101432]: osdmap e257: 8 total, 8 up, 8 in 2024-09-21T10:01:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:36 smithi145 ceph-mon[33206]: osdmap e257: 8 total, 8 up, 8 in 2024-09-21T10:01:37.290 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:36 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:37 smithi145 ceph-mon[33206]: pgmap v8: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:37 smithi145 ceph-mon[33206]: osdmap e258: 8 total, 8 up, 8 in 2024-09-21T10:01:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:37 smithi145 ceph-mon[33206]: mgrmap e37: y(active, since 4s), standbys: x 2024-09-21T10:01:37.864 INFO:teuthology.orchestra.run.smithi119.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:37.864 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:37 smithi119 ceph-mon[101432]: pgmap v8: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:37.864 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:37 smithi119 ceph-mon[101432]: osdmap e258: 8 total, 8 up, 8 in 2024-09-21T10:01:37.865 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:37 smithi119 ceph-mon[101432]: mgrmap e37: y(active, since 4s), standbys: x 2024-09-21T10:01:37.865 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:37 smithi119 ceph-mon[40785]: pgmap v8: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:37.865 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:37 smithi119 ceph-mon[40785]: osdmap e258: 8 total, 8 up, 8 in 2024-09-21T10:01:37.865 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:37 smithi119 ceph-mon[40785]: mgrmap e37: y(active, since 4s), standbys: x 2024-09-21T10:01:38.699 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: osdmap e259: 8 total, 8 up, 8 in 2024-09-21T10:01:38.699 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.699 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.699 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:38.700 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:38.700 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.700 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.700 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:38 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: osdmap e259: 8 total, 8 up, 8 in 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.838 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: osdmap e259: 8 total, 8 up, 8 in 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: from='client.34155 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:38.839 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:39.364 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:01:39.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T10:01:39.492 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T10:01:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:39 smithi119 ceph-mon[40785]: pgmap v11: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:39 smithi119 ceph-mon[40785]: osdmap e260: 8 total, 8 up, 8 in 2024-09-21T10:01:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:39 smithi119 ceph-mon[40785]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T10:01:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:39 smithi119 ceph-mon[101432]: pgmap v11: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:39 smithi119 ceph-mon[101432]: osdmap e260: 8 total, 8 up, 8 in 2024-09-21T10:01:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:39 smithi119 ceph-mon[101432]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T10:01:39.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:39 smithi145 ceph-mon[33206]: pgmap v11: 137 pgs: 137 active+clean; 845 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:01:39.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:39 smithi145 ceph-mon[33206]: osdmap e260: 8 total, 8 up, 8 in 2024-09-21T10:01:39.725 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:39 smithi145 ceph-mon[33206]: osdmap e261: 8 total, 8 up, 8 in 2024-09-21T10:01:40.108 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:01:40.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:40.438 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 0~8388609 2024-09-21T10:01:40.458 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 16777218~8388609 2024-09-21T10:01:40.558 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[101432]: pgmap v14: 137 pgs: 137 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 162 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[101432]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[40785]: pgmap v14: 137 pgs: 137 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 162 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[40785]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T10:01:41.709 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.709 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.709 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:41 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:41 smithi145 ceph-mon[33206]: pgmap v14: 137 pgs: 137 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 162 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-09-21T10:01:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:41 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:41 smithi145 ceph-mon[33206]: osdmap e262: 8 total, 8 up, 8 in 2024-09-21T10:01:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:41 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:41 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:41.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:41 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[40785]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[101432]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:42.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:42 smithi145 ceph-mon[33206]: osdmap e263: 8 total, 8 up, 8 in 2024-09-21T10:01:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:42 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:42 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:42 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:42 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T10:01:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T10:01:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: pgmap v17: 137 pgs: 137 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 162 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:43.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:43.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T10:01:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T10:01:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: pgmap v17: 137 pgs: 137 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 162 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.conf 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: pgmap v17: 137 pgs: 137 active+clean; 869 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 162 KiB/s rd, 4.0 MiB/s wr, 185 op/s 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: osdmap e264: 8 total, 8 up, 8 in 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: Updating smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:01:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:01:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:01:43.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:01:44.455 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:44 smithi145 ceph-mon[33206]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T10:01:44.849 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.AbortWithSnapshots (13812 ms) 2024-09-21T10:01:44.849 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV1Parent 2024-09-21T10:01:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:44 smithi119 ceph-mon[40785]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T10:01:44.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:44 smithi119 ceph-mon[101432]: osdmap e265: 8 total, 8 up, 8 in 2024-09-21T10:01:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:01:44] "GET /metrics HTTP/1.1" 200 36073 "" "Prometheus/2.51.0" 2024-09-21T10:01:44.879 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:44.890 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:01:44.899 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:45.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:45 smithi145 ceph-mon[33206]: pgmap v20: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T10:01:45.743 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:45 smithi145 ceph-mon[33206]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T10:01:45.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:45 smithi119 ceph-mon[101432]: pgmap v20: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T10:01:45.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:45 smithi119 ceph-mon[101432]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T10:01:45.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:45 smithi119 ceph-mon[40785]: pgmap v20: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T10:01:45.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:45 smithi119 ceph-mon[40785]: osdmap e266: 8 total, 8 up, 8 in 2024-09-21T10:01:46.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:45.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:46.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:46.483 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:01:46.543 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:46 smithi145 ceph-mon[33206]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T10:01:46.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:46 smithi119 ceph-mon[40785]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T10:01:46.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:46 smithi119 ceph-mon[101432]: osdmap e267: 8 total, 8 up, 8 in 2024-09-21T10:01:47.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:46 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:47.672 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:47.747 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:47 smithi145 ceph-mon[33206]: pgmap v23: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T10:01:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:47 smithi145 ceph-mon[33206]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T10:01:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:47 smithi119 ceph-mon[101432]: pgmap v23: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T10:01:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:47 smithi119 ceph-mon[101432]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T10:01:47.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:47 smithi119 ceph-mon[40785]: pgmap v23: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 42 MiB/s rd, 23 MiB/s wr, 389 op/s 2024-09-21T10:01:47.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:47 smithi119 ceph-mon[40785]: osdmap e268: 8 total, 8 up, 8 in 2024-09-21T10:01:48.591 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV1Parent (3742 ms) 2024-09-21T10:01:48.591 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV2Parent 2024-09-21T10:01:48.622 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:48.632 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:01:48.640 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:48 smithi145 ceph-mon[33206]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T10:01:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:48 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:48 smithi119 ceph-mon[101432]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T10:01:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:48 smithi119 ceph-mon[40785]: osdmap e269: 8 total, 8 up, 8 in 2024-09-21T10:01:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:48 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:01:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:49 smithi145 ceph-mon[33206]: pgmap v26: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:01:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:49 smithi145 ceph-mon[33206]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T10:01:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:49 smithi145 ceph-mon[33206]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T10:01:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:49 smithi119 ceph-mon[40785]: pgmap v26: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:01:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:49 smithi119 ceph-mon[40785]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T10:01:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:49 smithi119 ceph-mon[40785]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T10:01:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:49 smithi119 ceph-mon[101432]: pgmap v26: 137 pgs: 137 active+clean; 945 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:01:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:49 smithi119 ceph-mon[101432]: osdmap e270: 8 total, 8 up, 8 in 2024-09-21T10:01:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:49 smithi119 ceph-mon[101432]: osdmap e271: 8 total, 8 up, 8 in 2024-09-21T10:01:49.968 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:01:50.026 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:51.146 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:01:51.217 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:01:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:50 smithi145 ceph-mon[33206]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T10:01:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:50 smithi119 ceph-mon[101432]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T10:01:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:50 smithi119 ceph-mon[40785]: osdmap e272: 8 total, 8 up, 8 in 2024-09-21T10:01:52.089 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV2Parent (3498 ms) 2024-09-21T10:01:52.089 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV1ParentAbort 2024-09-21T10:01:52.122 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:52.134 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:01:52.143 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:52.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:51 smithi145 ceph-mon[33206]: pgmap v30: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 37 MiB/s rd, 22 KiB/s wr, 471 op/s 2024-09-21T10:01:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:51 smithi145 ceph-mon[33206]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T10:01:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:51 smithi119 ceph-mon[101432]: pgmap v30: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 37 MiB/s rd, 22 KiB/s wr, 471 op/s 2024-09-21T10:01:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:51 smithi119 ceph-mon[101432]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T10:01:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:51 smithi119 ceph-mon[40785]: pgmap v30: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 37 MiB/s rd, 22 KiB/s wr, 471 op/s 2024-09-21T10:01:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:51 smithi119 ceph-mon[40785]: osdmap e273: 8 total, 8 up, 8 in 2024-09-21T10:01:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:52 smithi145 ceph-mon[33206]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T10:01:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:52 smithi119 ceph-mon[40785]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T10:01:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:52 smithi119 ceph-mon[101432]: osdmap e274: 8 total, 8 up, 8 in 2024-09-21T10:01:53.996 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:01:54.061 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:53 smithi145 ceph-mon[33206]: pgmap v33: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:01:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:53 smithi145 ceph-mon[33206]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T10:01:54.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:53 smithi119 ceph-mon[101432]: pgmap v33: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:01:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:53 smithi119 ceph-mon[101432]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T10:01:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:53 smithi119 ceph-mon[40785]: pgmap v33: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:01:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:53 smithi119 ceph-mon[40785]: osdmap e275: 8 total, 8 up, 8 in 2024-09-21T10:01:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:01:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:01:54] "GET /metrics HTTP/1.1" 200 38912 "" "Prometheus/2.51.0" 2024-09-21T10:01:55.174 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:01:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:54 smithi145 ceph-mon[33206]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T10:01:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:54 smithi119 ceph-mon[40785]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T10:01:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:54 smithi119 ceph-mon[101432]: osdmap e276: 8 total, 8 up, 8 in 2024-09-21T10:01:56.000 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:56.000 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:56.130 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV1ParentAbort (4041 ms) 2024-09-21T10:01:56.130 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV2ParentAbort 2024-09-21T10:01:56.162 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:01:56.172 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:01:56.180 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:01:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:55 smithi145 ceph-mon[33206]: pgmap v36: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 382 KiB/s rd, 28 KiB/s wr, 492 op/s 2024-09-21T10:01:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:55 smithi145 ceph-mon[33206]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T10:01:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:55 smithi119 ceph-mon[40785]: pgmap v36: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 382 KiB/s rd, 28 KiB/s wr, 492 op/s 2024-09-21T10:01:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:55 smithi119 ceph-mon[40785]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T10:01:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:55 smithi119 ceph-mon[101432]: pgmap v36: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 382 KiB/s rd, 28 KiB/s wr, 492 op/s 2024-09-21T10:01:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:55 smithi119 ceph-mon[101432]: osdmap e277: 8 total, 8 up, 8 in 2024-09-21T10:01:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:57 smithi145 ceph-mon[33206]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T10:01:57.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:01:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:01:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:01:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:57 smithi119 ceph-mon[40785]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T10:01:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:57 smithi119 ceph-mon[101432]: osdmap e278: 8 total, 8 up, 8 in 2024-09-21T10:01:58.032 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:01:58.093 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:01:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:58 smithi145 ceph-mon[33206]: pgmap v39: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 382 KiB/s rd, 28 KiB/s wr, 492 op/s 2024-09-21T10:01:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:58 smithi145 ceph-mon[33206]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T10:01:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:58 smithi119 ceph-mon[101432]: pgmap v39: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 382 KiB/s rd, 28 KiB/s wr, 492 op/s 2024-09-21T10:01:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:58 smithi119 ceph-mon[101432]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T10:01:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:58 smithi119 ceph-mon[40785]: pgmap v39: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 382 KiB/s rd, 28 KiB/s wr, 492 op/s 2024-09-21T10:01:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:58 smithi119 ceph-mon[40785]: osdmap e279: 8 total, 8 up, 8 in 2024-09-21T10:01:59.110 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:01:59.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:59 smithi119 ceph-mon[101432]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T10:01:59.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:01:59 smithi119 ceph-mon[101432]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T10:01:59.268 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:59 smithi119 ceph-mon[40785]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T10:01:59.269 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:01:59 smithi119 ceph-mon[40785]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T10:01:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:59 smithi145 ceph-mon[33206]: osdmap e280: 8 total, 8 up, 8 in 2024-09-21T10:01:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:01:59 smithi145 ceph-mon[33206]: osdmap e281: 8 total, 8 up, 8 in 2024-09-21T10:02:00.131 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV2ParentAbort (4000 ms) 2024-09-21T10:02:00.131 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach 2024-09-21T10:02:00.161 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:00.172 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:02:00.180 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:00 smithi145 ceph-mon[33206]: pgmap v42: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:02:00.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:00 smithi119 ceph-mon[101432]: pgmap v42: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:02:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:00 smithi119 ceph-mon[40785]: pgmap v42: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:02:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:01 smithi145 ceph-mon[33206]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T10:02:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:01 smithi119 ceph-mon[101432]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T10:02:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:01 smithi119 ceph-mon[40785]: osdmap e282: 8 total, 8 up, 8 in 2024-09-21T10:02:02.052 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:02.115 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:02 smithi145 ceph-mon[33206]: pgmap v45: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 336 KiB/s rd, 23 KiB/s wr, 434 op/s 2024-09-21T10:02:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:02 smithi145 ceph-mon[33206]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T10:02:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:02 smithi119 ceph-mon[101432]: pgmap v45: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 336 KiB/s rd, 23 KiB/s wr, 434 op/s 2024-09-21T10:02:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:02 smithi119 ceph-mon[101432]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T10:02:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:02 smithi119 ceph-mon[40785]: pgmap v45: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 336 KiB/s rd, 23 KiB/s wr, 434 op/s 2024-09-21T10:02:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:02 smithi119 ceph-mon[40785]: osdmap e283: 8 total, 8 up, 8 in 2024-09-21T10:02:03.226 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:02:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:03 smithi119 ceph-mon[101432]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T10:02:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:03 smithi119 ceph-mon[101432]: pgmap v48: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 336 KiB/s rd, 23 KiB/s wr, 434 op/s 2024-09-21T10:02:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:03 smithi119 ceph-mon[40785]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T10:02:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:03 smithi119 ceph-mon[40785]: pgmap v48: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 336 KiB/s rd, 23 KiB/s wr, 434 op/s 2024-09-21T10:02:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:03 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:03 smithi145 ceph-mon[33206]: osdmap e284: 8 total, 8 up, 8 in 2024-09-21T10:02:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:03 smithi145 ceph-mon[33206]: pgmap v48: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 336 KiB/s rd, 23 KiB/s wr, 434 op/s 2024-09-21T10:02:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:03 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:04.166 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixIncompleteChildReattach (4036 ms) 2024-09-21T10:02:04.167 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV1ParentAbortFixParentReattach 2024-09-21T10:02:04.195 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:02:04.213 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:04 smithi119 ceph-mon[101432]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T10:02:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:04 smithi119 ceph-mon[40785]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T10:02:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:04 smithi145 ceph-mon[33206]: osdmap e285: 8 total, 8 up, 8 in 2024-09-21T10:02:04.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:04.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:02:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:02:04] "GET /metrics HTTP/1.1" 200 38911 "" "Prometheus/2.51.0" 2024-09-21T10:02:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:05 smithi119 ceph-mon[101432]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T10:02:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:05 smithi119 ceph-mon[101432]: pgmap v51: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 390 KiB/s rd, 24 KiB/s wr, 492 op/s 2024-09-21T10:02:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:05 smithi119 ceph-mon[40785]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T10:02:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:05 smithi119 ceph-mon[40785]: pgmap v51: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 390 KiB/s rd, 24 KiB/s wr, 492 op/s 2024-09-21T10:02:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:05 smithi145 ceph-mon[33206]: osdmap e286: 8 total, 8 up, 8 in 2024-09-21T10:02:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:05 smithi145 ceph-mon[33206]: pgmap v51: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 390 KiB/s rd, 24 KiB/s wr, 492 op/s 2024-09-21T10:02:06.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:06.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:06.086 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:06.142 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:06 smithi119 ceph-mon[101432]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T10:02:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:06 smithi119 ceph-mon[40785]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T10:02:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:06 smithi145 ceph-mon[33206]: osdmap e287: 8 total, 8 up, 8 in 2024-09-21T10:02:07.256 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:02:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:07 smithi145 ceph-mon[33206]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T10:02:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:07 smithi145 ceph-mon[33206]: pgmap v54: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 390 KiB/s rd, 24 KiB/s wr, 492 op/s 2024-09-21T10:02:07.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:07 smithi119 ceph-mon[101432]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T10:02:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:07 smithi119 ceph-mon[101432]: pgmap v54: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 390 KiB/s rd, 24 KiB/s wr, 492 op/s 2024-09-21T10:02:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:07 smithi119 ceph-mon[40785]: osdmap e288: 8 total, 8 up, 8 in 2024-09-21T10:02:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:07 smithi119 ceph-mon[40785]: pgmap v54: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 390 KiB/s rd, 24 KiB/s wr, 492 op/s 2024-09-21T10:02:08.209 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV1ParentAbortFixParentReattach (4043 ms) 2024-09-21T10:02:08.209 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV1ParentAbortRelinkNotNeeded 2024-09-21T10:02:08.240 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:08.250 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:02:08.259 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:08 smithi119 ceph-mon[101432]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T10:02:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:08 smithi119 ceph-mon[40785]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T10:02:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:08 smithi145 ceph-mon[33206]: osdmap e289: 8 total, 8 up, 8 in 2024-09-21T10:02:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:09 smithi119 ceph-mon[40785]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T10:02:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:09 smithi119 ceph-mon[40785]: pgmap v57: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:02:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:09 smithi119 ceph-mon[40785]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T10:02:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:09 smithi119 ceph-mon[101432]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T10:02:09.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:09 smithi119 ceph-mon[101432]: pgmap v57: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:02:09.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:09 smithi119 ceph-mon[101432]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T10:02:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:09 smithi145 ceph-mon[33206]: osdmap e290: 8 total, 8 up, 8 in 2024-09-21T10:02:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:09 smithi145 ceph-mon[33206]: pgmap v57: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:02:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:09 smithi145 ceph-mon[33206]: osdmap e291: 8 total, 8 up, 8 in 2024-09-21T10:02:09.989 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:10.052 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:11.161 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:02:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:10 smithi145 ceph-mon[33206]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T10:02:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:10 smithi119 ceph-mon[40785]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T10:02:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:10 smithi119 ceph-mon[101432]: osdmap e292: 8 total, 8 up, 8 in 2024-09-21T10:02:12.106 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV1ParentAbortRelinkNotNeeded (3897 ms) 2024-09-21T10:02:12.107 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach 2024-09-21T10:02:12.135 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:12.146 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:02:12.155 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:11 smithi145 ceph-mon[33206]: pgmap v60: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 313 KiB/s rd, 22 KiB/s wr, 410 op/s 2024-09-21T10:02:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:11 smithi145 ceph-mon[33206]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T10:02:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:11 smithi119 ceph-mon[40785]: pgmap v60: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 313 KiB/s rd, 22 KiB/s wr, 410 op/s 2024-09-21T10:02:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:11 smithi119 ceph-mon[40785]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T10:02:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:11 smithi119 ceph-mon[101432]: pgmap v60: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 313 KiB/s rd, 22 KiB/s wr, 410 op/s 2024-09-21T10:02:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:11 smithi119 ceph-mon[101432]: osdmap e293: 8 total, 8 up, 8 in 2024-09-21T10:02:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:12 smithi145 ceph-mon[33206]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T10:02:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:12 smithi119 ceph-mon[40785]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T10:02:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:12 smithi119 ceph-mon[101432]: osdmap e294: 8 total, 8 up, 8 in 2024-09-21T10:02:14.011 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:14.073 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:13 smithi145 ceph-mon[33206]: pgmap v63: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 313 KiB/s rd, 22 KiB/s wr, 410 op/s 2024-09-21T10:02:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:13 smithi145 ceph-mon[33206]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T10:02:14.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:13 smithi119 ceph-mon[40785]: pgmap v63: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 313 KiB/s rd, 22 KiB/s wr, 410 op/s 2024-09-21T10:02:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:13 smithi119 ceph-mon[40785]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T10:02:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:13 smithi119 ceph-mon[101432]: pgmap v63: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 313 KiB/s rd, 22 KiB/s wr, 410 op/s 2024-09-21T10:02:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:13 smithi119 ceph-mon[101432]: osdmap e295: 8 total, 8 up, 8 in 2024-09-21T10:02:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:02:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:02:14] "GET /metrics HTTP/1.1" 200 38911 "" "Prometheus/2.51.0" 2024-09-21T10:02:15.177 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:02:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:15 smithi145 ceph-mon[33206]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T10:02:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:15 smithi119 ceph-mon[40785]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T10:02:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:15 smithi119 ceph-mon[101432]: osdmap e296: 8 total, 8 up, 8 in 2024-09-21T10:02:16.015 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:16.015 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:16.140 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixIncompleteChildReattach (4034 ms) 2024-09-21T10:02:16.140 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV2ParentAbortFixParentReattach 2024-09-21T10:02:16.172 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:16.182 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:02:16.191 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:16 smithi145 ceph-mon[33206]: pgmap v66: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 269 KiB/s rd, 17 KiB/s wr, 345 op/s 2024-09-21T10:02:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:16 smithi145 ceph-mon[33206]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T10:02:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:16 smithi119 ceph-mon[101432]: pgmap v66: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 269 KiB/s rd, 17 KiB/s wr, 345 op/s 2024-09-21T10:02:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:16 smithi119 ceph-mon[101432]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T10:02:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:16 smithi119 ceph-mon[40785]: pgmap v66: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 269 KiB/s rd, 17 KiB/s wr, 345 op/s 2024-09-21T10:02:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:16 smithi119 ceph-mon[40785]: osdmap e297: 8 total, 8 up, 8 in 2024-09-21T10:02:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:17 smithi145 ceph-mon[33206]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T10:02:17.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:17 smithi119 ceph-mon[101432]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T10:02:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:17 smithi119 ceph-mon[40785]: osdmap e298: 8 total, 8 up, 8 in 2024-09-21T10:02:18.044 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:18.100 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:18 smithi145 ceph-mon[33206]: pgmap v69: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 269 KiB/s rd, 17 KiB/s wr, 345 op/s 2024-09-21T10:02:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:18 smithi145 ceph-mon[33206]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T10:02:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:18 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:18 smithi119 ceph-mon[40785]: pgmap v69: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 269 KiB/s rd, 17 KiB/s wr, 345 op/s 2024-09-21T10:02:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:18 smithi119 ceph-mon[40785]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T10:02:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:18 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:18 smithi119 ceph-mon[101432]: pgmap v69: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 269 KiB/s rd, 17 KiB/s wr, 345 op/s 2024-09-21T10:02:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:18 smithi119 ceph-mon[101432]: osdmap e299: 8 total, 8 up, 8 in 2024-09-21T10:02:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:19.135 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:02:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:19 smithi145 ceph-mon[33206]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T10:02:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:19 smithi145 ceph-mon[33206]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T10:02:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:19 smithi119 ceph-mon[40785]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T10:02:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:19 smithi119 ceph-mon[40785]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T10:02:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:19 smithi119 ceph-mon[101432]: osdmap e300: 8 total, 8 up, 8 in 2024-09-21T10:02:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:19 smithi119 ceph-mon[101432]: osdmap e301: 8 total, 8 up, 8 in 2024-09-21T10:02:20.172 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV2ParentAbortFixParentReattach (4032 ms) 2024-09-21T10:02:20.172 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.CloneV2ParentAbortRelinkNotNeeded 2024-09-21T10:02:20.202 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:20.213 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 0~10 2024-09-21T10:02:20.221 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:20 smithi145 ceph-mon[33206]: pgmap v72: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 184 KiB/s rd, 14 KiB/s wr, 225 op/s 2024-09-21T10:02:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:20 smithi119 ceph-mon[40785]: pgmap v72: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 184 KiB/s rd, 14 KiB/s wr, 225 op/s 2024-09-21T10:02:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:20 smithi119 ceph-mon[101432]: pgmap v72: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 184 KiB/s rd, 14 KiB/s wr, 225 op/s 2024-09-21T10:02:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:21 smithi119 ceph-mon[40785]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T10:02:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:21 smithi119 ceph-mon[40785]: pgmap v75: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 324 KiB/s rd, 26 KiB/s wr, 421 op/s 2024-09-21T10:02:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:21 smithi119 ceph-mon[101432]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T10:02:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:21 smithi119 ceph-mon[101432]: pgmap v75: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 324 KiB/s rd, 26 KiB/s wr, 421 op/s 2024-09-21T10:02:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:21 smithi145 ceph-mon[33206]: osdmap e302: 8 total, 8 up, 8 in 2024-09-21T10:02:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:21 smithi145 ceph-mon[33206]: pgmap v75: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 324 KiB/s rd, 26 KiB/s wr, 421 op/s 2024-09-21T10:02:22.078 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:22.133 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:22 smithi119 ceph-mon[40785]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:02:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:22 smithi119 ceph-mon[101432]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:02:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:22 smithi145 ceph-mon[33206]: osdmap e303: 8 total, 8 up, 8 in 2024-09-21T10:02:23.251 INFO:tasks.workunit.client.0.smithi119.stdout:migration_abort 2024-09-21T10:02:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:23 smithi119 ceph-mon[40785]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:02:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:23 smithi119 ceph-mon[40785]: pgmap v78: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 140 KiB/s rd, 12 KiB/s wr, 196 op/s 2024-09-21T10:02:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:23 smithi119 ceph-mon[101432]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:02:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:23 smithi119 ceph-mon[101432]: pgmap v78: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 140 KiB/s rd, 12 KiB/s wr, 196 op/s 2024-09-21T10:02:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:23 smithi145 ceph-mon[33206]: osdmap e304: 8 total, 8 up, 8 in 2024-09-21T10:02:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:23 smithi145 ceph-mon[33206]: pgmap v78: 137 pgs: 137 active+clean; 901 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 140 KiB/s rd, 12 KiB/s wr, 196 op/s 2024-09-21T10:02:24.215 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.CloneV2ParentAbortRelinkNotNeeded (4043 ms) 2024-09-21T10:02:24.215 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.StressNoMigrate 2024-09-21T10:02:24.249 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:24.259 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 8826979~7437178 2024-09-21T10:02:24.326 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 3136503~5536770 2024-09-21T10:02:24.328 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 40847804~2626458 2024-09-21T10:02:24.344 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 58277179~2118762 2024-09-21T10:02:24.344 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 47124081~5313432 2024-09-21T10:02:24.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:24 smithi119 ceph-mon[101432]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:02:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:24 smithi119 ceph-mon[40785]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:02:24.378 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 29289153~1400777 2024-09-21T10:02:24.378 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 10986989~2566320 2024-09-21T10:02:24.396 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 2730964~522456 2024-09-21T10:02:24.396 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap0 2024-09-21T10:02:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:24 smithi145 ceph-mon[33206]: osdmap e305: 8 total, 8 up, 8 in 2024-09-21T10:02:24.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:24.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:02:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:02:24] "GET /metrics HTTP/1.1" 200 38909 "" "Prometheus/2.51.0" 2024-09-21T10:02:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:25 smithi119 ceph-mon[40785]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:02:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:25 smithi119 ceph-mon[40785]: pgmap v81: 137 pgs: 137 active+clean; 901 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 362 KiB/s rd, 27 KiB/s wr, 463 op/s 2024-09-21T10:02:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:25 smithi119 ceph-mon[101432]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:02:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:25 smithi119 ceph-mon[101432]: pgmap v81: 137 pgs: 137 active+clean; 901 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 362 KiB/s rd, 27 KiB/s wr, 463 op/s 2024-09-21T10:02:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:25 smithi145 ceph-mon[33206]: osdmap e306: 8 total, 8 up, 8 in 2024-09-21T10:02:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:25 smithi145 ceph-mon[33206]: pgmap v81: 137 pgs: 137 active+clean; 901 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 362 KiB/s rd, 27 KiB/s wr, 463 op/s 2024-09-21T10:02:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:26.114 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap0 2024-09-21T10:02:26.130 INFO:tasks.workunit.client.0.smithi119.stdout:clone image159 -> image162 2024-09-21T10:02:26.229 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 88125323 2024-09-21T10:02:26.240 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 43416681~1978410 2024-09-21T10:02:26.276 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 68234774~4767024 2024-09-21T10:02:26.281 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 31765735~7415998 2024-09-21T10:02:26.343 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 53310884~2898757 2024-09-21T10:02:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:26 smithi119 ceph-mon[101432]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:02:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:26 smithi119 ceph-mon[40785]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:02:26.372 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 11515951~3363690 2024-09-21T10:02:26.435 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 59714230~7342384 2024-09-21T10:02:26.440 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 73900962~7134180 2024-09-21T10:02:26.476 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 30747874~6530745 2024-09-21T10:02:26.479 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:26 smithi145 ceph-mon[33206]: osdmap e307: 8 total, 8 up, 8 in 2024-09-21T10:02:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:27 smithi145 ceph-mon[33206]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:02:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:27 smithi145 ceph-mon[33206]: pgmap v84: 137 pgs: 137 active+clean; 901 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 362 KiB/s rd, 27 KiB/s wr, 463 op/s 2024-09-21T10:02:27.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:26 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:27 smithi119 ceph-mon[101432]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:02:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:27 smithi119 ceph-mon[101432]: pgmap v84: 137 pgs: 137 active+clean; 901 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 362 KiB/s rd, 27 KiB/s wr, 463 op/s 2024-09-21T10:02:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:27 smithi119 ceph-mon[40785]: osdmap e308: 8 total, 8 up, 8 in 2024-09-21T10:02:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:27 smithi119 ceph-mon[40785]: pgmap v84: 137 pgs: 137 active+clean; 901 MiB data, 4.7 GiB used, 710 GiB / 715 GiB avail; 362 KiB/s rd, 27 KiB/s wr, 463 op/s 2024-09-21T10:02:28.121 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:28.135 INFO:tasks.workunit.client.0.smithi119.stdout:clone image162 -> image164 2024-09-21T10:02:28.264 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 166724853 2024-09-21T10:02:28.275 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 56871058~1575529 2024-09-21T10:02:28.296 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 20800829~793271 2024-09-21T10:02:28.304 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 31667337~6483663 2024-09-21T10:02:28.353 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 124446631~5574687 2024-09-21T10:02:28.354 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 53970818~6090556 2024-09-21T10:02:28.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:28 smithi119 ceph-mon[101432]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:02:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:28 smithi119 ceph-mon[40785]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:02:28.402 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 155030749~5042482 2024-09-21T10:02:28.402 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 115747783~1438505 2024-09-21T10:02:28.415 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 90819479~6708215 2024-09-21T10:02:28.420 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:02:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:28 smithi145 ceph-mon[33206]: osdmap e309: 8 total, 8 up, 8 in 2024-09-21T10:02:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:29 smithi119 ceph-mon[101432]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:02:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:29 smithi119 ceph-mon[101432]: pgmap v87: 137 pgs: 137 active+clean; 934 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 100 op/s 2024-09-21T10:02:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:29 smithi119 ceph-mon[101432]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:02:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:29 smithi119 ceph-mon[40785]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:02:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:29 smithi119 ceph-mon[40785]: pgmap v87: 137 pgs: 137 active+clean; 934 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 100 op/s 2024-09-21T10:02:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:29 smithi119 ceph-mon[40785]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:02:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:29 smithi145 ceph-mon[33206]: osdmap e310: 8 total, 8 up, 8 in 2024-09-21T10:02:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:29 smithi145 ceph-mon[33206]: pgmap v87: 137 pgs: 137 active+clean; 934 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 MiB/s rd, 5.2 MiB/s wr, 100 op/s 2024-09-21T10:02:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:29 smithi145 ceph-mon[33206]: osdmap e311: 8 total, 8 up, 8 in 2024-09-21T10:02:29.980 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap2 2024-09-21T10:02:29.991 INFO:tasks.workunit.client.0.smithi119.stdout:clone image164 -> image166 2024-09-21T10:02:30.147 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 94175161~5642928 2024-09-21T10:02:30.192 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 112287069~254113 2024-09-21T10:02:30.196 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 52616890~8195550 2024-09-21T10:02:30.258 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 44273899~4085503 2024-09-21T10:02:30.299 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 42663932~7778832 2024-09-21T10:02:30.349 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 24734719~1217696 2024-09-21T10:02:30.355 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 102399062~1579809 2024-09-21T10:02:30.369 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 144979416~7186068 2024-09-21T10:02:30.375 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:02:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:30 smithi145 ceph-mon[33206]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:02:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:30 smithi119 ceph-mon[101432]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:02:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:30 smithi119 ceph-mon[40785]: osdmap e312: 8 total, 8 up, 8 in 2024-09-21T10:02:31.989 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 93807980 2024-09-21T10:02:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:31 smithi145 ceph-mon[33206]: pgmap v90: 137 pgs: 137 active+clean; 1.0 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 8.3 MiB/s rd, 30 MiB/s wr, 503 op/s 2024-09-21T10:02:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:31 smithi145 ceph-mon[33206]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:02:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:31 smithi119 ceph-mon[101432]: pgmap v90: 137 pgs: 137 active+clean; 1.0 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 8.3 MiB/s rd, 30 MiB/s wr, 503 op/s 2024-09-21T10:02:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:31 smithi119 ceph-mon[101432]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:02:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:31 smithi119 ceph-mon[40785]: pgmap v90: 137 pgs: 137 active+clean; 1.0 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 8.3 MiB/s rd, 30 MiB/s wr, 503 op/s 2024-09-21T10:02:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:31 smithi119 ceph-mon[40785]: osdmap e313: 8 total, 8 up, 8 in 2024-09-21T10:02:32.371 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.StressNoMigrate (8156 ms) 2024-09-21T10:02:32.371 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.Stress 2024-09-21T10:02:32.403 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:32.413 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 20675850~2955611 2024-09-21T10:02:32.439 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 2415706~3434620 2024-09-21T10:02:32.439 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 55707460~552508 2024-09-21T10:02:32.448 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 24025604~3854211 2024-09-21T10:02:32.450 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 6598382~3695419 2024-09-21T10:02:32.480 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 67694706~4583049 2024-09-21T10:02:32.480 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 72545079~3852703 2024-09-21T10:02:32.504 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 804026~839955 2024-09-21T10:02:32.504 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap0 2024-09-21T10:02:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:32 smithi145 ceph-mon[33206]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:02:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:32 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:32 smithi119 ceph-mon[101432]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:02:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:32 smithi119 ceph-mon[40785]: osdmap e314: 8 total, 8 up, 8 in 2024-09-21T10:02:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:32 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:34.012 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap0 2024-09-21T10:02:34.026 INFO:tasks.workunit.client.0.smithi119.stdout:clone image167 -> image170 2024-09-21T10:02:34.137 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 4442248~4889529 2024-09-21T10:02:34.192 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 2468500~1478267 2024-09-21T10:02:34.197 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 73412865~5659945 2024-09-21T10:02:34.241 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 58784818~4048309 2024-09-21T10:02:34.246 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 48364430~5101461 2024-09-21T10:02:34.275 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 10382584~5834094 2024-09-21T10:02:34.279 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 50732746~3765454 2024-09-21T10:02:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:33 smithi145 ceph-mon[33206]: pgmap v93: 137 pgs: 137 active+clean; 1.0 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 7.0 MiB/s rd, 25 MiB/s wr, 402 op/s 2024-09-21T10:02:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:33 smithi145 ceph-mon[33206]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:02:34.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:34 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:34.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:34.305 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 40754507~5892610 2024-09-21T10:02:34.311 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:33 smithi119 ceph-mon[101432]: pgmap v93: 137 pgs: 137 active+clean; 1.0 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 7.0 MiB/s rd, 25 MiB/s wr, 402 op/s 2024-09-21T10:02:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:33 smithi119 ceph-mon[101432]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:02:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:33 smithi119 ceph-mon[40785]: pgmap v93: 137 pgs: 137 active+clean; 1.0 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 7.0 MiB/s rd, 25 MiB/s wr, 402 op/s 2024-09-21T10:02:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:33 smithi119 ceph-mon[40785]: osdmap e315: 8 total, 8 up, 8 in 2024-09-21T10:02:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:02:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:02:34] "GET /metrics HTTP/1.1" 200 38914 "" "Prometheus/2.51.0" 2024-09-21T10:02:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:34 smithi145 ceph-mon[33206]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:02:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:34 smithi119 ceph-mon[101432]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:02:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:34 smithi119 ceph-mon[40785]: osdmap e316: 8 total, 8 up, 8 in 2024-09-21T10:02:36.008 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:36.008 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:36.020 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 57002663~4187279 2024-09-21T10:02:36.031 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 58214346~5304898 2024-09-21T10:02:36.055 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 9887948~4880789 2024-09-21T10:02:36.057 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 75842356~79932 2024-09-21T10:02:36.057 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 76417359~6935555 2024-09-21T10:02:36.066 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 3790105~1148910 2024-09-21T10:02:36.066 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 39592872~3007930 2024-09-21T10:02:36.067 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 46429237~8232354 2024-09-21T10:02:36.068 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:02:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:36 smithi145 ceph-mon[33206]: pgmap v96: 137 pgs: 137 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 436 op/s 2024-09-21T10:02:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:36 smithi145 ceph-mon[33206]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:02:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:36 smithi119 ceph-mon[40785]: pgmap v96: 137 pgs: 137 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 436 op/s 2024-09-21T10:02:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:36 smithi119 ceph-mon[40785]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:02:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:36 smithi119 ceph-mon[101432]: pgmap v96: 137 pgs: 137 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 436 op/s 2024-09-21T10:02:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:36 smithi119 ceph-mon[101432]: osdmap e317: 8 total, 8 up, 8 in 2024-09-21T10:02:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:37 smithi145 ceph-mon[33206]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:02:37.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:36 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:37 smithi119 ceph-mon[40785]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:02:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:37 smithi119 ceph-mon[101432]: osdmap e318: 8 total, 8 up, 8 in 2024-09-21T10:02:38.032 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 153396534 2024-09-21T10:02:38.044 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 61853005~4166642 2024-09-21T10:02:38.055 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 96807874~7503051 2024-09-21T10:02:38.055 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 76571977~3013740 2024-09-21T10:02:38.060 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 147431656~5964878 2024-09-21T10:02:38.061 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 45501633~7001029 2024-09-21T10:02:38.067 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 102067579~6231642 2024-09-21T10:02:38.068 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 115899715~575016 2024-09-21T10:02:38.068 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 70763399~1973830 2024-09-21T10:02:38.068 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:02:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:38 smithi145 ceph-mon[33206]: pgmap v99: 137 pgs: 137 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 436 op/s 2024-09-21T10:02:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:38 smithi145 ceph-mon[33206]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:02:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:38 smithi119 ceph-mon[40785]: pgmap v99: 137 pgs: 137 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 436 op/s 2024-09-21T10:02:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:38 smithi119 ceph-mon[40785]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:02:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:38 smithi119 ceph-mon[101432]: pgmap v99: 137 pgs: 137 active+clean; 1.2 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 50 MiB/s rd, 22 MiB/s wr, 436 op/s 2024-09-21T10:02:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:38 smithi119 ceph-mon[101432]: osdmap e319: 8 total, 8 up, 8 in 2024-09-21T10:02:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:39 smithi145 ceph-mon[33206]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:02:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:39 smithi145 ceph-mon[33206]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:02:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:39 smithi119 ceph-mon[40785]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:02:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:39 smithi119 ceph-mon[40785]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:02:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:39 smithi119 ceph-mon[101432]: osdmap e320: 8 total, 8 up, 8 in 2024-09-21T10:02:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:39 smithi119 ceph-mon[101432]: osdmap e321: 8 total, 8 up, 8 in 2024-09-21T10:02:39.985 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap3 2024-09-21T10:02:39.998 INFO:tasks.workunit.client.0.smithi119.stdout:clone image170 -> image172 2024-09-21T10:02:40.128 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:40 smithi145 ceph-mon[33206]: pgmap v102: 137 pgs: 137 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 4.0 MiB/s rd, 16 MiB/s wr, 189 op/s 2024-09-21T10:02:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:40 smithi145 ceph-mon[33206]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:02:40.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:40 smithi119 ceph-mon[101432]: pgmap v102: 137 pgs: 137 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 4.0 MiB/s rd, 16 MiB/s wr, 189 op/s 2024-09-21T10:02:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:40 smithi119 ceph-mon[101432]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:02:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:40 smithi119 ceph-mon[40785]: pgmap v102: 137 pgs: 137 active+clean; 1.3 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 4.0 MiB/s rd, 16 MiB/s wr, 189 op/s 2024-09-21T10:02:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:40 smithi119 ceph-mon[40785]: osdmap e322: 8 total, 8 up, 8 in 2024-09-21T10:02:40.519 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:02:40.661 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:02:40.897 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.Stress (8526 ms) 2024-09-21T10:02:40.897 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.Stress2 2024-09-21T10:02:40.925 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:02:40.934 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 51682763~149039 2024-09-21T10:02:40.943 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 26234751~2020117 2024-09-21T10:02:40.943 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 5810223~2500025 2024-09-21T10:02:40.967 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 33704289~4076857 2024-09-21T10:02:40.968 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 8563054~5653969 2024-09-21T10:02:41.012 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 53100700~268190 2024-09-21T10:02:41.015 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 54766480~2576036 2024-09-21T10:02:41.036 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 43633519~8337305 2024-09-21T10:02:41.036 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap0 2024-09-21T10:02:42.251 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:42 smithi119 ceph-mon[101432]: pgmap v105: 137 pgs: 137 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 4.1 MiB/s rd, 27 MiB/s wr, 350 op/s 2024-09-21T10:02:42.251 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:42 smithi119 ceph-mon[40785]: pgmap v105: 137 pgs: 137 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 4.1 MiB/s rd, 27 MiB/s wr, 350 op/s 2024-09-21T10:02:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:42 smithi145 ceph-mon[33206]: pgmap v105: 137 pgs: 137 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 4.1 MiB/s rd, 27 MiB/s wr, 350 op/s 2024-09-21T10:02:42.745 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T10:02:43.001 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:43.064 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 137016626 2024-09-21T10:02:43.075 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 120986545~4210526 2024-09-21T10:02:43.084 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 42517046~1411319 2024-09-21T10:02:43.084 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 28811526~4220309 2024-09-21T10:02:43.089 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 17082536~1173280 2024-09-21T10:02:43.090 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 20810415~54080 2024-09-21T10:02:43.090 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 76201566~1351981 2024-09-21T10:02:43.090 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 64197992~1391427 2024-09-21T10:02:43.090 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 77933568~5547368 2024-09-21T10:02:43.090 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:02:43.306 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:43 smithi119 ceph-mon[101432]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:02:43.306 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:43.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:43 smithi119 ceph-mon[40785]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:02:43.306 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:43 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:43.475 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:43 smithi145 ceph-mon[33206]: osdmap e323: 8 total, 8 up, 8 in 2024-09-21T10:02:43.475 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:43 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:44.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:44 smithi119 ceph-mon[40785]: pgmap v107: 137 pgs: 137 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.7 MiB/s rd, 24 MiB/s wr, 320 op/s 2024-09-21T10:02:44.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:44 smithi119 ceph-mon[40785]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:02:44.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:44 smithi119 ceph-mon[101432]: pgmap v107: 137 pgs: 137 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.7 MiB/s rd, 24 MiB/s wr, 320 op/s 2024-09-21T10:02:44.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:44 smithi119 ceph-mon[101432]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:02:44.418 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:44 smithi145 ceph-mon[33206]: pgmap v107: 137 pgs: 137 active+clean; 1.3 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 3.7 MiB/s rd, 24 MiB/s wr, 320 op/s 2024-09-21T10:02:44.418 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:44 smithi145 ceph-mon[33206]: osdmap e324: 8 total, 8 up, 8 in 2024-09-21T10:02:44.419 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:44.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:02:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:02:44] "GET /metrics HTTP/1.1" 200 38914 "" "Prometheus/2.51.0" 2024-09-21T10:02:44.988 INFO:teuthology.orchestra.run.smithi119.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:45.078 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:02:45.089 INFO:tasks.workunit.client.0.smithi119.stdout:clone image173 -> image176 2024-09-21T10:02:45.178 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 132366331 2024-09-21T10:02:45.192 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 114840439~5352979 2024-09-21T10:02:45.241 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 20432345~6711657 2024-09-21T10:02:45.253 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 19413055~3003396 2024-09-21T10:02:45.272 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 33672108~1667393 2024-09-21T10:02:45.277 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 19295388~2589564 2024-09-21T10:02:45.286 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[101432]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:02:45.286 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[101432]: pgmap v110: 137 pgs: 137 active+clean; 1.4 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 35 MiB/s rd, 9.0 MiB/s wr, 789 op/s 2024-09-21T10:02:45.286 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[101432]: from='client.26075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:45.286 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[101432]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:45.287 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:45.287 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[40785]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:02:45.287 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[40785]: pgmap v110: 137 pgs: 137 active+clean; 1.4 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 35 MiB/s rd, 9.0 MiB/s wr, 789 op/s 2024-09-21T10:02:45.287 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[40785]: from='client.26075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:45.287 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[40785]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:45.287 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:45 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:45.294 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 105068738~5914531 2024-09-21T10:02:45.300 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 46714615~4123567 2024-09-21T10:02:45.328 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 28789992~6799781 2024-09-21T10:02:45.333 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:02:45.434 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-mon[33206]: osdmap e325: 8 total, 8 up, 8 in 2024-09-21T10:02:45.434 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-mon[33206]: pgmap v110: 137 pgs: 137 active+clean; 1.4 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 35 MiB/s rd, 9.0 MiB/s wr, 789 op/s 2024-09-21T10:02:45.434 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-mon[33206]: from='client.26075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:45.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-mon[33206]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:45.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:46.030 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:46.030 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:46 smithi119 ceph-mon[40785]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:02:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:46 smithi119 ceph-mon[101432]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:02:46.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for majority of mons to upgrade' 2024-09-21T10:02:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:46 smithi145 ceph-mon[33206]: osdmap e326: 8 total, 8 up, 8 in 2024-09-21T10:02:46.915 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:47.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:46 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:47.087 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 162329604 2024-09-21T10:02:47.099 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 79082940~80650 2024-09-21T10:02:47.105 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 136798219~7137499 2024-09-21T10:02:47.105 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 13072386~72754 2024-09-21T10:02:47.105 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 37376362~4135727 2024-09-21T10:02:47.105 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 126329813~2600672 2024-09-21T10:02:47.105 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 36800211~200497 2024-09-21T10:02:47.105 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 69141113~6303220 2024-09-21T10:02:47.113 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 28252057~7460932 2024-09-21T10:02:47.113 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:02:47.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:47 smithi119 ceph-mon[101432]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:02:47.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:47 smithi119 ceph-mon[101432]: pgmap v113: 137 pgs: 137 active+clean; 1.4 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 39 MiB/s rd, 9.9 MiB/s wr, 874 op/s 2024-09-21T10:02:47.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:47 smithi119 ceph-mon[40785]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:02:47.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:47 smithi119 ceph-mon[40785]: pgmap v113: 137 pgs: 137 active+clean; 1.4 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 39 MiB/s rd, 9.9 MiB/s wr, 874 op/s 2024-09-21T10:02:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:47 smithi145 ceph-mon[33206]: osdmap e327: 8 total, 8 up, 8 in 2024-09-21T10:02:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:47 smithi145 ceph-mon[33206]: pgmap v113: 137 pgs: 137 active+clean; 1.4 GiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 39 MiB/s rd, 9.9 MiB/s wr, 874 op/s 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[101432]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[40785]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:48.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:48 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:48 smithi145 ceph-mon[33206]: osdmap e328: 8 total, 8 up, 8 in 2024-09-21T10:02:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:48 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:48 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:48 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:02:48.853 INFO:teuthology.orchestra.run.smithi119.stdout:wait for majority of mons to upgrade 2024-09-21T10:02:49.100 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap3 2024-09-21T10:02:49.113 INFO:tasks.workunit.client.0.smithi119.stdout:clone image176 -> image178 2024-09-21T10:02:49.227 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 121744590 2024-09-21T10:02:49.260 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:02:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:49 smithi119 ceph-mon[101432]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:02:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:49 smithi119 ceph-mon[101432]: pgmap v116: 137 pgs: 137 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 MiB/s rd, 11 MiB/s wr, 262 op/s 2024-09-21T10:02:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:49 smithi119 ceph-mon[40785]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:02:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:49 smithi119 ceph-mon[40785]: pgmap v116: 137 pgs: 137 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 MiB/s rd, 11 MiB/s wr, 262 op/s 2024-09-21T10:02:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:49 smithi145 ceph-mon[33206]: osdmap e329: 8 total, 8 up, 8 in 2024-09-21T10:02:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:49 smithi145 ceph-mon[33206]: pgmap v116: 137 pgs: 137 active+clean; 1.4 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.5 MiB/s rd, 11 MiB/s wr, 262 op/s 2024-09-21T10:02:49.641 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 38882647~6390331 2024-09-21T10:02:49.708 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 44542006~5116215 2024-09-21T10:02:49.730 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 3837964~7949435 2024-09-21T10:02:49.795 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 77554002~7819452 2024-09-21T10:02:49.829 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 6895708~5573006 2024-09-21T10:02:49.857 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 99219180~2852032 2024-09-21T10:02:49.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 | egrep '"'"': [23]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T10:02:49.885 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 107571248~7635003 2024-09-21T10:02:49.957 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 12273972~1828401 2024-09-21T10:02:49.983 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user0 2024-09-21T10:02:50.088 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:50 smithi119 ceph-mon[101432]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:02:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:50 smithi119 ceph-mon[40785]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:02:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:50 smithi145 ceph-mon[33206]: osdmap e330: 8 total, 8 up, 8 in 2024-09-21T10:02:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:50.795 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:02:51.120 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 95333058~1968720 2024-09-21T10:02:51.131 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 36586988~5090413 2024-09-21T10:02:51.131 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 101430580~5764995 2024-09-21T10:02:51.139 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 22784969~5318668 2024-09-21T10:02:51.139 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 19411824~5725752 2024-09-21T10:02:51.221 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 7407605~7215388 2024-09-21T10:02:51.222 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 11101274~4362402 2024-09-21T10:02:51.272 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 14872887~7165712 2024-09-21T10:02:51.286 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user1 2024-09-21T10:02:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:51 smithi119 ceph-mon[101432]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:02:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:51 smithi119 ceph-mon[101432]: pgmap v119: 137 pgs: 137 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 13 MiB/s wr, 300 op/s 2024-09-21T10:02:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:51 smithi119 ceph-mon[40785]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:02:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:51 smithi119 ceph-mon[40785]: pgmap v119: 137 pgs: 137 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 13 MiB/s wr, 300 op/s 2024-09-21T10:02:51.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:51 smithi145 ceph-mon[33206]: osdmap e331: 8 total, 8 up, 8 in 2024-09-21T10:02:51.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:51 smithi145 ceph-mon[33206]: pgmap v119: 137 pgs: 137 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 13 MiB/s wr, 300 op/s 2024-09-21T10:02:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:52 smithi119 ceph-mon[40785]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:02:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:52 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:52 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:52 smithi119 ceph-mon[101432]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:02:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:52.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:52 smithi145 ceph-mon[33206]: osdmap e332: 8 total, 8 up, 8 in 2024-09-21T10:02:52.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:52 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:52.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:52 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:53.133 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 160540074 2024-09-21T10:02:53.145 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 91863333~6484203 2024-09-21T10:02:53.169 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 7062201~2124203 2024-09-21T10:02:53.169 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 39763421~7181741 2024-09-21T10:02:53.177 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 153231839~7308235 2024-09-21T10:02:53.177 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 148448539~536342 2024-09-21T10:02:53.178 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 12209975~3701045 2024-09-21T10:02:53.178 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 36911153~6176062 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: pgmap v122: 137 pgs: 137 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-21T10:02:53.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[40785]: Upgrade: It appears safe to stop mon.c 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: pgmap v122: 137 pgs: 137 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-21T10:02:53.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:53 smithi119 ceph-mon[101432]: Upgrade: It appears safe to stop mon.c 2024-09-21T10:02:53.269 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 13872473~6030123 2024-09-21T10:02:53.270 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user2 2024-09-21T10:02:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: osdmap e333: 8 total, 8 up, 8 in 2024-09-21T10:02:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: pgmap v122: 137 pgs: 137 active+clean; 1.5 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 MiB/s wr, 37 op/s 2024-09-21T10:02:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:02:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:02:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T10:02:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["c"]} : dispatch 2024-09-21T10:02:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:53 smithi145 ceph-mon[33206]: Upgrade: It appears safe to stop mon.c 2024-09-21T10:02:53.551 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:02:54.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[101432]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:02:54.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3575806180' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:02:54.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[101432]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:54.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[101432]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:54.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[40785]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:02:54.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[40785]: from='client.? 172.21.15.119:0/3575806180' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:02:54.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[40785]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:54.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-mon[40785]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:54 smithi145 ceph-mon[33206]: osdmap e334: 8 total, 8 up, 8 in 2024-09-21T10:02:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:54 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3575806180' entity='client.admin' cmd=[{"prefix": "mon versions"}]: dispatch 2024-09-21T10:02:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:54 smithi145 ceph-mon[33206]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:54 smithi145 ceph-mon[33206]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:02:54.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:54.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:02:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:02:54] "GET /metrics HTTP/1.1" 200 38911 "" "Prometheus/2.51.0" 2024-09-21T10:02:55.148 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 217718742 2024-09-21T10:02:55.162 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 94520891~8168777 2024-09-21T10:02:55.187 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 99320873~7086117 2024-09-21T10:02:55.234 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 100883047~3171602 2024-09-21T10:02:55.237 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 100963890~6281303 2024-09-21T10:02:55.255 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 48926777~1554243 2024-09-21T10:02:55.256 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 54568782~6068771 2024-09-21T10:02:55.256 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 124685406~7751766 2024-09-21T10:02:55.266 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 203584085~3920539 2024-09-21T10:02:55.266 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user3 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: Upgrade: Updating mon.c 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: Deploying daemon mon.c on smithi119 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[40785]: pgmap v125: 137 pgs: 137 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 30 MiB/s rd, 51 MiB/s wr, 809 op/s 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:02:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: Upgrade: Updating mon.c 2024-09-21T10:02:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:02:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:02:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: Deploying daemon mon.c on smithi119 2024-09-21T10:02:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:55 smithi119 ceph-mon[101432]: pgmap v125: 137 pgs: 137 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 30 MiB/s rd, 51 MiB/s wr, 809 op/s 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: osdmap e335: 8 total, 8 up, 8 in 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: Upgrade: Updating mon.c 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: Deploying daemon mon.c on smithi119 2024-09-21T10:02:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-mon[33206]: pgmap v125: 137 pgs: 137 active+clean; 1.7 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 30 MiB/s rd, 51 MiB/s wr, 809 op/s 2024-09-21T10:02:56.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:56.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:56.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:56 smithi119 ceph-mon[101432]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:02:56.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:56 smithi119 systemd[1]: Stopping Ceph mon.c for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:02:56.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:56 smithi119 ceph-mon[40785]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:02:56.522 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:02:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:56 smithi145 ceph-mon[33206]: osdmap e336: 8 total, 8 up, 8 in 2024-09-21T10:02:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c[40762]: 2024-09-21T10:02:56.349+0000 7f478f035640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:02:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c[40762]: 2024-09-21T10:02:56.349+0000 7f478f035640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T10:02:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:56 smithi119 podman[105050]: 2024-09-21 10:02:56.421268745 +0000 UTC m=+0.205452431 container died d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T10:02:57.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:02:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:02:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:02:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 podman[105050]: 2024-09-21 10:02:57.007290879 +0000 UTC m=+0.791474566 container cleanup d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-21T10:02:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 bash[105050]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c 2024-09-21T10:02:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 podman[105090]: 2024-09-21 10:02:57.200617308 +0000 UTC m=+0.777923237 container remove d64b219d7298aeb0fa7abc2177de11c8926184a377cd086b1c02730f2b8fa453 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux ) 2024-09-21T10:02:57.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-21T10:02:57.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 systemd[1]: Stopped Ceph mon.c for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:02:57.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.c.service: Consumed 9.331s CPU time. 2024-09-21T10:02:57.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 systemd[1]: Starting Ceph mon.c for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:02:58.114 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 podman[105250]: 2024-09-21 10:02:57.727452644 +0000 UTC m=+0.016058677 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:02:58.114 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:57 smithi119 podman[105250]: 2024-09-21 10:02:57.854811632 +0000 UTC m=+0.143417652 container create d747bb0bb662b2a376e2b674aa205950dd7db2fe872038d168b993d4d5a0eef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-09-21T10:02:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 podman[105250]: 2024-09-21 10:02:58.112942561 +0000 UTC m=+0.401548580 container init d747bb0bb662b2a376e2b674aa205950dd7db2fe872038d168b993d4d5a0eef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:02:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 podman[105250]: 2024-09-21 10:02:58.117374062 +0000 UTC m=+0.405980082 container start d747bb0bb662b2a376e2b674aa205950dd7db2fe872038d168b993d4d5a0eef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: pidfile_write: ignore empty --pid-file 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: load: jerasure load: lrc 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: RocksDB version: 7.9.2 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Git sha 0 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: DB SUMMARY 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: DB Session ID: 5SNPHWW84MKZG4HZB801 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: CURRENT file: CURRENT 2024-09-21T10:02:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: MANIFEST file: MANIFEST-000009 size: 897 Bytes 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000024.sst 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000022.log size: 12211741 ; 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.error_if_exists: 0 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.create_if_missing: 0 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.paranoid_checks: 1 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.env: 0x5589847ad120 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.info_log: 0x5589858ab700 2024-09-21T10:02:58.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.statistics: (nil) 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.use_fsync: 0 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_log_file_size: 0 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T10:02:58.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.allow_fallocate: 1 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.use_direct_reads: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.db_log_dir: 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.wal_dir: 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T10:02:58.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.write_buffer_manager: 0x5589858ae1e0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.unordered_write: 0 2024-09-21T10:02:58.373 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.row_cache: None 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.wal_filter: None 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.two_write_queues: 0 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.wal_compression: 0 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.atomic_flush: 0 2024-09-21T10:02:58.374 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.log_readahead_size: 0 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_background_jobs: 2 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_background_compactions: -1 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_subcompactions: 1 2024-09-21T10:02:58.375 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_open_files: -1 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_background_flushes: -1 2024-09-21T10:02:58.376 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Compression algorithms supported: 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kZSTD supported: 0 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kXpressCompression supported: 0 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kBZip2Compression supported: 0 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kLZ4Compression supported: 1 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kZlibCompression supported: 1 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: kSnappyCompression supported: 1 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-21T10:02:58.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.merge_operator: 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_filter: None 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5589858ab320) 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-09-21T10:02:58.378 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_type: 0 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_block_index_type: 0 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_shortening: 1 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: checksum: 4 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: no_block_cache: 0 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache: 0x55898589f610 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_options: 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: capacity : 536870912 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: num_shard_bits : 4 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: strict_capacity_limit : 0 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-09-21T10:02:58.379 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_compressed: (nil) 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: persistent_cache: (nil) 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_size: 4096 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_size_deviation: 10 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_restart_interval: 16 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_block_restart_interval: 1 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: metadata_block_size: 4096 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: partition_filters: 0 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: use_delta_encoding: 1 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: filter_policy: bloomfilter 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: whole_key_filtering: 1 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: verify_compression: 0 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: format_version: 5 2024-09-21T10:02:58.380 INFO:journalctl@ceph.mon.c.smithi119.stdout: enable_index_compression: 1 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_align: 0 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout: max_auto_readahead_size: 262144 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout: prepopulate_block_cache: 0 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout: initial_auto_readahead_size: 8192 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression: NoCompression 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.num_levels: 7 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T10:02:58.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T10:02:58.382 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T10:02:58.383 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T10:02:58.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.inplace_update_support: 0 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T10:02:58.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.bloom_locality: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.max_successive_merges: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.ttl: 2592000 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.enable_blob_files: false 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.min_blob_size: 0 2024-09-21T10:02:58.386 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 13398, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2b353165-0970-4537-af19-91d6471d4537 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912978144336, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-09-21T10:02:58.387 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912978173633, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 7611569, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 13403, "largest_seqno": 17661, "table_properties": {"data_size": 7592020, "index_size": 13958, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 4549, "raw_key_size": 39844, "raw_average_key_size": 22, "raw_value_size": 7553149, "raw_average_value_size": 4200, "num_data_blocks": 601, "num_entries": 1798, "num_filter_entries": 1798, "num_deletions": 258, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726912978, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2b353165-0970-4537-af19-91d6471d4537", "db_session_id": "5SNPHWW84MKZG4HZB801", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726912978173758, "job": 1, "event": "recovery_finished"} 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5589858cee00 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: DB pointer 0x5589859d0000 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** DB Stats ** 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:02:58.388 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: L0 1/0 7.26 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 549.0 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: L6 1/0 7.80 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Sum 2/0 15.06 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 549.0 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 549.0 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-21T10:02:58.389 INFO:journalctl@ceph.mon.c.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 549.0 0.01 0.00 1 0.013 0 0 0.0 0.0 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: Flush(GB): cumulative 0.007, interval 0.007 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative compaction: 0.01 GB write, 224.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval compaction: 0.01 GB write, 224.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:02:58.390 INFO:journalctl@ceph.mon.c.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: Block cache BinnedLRUCache@0x55898589f610#7 capacity: 512.00 MB usage: 20.61 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,4.53 KB,0.000864267%) IndexBlock(1,16.08 KB,0.00306666%) Misc(1,0.00 KB,0%) 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: starting mon.c rank 2 at public addrs [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] at bind addrs [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???) e3 preinit fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).mds e9 new map 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).mds e9 print_map 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: e9 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: legacy client fscid: 1 2024-09-21T10:02:58.391 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: Filesystem 'foo' (1) 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: fs_name foo 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: epoch 8 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: created 2024-09-21T09:56:59.994919+0000 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: modified 2024-09-21T09:57:18.635502+0000 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: tableserver 0 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: root 0 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: session_timeout 60 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: session_autoclose 300 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: max_file_size 1099511627776 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: max_xattr_size 65536 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: required_client_features {} 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: last_failure 0 2024-09-21T10:02:58.392 INFO:journalctl@ceph.mon.c.smithi119.stdout: last_failure_osd_epoch 0 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: max_mds 1 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: in 0 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: up {0=24557} 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: failed 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: damaged 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: stopped 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_pools [4] 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: metadata_pool 3 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: inline_data disabled 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: balancer 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: bal_rank_mask -1 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: standby_count_wanted 1 2024-09-21T10:02:58.393 INFO:journalctl@ceph.mon.c.smithi119.stdout: qdb_cluster leader: 0 members: 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout: [mds.foo.smithi119.oikglq{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout: Standby daemons: 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout: [mds.foo.smithi145.tjomfr{-1:24457} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).osd e337 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).osd e337 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).osd e337 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).osd e337 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 ceph-mon[105288]: mon.c@-1(???).paxosservice(auth 1..20) refresh upgraded, format 0 -> 3 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 bash[105250]: d747bb0bb662b2a376e2b674aa205950dd7db2fe872038d168b993d4d5a0eef0 2024-09-21T10:02:58.394 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:58 smithi119 systemd[1]: Started Ceph mon.c for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:02:59.168 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:02:59.416 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 21771874~5442968 2024-09-21T10:02:59.421 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 23132616~2721484 2024-09-21T10:02:59.465 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 43543748~5442968 2024-09-21T10:02:59.466 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 44904490~2721484 2024-09-21T10:02:59.508 INFO:tasks.workunit.client.0.smithi119.stdout:write: 3 65315622~5442968 2024-09-21T10:02:59.510 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 66676364~2721484 2024-09-21T10:02:59.564 INFO:tasks.workunit.client.0.smithi119.stdout:write: 4 87087496~5442968 2024-09-21T10:02:59.566 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 88448238~2721484 2024-09-21T10:02:59.605 INFO:tasks.workunit.client.0.smithi119.stdout:write: 5 108859371~5442968 2024-09-21T10:02:59.608 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 110220113~2721484 2024-09-21T10:02:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: mon.c calling monitor election 2024-09-21T10:02:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: mon.a calling monitor election 2024-09-21T10:02:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: monmap epoch 3 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: last_changed 2024-09-21T09:51:41.651239+0000 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: min_mon_release 17 (quincy) 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: election_strategy: 1 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: mgrmap e37: y(active, since 86s), standbys: x 2024-09-21T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: pgmap v129: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 24 MiB/s rd, 49 MiB/s wr, 746 op/s 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3128014153' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: mon.c calling monitor election 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: mon.a calling monitor election 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:02:59.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: monmap epoch 3 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: last_changed 2024-09-21T09:51:41.651239+0000 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: min_mon_release 17 (quincy) 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: election_strategy: 1 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: mgrmap e37: y(active, since 86s), standbys: x 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:02:59.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: pgmap v129: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 24 MiB/s rd, 49 MiB/s wr, 746 op/s 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:59.621 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:02:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3128014153' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:02:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: mon.c calling monitor election 2024-09-21T10:02:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: mon.a calling monitor election 2024-09-21T10:02:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:02:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: monmap epoch 3 2024-09-21T10:02:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: last_changed 2024-09-21T09:51:41.651239+0000 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: min_mon_release 17 (quincy) 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: election_strategy: 1 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: osdmap e337: 8 total, 8 up, 8 in 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: mgrmap e37: y(active, since 86s), standbys: x 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: application not enabled on pool 'test-librbd-smithi119-84733-9' 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-21T10:02:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:59.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: pgmap v129: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 24 MiB/s rd, 49 MiB/s wr, 746 op/s 2024-09-21T10:02:59.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:02:59.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:02:59.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:02:59 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/3128014153' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:03:00.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:03:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:00 smithi145 ceph-mon[33206]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:03:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:00 smithi119 ceph-mon[105288]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:03:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:00 smithi119 ceph-mon[101432]: osdmap e338: 8 total, 8 up, 8 in 2024-09-21T10:03:00.890 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:03:01.058 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:03:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:01 smithi119 ceph-mon[101432]: pgmap v131: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 9.3 MiB/s wr, 118 op/s 2024-09-21T10:03:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:01 smithi119 ceph-mon[105288]: pgmap v131: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 9.3 MiB/s wr, 118 op/s 2024-09-21T10:03:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:01 smithi145 ceph-mon[33206]: pgmap v131: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.6 MiB/s rd, 9.3 MiB/s wr, 118 op/s 2024-09-21T10:03:02.352 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.Stress2 (21453 ms) 2024-09-21T10:03:02.503 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMigration.StressLive 2024-09-21T10:03:02.504 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 83886080 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 69886172~2713673 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 71695180~7540197 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 80572429~3313651 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 4529020~8287361 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 30734342~3644815 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 40077114~2170535 2024-09-21T10:03:02.505 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 39131477~7419068 2024-09-21T10:03:02.519 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 56531325~6302851 2024-09-21T10:03:02.519 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap0 2024-09-21T10:03:03.413 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:03:03.184+0000 7fc0be6c7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T10:03:03.452 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:03:03.452 INFO:teuthology.orchestra.run.smithi119.stdout:alertmanager.a smithi119 *:9093,9094 running (2m) 11s ago 7m 20.2M - 0.27.0 11f11916f8cd e4cf1f6d8355 2024-09-21T10:03:03.452 INFO:teuthology.orchestra.run.smithi119.stdout:grafana.a smithi145 *:3000 running (2m) 16s ago 6m 76.7M - 9.4.7 954c08fa6188 e033bb16a7cd 2024-09-21T10:03:03.452 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi119.oikglq smithi119 running (5m) 11s ago 5m 17.9M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4ff2238155bf 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi145.tjomfr smithi145 running (5m) 16s ago 5m 15.6M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b1338cf49871 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi145 *:8443,9283,8765 running (114s) 16s ago 11m 453M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 1346375dd747 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi119 *:8443,9283,8765 running (2m) 11s ago 12m 508M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d7bcb26ddb62 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (99s) 11s ago 12m 41.8M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 030d68226d6c 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi145 running (11m) 16s ago 11m 63.7M 2048M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be f45b53ab2476 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi119 starting - - - 2048M 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.a smithi119 *:9100 running (2m) 11s ago 6m 10.3M - 1.7.0 72c9c2088986 05912f78c070 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.b smithi145 *:9100 running (2m) 16s ago 6m 9601k - 1.7.0 72c9c2088986 d906bafb8d86 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (10m) 11s ago 10m 138M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ce07d9950c3f 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (10m) 11s ago 10m 130M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b8f7b7d5ac64 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (9m) 11s ago 9m 116M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 87104f5801c3 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (9m) 11s ago 9m 137M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 700faab8401d 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi145 running (9m) 16s ago 9m 138M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 18086eb220ec 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi145 running (8m) 16s ago 8m 101M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 90ab6d4f8571 2024-09-21T10:03:03.453 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi145 running (8m) 16s ago 8m 155M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 772774e9be4a 2024-09-21T10:03:03.454 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi145 running (7m) 16s ago 7m 117M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1a18c8e91654 2024-09-21T10:03:03.454 INFO:teuthology.orchestra.run.smithi119.stdout:prometheus.a smithi145 *:9095 running (119s) 16s ago 7m 48.5M - 2.51.0 1d3b7f56885b d2de6cb75d75 2024-09-21T10:03:03.792 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:03 smithi119 ceph-mon[105288]: pgmap v132: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.4 MiB/s rd, 7.7 MiB/s wr, 97 op/s 2024-09-21T10:03:03.793 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:03.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:03 smithi119 ceph-mon[101432]: pgmap v132: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.4 MiB/s rd, 7.7 MiB/s wr, 97 op/s 2024-09-21T10:03:03.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:03 smithi145 ceph-mon[33206]: pgmap v132: 137 pgs: 137 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.4 MiB/s rd, 7.7 MiB/s wr, 97 op/s 2024-09-21T10:03:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:03 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:04.421 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:04.439 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap0 2024-09-21T10:03:04.449 INFO:tasks.workunit.client.0.smithi119.stdout:clone image179 -> image182 2024-09-21T10:03:04.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:04 smithi119 ceph-mon[101432]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:03:04.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:04 smithi119 ceph-mon[101432]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:04.526 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:04 smithi119 ceph-mon[105288]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:03:04.526 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:04 smithi119 ceph-mon[105288]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:04.545 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 118423183 2024-09-21T10:03:04.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T10:03:04.556 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 95097459~1798843 2024-09-21T10:03:04.579 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 80360469~128067 2024-09-21T10:03:04.619 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 45191866~6371267 2024-09-21T10:03:04.688 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 58558655~5664890 2024-09-21T10:03:04.695 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 82134888~1972597 2024-09-21T10:03:04.711 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 56011460~7221786 2024-09-21T10:03:04.713 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 65771338~3910494 2024-09-21T10:03:04.761 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 111652689~1356149 2024-09-21T10:03:04.761 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap1 2024-09-21T10:03:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:04 smithi145 ceph-mon[33206]: osdmap e339: 8 total, 8 up, 8 in 2024-09-21T10:03:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:04 smithi145 ceph-mon[33206]: from='client.34209 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:04.845 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:03:04] "GET /metrics HTTP/1.1" 200 38837 "" "Prometheus/2.51.0" 2024-09-21T10:03:05.310 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[101432]: pgmap v134: 137 pgs: 137 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s rd, 19 MiB/s wr, 513 op/s 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[101432]: mgrmap e38: y(active, since 92s), standbys: x 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[101432]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[105288]: pgmap v134: 137 pgs: 137 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s rd, 19 MiB/s wr, 513 op/s 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[105288]: mgrmap e38: y(active, since 92s), standbys: x 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[105288]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:05.684 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:05.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-mon[33206]: pgmap v134: 137 pgs: 137 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 170 MiB/s rd, 19 MiB/s wr, 513 op/s 2024-09-21T10:03:05.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-mon[33206]: mgrmap e38: y(active, since 92s), standbys: x 2024-09-21T10:03:05.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-mon[33206]: osdmap e340: 8 total, 8 up, 8 in 2024-09-21T10:03:05.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:05.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:06.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:06.442 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap1 2024-09-21T10:03:06.451 INFO:tasks.workunit.client.0.smithi119.stdout:clone image182 -> image184 2024-09-21T10:03:06.576 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 112609036~6956 2024-09-21T10:03:06.587 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 105256100~2469717 2024-09-21T10:03:06.592 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 9519639~3235085 2024-09-21T10:03:06.626 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 31029941~4605935 2024-09-21T10:03:06.653 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 45414524~6906642 2024-09-21T10:03:06.680 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:06 smithi119 ceph-mon[101432]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:03:06.680 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.680 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.681 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:06 smithi119 ceph-mon[105288]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:03:06.681 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.681 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.721 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 41915897~4698076 2024-09-21T10:03:06.768 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 115023507~3399676 2024-09-21T10:03:06.791 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 83027708~3251059 2024-09-21T10:03:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:06 smithi145 ceph-mon[33206]: osdmap e341: 8 total, 8 up, 8 in 2024-09-21T10:03:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:06 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:06 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:06.824 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap2 2024-09-21T10:03:07.220 INFO:teuthology.orchestra.run.smithi119.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:03:07.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:06.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:07.614 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:07 smithi119 ceph-mon[101432]: pgmap v137: 137 pgs: 137 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 225 MiB/s rd, 16 MiB/s wr, 566 op/s 2024-09-21T10:03:07.615 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:07 smithi119 ceph-mon[101432]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:03:07.615 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:07.615 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:07 smithi119 ceph-mon[105288]: pgmap v137: 137 pgs: 137 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 225 MiB/s rd, 16 MiB/s wr, 566 op/s 2024-09-21T10:03:07.615 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:07 smithi119 ceph-mon[105288]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:03:07.615 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:07 smithi145 ceph-mon[33206]: pgmap v137: 137 pgs: 137 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 225 MiB/s rd, 16 MiB/s wr, 566 op/s 2024-09-21T10:03:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:07 smithi145 ceph-mon[33206]: osdmap e342: 8 total, 8 up, 8 in 2024-09-21T10:03:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:07 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:08.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T10:03:08.454 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap2 2024-09-21T10:03:08.466 INFO:tasks.workunit.client.0.smithi119.stdout:clone image184 -> image186 2024-09-21T10:03:08.608 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 93682471 2024-09-21T10:03:08.631 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 12366916~7126287 2024-09-21T10:03:08.697 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 19294922~3308572 2024-09-21T10:03:08.703 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 80952513~5801323 2024-09-21T10:03:08.740 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 68773362~3844995 2024-09-21T10:03:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 86813191~6869280 2024-09-21T10:03:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:08 smithi145 ceph-mon[33206]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:08 smithi145 ceph-mon[33206]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:03:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:08 smithi145 ceph-mon[33206]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:03:08.811 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 30685736~3947340 2024-09-21T10:03:08.822 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 32819010~3960154 2024-09-21T10:03:08.844 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 54119965~8163899 2024-09-21T10:03:08.866 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: snap3 2024-09-21T10:03:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:08 smithi119 ceph-mon[105288]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:08 smithi119 ceph-mon[105288]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:03:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:08 smithi119 ceph-mon[105288]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:03:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:08 smithi119 ceph-mon[101432]: from='client.44108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:03:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:08 smithi119 ceph-mon[101432]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:03:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:08 smithi119 ceph-mon[101432]: osdmap e343: 8 total, 8 up, 8 in 2024-09-21T10:03:09.113 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:03:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:09 smithi119 ceph-mon[105288]: pgmap v140: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 23 MiB/s wr, 418 op/s 2024-09-21T10:03:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:09 smithi119 ceph-mon[105288]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:03:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:09 smithi119 ceph-mon[101432]: pgmap v140: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 23 MiB/s wr, 418 op/s 2024-09-21T10:03:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:09 smithi119 ceph-mon[101432]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:03:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:09 smithi145 ceph-mon[33206]: pgmap v140: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 23 MiB/s wr, 418 op/s 2024-09-21T10:03:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:09 smithi145 ceph-mon[33206]: osdmap e344: 8 total, 8 up, 8 in 2024-09-21T10:03:10.463 INFO:tasks.workunit.client.0.smithi119.stdout:snap_protect: snap3 2024-09-21T10:03:10.470 INFO:tasks.workunit.client.0.smithi119.stdout:clone image186 -> image188 2024-09-21T10:03:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:10 smithi145 ceph-mon[33206]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:03:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:10.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:10 smithi119 ceph-mon[105288]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:03:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:10 smithi119 ceph-mon[101432]: osdmap e345: 8 total, 8 up, 8 in 2024-09-21T10:03:10.646 INFO:tasks.workunit.client.0.smithi119.stdout:migration_prepare 2024-09-21T10:03:11.090 INFO:tasks.workunit.client.0.smithi119.stdout:migration_execute 2024-09-21T10:03:11.090 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 87929515~5752956 2024-09-21T10:03:11.156 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 45889652~7285725 2024-09-21T10:03:11.186 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 74750735~4979965 2024-09-21T10:03:11.236 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 14234360~3181029 2024-09-21T10:03:11.280 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 38855421~4086376 2024-09-21T10:03:11.328 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 59642732~4549673 2024-09-21T10:03:11.364 INFO:tasks.workunit.client.0.smithi119.stdout:write: a 29686689~8044747 2024-09-21T10:03:11.428 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 2964711~3899631 2024-09-21T10:03:11.433 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user0 2024-09-21T10:03:11.737 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:11 smithi119 ceph-mon[105288]: pgmap v143: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 23 MiB/s wr, 418 op/s 2024-09-21T10:03:11.737 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:11 smithi119 ceph-mon[105288]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:03:11.738 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:11 smithi119 ceph-mon[101432]: pgmap v143: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 23 MiB/s wr, 418 op/s 2024-09-21T10:03:11.738 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:11 smithi119 ceph-mon[101432]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:03:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:11 smithi145 ceph-mon[33206]: pgmap v143: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 12 MiB/s rd, 23 MiB/s wr, 418 op/s 2024-09-21T10:03:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:11 smithi145 ceph-mon[33206]: osdmap e346: 8 total, 8 up, 8 in 2024-09-21T10:03:12.480 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 83917224~1312346 2024-09-21T10:03:12.486 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 86615624~7066847 2024-09-21T10:03:12.486 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 86723956~6958515 2024-09-21T10:03:12.500 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 51780171~4282888 2024-09-21T10:03:12.501 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 87090879~6591592 2024-09-21T10:03:12.539 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 73323514~4039381 2024-09-21T10:03:12.539 INFO:tasks.workunit.client.0.smithi119.stdout:write: b 21195679~764537 2024-09-21T10:03:12.540 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 70796586~5102962 2024-09-21T10:03:12.573 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user1 2024-09-21T10:03:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:03:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:12.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:12.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:12.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.813 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:03:12.813 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.813 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: osdmap e347: 8 total, 8 up, 8 in 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.814 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.815 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:12.815 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:12.815 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:03:12.815 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:03:12.815 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:03:12.815 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:13 smithi145 ceph-mon[33206]: pgmap v146: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:03:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:13 smithi145 ceph-mon[33206]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:03:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:13 smithi119 ceph-mon[101432]: pgmap v146: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:03:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:13 smithi119 ceph-mon[101432]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:03:13.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:13 smithi119 ceph-mon[105288]: pgmap v146: 137 pgs: 137 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:03:13.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:13 smithi119 ceph-mon[105288]: osdmap e348: 8 total, 8 up, 8 in 2024-09-21T10:03:14.484 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:14.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:14.497 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 5910211~4884315 2024-09-21T10:03:14.512 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 6599471~4150897 2024-09-21T10:03:14.545 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 66957529~726059 2024-09-21T10:03:14.546 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 87296963~6385508 2024-09-21T10:03:14.546 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 53499639~5168760 2024-09-21T10:03:14.548 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 25671337~174476 2024-09-21T10:03:14.550 INFO:tasks.workunit.client.0.smithi119.stdout:write: c 6153805~617750 2024-09-21T10:03:14.551 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 46699104~1992130 2024-09-21T10:03:14.551 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user2 2024-09-21T10:03:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:14 smithi145 ceph-mon[33206]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:03:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:14 smithi119 ceph-mon[101432]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:03:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:14 smithi119 ceph-mon[105288]: osdmap e349: 8 total, 8 up, 8 in 2024-09-21T10:03:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:03:14] "GET /metrics HTTP/1.1" 200 38837 "" "Prometheus/2.51.0" 2024-09-21T10:03:15.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:15 smithi145 ceph-mon[33206]: pgmap v149: 137 pgs: 137 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 41 MiB/s rd, 47 MiB/s wr, 1.21k op/s 2024-09-21T10:03:15.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:15 smithi145 ceph-mon[33206]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:03:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:15 smithi119 ceph-mon[105288]: pgmap v149: 137 pgs: 137 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 41 MiB/s rd, 47 MiB/s wr, 1.21k op/s 2024-09-21T10:03:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:15 smithi119 ceph-mon[105288]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:03:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:15 smithi119 ceph-mon[101432]: pgmap v149: 137 pgs: 137 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 41 MiB/s rd, 47 MiB/s wr, 1.21k op/s 2024-09-21T10:03:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:15 smithi119 ceph-mon[101432]: osdmap e350: 8 total, 8 up, 8 in 2024-09-21T10:03:16.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:16.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:15.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:16.509 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 60580354~2327678 2024-09-21T10:03:16.517 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 2271038~4706006 2024-09-21T10:03:16.517 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 8365852~3540981 2024-09-21T10:03:16.519 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 1891140~7516737 2024-09-21T10:03:16.542 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 80963181~2258995 2024-09-21T10:03:16.543 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 47897214~3610218 2024-09-21T10:03:16.543 INFO:tasks.workunit.client.0.smithi119.stdout:write: d 34849098~939093 2024-09-21T10:03:16.544 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 71200693~3857664 2024-09-21T10:03:16.544 INFO:tasks.workunit.client.0.smithi119.stdout:snap_create: user3 2024-09-21T10:03:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:16 smithi145 ceph-mon[33206]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:03:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:16 smithi119 ceph-mon[105288]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:03:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:16 smithi119 ceph-mon[101432]: osdmap e351: 8 total, 8 up, 8 in 2024-09-21T10:03:17.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:17.499 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:17 smithi145 ceph-mon[33206]: pgmap v152: 137 pgs: 137 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 41 MiB/s rd, 47 MiB/s wr, 1.21k op/s 2024-09-21T10:03:17.499 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:17 smithi145 ceph-mon[33206]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:03:17.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:17 smithi119 ceph-mon[101432]: pgmap v152: 137 pgs: 137 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 41 MiB/s rd, 47 MiB/s wr, 1.21k op/s 2024-09-21T10:03:17.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:17 smithi119 ceph-mon[101432]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:03:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:17 smithi119 ceph-mon[105288]: pgmap v152: 137 pgs: 137 active+clean; 2.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 41 MiB/s rd, 47 MiB/s wr, 1.21k op/s 2024-09-21T10:03:17.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:17 smithi119 ceph-mon[105288]: osdmap e352: 8 total, 8 up, 8 in 2024-09-21T10:03:18.525 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 153063471 2024-09-21T10:03:18.547 INFO:tasks.workunit.client.0.smithi119.stdout:write: 1 15306347~3826586 2024-09-21T10:03:18.548 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16262993~1913293 2024-09-21T10:03:18.568 INFO:tasks.workunit.client.0.smithi119.stdout:write: 2 30612694~3826586 2024-09-21T10:03:18.569 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 31569340~1913293 2024-09-21T10:03:18.592 INFO:tasks.workunit.client.0.smithi119.stdout:write: 3 45919041~3826586 2024-09-21T10:03:18.594 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 46875687~1913293 2024-09-21T10:03:18.618 INFO:tasks.workunit.client.0.smithi119.stdout:write: 4 61225388~3826586 2024-09-21T10:03:18.619 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 62182034~1913293 2024-09-21T10:03:18.641 INFO:tasks.workunit.client.0.smithi119.stdout:write: 5 76531735~3826586 2024-09-21T10:03:18.643 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 77488381~1913293 2024-09-21T10:03:18.765 INFO:tasks.workunit.client.0.smithi119.stdout:migration_commit 2024-09-21T10:03:18.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:18 smithi145 ceph-mon[33206]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:03:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:18 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:18 smithi119 ceph-mon[101432]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:03:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:18.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:18 smithi119 ceph-mon[105288]: osdmap e353: 8 total, 8 up, 8 in 2024-09-21T10:03:18.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:19.612 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMigration.StressLive (17262 ms) 2024-09-21T10:03:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:19 smithi145 ceph-mon[33206]: pgmap v155: 137 pgs: 137 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 171 op/s 2024-09-21T10:03:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:19 smithi145 ceph-mon[33206]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:03:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:19 smithi119 ceph-mon[101432]: pgmap v155: 137 pgs: 137 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 171 op/s 2024-09-21T10:03:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:19 smithi119 ceph-mon[101432]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:03:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:19 smithi119 ceph-mon[105288]: pgmap v155: 137 pgs: 137 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.6 MiB/s rd, 11 MiB/s wr, 171 op/s 2024-09-21T10:03:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:19 smithi119 ceph-mon[105288]: osdmap e354: 8 total, 8 up, 8 in 2024-09-21T10:03:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:21.531 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 57 tests from TestMigration (266146 ms total) 2024-09-21T10:03:21.532 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:03:21.532 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 6 tests from TestGroup 2024-09-21T10:03:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:21 smithi145 ceph-mon[33206]: pgmap v157: 137 pgs: 137 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 MiB/s rd, 9.1 MiB/s wr, 139 op/s 2024-09-21T10:03:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:21 smithi145 ceph-mon[33206]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:03:21.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:21 smithi119 ceph-mon[105288]: pgmap v157: 137 pgs: 137 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 MiB/s rd, 9.1 MiB/s wr, 139 op/s 2024-09-21T10:03:21.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:21 smithi119 ceph-mon[105288]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:03:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:21 smithi119 ceph-mon[101432]: pgmap v157: 137 pgs: 137 active+clean; 2.2 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.3 MiB/s rd, 9.1 MiB/s wr, 139 op/s 2024-09-21T10:03:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:21 smithi119 ceph-mon[101432]: osdmap e355: 8 total, 8 up, 8 in 2024-09-21T10:03:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:22 smithi145 ceph-mon[33206]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:03:22.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:22 smithi119 ceph-mon[101432]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:03:22.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:22 smithi119 ceph-mon[105288]: osdmap e356: 8 total, 8 up, 8 in 2024-09-21T10:03:23.546 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestGroup.group_create 2024-09-21T10:03:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:23 smithi145 ceph-mon[33206]: pgmap v160: 73 pgs: 73 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:03:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:23 smithi145 ceph-mon[33206]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:03:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:23 smithi145 ceph-mon[33206]: Cluster is now healthy 2024-09-21T10:03:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:23 smithi145 ceph-mon[33206]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:03:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:23 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/743597933' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:23 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[101432]: pgmap v160: 73 pgs: 73 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[101432]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/743597933' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[105288]: pgmap v160: 73 pgs: 73 active+clean; 582 KiB data, 7.4 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:03:23.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:03:23.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:03:23.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[105288]: osdmap e357: 8 total, 8 up, 8 in 2024-09-21T10:03:23.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/743597933' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:23.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:24.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:24.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:03:24] "GET /metrics HTTP/1.1" 200 38904 "" "Prometheus/2.51.0" 2024-09-21T10:03:24.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:24 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:24.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:24 smithi119 ceph-mon[105288]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:03:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:24 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:24 smithi119 ceph-mon[101432]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:03:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:24 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:24 smithi145 ceph-mon[33206]: osdmap e358: 8 total, 8 up, 8 in 2024-09-21T10:03:25.587 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestGroup.group_create (2041 ms) 2024-09-21T10:03:25.587 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestGroup.group_createPP 2024-09-21T10:03:25.604 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestGroup.group_createPP (17 ms) 2024-09-21T10:03:25.604 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestGroup.add_image 2024-09-21T10:03:25.641 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestGroup.add_image (37 ms) 2024-09-21T10:03:25.641 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestGroup.add_imagePP 2024-09-21T10:03:25.675 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestGroup.add_imagePP (33 ms) 2024-09-21T10:03:25.675 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestGroup.add_snapshot 2024-09-21T10:03:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:25 smithi119 ceph-mon[101432]: pgmap v163: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:03:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:25 smithi119 ceph-mon[101432]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:03:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:25 smithi119 ceph-mon[105288]: pgmap v163: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:03:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:25 smithi119 ceph-mon[105288]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:03:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:25 smithi145 ceph-mon[33206]: pgmap v163: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:03:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:25 smithi145 ceph-mon[33206]: osdmap e359: 8 total, 8 up, 8 in 2024-09-21T10:03:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:26 smithi119 ceph-mon[101432]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:03:26.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:26 smithi119 ceph-mon[105288]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:03:26.951 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:26 smithi145 ceph-mon[33206]: osdmap e360: 8 total, 8 up, 8 in 2024-09-21T10:03:27.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:26 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:27 smithi119 ceph-mon[101432]: pgmap v166: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:03:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:27 smithi119 ceph-mon[101432]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:03:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:27 smithi119 ceph-mon[105288]: pgmap v166: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:03:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:27 smithi119 ceph-mon[105288]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:03:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:27 smithi145 ceph-mon[33206]: pgmap v166: 105 pgs: 32 creating+peering, 73 active+clean; 582 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:03:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:27 smithi145 ceph-mon[33206]: osdmap e361: 8 total, 8 up, 8 in 2024-09-21T10:03:28.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:28 smithi119 ceph-mon[101432]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:03:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:28 smithi119 ceph-mon[105288]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:03:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:28 smithi145 ceph-mon[33206]: osdmap e362: 8 total, 8 up, 8 in 2024-09-21T10:03:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:29 smithi119 ceph-mon[101432]: pgmap v169: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 139 KiB/s rd, 16 KiB/s wr, 188 op/s 2024-09-21T10:03:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:29 smithi119 ceph-mon[101432]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:03:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:29 smithi119 ceph-mon[101432]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:03:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:29 smithi119 ceph-mon[105288]: pgmap v169: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 139 KiB/s rd, 16 KiB/s wr, 188 op/s 2024-09-21T10:03:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:29 smithi119 ceph-mon[105288]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:03:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:29 smithi119 ceph-mon[105288]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:03:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:29 smithi145 ceph-mon[33206]: pgmap v169: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 139 KiB/s rd, 16 KiB/s wr, 188 op/s 2024-09-21T10:03:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:29 smithi145 ceph-mon[33206]: osdmap e363: 8 total, 8 up, 8 in 2024-09-21T10:03:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:29 smithi145 ceph-mon[33206]: osdmap e364: 8 total, 8 up, 8 in 2024-09-21T10:03:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:31.026 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestGroup.add_snapshot (5352 ms) 2024-09-21T10:03:31.026 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestGroup.add_snapshotPP 2024-09-21T10:03:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:30 smithi145 ceph-mon[33206]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:03:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:30 smithi119 ceph-mon[105288]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:03:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:30 smithi119 ceph-mon[101432]: osdmap e365: 8 total, 8 up, 8 in 2024-09-21T10:03:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:31 smithi145 ceph-mon[33206]: pgmap v173: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 145 KiB/s rd, 17 KiB/s wr, 196 op/s 2024-09-21T10:03:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:31 smithi145 ceph-mon[33206]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:03:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:31 smithi119 ceph-mon[105288]: pgmap v173: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 145 KiB/s rd, 17 KiB/s wr, 196 op/s 2024-09-21T10:03:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:31 smithi119 ceph-mon[105288]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:03:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:31 smithi119 ceph-mon[101432]: pgmap v173: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 145 KiB/s rd, 17 KiB/s wr, 196 op/s 2024-09-21T10:03:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:31 smithi119 ceph-mon[101432]: osdmap e366: 8 total, 8 up, 8 in 2024-09-21T10:03:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:33 smithi145 ceph-mon[33206]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:03:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:33 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:33 smithi119 ceph-mon[101432]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:03:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:33 smithi119 ceph-mon[105288]: osdmap e367: 8 total, 8 up, 8 in 2024-09-21T10:03:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:34 smithi145 ceph-mon[33206]: pgmap v176: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:34 smithi145 ceph-mon[33206]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:03:34.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:34 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:34 smithi119 ceph-mon[101432]: pgmap v176: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:34 smithi119 ceph-mon[101432]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:03:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:34 smithi119 ceph-mon[105288]: pgmap v176: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:34 smithi119 ceph-mon[105288]: osdmap e368: 8 total, 8 up, 8 in 2024-09-21T10:03:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:03:34] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:03:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:35 smithi145 ceph-mon[33206]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:03:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:35 smithi119 ceph-mon[101432]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:03:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:35 smithi119 ceph-mon[105288]: osdmap e369: 8 total, 8 up, 8 in 2024-09-21T10:03:36.019 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:36.019 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:36 smithi145 ceph-mon[33206]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-09-21T10:03:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:36 smithi145 ceph-mon[33206]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:03:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:36 smithi119 ceph-mon[101432]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-09-21T10:03:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:36 smithi119 ceph-mon[101432]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:03:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:36 smithi119 ceph-mon[105288]: pgmap v179: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-09-21T10:03:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:36 smithi119 ceph-mon[105288]: osdmap e370: 8 total, 8 up, 8 in 2024-09-21T10:03:37.061 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestGroup.add_snapshotPP (6035 ms) 2024-09-21T10:03:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:37 smithi145 ceph-mon[33206]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:03:37.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:36 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:37 smithi119 ceph-mon[105288]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:03:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:37 smithi119 ceph-mon[101432]: osdmap e371: 8 total, 8 up, 8 in 2024-09-21T10:03:38.057 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 6 tests from TestGroup (13515 ms total) 2024-09-21T10:03:38.057 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:03:38.057 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 31 tests from TestDeepCopy 2024-09-21T10:03:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:38 smithi145 ceph-mon[33206]: pgmap v182: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-09-21T10:03:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:38 smithi145 ceph-mon[33206]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:03:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:38 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:38 smithi119 ceph-mon[101432]: pgmap v182: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-09-21T10:03:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:38 smithi119 ceph-mon[101432]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:03:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:38.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:38 smithi119 ceph-mon[105288]: pgmap v182: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 119 KiB/s rd, 12 KiB/s wr, 161 op/s 2024-09-21T10:03:38.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:38 smithi119 ceph-mon[105288]: osdmap e372: 8 total, 8 up, 8 in 2024-09-21T10:03:38.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:39 smithi119 ceph-mon[105288]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:03:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:39 smithi119 ceph-mon[101432]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:03:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:39 smithi145 ceph-mon[33206]: osdmap e373: 8 total, 8 up, 8 in 2024-09-21T10:03:40.064 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Empty 2024-09-21T10:03:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[105288]: pgmap v185: 73 pgs: 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[105288]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:03:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2068046253' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[101432]: pgmap v185: 73 pgs: 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[101432]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:03:40.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2068046253' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:40.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:40 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:40 smithi145 ceph-mon[33206]: pgmap v185: 73 pgs: 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:40 smithi145 ceph-mon[33206]: osdmap e374: 8 total, 8 up, 8 in 2024-09-21T10:03:40.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:40 smithi145 ceph-mon[33206]: from='client.? 172.21.15.119:0/2068046253' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:40.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:40 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:03:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:41 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:41 smithi119 ceph-mon[101432]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:03:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:41 smithi119 ceph-mon[101432]: pgmap v188: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:41 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:41 smithi119 ceph-mon[105288]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:03:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:41 smithi119 ceph-mon[105288]: pgmap v188: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:41 smithi145 ceph-mon[33206]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:03:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:41 smithi145 ceph-mon[33206]: osdmap e375: 8 total, 8 up, 8 in 2024-09-21T10:03:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:41 smithi145 ceph-mon[33206]: pgmap v188: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:42.184 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Empty (2120 ms) 2024-09-21T10:03:42.184 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.NoSnaps 2024-09-21T10:03:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:42 smithi119 ceph-mon[105288]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:03:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:42 smithi119 ceph-mon[101432]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:03:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:42 smithi145 ceph-mon[33206]: osdmap e376: 8 total, 8 up, 8 in 2024-09-21T10:03:42.447 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.NoSnaps (263 ms) 2024-09-21T10:03:42.447 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Snaps 2024-09-21T10:03:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:43 smithi119 ceph-mon[105288]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:03:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:43 smithi119 ceph-mon[105288]: pgmap v191: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:43 smithi119 ceph-mon[101432]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:03:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:43 smithi119 ceph-mon[101432]: pgmap v191: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:43 smithi145 ceph-mon[33206]: osdmap e377: 8 total, 8 up, 8 in 2024-09-21T10:03:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:43 smithi145 ceph-mon[33206]: pgmap v191: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:44 smithi119 ceph-mon[105288]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:03:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:44 smithi119 ceph-mon[101432]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:03:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:44 smithi145 ceph-mon[33206]: osdmap e378: 8 total, 8 up, 8 in 2024-09-21T10:03:44.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:03:44] "GET /metrics HTTP/1.1" 200 36447 "" "Prometheus/2.51.0" 2024-09-21T10:03:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:45 smithi119 ceph-mon[105288]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:03:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:45 smithi119 ceph-mon[105288]: pgmap v194: 105 pgs: 105 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 146 op/s 2024-09-21T10:03:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:45 smithi119 ceph-mon[101432]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:03:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:45 smithi119 ceph-mon[101432]: pgmap v194: 105 pgs: 105 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 146 op/s 2024-09-21T10:03:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:45 smithi145 ceph-mon[33206]: osdmap e379: 8 total, 8 up, 8 in 2024-09-21T10:03:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:45 smithi145 ceph-mon[33206]: pgmap v194: 105 pgs: 105 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 146 op/s 2024-09-21T10:03:46.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:46.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:45.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:46.367 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Snaps (3919 ms) 2024-09-21T10:03:46.367 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.SnapDiscard 2024-09-21T10:03:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:46 smithi119 ceph-mon[101432]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:03:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:46 smithi119 ceph-mon[105288]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:03:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:46 smithi145 ceph-mon[33206]: osdmap e380: 8 total, 8 up, 8 in 2024-09-21T10:03:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:47 smithi145 ceph-mon[33206]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:03:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:47 smithi145 ceph-mon[33206]: pgmap v197: 105 pgs: 105 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 146 op/s 2024-09-21T10:03:47.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:46 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:46.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:47 smithi119 ceph-mon[101432]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:03:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:47 smithi119 ceph-mon[101432]: pgmap v197: 105 pgs: 105 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 146 op/s 2024-09-21T10:03:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:47 smithi119 ceph-mon[105288]: osdmap e381: 8 total, 8 up, 8 in 2024-09-21T10:03:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:47 smithi119 ceph-mon[105288]: pgmap v197: 105 pgs: 105 active+clean; 49 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 146 op/s 2024-09-21T10:03:48.218 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.SnapDiscard (1852 ms) 2024-09-21T10:03:48.219 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneDiscard 2024-09-21T10:03:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:48 smithi119 ceph-mon[105288]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:03:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:48 smithi119 ceph-mon[101432]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:03:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:48 smithi145 ceph-mon[33206]: osdmap e382: 8 total, 8 up, 8 in 2024-09-21T10:03:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:48 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:03:49.170 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneDiscard (951 ms) 2024-09-21T10:03:49.170 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneShrink 2024-09-21T10:03:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:49 smithi119 ceph-mon[105288]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:03:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:49 smithi119 ceph-mon[105288]: pgmap v200: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-21T10:03:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:49 smithi119 ceph-mon[105288]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:03:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:49 smithi119 ceph-mon[101432]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:03:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:49 smithi119 ceph-mon[101432]: pgmap v200: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-21T10:03:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:49 smithi119 ceph-mon[101432]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:03:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:49 smithi145 ceph-mon[33206]: osdmap e383: 8 total, 8 up, 8 in 2024-09-21T10:03:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:49 smithi145 ceph-mon[33206]: pgmap v200: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-21T10:03:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:49 smithi145 ceph-mon[33206]: osdmap e384: 8 total, 8 up, 8 in 2024-09-21T10:03:50.300 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneShrink (1130 ms) 2024-09-21T10:03:50.300 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneExpand 2024-09-21T10:03:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:51.351 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneExpand (1052 ms) 2024-09-21T10:03:51.352 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneHideParent 2024-09-21T10:03:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:51 smithi145 ceph-mon[33206]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:03:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:51 smithi145 ceph-mon[33206]: pgmap v203: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-21T10:03:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:51 smithi119 ceph-mon[101432]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:03:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:51 smithi119 ceph-mon[101432]: pgmap v203: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-21T10:03:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:51 smithi119 ceph-mon[105288]: osdmap e385: 8 total, 8 up, 8 in 2024-09-21T10:03:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:51 smithi119 ceph-mon[105288]: pgmap v203: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 24 MiB/s rd, 4.0 MiB/s wr, 164 op/s 2024-09-21T10:03:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:52 smithi145 ceph-mon[33206]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:03:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:52 smithi119 ceph-mon[105288]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:03:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:52 smithi119 ceph-mon[101432]: osdmap e386: 8 total, 8 up, 8 in 2024-09-21T10:03:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:53 smithi145 ceph-mon[33206]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:03:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:53 smithi145 ceph-mon[33206]: pgmap v206: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:53 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:53 smithi119 ceph-mon[105288]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:03:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:53 smithi119 ceph-mon[105288]: pgmap v206: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:53 smithi119 ceph-mon[101432]: osdmap e387: 8 total, 8 up, 8 in 2024-09-21T10:03:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:53 smithi119 ceph-mon[101432]: pgmap v206: 105 pgs: 105 active+clean; 65 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:03:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:03:54.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:54 smithi119 ceph-mon[105288]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:03:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:54 smithi119 ceph-mon[101432]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:03:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:54 smithi145 ceph-mon[33206]: osdmap e388: 8 total, 8 up, 8 in 2024-09-21T10:03:54.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:54.148Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:03:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:03:54] "GET /metrics HTTP/1.1" 200 36474 "" "Prometheus/2.51.0" 2024-09-21T10:03:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:55 smithi145 ceph-mon[33206]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:03:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:55 smithi145 ceph-mon[33206]: pgmap v209: 105 pgs: 105 active+clean; 65 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 393 KiB/s rd, 30 KiB/s wr, 528 op/s 2024-09-21T10:03:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:55 smithi119 ceph-mon[105288]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:03:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:55 smithi119 ceph-mon[105288]: pgmap v209: 105 pgs: 105 active+clean; 65 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 393 KiB/s rd, 30 KiB/s wr, 528 op/s 2024-09-21T10:03:55.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:55 smithi119 ceph-mon[101432]: osdmap e389: 8 total, 8 up, 8 in 2024-09-21T10:03:55.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:55 smithi119 ceph-mon[101432]: pgmap v209: 105 pgs: 105 active+clean; 65 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 393 KiB/s rd, 30 KiB/s wr, 528 op/s 2024-09-21T10:03:56.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:56.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:56 smithi145 ceph-mon[33206]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:03:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:56 smithi119 ceph-mon[105288]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:03:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:56 smithi119 ceph-mon[101432]: osdmap e390: 8 total, 8 up, 8 in 2024-09-21T10:03:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:57 smithi145 ceph-mon[33206]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:03:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:57 smithi145 ceph-mon[33206]: pgmap v212: 105 pgs: 105 active+clean; 65 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 393 KiB/s rd, 30 KiB/s wr, 528 op/s 2024-09-21T10:03:57.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:03:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:03:56.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:03:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:57 smithi119 ceph-mon[105288]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:03:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:57 smithi119 ceph-mon[105288]: pgmap v212: 105 pgs: 105 active+clean; 65 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 393 KiB/s rd, 30 KiB/s wr, 528 op/s 2024-09-21T10:03:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:57 smithi119 ceph-mon[101432]: osdmap e391: 8 total, 8 up, 8 in 2024-09-21T10:03:57.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:57 smithi119 ceph-mon[101432]: pgmap v212: 105 pgs: 105 active+clean; 65 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 393 KiB/s rd, 30 KiB/s wr, 528 op/s 2024-09-21T10:03:58.307 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneHideParent (6956 ms) 2024-09-21T10:03:58.307 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Clone 2024-09-21T10:03:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:58 smithi145 ceph-mon[33206]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:03:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:58 smithi119 ceph-mon[105288]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:03:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:58 smithi119 ceph-mon[101432]: osdmap e392: 8 total, 8 up, 8 in 2024-09-21T10:03:59.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:59 smithi119 ceph-mon[105288]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:03:59.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:03:59 smithi119 ceph-mon[105288]: pgmap v215: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 116 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-09-21T10:03:59.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:59 smithi119 ceph-mon[101432]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:03:59.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:03:59 smithi119 ceph-mon[101432]: pgmap v215: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 116 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-09-21T10:03:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:59 smithi145 ceph-mon[33206]: osdmap e393: 8 total, 8 up, 8 in 2024-09-21T10:03:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:03:59 smithi145 ceph-mon[33206]: pgmap v215: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 116 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-09-21T10:04:00.516 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Clone (2209 ms) 2024-09-21T10:04:00.516 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneFlatten 2024-09-21T10:04:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:00 smithi145 ceph-mon[33206]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:04:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:00 smithi119 ceph-mon[101432]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:04:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:00 smithi119 ceph-mon[105288]: osdmap e394: 8 total, 8 up, 8 in 2024-09-21T10:04:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:01 smithi145 ceph-mon[33206]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:04:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:01 smithi145 ceph-mon[33206]: pgmap v218: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 116 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-09-21T10:04:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:01 smithi119 ceph-mon[105288]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:04:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:01 smithi119 ceph-mon[105288]: pgmap v218: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 116 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-09-21T10:04:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:01 smithi119 ceph-mon[101432]: osdmap e395: 8 total, 8 up, 8 in 2024-09-21T10:04:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:01 smithi119 ceph-mon[101432]: pgmap v218: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 116 KiB/s rd, 4.0 MiB/s wr, 175 op/s 2024-09-21T10:04:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:02 smithi145 ceph-mon[33206]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:04:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:02 smithi119 ceph-mon[101432]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:04:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:02 smithi119 ceph-mon[105288]: osdmap e396: 8 total, 8 up, 8 in 2024-09-21T10:04:02.681 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneFlatten (2165 ms) 2024-09-21T10:04:02.681 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Stress 2024-09-21T10:04:02.704 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 19313968~7691544 2024-09-21T10:04:02.728 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 12967277~1531301 2024-09-21T10:04:02.728 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 7572948~7705512 2024-09-21T10:04:02.758 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 33551120~4753145 2024-09-21T10:04:02.758 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 58481122~8190699 2024-09-21T10:04:02.786 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 34063508~1988818 2024-09-21T10:04:02.786 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 37870928~1835435 2024-09-21T10:04:02.796 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 14930049~8174576 2024-09-21T10:04:02.799 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap0 2024-09-21T10:04:03.224 INFO:tasks.workunit.client.0.smithi119.stdout:clone image234 -> image235 2024-09-21T10:04:03.270 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 110113727 2024-09-21T10:04:03.274 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 64212477~1666585 2024-09-21T10:04:03.294 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 99992926~3320818 2024-09-21T10:04:03.295 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 33618978~6128922 2024-09-21T10:04:03.313 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 57015430~2498345 2024-09-21T10:04:03.334 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 60645349~6151832 2024-09-21T10:04:03.351 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 102759527~7354200 2024-09-21T10:04:03.351 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 83052826~378159 2024-09-21T10:04:03.354 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 16318953~2828926 2024-09-21T10:04:03.370 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:04:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:03 smithi145 ceph-mon[33206]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T10:04:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:03 smithi145 ceph-mon[33206]: pgmap v221: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:04:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:03 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:03 smithi119 ceph-mon[101432]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T10:04:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:03 smithi119 ceph-mon[101432]: pgmap v221: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:04:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:03 smithi119 ceph-mon[105288]: osdmap e397: 8 total, 8 up, 8 in 2024-09-21T10:04:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:03 smithi119 ceph-mon[105288]: pgmap v221: 105 pgs: 105 active+clean; 81 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:04:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:04.235 INFO:tasks.workunit.client.0.smithi119.stdout:clone image235 -> image236 2024-09-21T10:04:04.303 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 18549000~1321365 2024-09-21T10:04:04.313 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 53431550~1815357 2024-09-21T10:04:04.317 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 101819930~2804755 2024-09-21T10:04:04.329 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 30163429~2917667 2024-09-21T10:04:04.332 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 84744628~2110401 2024-09-21T10:04:04.340 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 11490361~1271099 2024-09-21T10:04:04.360 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 100415511~2132323 2024-09-21T10:04:04.369 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 10550304~7956055 2024-09-21T10:04:04.370 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:04:04.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:04 smithi119 ceph-mon[101432]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T10:04:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:04 smithi119 ceph-mon[105288]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T10:04:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:04 smithi145 ceph-mon[33206]: osdmap e398: 8 total, 8 up, 8 in 2024-09-21T10:04:04.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:04:04] "GET /metrics HTTP/1.1" 200 36404 "" "Prometheus/2.51.0" 2024-09-21T10:04:05.231 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 182846254 2024-09-21T10:04:05.237 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 37231427~6995281 2024-09-21T10:04:05.240 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 77136494~5510534 2024-09-21T10:04:05.241 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 152947741~939465 2024-09-21T10:04:05.241 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 56179702~60055 2024-09-21T10:04:05.242 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 48790047~1248477 2024-09-21T10:04:05.242 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 124108499~3261059 2024-09-21T10:04:05.245 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 160181306~6338248 2024-09-21T10:04:05.246 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 31137186~5077101 2024-09-21T10:04:05.289 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap3 2024-09-21T10:04:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:05 smithi145 ceph-mon[33206]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T10:04:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:05 smithi145 ceph-mon[33206]: pgmap v224: 105 pgs: 105 active+clean; 261 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s rd, 43 MiB/s wr, 623 op/s 2024-09-21T10:04:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:05 smithi119 ceph-mon[101432]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T10:04:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:05 smithi119 ceph-mon[101432]: pgmap v224: 105 pgs: 105 active+clean; 261 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s rd, 43 MiB/s wr, 623 op/s 2024-09-21T10:04:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:05 smithi119 ceph-mon[105288]: osdmap e399: 8 total, 8 up, 8 in 2024-09-21T10:04:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:05 smithi119 ceph-mon[105288]: pgmap v224: 105 pgs: 105 active+clean; 261 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s rd, 43 MiB/s wr, 623 op/s 2024-09-21T10:04:06.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:05.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:06.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:05.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:06.247 INFO:tasks.workunit.client.0.smithi119.stdout:clone image236 -> image237 2024-09-21T10:04:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:06 smithi145 ceph-mon[33206]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T10:04:06.607 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Stress (3926 ms) 2024-09-21T10:04:06.607 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.NoSnaps_LargerDstObjSize 2024-09-21T10:04:06.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:06 smithi119 ceph-mon[101432]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T10:04:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:06 smithi119 ceph-mon[105288]: osdmap e400: 8 total, 8 up, 8 in 2024-09-21T10:04:06.848 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.NoSnaps_LargerDstObjSize (241 ms) 2024-09-21T10:04:06.848 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Snaps_LargerDstObjSize 2024-09-21T10:04:07.231 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:07 smithi145 ceph-mon[33206]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T10:04:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:07 smithi145 ceph-mon[33206]: pgmap v227: 105 pgs: 105 active+clean; 261 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s rd, 43 MiB/s wr, 623 op/s 2024-09-21T10:04:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:07 smithi119 ceph-mon[101432]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T10:04:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:07 smithi119 ceph-mon[101432]: pgmap v227: 105 pgs: 105 active+clean; 261 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s rd, 43 MiB/s wr, 623 op/s 2024-09-21T10:04:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:07 smithi119 ceph-mon[105288]: osdmap e401: 8 total, 8 up, 8 in 2024-09-21T10:04:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:07 smithi119 ceph-mon[105288]: pgmap v227: 105 pgs: 105 active+clean; 261 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 MiB/s rd, 43 MiB/s wr, 623 op/s 2024-09-21T10:04:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:08 smithi145 ceph-mon[33206]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:04:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:08 smithi119 ceph-mon[101432]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:04:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:08 smithi119 ceph-mon[105288]: osdmap e402: 8 total, 8 up, 8 in 2024-09-21T10:04:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:09 smithi145 ceph-mon[33206]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:04:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:09 smithi145 ceph-mon[33206]: pgmap v230: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 MiB/s rd, 21 MiB/s wr, 529 op/s 2024-09-21T10:04:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:09 smithi119 ceph-mon[101432]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:04:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:09 smithi119 ceph-mon[101432]: pgmap v230: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 MiB/s rd, 21 MiB/s wr, 529 op/s 2024-09-21T10:04:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:09 smithi119 ceph-mon[105288]: osdmap e403: 8 total, 8 up, 8 in 2024-09-21T10:04:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:09 smithi119 ceph-mon[105288]: pgmap v230: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 MiB/s rd, 21 MiB/s wr, 529 op/s 2024-09-21T10:04:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:10 smithi145 ceph-mon[33206]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:04:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:10.565 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Snaps_LargerDstObjSize (3717 ms) 2024-09-21T10:04:10.565 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Clone_LargerDstObjSize 2024-09-21T10:04:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:10 smithi119 ceph-mon[101432]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:04:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:10 smithi119 ceph-mon[105288]: osdmap e404: 8 total, 8 up, 8 in 2024-09-21T10:04:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:11 smithi145 ceph-mon[33206]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:04:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:11 smithi145 ceph-mon[33206]: pgmap v233: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 MiB/s rd, 21 MiB/s wr, 529 op/s 2024-09-21T10:04:11.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:11 smithi119 ceph-mon[105288]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:04:11.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:11 smithi119 ceph-mon[105288]: pgmap v233: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 MiB/s rd, 21 MiB/s wr, 529 op/s 2024-09-21T10:04:11.597 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:11 smithi119 ceph-mon[101432]: osdmap e405: 8 total, 8 up, 8 in 2024-09-21T10:04:11.597 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:11 smithi119 ceph-mon[101432]: pgmap v233: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 34 MiB/s rd, 21 MiB/s wr, 529 op/s 2024-09-21T10:04:11.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T10:04:12.223 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:12.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:12 smithi119 ceph-mon[101432]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:04:12.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:12.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:12 smithi119 ceph-mon[105288]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:04:12.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:12 smithi145 ceph-mon[33206]: osdmap e406: 8 total, 8 up, 8 in 2024-09-21T10:04:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:12 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:12.613 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Clone_LargerDstObjSize (2047 ms) 2024-09-21T10:04:12.613 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneFlatten_LargerDstObjSize 2024-09-21T10:04:13.488 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:13 smithi119 ceph-mon[101432]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:04:13.489 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:13 smithi119 ceph-mon[101432]: pgmap v236: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:04:13.489 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:13 smithi119 ceph-mon[105288]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:04:13.489 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:13 smithi119 ceph-mon[105288]: pgmap v236: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:04:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:13 smithi145 ceph-mon[33206]: osdmap e407: 8 total, 8 up, 8 in 2024-09-21T10:04:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:13 smithi145 ceph-mon[33206]: pgmap v236: 105 pgs: 105 active+clean; 366 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:04:14.269 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:14 smithi119 ceph-mon[101432]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:04:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:14 smithi119 ceph-mon[105288]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:04:14.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:14 smithi145 ceph-mon[33206]: osdmap e408: 8 total, 8 up, 8 in 2024-09-21T10:04:14.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:14.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:14.764 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneFlatten_LargerDstObjSize (2151 ms) 2024-09-21T10:04:14.764 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Stress_LargerDstObjSize 2024-09-21T10:04:14.788 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 48828533~3020039 2024-09-21T10:04:14.800 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 77325401~6560679 2024-09-21T10:04:14.800 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 48071166~1434569 2024-09-21T10:04:14.804 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 47874545~2216896 2024-09-21T10:04:14.805 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 35130029~3435887 2024-09-21T10:04:14.815 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 22186976~4538179 2024-09-21T10:04:14.816 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 29385899~7086131 2024-09-21T10:04:14.833 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 5505682~6349226 2024-09-21T10:04:14.833 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap0 2024-09-21T10:04:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:04:14] "GET /metrics HTTP/1.1" 200 36404 "" "Prometheus/2.51.0" 2024-09-21T10:04:14.914 INFO:teuthology.orchestra.run.smithi119.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:15.302 INFO:tasks.workunit.client.0.smithi119.stdout:clone image256 -> image257 2024-09-21T10:04:15.344 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 34462837~6754159 2024-09-21T10:04:15.368 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 68736159~3663487 2024-09-21T10:04:15.372 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 66460739~7246653 2024-09-21T10:04:15.394 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 56610174~6704396 2024-09-21T10:04:15.398 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 47660362~6233626 2024-09-21T10:04:15.417 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 28625236~8228065 2024-09-21T10:04:15.421 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 11248105~6840650 2024-09-21T10:04:15.440 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 13401329~1956298 2024-09-21T10:04:15.441 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:04:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-mon[33206]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:04:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-mon[33206]: pgmap v239: 105 pgs: 105 active+clean; 446 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 41 MiB/s rd, 20 MiB/s wr, 417 op/s 2024-09-21T10:04:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-mon[33206]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-mon[33206]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[105288]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:04:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[105288]: pgmap v239: 105 pgs: 105 active+clean; 446 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 41 MiB/s rd, 20 MiB/s wr, 417 op/s 2024-09-21T10:04:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[105288]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[105288]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[101432]: osdmap e409: 8 total, 8 up, 8 in 2024-09-21T10:04:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[101432]: pgmap v239: 105 pgs: 105 active+clean; 446 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 41 MiB/s rd, 20 MiB/s wr, 417 op/s 2024-09-21T10:04:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[101432]: from='client.34245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[101432]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:16.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:15.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:16.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:15.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:16.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for all mons to upgrade' 2024-09-21T10:04:16.310 INFO:tasks.workunit.client.0.smithi119.stdout:clone image257 -> image258 2024-09-21T10:04:16.374 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 145358575 2024-09-21T10:04:16.380 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 23347350~1840147 2024-09-21T10:04:16.393 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 76160978~6188126 2024-09-21T10:04:16.397 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 131922529~5471944 2024-09-21T10:04:16.415 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 96989793~2113268 2024-09-21T10:04:16.415 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 74991597~1169897 2024-09-21T10:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 130285575~5583541 2024-09-21T10:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 137957435~5984502 2024-09-21T10:04:16.447 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 91761659~5235812 2024-09-21T10:04:16.447 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:04:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:16 smithi145 ceph-mon[33206]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:04:16.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:16 smithi119 ceph-mon[101432]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:04:16.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:16 smithi119 ceph-mon[105288]: osdmap e410: 8 total, 8 up, 8 in 2024-09-21T10:04:17.080 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:17.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:16.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:17.316 INFO:tasks.workunit.client.0.smithi119.stdout:clone image258 -> image259 2024-09-21T10:04:17.384 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 132152378 2024-09-21T10:04:17.392 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 10312481~3239909 2024-09-21T10:04:17.416 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 4890301~3256300 2024-09-21T10:04:17.419 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 55789066~5892117 2024-09-21T10:04:17.439 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 53922155~5938864 2024-09-21T10:04:17.458 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 101840295~6721064 2024-09-21T10:04:17.475 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 67939143~2850675 2024-09-21T10:04:17.493 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 20561656~7402263 2024-09-21T10:04:17.521 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 124879556~7272822 2024-09-21T10:04:17.524 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap3 2024-09-21T10:04:17.557 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[101432]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[101432]: pgmap v242: 105 pgs: 105 active+clean; 446 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 41 MiB/s rd, 20 MiB/s wr, 417 op/s 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[105288]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[105288]: pgmap v242: 105 pgs: 105 active+clean; 446 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 41 MiB/s rd, 20 MiB/s wr, 417 op/s 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:17.558 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:17 smithi145 ceph-mon[33206]: osdmap e411: 8 total, 8 up, 8 in 2024-09-21T10:04:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:17 smithi145 ceph-mon[33206]: pgmap v242: 105 pgs: 105 active+clean; 446 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 41 MiB/s rd, 20 MiB/s wr, 417 op/s 2024-09-21T10:04:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:17 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:17 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:18.316 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 193095536 2024-09-21T10:04:18.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:18 smithi119 ceph-mon[101432]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:04:18.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:18.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:18 smithi119 ceph-mon[105288]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:04:18.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:18 smithi145 ceph-mon[33206]: osdmap e412: 8 total, 8 up, 8 in 2024-09-21T10:04:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:18 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:19.517 INFO:teuthology.orchestra.run.smithi119.stdout:wait for all mons to upgrade 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: pgmap v245: 105 pgs: 105 active+clean; 645 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 45 MiB/s wr, 611 op/s 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: pgmap v245: 105 pgs: 105 active+clean; 645 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 45 MiB/s wr, 611 op/s 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:19.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:04:19.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T10:04:19.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-21T10:04:19.628 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: osdmap e413: 8 total, 8 up, 8 in 2024-09-21T10:04:19.628 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: pgmap v245: 105 pgs: 105 active+clean; 645 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 45 MiB/s wr, 611 op/s 2024-09-21T10:04:19.628 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.628 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.628 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:19.629 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:19.629 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:19.629 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:19.629 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:04:19.629 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "quorum_status"} : dispatch 2024-09-21T10:04:19.629 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:19 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon ok-to-stop", "ids": ["b"]} : dispatch 2024-09-21T10:04:19.844 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Stress_LargerDstObjSize (5081 ms) 2024-09-21T10:04:19.845 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.NoSnaps_SmallerDstObjSize 2024-09-21T10:04:20.060 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.NoSnaps_SmallerDstObjSize (216 ms) 2024-09-21T10:04:20.060 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Snaps_SmallerDstObjSize 2024-09-21T10:04:20.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph mon versions | grep $sha1 | grep '"'"': 3'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T10:04:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:20.476 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:20.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:20 smithi119 ceph-mon[105288]: Upgrade: It appears safe to stop mon.b 2024-09-21T10:04:20.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:20 smithi119 ceph-mon[105288]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:04:20.598 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:20 smithi119 ceph-mon[101432]: Upgrade: It appears safe to stop mon.b 2024-09-21T10:04:20.598 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:20 smithi119 ceph-mon[101432]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:04:20.748 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:20 smithi145 ceph-mon[33206]: Upgrade: It appears safe to stop mon.b 2024-09-21T10:04:20.748 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:20 smithi145 ceph-mon[33206]: osdmap e414: 8 total, 8 up, 8 in 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: pgmap v248: 105 pgs: 105 active+clean; 645 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 45 MiB/s wr, 611 op/s 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: Upgrade: Updating mon.b 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:21.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[101432]: Deploying daemon mon.b on smithi145 2024-09-21T10:04:21.434 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: pgmap v248: 105 pgs: 105 active+clean; 645 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 45 MiB/s wr, 611 op/s 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: Upgrade: Updating mon.b 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:21.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:21 smithi145 ceph-mon[33206]: Deploying daemon mon.b on smithi145 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: osdmap e415: 8 total, 8 up, 8 in 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: pgmap v248: 105 pgs: 105 active+clean; 645 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 37 MiB/s rd, 45 MiB/s wr, 611 op/s 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: Upgrade: Updating mon.b 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:21 smithi119 ceph-mon[105288]: Deploying daemon mon.b on smithi145 2024-09-21T10:04:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:22 smithi145 ceph-mon[33206]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:04:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:22 smithi119 ceph-mon[105288]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:04:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:22 smithi119 ceph-mon[101432]: osdmap e416: 8 total, 8 up, 8 in 2024-09-21T10:04:22.755 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:04:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:22 smithi145 systemd[1]: Stopping Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:04:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b[33175]: 2024-09-21T10:04:23.145+0000 7f46dcfd3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:04:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b[33175]: 2024-09-21T10:04:23.145+0000 7f46dcfd3640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T10:04:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:23 smithi145 podman[89167]: 2024-09-21 10:04:23.288639362 +0000 UTC m=+0.591693171 container died f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:04:24.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:23 smithi145 podman[89167]: 2024-09-21 10:04:23.892720549 +0000 UTC m=+1.195774412 container cleanup f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T10:04:24.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:23 smithi145 bash[89167]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b 2024-09-21T10:04:24.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 podman[89184]: 2024-09-21 10:04:24.093933257 +0000 UTC m=+0.806433472 container remove f45b53ab247641533cc8a9667a3f155e2e3591fcdfd7110348b32f6aa4c3e260 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a) 2024-09-21T10:04:24.534 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:24 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:24.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:24.534 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-21T10:04:24.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 systemd[1]: Stopped Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:04:24.535 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.b.service: Consumed 11.089s CPU time. 2024-09-21T10:04:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 systemd[1]: Starting Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:04:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 podman[89321]: 2024-09-21 10:04:24.64170701 +0000 UTC m=+0.015724055 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:24 smithi145 podman[89321]: 2024-09-21 10:04:24.753221001 +0000 UTC m=+0.127238046 container create 550baecb68e011536364fe641567a51415adcd8cd4f9aa34b5b4313c6c508c09 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T10:04:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:04:24] "GET /metrics HTTP/1.1" 200 36485 "" "Prometheus/2.51.0" 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 podman[89321]: 2024-09-21 10:04:25.005624029 +0000 UTC m=+0.379641074 container init 550baecb68e011536364fe641567a51415adcd8cd4f9aa34b5b4313c6c508c09 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 podman[89321]: 2024-09-21 10:04:25.009875462 +0000 UTC m=+0.383892508 container start 550baecb68e011536364fe641567a51415adcd8cd4f9aa34b5b4313c6c508c09 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev), process ceph-mon, pid 7 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: pidfile_write: ignore empty --pid-file 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: load: jerasure load: lrc 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: RocksDB version: 7.9.2 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Git sha 0 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Compile date 2024-09-20 20:32:28 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: DB SUMMARY 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: DB Session ID: J3364XG8EJCVL1G0A6AP 2024-09-21T10:04:25.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: CURRENT file: CURRENT 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1087 Bytes 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1164926 ; 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.error_if_exists: 0 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.create_if_missing: 0 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.paranoid_checks: 1 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.env: 0x563d80df3120 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.info_log: 0x563d82a4f700 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T10:04:25.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.statistics: (nil) 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.use_fsync: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_log_file_size: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.allow_fallocate: 1 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.use_direct_reads: 0 2024-09-21T10:04:25.298 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.db_log_dir: 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.wal_dir: 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.write_buffer_manager: 0x563d82a521e0 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T10:04:25.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.unordered_write: 0 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.row_cache: None 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.wal_filter: None 2024-09-21T10:04:25.300 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.two_write_queues: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.wal_compression: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.atomic_flush: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.log_readahead_size: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T10:04:25.301 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_background_jobs: 2 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_background_compactions: -1 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_subcompactions: 1 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T10:04:25.302 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_open_files: -1 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_background_flushes: -1 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Compression algorithms supported: 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kZSTD supported: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kXpressCompression supported: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kBZip2Compression supported: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kLZ4Compression supported: 1 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kZlibCompression supported: 1 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T10:04:25.303 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: kSnappyCompression supported: 1 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.merge_operator: 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_filter: None 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d82a4f320) 2024-09-21T10:04:25.304 INFO:journalctl@ceph.mon.b.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: index_type: 0 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: data_block_index_type: 0 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: index_shortening: 1 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: checksum: 4 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: no_block_cache: 0 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache: 0x563d82a43610 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache_options: 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: capacity : 536870912 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: num_shard_bits : 4 2024-09-21T10:04:25.305 INFO:journalctl@ceph.mon.b.smithi145.stdout: strict_capacity_limit : 0 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_cache_compressed: (nil) 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: persistent_cache: (nil) 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_size: 4096 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_size_deviation: 10 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_restart_interval: 16 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: index_block_restart_interval: 1 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: metadata_block_size: 4096 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: partition_filters: 0 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: use_delta_encoding: 1 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: filter_policy: bloomfilter 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: whole_key_filtering: 1 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: verify_compression: 0 2024-09-21T10:04:25.306 INFO:journalctl@ceph.mon.b.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: format_version: 5 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: enable_index_compression: 1 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: block_align: 0 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: max_auto_readahead_size: 262144 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: prepopulate_block_cache: 0 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: initial_auto_readahead_size: 8192 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression: NoCompression 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T10:04:25.307 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.num_levels: 7 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T10:04:25.308 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T10:04:25.309 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T10:04:25.310 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.inplace_update_support: 0 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T10:04:25.311 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.bloom_locality: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.max_successive_merges: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.ttl: 2592000 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T10:04:25.312 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.enable_blob_files: false 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.min_blob_size: 0 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 19476, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-21T10:04:25.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 79669804-cb43-4794-a416-e99ba43af622 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726913065037392, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726913065042797, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 768264, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 19481, "largest_seqno": 20075, "table_properties": {"data_size": 764466, "index_size": 2118, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 5268, "raw_average_key_size": 22, "raw_value_size": 758795, "raw_average_value_size": 3284, "num_data_blocks": 91, "num_entries": 231, "num_filter_entries": 231, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726913065, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "79669804-cb43-4794-a416-e99ba43af622", "db_session_id": "J3364XG8EJCVL1G0A6AP", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726913065042945, "job": 1, "event": "recovery_finished"} 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563d82a72e00 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: DB pointer 0x563d82b74000 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** DB Stats ** 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:04:25.314 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** Compaction Stats [default] ** 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: L0 1/0 750.26 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 269.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: L6 1/0 13.30 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Sum 2/0 14.03 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 269.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 269.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T10:04:25.315 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.316 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** Compaction Stats [default] ** 2024-09-21T10:04:25.316 INFO:journalctl@ceph.mon.b.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T10:04:25.316 INFO:journalctl@ceph.mon.b.smithi145.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T10:04:25.317 INFO:journalctl@ceph.mon.b.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 269.5 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-09-21T10:04:25.317 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.317 INFO:journalctl@ceph.mon.b.smithi145.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T10:04:25.317 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.317 INFO:journalctl@ceph.mon.b.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: Cumulative compaction: 0.00 GB write, 90.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: Interval compaction: 0.00 GB write, 90.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: Block cache BinnedLRUCache@0x563d82a43610#7 capacity: 512.00 MB usage: 3.05 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.72 KB,0.000137091%) IndexBlock(1,2.33 KB,0.000444055%) Misc(1,0.00 KB,0%) 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: starting mon.b rank 1 at public addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] at bind addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???) e3 preinit fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:25.318 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).mds e9 new map 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).mds e9 print_map 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: e9 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: legacy client fscid: 1 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: Filesystem 'foo' (1) 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: fs_name foo 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: epoch 8 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: created 2024-09-21T09:56:59.994919+0000 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: modified 2024-09-21T09:57:18.635502+0000 2024-09-21T10:04:25.319 INFO:journalctl@ceph.mon.b.smithi145.stdout: tableserver 0 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: root 0 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: session_timeout 60 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: session_autoclose 300 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: max_file_size 1099511627776 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: max_xattr_size 65536 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: required_client_features {} 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: last_failure 0 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: last_failure_osd_epoch 0 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: max_mds 1 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: in 0 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: up {0=24557} 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: failed 2024-09-21T10:04:25.320 INFO:journalctl@ceph.mon.b.smithi145.stdout: damaged 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: stopped 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: data_pools [4] 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: metadata_pool 3 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: inline_data disabled 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: balancer 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: bal_rank_mask -1 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: standby_count_wanted 1 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: qdb_cluster leader: 0 members: 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: [mds.foo.smithi119.oikglq{0:24557} state up:active seq 3 join_fscid=1 addr [v2:172.21.15.119:6834/4150717344,v1:172.21.15.119:6835/4150717344] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: Standby daemons: 2024-09-21T10:04:25.321 INFO:journalctl@ceph.mon.b.smithi145.stdout: 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout: [mds.foo.smithi145.tjomfr{-1:24457} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.145:6832/972108081,v1:172.21.15.145:6833/972108081] compat {c=[1],r=[1],i=[7ff]}] 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).osd e417 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).osd e417 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).osd e417 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).osd e417 crush map has features 432629239337189376, adjusting msgr requires 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-mon[89359]: mon.b@-1(???).paxosservice(auth 1..21) refresh upgraded, format 0 -> 3 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 bash[89321]: 550baecb68e011536364fe641567a51415adcd8cd4f9aa34b5b4313c6c508c09 2024-09-21T10:04:25.322 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:25 smithi145 systemd[1]: Started Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:04:25.674 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:04:26.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:25.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:26.045 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:25 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:25.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:27.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:26 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:26.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:30.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:30 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:04:30.053+0000 7fc0be6c7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T10:04:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:04:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:04:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:04:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: mon.a calling monitor election 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: mon.b calling monitor election 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: pgmap v253: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 45 MiB/s rd, 16 MiB/s wr, 176 op/s 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: pgmap v254: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 35 MiB/s rd, 12 MiB/s wr, 134 op/s 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: pgmap v255: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 11 MiB/s wr, 118 op/s 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: monmap epoch 4 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: last_changed 2024-09-21T10:04:26.345994+0000 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: min_mon_release 19 (squid) 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: election_strategy: 1 2024-09-21T10:04:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: mon.c (rank 2) addr [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] is down (out of quorum) 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:31.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:04:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:04:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: mon.a calling monitor election 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: mon.b calling monitor election 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: pgmap v253: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 45 MiB/s rd, 16 MiB/s wr, 176 op/s 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: pgmap v254: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 35 MiB/s rd, 12 MiB/s wr, 134 op/s 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: pgmap v255: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 11 MiB/s wr, 118 op/s 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: monmap epoch 4 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: last_changed 2024-09-21T10:04:26.345994+0000 2024-09-21T10:04:31.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: min_mon_release 19 (squid) 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: election_strategy: 1 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T10:04:31.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T10:04:31.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T10:04:31.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: mon.c (rank 2) addr [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] is down (out of quorum) 2024-09-21T10:04:31.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:31.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:31.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:31.920 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:04:32.479 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1011688271' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:04:32.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:04:32.754 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:32.770 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1011688271' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:04:32.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-21T10:04:32.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-21T10:04:32.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-21T10:04:32.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: mon.a calling monitor election 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: mon.b calling monitor election 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: pgmap v253: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 45 MiB/s rd, 16 MiB/s wr, 176 op/s 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: pgmap v254: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 35 MiB/s rd, 12 MiB/s wr, 134 op/s 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: pgmap v255: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 30 MiB/s rd, 11 MiB/s wr, 118 op/s 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: monmap epoch 4 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: last_changed 2024-09-21T10:04:26.345994+0000 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: min_mon_release 19 (squid) 2024-09-21T10:04:32.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: election_strategy: 1 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: mgrmap e38: y(active, since 2m), standbys: x 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: mon.c (rank 2) addr [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] is down (out of quorum) 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:32.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:04:32.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1011688271' entity='client.admin' cmd={"prefix": "mon versions"} : dispatch 2024-09-21T10:04:33.659 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Snaps_SmallerDstObjSize (13599 ms) 2024-09-21T10:04:33.659 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Clone_SmallerDstObjSize 2024-09-21T10:04:33.783 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: mon.c calling monitor election 2024-09-21T10:04:33.783 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: pgmap v256: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 27 MiB/s rd, 9.7 MiB/s wr, 106 op/s 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: mon.b calling monitor election 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: mon.a calling monitor election 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: monmap epoch 4 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: last_changed 2024-09-21T10:04:26.345994+0000 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: min_mon_release 19 (squid) 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: election_strategy: 1 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:04:33.784 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:04:33.785 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:04:33.785 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: mgrmap e38: y(active, since 3m), standbys: x 2024-09-21T10:04:33.785 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T10:04:33.785 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:04:33.785 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: overall HEALTH_OK 2024-09-21T10:04:33.785 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: mon.c calling monitor election 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: pgmap v256: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 27 MiB/s rd, 9.7 MiB/s wr, 106 op/s 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: mon.b calling monitor election 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: mon.a calling monitor election 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: monmap epoch 4 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: last_changed 2024-09-21T10:04:26.345994+0000 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: min_mon_release 19 (squid) 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: election_strategy: 1 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: mgrmap e38: y(active, since 3m), standbys: x 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: overall HEALTH_OK 2024-09-21T10:04:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: mon.c calling monitor election 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: pgmap v256: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 27 MiB/s rd, 9.7 MiB/s wr, 106 op/s 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: mon.b calling monitor election 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: mon.a calling monitor election 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: monmap epoch 4 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: last_changed 2024-09-21T10:04:26.345994+0000 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: created 2024-09-21T09:50:05.190645+0000 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: min_mon_release 19 (squid) 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: election_strategy: 1 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: 0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: 1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-21T10:04:33.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: 2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: fsmap foo:1 {0=foo.smithi119.oikglq=up:active} 1 up:standby 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: osdmap e417: 8 total, 8 up, 8 in 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: mgrmap e38: y(active, since 3m), standbys: x 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: overall HEALTH_OK 2024-09-21T10:04:33.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:34.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:34 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:34.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:34.439 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:34 smithi119 ceph-mon[101432]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:alertmanager.a smithi119 *:9093,9094 running (4m) 16s ago 8m 18.5M - 0.27.0 11f11916f8cd e4cf1f6d8355 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:grafana.a smithi145 *:3000 running (4m) 17s ago 8m 81.5M - 9.4.7 954c08fa6188 e033bb16a7cd 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi119.oikglq smithi119 running (7m) 16s ago 7m 18.7M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4ff2238155bf 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi145.tjomfr smithi145 running (7m) 17s ago 7m 15.6M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b1338cf49871 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi145 *:8443,9283,8765 running (3m) 17s ago 12m 454M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 1346375dd747 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi119 *:8443,9283,8765 running (3m) 16s ago 14m 511M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d7bcb26ddb62 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (3m) 16s ago 14m 47.7M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 030d68226d6c 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi145 starting - - - 2048M 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi119 running (96s) 16s ago 12m 38.9M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d747bb0bb662 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.a smithi119 *:9100 running (4m) 16s ago 8m 10.3M - 1.7.0 72c9c2088986 05912f78c070 2024-09-21T10:04:34.581 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.b smithi145 *:9100 running (3m) 17s ago 8m 11.3M - 1.7.0 72c9c2088986 d906bafb8d86 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (12m) 16s ago 12m 117M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be ce07d9950c3f 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (11m) 16s ago 11m 125M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b8f7b7d5ac64 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (11m) 16s ago 11m 155M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 87104f5801c3 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (11m) 16s ago 11m 176M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 700faab8401d 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi145 running (10m) 17s ago 10m 179M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 18086eb220ec 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi145 running (10m) 17s ago 10m 139M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 90ab6d4f8571 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi145 running (9m) 17s ago 9m 132M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 772774e9be4a 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi145 running (9m) 17s ago 9m 113M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1a18c8e91654 2024-09-21T10:04:34.582 INFO:teuthology.orchestra.run.smithi119.stdout:prometheus.a smithi145 *:9095 running (3m) 17s ago 8m 54.0M - 2.51.0 1d3b7f56885b d2de6cb75d75 2024-09-21T10:04:34.713 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:34 smithi145 ceph-mon[89359]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:04:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:34 smithi119 ceph-mon[105288]: osdmap e418: 8 total, 8 up, 8 in 2024-09-21T10:04:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:04:34] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-09-21T10:04:35.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T10:04:35.438 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:35 smithi119 ceph-mon[101432]: pgmap v258: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:35.438 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:35 smithi119 ceph-mon[101432]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:04:35.438 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:35 smithi119 ceph-mon[101432]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:35.443 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:35 smithi145 ceph-mon[89359]: pgmap v258: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:35 smithi145 ceph-mon[89359]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:04:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:35 smithi145 ceph-mon[89359]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:35.744 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Clone_SmallerDstObjSize (2085 ms) 2024-09-21T10:04:35.744 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneFlatten_SmallerDstObjSize 2024-09-21T10:04:35.745 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:35 smithi119 ceph-mon[105288]: pgmap v258: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:35.745 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:35 smithi119 ceph-mon[105288]: osdmap e419: 8 total, 8 up, 8 in 2024-09-21T10:04:35.745 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:35 smithi119 ceph-mon[105288]: from='client.34293 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:36.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:35.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:36.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:35.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[105288]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:04:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[101432]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:04:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.736 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:36 smithi145 ceph-mon[89359]: osdmap e420: 8 total, 8 up, 8 in 2024-09-21T10:04:36.736 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.736 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.736 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:36.736 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:37.005 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:36 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:36.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:37.414 INFO:teuthology.orchestra.run.smithi119.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:38.030 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:37 smithi145 ceph-mon[89359]: pgmap v261: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:38.030 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:37 smithi145 ceph-mon[89359]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:04:38.030 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:37 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:37 smithi119 ceph-mon[105288]: pgmap v261: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:37 smithi119 ceph-mon[105288]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:04:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:37 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:37 smithi119 ceph-mon[101432]: pgmap v261: 105 pgs: 105 active+clean; 752 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:37 smithi119 ceph-mon[101432]: osdmap e421: 8 total, 8 up, 8 in 2024-09-21T10:04:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:37 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:38.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'sleep 60' 2024-09-21T10:04:38.210 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneFlatten_SmallerDstObjSize (2466 ms) 2024-09-21T10:04:38.210 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Stress_SmallerDstObjSize 2024-09-21T10:04:38.233 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 76131352~7754728 2024-09-21T10:04:38.259 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 5858066~1368906 2024-09-21T10:04:38.259 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 50923743~1682919 2024-09-21T10:04:38.267 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 23047822~2819449 2024-09-21T10:04:38.268 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 39613791~6519681 2024-09-21T10:04:38.288 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 76884845~314703 2024-09-21T10:04:38.290 INFO:tasks.workunit.client.0.smithi119.stdout:write: A 39761252~750118 2024-09-21T10:04:38.294 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 73797846~3095715 2024-09-21T10:04:38.295 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap0 2024-09-21T10:04:38.438 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:04:38.792 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 163246041 2024-09-21T10:04:38.799 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 47110427~128695 2024-09-21T10:04:38.801 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 33060739~2876144 2024-09-21T10:04:38.801 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 123409686~1397081 2024-09-21T10:04:38.801 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 49415394~4780911 2024-09-21T10:04:38.802 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 82449570~1459605 2024-09-21T10:04:38.802 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 74332370~6753964 2024-09-21T10:04:38.803 INFO:tasks.workunit.client.0.smithi119.stdout:write: B 149125557~6318218 2024-09-21T10:04:38.804 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 61963593~6601522 2024-09-21T10:04:38.828 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:04:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:38 smithi119 ceph-mon[101432]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:38 smithi119 ceph-mon[101432]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:38 smithi119 ceph-mon[101432]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:04:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:38 smithi119 ceph-mon[105288]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:38 smithi119 ceph-mon[105288]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:38 smithi119 ceph-mon[105288]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:04:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:38 smithi145 ceph-mon[89359]: from='client.44186 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:04:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:38 smithi145 ceph-mon[89359]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:04:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:38 smithi145 ceph-mon[89359]: osdmap e422: 8 total, 8 up, 8 in 2024-09-21T10:04:39.809 INFO:tasks.workunit.client.0.smithi119.stdout:clone image278 -> image279 2024-09-21T10:04:39.858 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 7292107~6607125 2024-09-21T10:04:39.882 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 134633115~5324691 2024-09-21T10:04:39.886 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 156499039~3080691 2024-09-21T10:04:39.900 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 42297604~5027994 2024-09-21T10:04:39.909 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 105982554~1419171 2024-09-21T10:04:39.916 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 67420978~6565531 2024-09-21T10:04:39.920 INFO:tasks.workunit.client.0.smithi119.stdout:write: C 1623768~7190858 2024-09-21T10:04:39.938 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 14593121~5360210 2024-09-21T10:04:39.941 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:04:39.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: pgmap v264: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 44 MiB/s wr, 702 op/s 2024-09-21T10:04:39.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:04:39.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:39.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:39.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: pgmap v264: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 44 MiB/s wr, 702 op/s 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:39.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:39.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:39.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:39.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:39.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: pgmap v264: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 44 MiB/s wr, 702 op/s 2024-09-21T10:04:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: osdmap e423: 8 total, 8 up, 8 in 2024-09-21T10:04:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:40.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:04:40.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:40.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:40.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:40.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:40.810 INFO:tasks.workunit.client.0.smithi119.stdout:clone image279 -> image280 2024-09-21T10:04:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:40 smithi119 ceph-mon[105288]: Reconfiguring mon.a (monmap changed)... 2024-09-21T10:04:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:40 smithi119 ceph-mon[105288]: Reconfiguring daemon mon.a on smithi119 2024-09-21T10:04:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:40 smithi119 ceph-mon[105288]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:04:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:40 smithi119 ceph-mon[101432]: Reconfiguring mon.a (monmap changed)... 2024-09-21T10:04:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:40 smithi119 ceph-mon[101432]: Reconfiguring daemon mon.a on smithi119 2024-09-21T10:04:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:40 smithi119 ceph-mon[101432]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:04:40.868 INFO:tasks.workunit.client.0.smithi119.stdout:resize: 221888888 2024-09-21T10:04:40.873 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 123903297~6999664 2024-09-21T10:04:40.904 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 14770210~54936 2024-09-21T10:04:40.913 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 220424728~224613 2024-09-21T10:04:40.916 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 146850468~3492764 2024-09-21T10:04:40.925 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 73878016~3500467 2024-09-21T10:04:40.937 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 198517253~8309674 2024-09-21T10:04:40.937 INFO:tasks.workunit.client.0.smithi119.stdout:write: D 155318894~5498958 2024-09-21T10:04:40.965 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 115862106~7630168 2024-09-21T10:04:40.968 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap3 2024-09-21T10:04:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:40 smithi145 ceph-mon[89359]: Reconfiguring mon.a (monmap changed)... 2024-09-21T10:04:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:40 smithi145 ceph-mon[89359]: Reconfiguring daemon mon.a on smithi119 2024-09-21T10:04:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:40 smithi145 ceph-mon[89359]: osdmap e424: 8 total, 8 up, 8 in 2024-09-21T10:04:41.817 INFO:tasks.workunit.client.0.smithi119.stdout:clone image280 -> image281 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: pgmap v267: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 44 MiB/s wr, 702 op/s 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:04:42.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: pgmap v267: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 44 MiB/s wr, 702 op/s 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:04:42.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:41 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:42.179 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Stress_SmallerDstObjSize (3969 ms) 2024-09-21T10:04:42.179 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: pgmap v267: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 70 MiB/s rd, 44 MiB/s wr, 702 op/s 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: osdmap e425: 8 total, 8 up, 8 in 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:04:42.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:41 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:42.203 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.275 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.NoSnaps_StrippingLargerDstObjSize (96 ms) 2024-09-21T10:04:42.276 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingLargerDstObjSize 2024-09-21T10:04:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.361 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Snaps_StrippingLargerDstObjSize (86 ms) 2024-09-21T10:04:42.361 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Clone_StrippingLargerDstObjSize 2024-09-21T10:04:42.383 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.453 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Clone_StrippingLargerDstObjSize (92 ms) 2024-09-21T10:04:42.453 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize 2024-09-21T10:04:42.474 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.541 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneFlatten_StrippingLargerDstObjSize (88 ms) 2024-09-21T10:04:42.542 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Stress_StrippingLargerDstObjSize 2024-09-21T10:04:42.562 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.627 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Stress_StrippingLargerDstObjSize (86 ms) 2024-09-21T10:04:42.627 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize 2024-09-21T10:04:42.648 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.715 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.NoSnaps_StrippingSmallerDstObjSize (88 ms) 2024-09-21T10:04:42.715 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize 2024-09-21T10:04:42.736 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.808 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Snaps_StrippingSmallerDstObjSize (93 ms) 2024-09-21T10:04:42.808 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Clone_StrippingSmallerDstObjSize 2024-09-21T10:04:42.830 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.904 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Clone_StrippingSmallerDstObjSize (96 ms) 2024-09-21T10:04:42.904 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize 2024-09-21T10:04:42.926 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:42.997 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.CloneFlatten_StrippingSmallerDstObjSize (93 ms) 2024-09-21T10:04:42.997 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestDeepCopy.Stress_StrippingSmallerDstObjSize 2024-09-21T10:04:43.018 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:43.089 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestDeepCopy.Stress_StrippingSmallerDstObjSize (92 ms) 2024-09-21T10:04:43.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:42 smithi119 ceph-mon[101432]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T10:04:43.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:42 smithi119 ceph-mon[101432]: Reconfiguring daemon mgr.y on smithi119 2024-09-21T10:04:43.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:42 smithi119 ceph-mon[101432]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:04:43.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:42 smithi119 ceph-mon[105288]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T10:04:43.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:42 smithi119 ceph-mon[105288]: Reconfiguring daemon mgr.y on smithi119 2024-09-21T10:04:43.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:42 smithi119 ceph-mon[105288]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:04:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:42 smithi145 ceph-mon[89359]: Reconfiguring mgr.y (monmap changed)... 2024-09-21T10:04:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:42 smithi145 ceph-mon[89359]: Reconfiguring daemon mgr.y on smithi119 2024-09-21T10:04:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:42 smithi145 ceph-mon[89359]: osdmap e426: 8 total, 8 up, 8 in 2024-09-21T10:04:43.898 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 31 tests from TestDeepCopy (63025 ms total) 2024-09-21T10:04:43.898 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:04:43.898 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 51 tests from TestMirroring 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: pgmap v270: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: Reconfiguring mon.c (monmap changed)... 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:44.149 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:44.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:43 smithi145 ceph-mon[89359]: Reconfiguring daemon mon.c on smithi119 2024-09-21T10:04:44.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: pgmap v270: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:44.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:44.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:44.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: Reconfiguring mon.c (monmap changed)... 2024-09-21T10:04:44.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[101432]: Reconfiguring daemon mon.c on smithi119 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: pgmap v270: 105 pgs: 105 active+clean; 930 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: Reconfiguring mon.c (monmap changed)... 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:44.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:43 smithi119 ceph-mon[105288]: Reconfiguring daemon mon.c on smithi119 2024-09-21T10:04:44.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:44 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:44.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:44.797 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:04:44] "GET /metrics HTTP/1.1" 200 36484 "" "Prometheus/2.51.0" 2024-09-21T10:04:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[105288]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:04:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:04:45.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[101432]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:04:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:04:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:44 smithi145 ceph-mon[89359]: osdmap e427: 8 total, 8 up, 8 in 2024-09-21T10:04:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:04:45.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:45.905 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage 2024-09-21T10:04:45.907 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T10:04:45.907 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.0 on smithi119 2024-09-21T10:04:45.907 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: pgmap v272: 73 pgs: 73 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:45.907 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:04:45.908 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:45.908 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.908 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:45.908 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:04:45.908 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:46.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:45.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:46.044 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:45.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T10:04:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.0 on smithi119 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: pgmap v272: 73 pgs: 73 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:04:46.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:46.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: Reconfiguring osd.0 (monmap changed)... 2024-09-21T10:04:46.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.0 on smithi119 2024-09-21T10:04:46.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: pgmap v272: 73 pgs: 73 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:46.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: osdmap e428: 8 total, 8 up, 8 in 2024-09-21T10:04:46.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-12","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:04:46.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:46.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:46.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:04:46.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:47.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[101432]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T10:04:47.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.1 on smithi119 2024-09-21T10:04:47.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:47.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[101432]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:04:47.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[105288]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T10:04:47.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.1 on smithi119 2024-09-21T10:04:47.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:47.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:46 smithi119 ceph-mon[105288]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:04:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:46 smithi145 ceph-mon[89359]: Reconfiguring osd.1 (monmap changed)... 2024-09-21T10:04:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:46 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.1 on smithi119 2024-09-21T10:04:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:04:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:46 smithi145 ceph-mon[89359]: osdmap e429: 8 total, 8 up, 8 in 2024-09-21T10:04:47.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:46 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:46.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:48.021 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage (2116 ms) 2024-09-21T10:04:48.021 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool 2024-09-21T10:04:48.120 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool (98 ms) 2024-09-21T10:04:48.120 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled 2024-09-21T10:04:48.191 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled (72 ms) 2024-09-21T10:04:48.191 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage 2024-09-21T10:04:48.287 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage (96 ms) 2024-09-21T10:04:48.287 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: pgmap v275: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: pgmap v275: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: pgmap v275: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: osdmap e430: 8 total, 8 up, 8 in 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:04:48.374 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeImage_NoObjectMap (87 ms) 2024-09-21T10:04:48.374 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModePool 2024-09-21T10:04:48.464 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModePool (90 ms) 2024-09-21T10:04:48.464 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled 2024-09-21T10:04:48.568 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableImageMirror_In_MirrorModeDisabled (104 ms) 2024-09-21T10:04:48.568 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableImageMirrorWithPeer 2024-09-21T10:04:48.568 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:48.568 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableImageMirrorWithPeer (0 ms) 2024-09-21T10:04:48.569 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableJournalingWithPeer 2024-09-21T10:04:48.569 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:04:48.569 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableJournalingWithPeer (0 ms) 2024-09-21T10:04:48.569 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling 2024-09-21T10:04:48.633 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeDisabled_WithoutJournaling (65 ms) 2024-09-21T10:04:48.633 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling 2024-09-21T10:04:48.706 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModePool_WithoutJournaling (72 ms) 2024-09-21T10:04:48.706 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling 2024-09-21T10:04:49.036 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutJournaling (331 ms) 2024-09-21T10:04:49.036 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.2 on smithi119 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.2 on smithi119 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:04:49.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:49.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:49.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:04:49.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: Reconfiguring osd.2 (monmap changed)... 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.2 on smithi119 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: osdmap e431: 8 total, 8 up, 8 in 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:04:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[101432]: pgmap v278: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 224 KiB/s rd, 36 KiB/s wr, 324 op/s 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[101432]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.3 on smithi119 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[101432]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[101432]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[105288]: pgmap v278: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 224 KiB/s rd, 36 KiB/s wr, 324 op/s 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[105288]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.3 on smithi119 2024-09-21T10:04:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[105288]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:04:50.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:49 smithi119 ceph-mon[105288]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:04:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:49 smithi145 ceph-mon[89359]: pgmap v278: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 224 KiB/s rd, 36 KiB/s wr, 324 op/s 2024-09-21T10:04:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:49 smithi145 ceph-mon[89359]: Reconfiguring osd.3 (monmap changed)... 2024-09-21T10:04:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:49 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.3 on smithi119 2024-09-21T10:04:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:49 smithi145 ceph-mon[89359]: osdmap e432: 8 total, 8 up, 8 in 2024-09-21T10:04:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:49 smithi145 ceph-mon[89359]: osdmap e433: 8 total, 8 up, 8 in 2024-09-21T10:04:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:51.041 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableImageMirror_In_MirrorModeImage_WithoutExclusiveLock (2005 ms) 2024-09-21T10:04:51.041 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeDisabled 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[105288]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[101432]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:04:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:04:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:51.118 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeDisabled (78 ms) 2024-09-21T10:04:51.119 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage 2024-09-21T10:04:51.203 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage (84 ms) 2024-09-21T10:04:51.203 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool 2024-09-21T10:04:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:51 smithi145 ceph-mon[89359]: osdmap e434: 8 total, 8 up, 8 in 2024-09-21T10:04:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi119.oikglq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:04:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:51.295 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool (91 ms) 2024-09-21T10:04:51.295 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling 2024-09-21T10:04:51.369 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModePool_WithoutJournaling (75 ms) 2024-09-21T10:04:51.370 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling 2024-09-21T10:04:51.440 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.CreateImage_In_MirrorModeImage_WithoutJournaling (71 ms) 2024-09-21T10:04:51.440 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeDisabled 2024-09-21T10:04:51.532 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeDisabled (91 ms) 2024-09-21T10:04:51.532 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage 2024-09-21T10:04:51.620 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage (89 ms) 2024-09-21T10:04:51.620 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: Reconfiguring mds.foo.smithi119.oikglq (monmap changed)... 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: Reconfiguring daemon mds.foo.smithi119.oikglq on smithi119 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: pgmap v282: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 KiB/s rd, 41 KiB/s wr, 368 op/s 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:52.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: Reconfiguring mds.foo.smithi119.oikglq (monmap changed)... 2024-09-21T10:04:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: Reconfiguring daemon mds.foo.smithi119.oikglq on smithi119 2024-09-21T10:04:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: pgmap v282: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 KiB/s rd, 41 KiB/s wr, 368 op/s 2024-09-21T10:04:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:04:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: Reconfiguring mds.foo.smithi119.oikglq (monmap changed)... 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: Reconfiguring daemon mds.foo.smithi119.oikglq on smithi119 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: pgmap v282: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 KiB/s rd, 41 KiB/s wr, 368 op/s 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: osdmap e435: 8 total, 8 up, 8 in 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-21T10:04:52.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:53 smithi145 ceph-mon[89359]: Reconfiguring mon.b (monmap changed)... 2024-09-21T10:04:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:53 smithi145 ceph-mon[89359]: Reconfiguring daemon mon.b on smithi145 2024-09-21T10:04:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:53 smithi145 ceph-mon[89359]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T10:04:53.277 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModeImage_SnapshotMirroringEnabled (1657 ms) 2024-09-21T10:04:53.277 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.EnableJournaling_In_MirrorModePool 2024-09-21T10:04:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:53 smithi119 ceph-mon[101432]: Reconfiguring mon.b (monmap changed)... 2024-09-21T10:04:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:53 smithi119 ceph-mon[101432]: Reconfiguring daemon mon.b on smithi145 2024-09-21T10:04:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:53 smithi119 ceph-mon[101432]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T10:04:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:53 smithi119 ceph-mon[105288]: Reconfiguring mon.b (monmap changed)... 2024-09-21T10:04:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:53 smithi119 ceph-mon[105288]: Reconfiguring daemon mon.b on smithi145 2024-09-21T10:04:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:53 smithi119 ceph-mon[105288]: osdmap e436: 8 total, 8 up, 8 in 2024-09-21T10:04:53.381 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.EnableJournaling_In_MirrorModePool (104 ms) 2024-09-21T10:04:53.381 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModePool 2024-09-21T10:04:53.480 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModePool (99 ms) 2024-09-21T10:04:53.481 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.DisableJournaling_In_MirrorModeImage 2024-09-21T10:04:53.580 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.DisableJournaling_In_MirrorModeImage (100 ms) 2024-09-21T10:04:53.580 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode 2024-09-21T10:04:53.723 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_PoolMode (143 ms) 2024-09-21T10:04:53.723 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode 2024-09-21T10:04:53.865 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_DisabledMode (142 ms) 2024-09-21T10:04:53.866 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode 2024-09-21T10:04:53.990 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorModeSet_DisabledMode_To_ImageMode (125 ms) 2024-09-21T10:04:53.990 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode 2024-09-21T10:04:54.122 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorModeSet_PoolMode_To_ImageMode (132 ms) 2024-09-21T10:04:54.122 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: pgmap v285: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:54.150 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-mon[89359]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T10:04:54.266 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_PoolMode (144 ms) 2024-09-21T10:04:54.266 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: pgmap v285: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[105288]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T10:04:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: pgmap v285: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:04:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-21T10:04:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-21T10:04:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-mon[101432]: osdmap e437: 8 total, 8 up, 8 in 2024-09-21T10:04:54.425 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:54.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:54.425 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorModeSet_ImageMode_To_DisabledMode (159 ms) 2024-09-21T10:04:54.425 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageEnabled 2024-09-21T10:04:54.503 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageEnabled (78 ms) 2024-09-21T10:04:54.503 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDisabled 2024-09-21T10:04:54.572 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDisabled (69 ms) 2024-09-21T10:04:54.573 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.RemoveImage_With_ImageWithoutJournal 2024-09-21T10:04:54.632 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.RemoveImage_With_ImageWithoutJournal (59 ms) 2024-09-21T10:04:54.632 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.RemoveImage_With_MirrorImageDemoted 2024-09-21T10:04:54.740 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.RemoveImage_With_MirrorImageDemoted (109 ms) 2024-09-21T10:04:54.740 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.TrashMoveRestore_PoolMode 2024-09-21T10:04:54.853 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.TrashMoveRestore_PoolMode (113 ms) 2024-09-21T10:04:54.853 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled 2024-09-21T10:04:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:04:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:04:54] "GET /metrics HTTP/1.1" 200 36407 "" "Prometheus/2.51.0" 2024-09-21T10:04:54.956 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringDisabled (103 ms) 2024-09-21T10:04:54.956 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled 2024-09-21T10:04:55.065 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.TrashMoveRestore_ImageMode_MirroringEnabled (109 ms) 2024-09-21T10:04:55.065 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.MirrorStatusList 2024-09-21T10:04:55.133 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.MirrorStatusList (67 ms) 2024-09-21T10:04:55.133 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.RemoveBootstrapped 2024-09-21T10:04:55.220 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.RemoveBootstrapped (87 ms) 2024-09-21T10:04:55.221 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.AioPromoteDemote 2024-09-21T10:04:55.287 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-mon[89359]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T10:04:55.287 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-mon[89359]: Reconfiguring daemon mgr.x on smithi145 2024-09-21T10:04:55.287 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:55.288 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:55.288 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T10:04:55.288 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[101432]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T10:04:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[101432]: Reconfiguring daemon mgr.x on smithi145 2024-09-21T10:04:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T10:04:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:55.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[105288]: Reconfiguring mgr.x (monmap changed)... 2024-09-21T10:04:55.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[105288]: Reconfiguring daemon mgr.x on smithi145 2024-09-21T10:04:55.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:55.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:55.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-21T10:04:55.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:55.554 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.AioPromoteDemote (334 ms) 2024-09-21T10:04:55.555 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.AioGetInfo 2024-09-21T10:04:55.886 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.AioGetInfo (331 ms) 2024-09-21T10:04:55.886 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.AioGetStatus 2024-09-21T10:04:56.012 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:55.740Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:56.013 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:55.740Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:56.094 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.AioGetStatus (208 ms) 2024-09-21T10:04:56.094 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.SiteName 2024-09-21T10:04:56.102 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.SiteName (9 ms) 2024-09-21T10:04:56.102 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.Bootstrap 2024-09-21T10:04:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:56 smithi145 ceph-mon[89359]: pgmap v287: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 68 KiB/s wr, 803 op/s 2024-09-21T10:04:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:56 smithi145 ceph-mon[89359]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T10:04:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:56 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.4 on smithi145 2024-09-21T10:04:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:56 smithi119 ceph-mon[101432]: pgmap v287: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 68 KiB/s wr, 803 op/s 2024-09-21T10:04:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:56 smithi119 ceph-mon[101432]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T10:04:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:56 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.4 on smithi145 2024-09-21T10:04:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:56 smithi119 ceph-mon[105288]: pgmap v287: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 550 KiB/s rd, 68 KiB/s wr, 803 op/s 2024-09-21T10:04:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:56 smithi119 ceph-mon[105288]: Reconfiguring osd.4 (monmap changed)... 2024-09-21T10:04:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:56 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.4 on smithi145 2024-09-21T10:04:56.869 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.Bootstrap (767 ms) 2024-09-21T10:04:56.870 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.PeerDirection 2024-09-21T10:04:56.879 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.PeerDirection (10 ms) 2024-09-21T10:04:56.879 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.Snapshot 2024-09-21T10:04:57.222 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:04:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:04:56.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:04:57.222 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' 2024-09-21T10:04:57.222 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: pgmap v288: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 407 KiB/s rd, 50 KiB/s wr, 594 op/s 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.5 on smithi145 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-09-21T10:04:57.223 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-09-21T10:04:57.224 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.224 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/5276a0ac-6f6d-45c6-a99b-6f9fb913541e"} : dispatch 2024-09-21T10:04:57.224 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/5276a0ac-6f6d-45c6-a99b-6f9fb913541e"}]': finished 2024-09-21T10:04:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' 2024-09-21T10:04:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: pgmap v288: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 407 KiB/s rd, 50 KiB/s wr, 594 op/s 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.5 on smithi145 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/5276a0ac-6f6d-45c6-a99b-6f9fb913541e"} : dispatch 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/5276a0ac-6f6d-45c6-a99b-6f9fb913541e"}]': finished 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/site_name"}]': finished 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: pgmap v288: 105 pgs: 105 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 407 KiB/s rd, 50 KiB/s wr, 594 op/s 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: Reconfiguring osd.5 (monmap changed)... 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.5 on smithi145 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer_client_id"} : dispatch 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"} : dispatch 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{ "prefix": "auth get-or-create", "entity": "client.rbd-mirror-peer", "caps": [ "mon", "profile rbd-mirror-peer", "osd", "profile rbd"], "format": "json"}]': finished 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/site_name"} : dispatch 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/5276a0ac-6f6d-45c6-a99b-6f9fb913541e"} : dispatch 2024-09-21T10:04:57.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/5276a0ac-6f6d-45c6-a99b-6f9fb913541e"}]': finished 2024-09-21T10:04:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:58 smithi145 ceph-mon[89359]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T10:04:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T10:04:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[101432]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[105288]: osdmap e438: 8 total, 8 up, 8 in 2024-09-21T10:04:59.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:04:59.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-21T10:04:59.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.6 on smithi145 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: pgmap v290: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 240 KiB/s wr, 2.34k op/s 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T10:05:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:04:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.6 on smithi145 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: pgmap v290: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 240 KiB/s wr, 2.34k op/s 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T10:05:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: Reconfiguring osd.6 (monmap changed)... 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.6 on smithi145 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: pgmap v290: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 240 KiB/s wr, 2.34k op/s 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: osdmap e439: 8 total, 8 up, 8 in 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: osdmap e440: 8 total, 8 up, 8 in 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-21T10:05:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:04:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:00.486 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:01.275 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[105288]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T10:05:01.276 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[105288]: Reconfiguring daemon osd.7 on smithi145 2024-09-21T10:05:01.276 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[105288]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T10:05:01.276 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:00 smithi119 ceph-mon[101432]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T10:05:01.276 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:00 smithi119 ceph-mon[101432]: Reconfiguring daemon osd.7 on smithi145 2024-09-21T10:05:01.276 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:00 smithi119 ceph-mon[101432]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T10:05:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:00 smithi145 ceph-mon[89359]: Reconfiguring osd.7 (monmap changed)... 2024-09-21T10:05:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:00 smithi145 ceph-mon[89359]: Reconfiguring daemon osd.7 on smithi145 2024-09-21T10:05:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:00 smithi145 ceph-mon[89359]: osdmap e441: 8 total, 8 up, 8 in 2024-09-21T10:05:02.252 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[101432]: pgmap v294: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 296 KiB/s wr, 2.74k op/s 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[101432]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:02 smithi119 ceph-mon[105288]: pgmap v294: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 296 KiB/s wr, 2.74k op/s 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:02 smithi119 ceph-mon[105288]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:02.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:05:02.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:01 smithi145 ceph-mon[89359]: pgmap v294: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 MiB/s rd, 296 KiB/s wr, 2.74k op/s 2024-09-21T10:05:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:01 smithi145 ceph-mon[89359]: osdmap e442: 8 total, 8 up, 8 in 2024-09-21T10:05:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mds.foo.smithi145.tjomfr", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]} : dispatch 2024-09-21T10:05:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: Reconfiguring mds.foo.smithi145.tjomfr (monmap changed)... 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: Reconfiguring daemon mds.foo.smithi145.tjomfr on smithi145 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:03.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:03.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:03.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: Reconfiguring mds.foo.smithi145.tjomfr (monmap changed)... 2024-09-21T10:05:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: Reconfiguring daemon mds.foo.smithi145.tjomfr on smithi145 2024-09-21T10:05:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T10:05:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: Reconfiguring mds.foo.smithi145.tjomfr (monmap changed)... 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: Reconfiguring daemon mds.foo.smithi145.tjomfr on smithi145 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: osdmap e443: 8 total, 8 up, 8 in 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:03.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:03.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:04 smithi145 ceph-mon[89359]: pgmap v297: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:04 smithi145 ceph-mon[89359]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T10:05:04.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:04.147Z caller=group.go:483 level=warn name=CephOSDFlapping index=13 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)\", cluster_addr=\"172.21.15.119\", device_class=\"ssd\", hostname=\"smithi119\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.119\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:04 smithi119 ceph-mon[105288]: pgmap v297: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:04 smithi119 ceph-mon[105288]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T10:05:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:04 smithi119 ceph-mon[101432]: pgmap v297: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:04.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:04 smithi119 ceph-mon[101432]: osdmap e444: 8 total, 8 up, 8 in 2024-09-21T10:05:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:05:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:05:04] "GET /metrics HTTP/1.1" 200 36451 "" "Prometheus/2.51.0" 2024-09-21T10:05:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:05 smithi145 ceph-mon[89359]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T10:05:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:05 smithi119 ceph-mon[105288]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T10:05:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:05 smithi119 ceph-mon[101432]: osdmap e445: 8 total, 8 up, 8 in 2024-09-21T10:05:06.029 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:05.741Z caller=group.go:483 level=warn name=CephPGsInactive index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsInactive\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_active)\n > 0\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.1\n severity: critical\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been inactive for more than 5 minutes in pool\n {{ $labels.name }}. Inactive placement groups are not able to serve read/write\n requests.'\n summary: One or more placement groups are inactive\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:06.029 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:05.741Z caller=group.go:483 level=warn name=CephPGsUnclean index=1 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pgs msg="Evaluating rule failed" rule="alert: CephPGsUnclean\nexpr: ceph_pool_metadata * on (pool_id, instance) group_left () (ceph_pg_total - ceph_pg_clean)\n > 0\nfor: 15m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.7.2\n severity: warning\n type: ceph_default\nannotations:\n description: '{{ $value }} PGs have been unclean for more than 15 minutes in pool\n {{ $labels.name }}. Unclean PGs have not recovered from a previous failure.'\n summary: One or more placement groups are marked unclean\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the right hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:06 smithi145 ceph-mon[89359]: pgmap v300: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 7.2 KiB/s wr, 86 op/s 2024-09-21T10:05:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:06 smithi145 ceph-mon[89359]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T10:05:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:06 smithi119 ceph-mon[105288]: pgmap v300: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 7.2 KiB/s wr, 86 op/s 2024-09-21T10:05:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:06 smithi119 ceph-mon[105288]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T10:05:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:06 smithi119 ceph-mon[101432]: pgmap v300: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 7.2 KiB/s wr, 86 op/s 2024-09-21T10:05:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:06 smithi119 ceph-mon[101432]: osdmap e446: 8 total, 8 up, 8 in 2024-09-21T10:05:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:07 smithi145 ceph-mon[89359]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T10:05:07.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:06.948Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:07 smithi119 ceph-mon[101432]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T10:05:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:07 smithi119 ceph-mon[105288]: osdmap e447: 8 total, 8 up, 8 in 2024-09-21T10:05:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:08 smithi145 ceph-mon[89359]: pgmap v303: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 7.2 KiB/s wr, 86 op/s 2024-09-21T10:05:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:08 smithi145 ceph-mon[89359]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T10:05:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:08 smithi119 ceph-mon[105288]: pgmap v303: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 7.2 KiB/s wr, 86 op/s 2024-09-21T10:05:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:08 smithi119 ceph-mon[105288]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T10:05:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:08 smithi119 ceph-mon[101432]: pgmap v303: 105 pgs: 105 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 7.2 KiB/s wr, 86 op/s 2024-09-21T10:05:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:08 smithi119 ceph-mon[101432]: osdmap e448: 8 total, 8 up, 8 in 2024-09-21T10:05:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:09 smithi119 ceph-mon[101432]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T10:05:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:09 smithi119 ceph-mon[101432]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:05:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:09 smithi119 ceph-mon[105288]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T10:05:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:09 smithi119 ceph-mon[105288]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:05:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:09 smithi145 ceph-mon[89359]: osdmap e449: 8 total, 8 up, 8 in 2024-09-21T10:05:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:09 smithi145 ceph-mon[89359]: osdmap e450: 8 total, 8 up, 8 in 2024-09-21T10:05:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:10 smithi119 ceph-mon[101432]: pgmap v306: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 6.2 KiB/s wr, 96 op/s 2024-09-21T10:05:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:10 smithi119 ceph-mon[101432]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:05:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:10 smithi119 ceph-mon[105288]: pgmap v306: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 6.2 KiB/s wr, 96 op/s 2024-09-21T10:05:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:10 smithi119 ceph-mon[105288]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:05:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:10 smithi145 ceph-mon[89359]: pgmap v306: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 6.2 KiB/s wr, 96 op/s 2024-09-21T10:05:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:10 smithi145 ceph-mon[89359]: osdmap e451: 8 total, 8 up, 8 in 2024-09-21T10:05:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:12.096 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.Snapshot (15216 ms) 2024-09-21T10:05:12.096 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.SnapshotRemoveOnDisable 2024-09-21T10:05:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:11 smithi119 ceph-mon[101432]: pgmap v309: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 6.2 KiB/s wr, 96 op/s 2024-09-21T10:05:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:11 smithi119 ceph-mon[101432]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:05:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:11 smithi119 ceph-mon[105288]: pgmap v309: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 6.2 KiB/s wr, 96 op/s 2024-09-21T10:05:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:11 smithi119 ceph-mon[105288]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:05:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:11 smithi145 ceph-mon[89359]: pgmap v309: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 6.2 KiB/s wr, 96 op/s 2024-09-21T10:05:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:11 smithi145 ceph-mon[89359]: osdmap e452: 8 total, 8 up, 8 in 2024-09-21T10:05:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:13 smithi145 ceph-mon[89359]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:05:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:13 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e1e48326-3e91-4ade-8098-378aedb85b91"} : dispatch 2024-09-21T10:05:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:13 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e1e48326-3e91-4ade-8098-378aedb85b91"}]': finished 2024-09-21T10:05:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:13 smithi119 ceph-mon[101432]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:05:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:13 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e1e48326-3e91-4ade-8098-378aedb85b91"} : dispatch 2024-09-21T10:05:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:13 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e1e48326-3e91-4ade-8098-378aedb85b91"}]': finished 2024-09-21T10:05:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:13 smithi119 ceph-mon[105288]: osdmap e453: 8 total, 8 up, 8 in 2024-09-21T10:05:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:13 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e1e48326-3e91-4ade-8098-378aedb85b91"} : dispatch 2024-09-21T10:05:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:13 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/e1e48326-3e91-4ade-8098-378aedb85b91"}]': finished 2024-09-21T10:05:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:14 smithi145 ceph-mon[89359]: pgmap v312: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:14 smithi145 ceph-mon[89359]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:05:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:14 smithi119 ceph-mon[101432]: pgmap v312: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:14 smithi119 ceph-mon[101432]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:05:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:14 smithi119 ceph-mon[105288]: pgmap v312: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:14 smithi119 ceph-mon[105288]: osdmap e454: 8 total, 8 up, 8 in 2024-09-21T10:05:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:05:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:05:14] "GET /metrics HTTP/1.1" 200 36451 "" "Prometheus/2.51.0" 2024-09-21T10:05:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:15 smithi119 ceph-mon[105288]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:05:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:15 smithi119 ceph-mon[105288]: pgmap v315: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 8.0 KiB/s wr, 107 op/s 2024-09-21T10:05:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:15 smithi119 ceph-mon[101432]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:05:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:15 smithi119 ceph-mon[101432]: pgmap v315: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 8.0 KiB/s wr, 107 op/s 2024-09-21T10:05:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:15 smithi145 ceph-mon[89359]: osdmap e455: 8 total, 8 up, 8 in 2024-09-21T10:05:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:15 smithi145 ceph-mon[89359]: pgmap v315: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 8.0 KiB/s wr, 107 op/s 2024-09-21T10:05:16.199 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.SnapshotRemoveOnDisable (4102 ms) 2024-09-21T10:05:16.199 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.SnapshotUnlinkPeer 2024-09-21T10:05:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:16 smithi119 ceph-mon[105288]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:05:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:16 smithi119 ceph-mon[101432]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:05:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:16 smithi145 ceph-mon[89359]: osdmap e456: 8 total, 8 up, 8 in 2024-09-21T10:05:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:17 smithi145 ceph-mon[89359]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:05:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/179786d8-7d05-4614-8264-1b1df072b7b3"} : dispatch 2024-09-21T10:05:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/179786d8-7d05-4614-8264-1b1df072b7b3"}]': finished 2024-09-21T10:05:17.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:17 smithi145 ceph-mon[89359]: pgmap v318: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 8.0 KiB/s wr, 107 op/s 2024-09-21T10:05:17.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:16.949Z caller=group.go:483 level=warn name=CephNodeDiskspaceWarning index=4 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi119\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi119\", job=\"node\", machine=\"x86_64\", nodename=\"smithi119\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[101432]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:05:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/179786d8-7d05-4614-8264-1b1df072b7b3"} : dispatch 2024-09-21T10:05:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/179786d8-7d05-4614-8264-1b1df072b7b3"}]': finished 2024-09-21T10:05:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[101432]: pgmap v318: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 8.0 KiB/s wr, 107 op/s 2024-09-21T10:05:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[105288]: osdmap e457: 8 total, 8 up, 8 in 2024-09-21T10:05:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/179786d8-7d05-4614-8264-1b1df072b7b3"} : dispatch 2024-09-21T10:05:17.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/179786d8-7d05-4614-8264-1b1df072b7b3"}]': finished 2024-09-21T10:05:17.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:17 smithi119 ceph-mon[105288]: pgmap v318: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 8.0 KiB/s wr, 107 op/s 2024-09-21T10:05:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:18 smithi145 ceph-mon[89359]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:05:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:18 smithi119 ceph-mon[105288]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:05:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:18 smithi119 ceph-mon[101432]: osdmap e458: 8 total, 8 up, 8 in 2024-09-21T10:05:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:19 smithi145 ceph-mon[89359]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:05:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:19 smithi145 ceph-mon[89359]: pgmap v321: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 9.7 KiB/s wr, 116 op/s 2024-09-21T10:05:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:19 smithi145 ceph-mon[89359]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:05:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:19 smithi119 ceph-mon[105288]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:05:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:19 smithi119 ceph-mon[105288]: pgmap v321: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 9.7 KiB/s wr, 116 op/s 2024-09-21T10:05:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:19 smithi119 ceph-mon[105288]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:05:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:19 smithi119 ceph-mon[101432]: osdmap e459: 8 total, 8 up, 8 in 2024-09-21T10:05:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:19 smithi119 ceph-mon[101432]: pgmap v321: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 9.7 KiB/s wr, 116 op/s 2024-09-21T10:05:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:19 smithi119 ceph-mon[101432]: osdmap e460: 8 total, 8 up, 8 in 2024-09-21T10:05:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:21 smithi145 ceph-mon[89359]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:05:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:21 smithi119 ceph-mon[105288]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:05:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:21 smithi119 ceph-mon[101432]: osdmap e461: 8 total, 8 up, 8 in 2024-09-21T10:05:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:22 smithi145 ceph-mon[89359]: pgmap v324: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 9.7 KiB/s wr, 116 op/s 2024-09-21T10:05:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:22 smithi145 ceph-mon[89359]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:05:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/960aa5fa-c30d-42a0-96d2-c4579fbc3059"} : dispatch 2024-09-21T10:05:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/960aa5fa-c30d-42a0-96d2-c4579fbc3059"}]': finished 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[105288]: pgmap v324: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 9.7 KiB/s wr, 116 op/s 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[105288]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/960aa5fa-c30d-42a0-96d2-c4579fbc3059"} : dispatch 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/960aa5fa-c30d-42a0-96d2-c4579fbc3059"}]': finished 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[101432]: pgmap v324: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 9.7 KiB/s wr, 116 op/s 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[101432]: osdmap e462: 8 total, 8 up, 8 in 2024-09-21T10:05:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/960aa5fa-c30d-42a0-96d2-c4579fbc3059"} : dispatch 2024-09-21T10:05:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/960aa5fa-c30d-42a0-96d2-c4579fbc3059"}]': finished 2024-09-21T10:05:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:23 smithi145 ceph-mon[89359]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:05:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:23 smithi119 ceph-mon[105288]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:05:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:23 smithi119 ceph-mon[101432]: osdmap e463: 8 total, 8 up, 8 in 2024-09-21T10:05:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:24 smithi145 ceph-mon[89359]: pgmap v327: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:24 smithi145 ceph-mon[89359]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:05:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:24 smithi119 ceph-mon[105288]: pgmap v327: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:24 smithi119 ceph-mon[105288]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:05:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:24 smithi119 ceph-mon[101432]: pgmap v327: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:24 smithi119 ceph-mon[101432]: osdmap e464: 8 total, 8 up, 8 in 2024-09-21T10:05:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:05:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:05:24] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:05:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:25 smithi119 ceph-mon[105288]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:05:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:25 smithi119 ceph-mon[101432]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:05:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:25 smithi145 ceph-mon[89359]: osdmap e465: 8 total, 8 up, 8 in 2024-09-21T10:05:26.122 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.SnapshotUnlinkPeer (9922 ms) 2024-09-21T10:05:26.122 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.SnapshotImageState 2024-09-21T10:05:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:26 smithi119 ceph-mon[105288]: pgmap v330: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 176 KiB/s rd, 11 KiB/s wr, 232 op/s 2024-09-21T10:05:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:26 smithi119 ceph-mon[105288]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:05:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:26 smithi119 ceph-mon[101432]: pgmap v330: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 176 KiB/s rd, 11 KiB/s wr, 232 op/s 2024-09-21T10:05:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:26 smithi119 ceph-mon[101432]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:05:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:26 smithi145 ceph-mon[89359]: pgmap v330: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 176 KiB/s rd, 11 KiB/s wr, 232 op/s 2024-09-21T10:05:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:26 smithi145 ceph-mon[89359]: osdmap e466: 8 total, 8 up, 8 in 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[105288]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/c19ac9a1-d614-436b-a95a-c3ed7e6b749f"} : dispatch 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/c19ac9a1-d614-436b-a95a-c3ed7e6b749f"}]': finished 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/b60381b4-ec3e-45f7-88b9-3b1fcf06c11b"} : dispatch 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/b60381b4-ec3e-45f7-88b9-3b1fcf06c11b"}]': finished 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[101432]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/c19ac9a1-d614-436b-a95a-c3ed7e6b749f"} : dispatch 2024-09-21T10:05:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/c19ac9a1-d614-436b-a95a-c3ed7e6b749f"}]': finished 2024-09-21T10:05:27.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/b60381b4-ec3e-45f7-88b9-3b1fcf06c11b"} : dispatch 2024-09-21T10:05:27.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/b60381b4-ec3e-45f7-88b9-3b1fcf06c11b"}]': finished 2024-09-21T10:05:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:27 smithi145 ceph-mon[89359]: osdmap e467: 8 total, 8 up, 8 in 2024-09-21T10:05:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/c19ac9a1-d614-436b-a95a-c3ed7e6b749f"} : dispatch 2024-09-21T10:05:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/c19ac9a1-d614-436b-a95a-c3ed7e6b749f"}]': finished 2024-09-21T10:05:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/b60381b4-ec3e-45f7-88b9-3b1fcf06c11b"} : dispatch 2024-09-21T10:05:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/b60381b4-ec3e-45f7-88b9-3b1fcf06c11b"}]': finished 2024-09-21T10:05:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:28 smithi119 ceph-mon[105288]: pgmap v333: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 176 KiB/s rd, 11 KiB/s wr, 232 op/s 2024-09-21T10:05:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:28 smithi119 ceph-mon[105288]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:05:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:28 smithi119 ceph-mon[101432]: pgmap v333: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 176 KiB/s rd, 11 KiB/s wr, 232 op/s 2024-09-21T10:05:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:28 smithi119 ceph-mon[101432]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:05:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:28 smithi145 ceph-mon[89359]: pgmap v333: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 176 KiB/s rd, 11 KiB/s wr, 232 op/s 2024-09-21T10:05:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:28 smithi145 ceph-mon[89359]: osdmap e468: 8 total, 8 up, 8 in 2024-09-21T10:05:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:29 smithi145 ceph-mon[89359]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:05:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:29 smithi145 ceph-mon[89359]: pgmap v336: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 25 KiB/s wr, 244 op/s 2024-09-21T10:05:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:29 smithi145 ceph-mon[89359]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:05:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:29 smithi119 ceph-mon[105288]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:05:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:29 smithi119 ceph-mon[105288]: pgmap v336: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 25 KiB/s wr, 244 op/s 2024-09-21T10:05:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:29 smithi119 ceph-mon[105288]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:05:29.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:29 smithi119 ceph-mon[101432]: osdmap e469: 8 total, 8 up, 8 in 2024-09-21T10:05:29.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:29 smithi119 ceph-mon[101432]: pgmap v336: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 25 KiB/s wr, 244 op/s 2024-09-21T10:05:29.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:29 smithi119 ceph-mon[101432]: osdmap e470: 8 total, 8 up, 8 in 2024-09-21T10:05:30.045 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.SnapshotImageState (3924 ms) 2024-09-21T10:05:30.045 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.SnapshotPromoteDemote 2024-09-21T10:05:30.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:31 smithi145 ceph-mon[89359]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:05:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:31 smithi119 ceph-mon[105288]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:05:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:31 smithi119 ceph-mon[101432]: osdmap e471: 8 total, 8 up, 8 in 2024-09-21T10:05:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:32 smithi145 ceph-mon[89359]: pgmap v339: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 25 KiB/s wr, 244 op/s 2024-09-21T10:05:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:32 smithi145 ceph-mon[89359]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T10:05:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:32 smithi119 ceph-mon[105288]: pgmap v339: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 25 KiB/s wr, 244 op/s 2024-09-21T10:05:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:32 smithi119 ceph-mon[105288]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T10:05:32.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:32 smithi119 ceph-mon[101432]: pgmap v339: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 167 KiB/s rd, 25 KiB/s wr, 244 op/s 2024-09-21T10:05:32.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:32 smithi119 ceph-mon[101432]: osdmap e472: 8 total, 8 up, 8 in 2024-09-21T10:05:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:33 smithi145 ceph-mon[89359]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T10:05:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:33 smithi119 ceph-mon[105288]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T10:05:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:33 smithi119 ceph-mon[101432]: osdmap e473: 8 total, 8 up, 8 in 2024-09-21T10:05:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:34 smithi145 ceph-mon[89359]: pgmap v342: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:34 smithi145 ceph-mon[89359]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T10:05:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:34 smithi119 ceph-mon[101432]: pgmap v342: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:34 smithi119 ceph-mon[101432]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T10:05:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:34 smithi119 ceph-mon[105288]: pgmap v342: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:34 smithi119 ceph-mon[105288]: osdmap e474: 8 total, 8 up, 8 in 2024-09-21T10:05:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:05:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:05:34] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:05:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:35 smithi145 ceph-mon[89359]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T10:05:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:35 smithi119 ceph-mon[101432]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T10:05:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:35 smithi119 ceph-mon[105288]: osdmap e475: 8 total, 8 up, 8 in 2024-09-21T10:05:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:36 smithi145 ceph-mon[89359]: pgmap v345: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 8.7 KiB/s wr, 135 op/s 2024-09-21T10:05:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:36 smithi145 ceph-mon[89359]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T10:05:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:36 smithi119 ceph-mon[105288]: pgmap v345: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 8.7 KiB/s wr, 135 op/s 2024-09-21T10:05:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:36 smithi119 ceph-mon[105288]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T10:05:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:36 smithi119 ceph-mon[101432]: pgmap v345: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 8.7 KiB/s wr, 135 op/s 2024-09-21T10:05:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:36 smithi119 ceph-mon[101432]: osdmap e476: 8 total, 8 up, 8 in 2024-09-21T10:05:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:37 smithi145 ceph-mon[89359]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T10:05:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:37 smithi119 ceph-mon[105288]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T10:05:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:37 smithi119 ceph-mon[101432]: osdmap e477: 8 total, 8 up, 8 in 2024-09-21T10:05:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:38 smithi119 ceph-mon[105288]: pgmap v348: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 8.7 KiB/s wr, 135 op/s 2024-09-21T10:05:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:38 smithi119 ceph-mon[105288]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T10:05:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:38 smithi119 ceph-mon[101432]: pgmap v348: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 8.7 KiB/s wr, 135 op/s 2024-09-21T10:05:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:38 smithi119 ceph-mon[101432]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T10:05:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:38 smithi145 ceph-mon[89359]: pgmap v348: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 99 KiB/s rd, 8.7 KiB/s wr, 135 op/s 2024-09-21T10:05:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:38 smithi145 ceph-mon[89359]: osdmap e478: 8 total, 8 up, 8 in 2024-09-21T10:05:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:39 smithi119 ceph-mon[105288]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T10:05:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:39 smithi119 ceph-mon[105288]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T10:05:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:39 smithi119 ceph-mon[101432]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T10:05:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:39 smithi119 ceph-mon[101432]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T10:05:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:39 smithi145 ceph-mon[89359]: osdmap e479: 8 total, 8 up, 8 in 2024-09-21T10:05:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:39 smithi145 ceph-mon[89359]: osdmap e480: 8 total, 8 up, 8 in 2024-09-21T10:05:40.062 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.SnapshotPromoteDemote (10017 ms) 2024-09-21T10:05:40.062 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestMirroring.AioSnapshotCreate 2024-09-21T10:05:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[101432]: pgmap v351: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 7.0 KiB/s wr, 107 op/s 2024-09-21T10:05:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[101432]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T10:05:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/2f6f4e46-3a1f-490c-b0f5-161193824629"} : dispatch 2024-09-21T10:05:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/2f6f4e46-3a1f-490c-b0f5-161193824629"}]': finished 2024-09-21T10:05:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[105288]: pgmap v351: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 7.0 KiB/s wr, 107 op/s 2024-09-21T10:05:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[105288]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T10:05:40.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/2f6f4e46-3a1f-490c-b0f5-161193824629"} : dispatch 2024-09-21T10:05:40.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/2f6f4e46-3a1f-490c-b0f5-161193824629"}]': finished 2024-09-21T10:05:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:40 smithi145 ceph-mon[89359]: pgmap v351: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 7.0 KiB/s wr, 107 op/s 2024-09-21T10:05:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:40 smithi145 ceph-mon[89359]: osdmap e481: 8 total, 8 up, 8 in 2024-09-21T10:05:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/2f6f4e46-3a1f-490c-b0f5-161193824629"} : dispatch 2024-09-21T10:05:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/2f6f4e46-3a1f-490c-b0f5-161193824629"}]': finished 2024-09-21T10:05:40.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade resume' 2024-09-21T10:05:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:42.346 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:42 smithi145 ceph-mon[89359]: pgmap v354: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 7.0 KiB/s wr, 107 op/s 2024-09-21T10:05:42.346 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:42 smithi145 ceph-mon[89359]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T10:05:42.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:42 smithi119 ceph-mon[101432]: pgmap v354: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 7.0 KiB/s wr, 107 op/s 2024-09-21T10:05:42.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:42 smithi119 ceph-mon[101432]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T10:05:42.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:42 smithi119 ceph-mon[105288]: pgmap v354: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 7.0 KiB/s wr, 107 op/s 2024-09-21T10:05:42.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:42 smithi119 ceph-mon[105288]: osdmap e482: 8 total, 8 up, 8 in 2024-09-21T10:05:42.958 INFO:teuthology.orchestra.run.smithi119.stdout:Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:05:43.297 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[105288]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[105288]: pgmap v357: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[105288]: from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[105288]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[101432]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[101432]: pgmap v357: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[101432]: from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:43.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[101432]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:05:43.299 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:43.299 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:43 smithi145 ceph-mon[89359]: osdmap e483: 8 total, 8 up, 8 in 2024-09-21T10:05:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:43 smithi145 ceph-mon[89359]: pgmap v357: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:43 smithi145 ceph-mon[89359]: from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade resume", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:43 smithi145 ceph-mon[89359]: Upgrade: Resumed upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:05:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:43.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'echo wait for half of osds to upgrade' 2024-09-21T10:05:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:44 smithi119 ceph-mon[105288]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T10:05:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:44 smithi119 ceph-mon[101432]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T10:05:44.375 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:44 smithi145 ceph-mon[89359]: osdmap e484: 8 total, 8 up, 8 in 2024-09-21T10:05:44.432 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:05:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:05:44] "GET /metrics HTTP/1.1" 200 36455 "" "Prometheus/2.51.0" 2024-09-21T10:05:45.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:45 smithi119 ceph-mon[101432]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T10:05:45.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:45 smithi119 ceph-mon[101432]: pgmap v360: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s rd, 12 KiB/s wr, 112 op/s 2024-09-21T10:05:45.225 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:45 smithi119 ceph-mon[105288]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T10:05:45.225 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:45 smithi119 ceph-mon[105288]: pgmap v360: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s rd, 12 KiB/s wr, 112 op/s 2024-09-21T10:05:45.414 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:45 smithi145 ceph-mon[89359]: osdmap e485: 8 total, 8 up, 8 in 2024-09-21T10:05:45.414 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:45 smithi145 ceph-mon[89359]: pgmap v360: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s rd, 12 KiB/s wr, 112 op/s 2024-09-21T10:05:46.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:46 smithi119 ceph-mon[101432]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T10:05:46.273 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:46 smithi119 ceph-mon[105288]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T10:05:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:46 smithi145 ceph-mon[89359]: osdmap e486: 8 total, 8 up, 8 in 2024-09-21T10:05:46.556 INFO:teuthology.orchestra.run.smithi119.stdout:wait for half of osds to upgrade 2024-09-21T10:05:47.238 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:47 smithi119 ceph-mon[101432]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T10:05:47.238 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:47 smithi119 ceph-mon[101432]: pgmap v363: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s rd, 12 KiB/s wr, 112 op/s 2024-09-21T10:05:47.238 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:47 smithi119 ceph-mon[105288]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T10:05:47.238 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:47 smithi119 ceph-mon[105288]: pgmap v363: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s rd, 12 KiB/s wr, 112 op/s 2024-09-21T10:05:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:47 smithi145 ceph-mon[89359]: osdmap e487: 8 total, 8 up, 8 in 2024-09-21T10:05:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:47 smithi145 ceph-mon[89359]: pgmap v363: 105 pgs: 105 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 75 KiB/s rd, 12 KiB/s wr, 112 op/s 2024-09-21T10:05:47.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'while ! ceph osd versions | grep $sha1 | egrep '"'"': [45678]'"'"' && ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error; do sleep 2 ; done' 2024-09-21T10:05:48.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[105288]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T10:05:48.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:48.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:48.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:48.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[101432]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T10:05:48.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:48.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:48.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:48.531 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:05:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:48 smithi145 ceph-mon[89359]: osdmap e488: 8 total, 8 up, 8 in 2024-09-21T10:05:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:05:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:49 smithi119 ceph-mon[105288]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T10:05:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:49 smithi119 ceph-mon[105288]: pgmap v366: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:05:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:49 smithi119 ceph-mon[105288]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T10:05:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:49 smithi119 ceph-mon[101432]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T10:05:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:49 smithi119 ceph-mon[101432]: pgmap v366: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:05:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:49 smithi119 ceph-mon[101432]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T10:05:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:49 smithi145 ceph-mon[89359]: osdmap e489: 8 total, 8 up, 8 in 2024-09-21T10:05:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:49 smithi145 ceph-mon[89359]: pgmap v366: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:05:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:49 smithi145 ceph-mon[89359]: osdmap e490: 8 total, 8 up, 8 in 2024-09-21T10:05:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:05:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:05:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:05:51.245 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T10:05:51.245 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.245 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.245 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T10:05:51.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: osdmap e491: 8 total, 8 up, 8 in 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.a"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.b"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config rm", "name": "container_image", "who": "mon.c"} : dispatch 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-21T10:05:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:51.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16} : dispatch 2024-09-21T10:05:51.380 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: pgmap v369: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: Upgrade: Setting container_image for all mon 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: Upgrade: Setting container_image for all crash 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: Upgrade: osd.0 is safe to restart 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2969576634' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:52.176 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: pgmap v369: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:05:52.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: Upgrade: Setting container_image for all mon 2024-09-21T10:05:52.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: Upgrade: Setting container_image for all crash 2024-09-21T10:05:52.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T10:05:52.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: Upgrade: osd.0 is safe to restart 2024-09-21T10:05:52.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T10:05:52.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2969576634' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: pgmap v369: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 82 KiB/s rd, 14 KiB/s wr, 119 op/s 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: Upgrade: Setting container_image for all mon 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: Upgrade: Setting container_image for all crash 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: Upgrade: osd.0 is safe to restart 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: osdmap e492: 8 total, 8 up, 8 in 2024-09-21T10:05:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2969576634' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[105288]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[105288]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[105288]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T10:05:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:05:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[101432]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[101432]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[101432]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T10:05:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:05:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:53 smithi145 ceph-mon[89359]: from='client.34335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:53 smithi145 ceph-mon[89359]: from='client.34341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:53.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:53 smithi145 ceph-mon[89359]: osdmap e493: 8 total, 8 up, 8 in 2024-09-21T10:05:53.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:05:53.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-21T10:05:53.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:05:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[101432]: Upgrade: Updating osd.0 2024-09-21T10:05:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[101432]: Deploying daemon osd.0 on smithi119 2024-09-21T10:05:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[101432]: pgmap v372: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[101432]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T10:05:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1980943640' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[105288]: Upgrade: Updating osd.0 2024-09-21T10:05:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[105288]: Deploying daemon osd.0 on smithi119 2024-09-21T10:05:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[105288]: pgmap v372: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[105288]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T10:05:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1980943640' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:54.286 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:05:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:54 smithi145 ceph-mon[89359]: Upgrade: Updating osd.0 2024-09-21T10:05:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:54 smithi145 ceph-mon[89359]: Deploying daemon osd.0 on smithi119 2024-09-21T10:05:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:54 smithi145 ceph-mon[89359]: pgmap v372: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:05:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:54 smithi145 ceph-mon[89359]: osdmap e494: 8 total, 8 up, 8 in 2024-09-21T10:05:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1980943640' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:54.784 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:05:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:05:54] "GET /metrics HTTP/1.1" 200 36391 "" "Prometheus/2.51.0" 2024-09-21T10:05:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:55 smithi119 ceph-mon[105288]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T10:05:55.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:55 smithi119 systemd[1]: Stopping Ceph osd.0 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:05:55.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:55 smithi119 ceph-mon[101432]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T10:05:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:55 smithi145 ceph-mon[89359]: osdmap e495: 8 total, 8 up, 8 in 2024-09-21T10:05:55.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[45163]: 2024-09-21T10:05:55.398+0000 7fd84bf3a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:05:55.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[45163]: 2024-09-21T10:05:55.398+0000 7fd84bf3a640 -1 osd.0 496 *** Got signal Terminated *** 2024-09-21T10:05:55.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[45163]: 2024-09-21T10:05:55.398+0000 7fd84bf3a640 -1 osd.0 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[101432]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[101432]: pgmap v375: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[101432]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[101432]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[101432]: osd.0 marked itself down and dead 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[105288]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[105288]: pgmap v375: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:05:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[105288]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[105288]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T10:05:56.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:56 smithi119 ceph-mon[105288]: osd.0 marked itself down and dead 2024-09-21T10:05:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:56 smithi145 ceph-mon[89359]: from='client.34353 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:56 smithi145 ceph-mon[89359]: pgmap v375: 105 pgs: 105 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:05:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:56 smithi145 ceph-mon[89359]: from='client.34359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:56 smithi145 ceph-mon[89359]: osdmap e496: 8 total, 8 up, 8 in 2024-09-21T10:05:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:56 smithi145 ceph-mon[89359]: osd.0 marked itself down and dead 2024-09-21T10:05:56.865 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:56 smithi119 podman[114251]: 2024-09-21 10:05:56.604615108 +0000 UTC m=+1.327692150 container died ce07d9950c3f2860532560c56531c22961c2aa63e436b37d58708428e9ed41a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-21T10:05:57.117 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 podman[114251]: 2024-09-21 10:05:57.028502574 +0000 UTC m=+1.751579617 container cleanup ce07d9950c3f2860532560c56531c22961c2aa63e436b37d58708428e9ed41a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:05:57.117 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 bash[114251]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0 2024-09-21T10:05:57.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:57 smithi119 ceph-mon[101432]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:05:57.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:57 smithi119 ceph-mon[101432]: osdmap e497: 8 total, 7 up, 8 in 2024-09-21T10:05:57.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2249991725' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:57.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:57 smithi119 ceph-mon[105288]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:05:57.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:57 smithi119 ceph-mon[105288]: osdmap e497: 8 total, 7 up, 8 in 2024-09-21T10:05:57.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2249991725' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:57.163 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:05:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:57 smithi145 ceph-mon[89359]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:05:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:57 smithi145 ceph-mon[89359]: osdmap e497: 8 total, 7 up, 8 in 2024-09-21T10:05:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2249991725' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:05:57.617 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 podman[114274]: 2024-09-21 10:05:57.36424154 +0000 UTC m=+0.753433641 container remove ce07d9950c3f2860532560c56531c22961c2aa63e436b37d58708428e9ed41a9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T10:05:57.961 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 podman[114435]: 2024-09-21 10:05:57.608819453 +0000 UTC m=+0.015950565 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:05:57.961 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 podman[114435]: 2024-09-21 10:05:57.731928514 +0000 UTC m=+0.139059626 container create 7c7cfc7fa0f306d17bad1a985d8063a5436ef94b062e141a6a0f4107e3bc3d18 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T10:05:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:58 smithi119 ceph-mon[101432]: pgmap v378: 105 pgs: 18 stale+active+clean, 87 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:05:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:58 smithi119 ceph-mon[101432]: osdmap e498: 8 total, 7 up, 8 in 2024-09-21T10:05:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:58 smithi119 ceph-mon[105288]: pgmap v378: 105 pgs: 18 stale+active+clean, 87 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:05:58.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:58 smithi119 ceph-mon[105288]: osdmap e498: 8 total, 7 up, 8 in 2024-09-21T10:05:58.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 podman[114435]: 2024-09-21 10:05:57.959832957 +0000 UTC m=+0.366964077 container init 7c7cfc7fa0f306d17bad1a985d8063a5436ef94b062e141a6a0f4107e3bc3d18 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:05:58.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:57 smithi119 podman[114435]: 2024-09-21 10:05:57.96512784 +0000 UTC m=+0.372258953 container start 7c7cfc7fa0f306d17bad1a985d8063a5436ef94b062e141a6a0f4107e3bc3d18 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T10:05:58.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 podman[114435]: 2024-09-21 10:05:58.059905419 +0000 UTC m=+0.467036540 container attach 7c7cfc7fa0f306d17bad1a985d8063a5436ef94b062e141a6a0f4107e3bc3d18 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T10:05:58.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 podman[114435]: 2024-09-21 10:05:58.171261789 +0000 UTC m=+0.578392909 container died 7c7cfc7fa0f306d17bad1a985d8063a5436ef94b062e141a6a0f4107e3bc3d18 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-21T10:05:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:58 smithi145 ceph-mon[89359]: pgmap v378: 105 pgs: 18 stale+active+clean, 87 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 187 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:05:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:58 smithi145 ceph-mon[89359]: osdmap e498: 8 total, 7 up, 8 in 2024-09-21T10:05:59.074 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 podman[114478]: 2024-09-21 10:05:58.738625198 +0000 UTC m=+0.561464926 container remove 7c7cfc7fa0f306d17bad1a985d8063a5436ef94b062e141a6a0f4107e3bc3d18 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T10:05:59.074 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-21T10:05:59.074 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 systemd[1]: Stopped Ceph osd.0 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:05:59.074 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.0.service: Consumed 25.060s CPU time. 2024-09-21T10:05:59.074 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:58 smithi119 systemd[1]: Starting Ceph osd.0 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[101432]: from='client.44255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[101432]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[101432]: osdmap e499: 8 total, 7 up, 8 in 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[101432]: osdmap e500: 8 total, 7 up, 8 in 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[105288]: from='client.44255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[105288]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[105288]: osdmap e499: 8 total, 7 up, 8 in 2024-09-21T10:05:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-mon[105288]: osdmap e500: 8 total, 7 up, 8 in 2024-09-21T10:05:59.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 podman[114585]: 2024-09-21 10:05:59.062213755 +0000 UTC m=+0.015577899 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:05:59.368 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 podman[114585]: 2024-09-21 10:05:59.19852249 +0000 UTC m=+0.151886663 container create 42c71906d43c79d7195af0a667c8d2c3ad03f6b59a3fc03a33a7f174efe0f664 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T10:05:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:59 smithi145 ceph-mon[89359]: from='client.44255 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:59 smithi145 ceph-mon[89359]: from='client.44261 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:05:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:59 smithi145 ceph-mon[89359]: osdmap e499: 8 total, 7 up, 8 in 2024-09-21T10:05:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:05:59 smithi145 ceph-mon[89359]: osdmap e500: 8 total, 7 up, 8 in 2024-09-21T10:05:59.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 podman[114585]: 2024-09-21 10:05:59.426405917 +0000 UTC m=+0.379770061 container init 42c71906d43c79d7195af0a667c8d2c3ad03f6b59a3fc03a33a7f174efe0f664 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:05:59.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 podman[114585]: 2024-09-21 10:05:59.431292275 +0000 UTC m=+0.384656425 container start 42c71906d43c79d7195af0a667c8d2c3ad03f6b59a3fc03a33a7f174efe0f664 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T10:05:59.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 podman[114585]: 2024-09-21 10:05:59.509506719 +0000 UTC m=+0.462870864 container attach 42c71906d43c79d7195af0a667c8d2c3ad03f6b59a3fc03a33a7f174efe0f664 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T10:05:59.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:05:59.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 bash[114585]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:05:59.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:05:59.868 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:05:59 smithi119 bash[114585]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:06:00.038 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:00.359 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[101432]: pgmap v381: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 6.5 KiB/s wr, 102 op/s; 68/444 objects degraded (15.315%) 2024-09-21T10:06:00.359 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 68/444 objects degraded (15.315%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:00.359 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1806610115' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:00.359 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[101432]: osdmap e501: 8 total, 7 up, 8 in 2024-09-21T10:06:00.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[105288]: pgmap v381: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 6.5 KiB/s wr, 102 op/s; 68/444 objects degraded (15.315%) 2024-09-21T10:06:00.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 68/444 objects degraded (15.315%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:00.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1806610115' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:00.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-mon[105288]: osdmap e501: 8 total, 7 up, 8 in 2024-09-21T10:06:00.360 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:06:00.360 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:06:00.360 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:06:00.360 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:06:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:00 smithi145 ceph-mon[89359]: pgmap v381: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 6.5 KiB/s wr, 102 op/s; 68/444 objects degraded (15.315%) 2024-09-21T10:06:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:00 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 68/444 objects degraded (15.315%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1806610115' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:00 smithi145 ceph-mon[89359]: osdmap e501: 8 total, 7 up, 8 in 2024-09-21T10:06:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:06:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:06:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:00.617 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:06:00.617 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:06:00.617 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:06:00.617 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:06:00.617 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T10:06:00.618 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T10:06:00.618 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:06:00.618 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T10:06:00.618 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate[114600]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T10:06:00.618 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 bash[114585]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T10:06:00.618 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 podman[114585]: 2024-09-21 10:06:00.385419124 +0000 UTC m=+1.338783269 container died 42c71906d43c79d7195af0a667c8d2c3ad03f6b59a3fc03a33a7f174efe0f664 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T10:06:01.216 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:01 smithi119 ceph-mon[105288]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:01.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:01 smithi119 ceph-mon[101432]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:01.217 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:00 smithi119 podman[114818]: 2024-09-21 10:06:00.963450023 +0000 UTC m=+0.568820139 container remove 42c71906d43c79d7195af0a667c8d2c3ad03f6b59a3fc03a33a7f174efe0f664 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=main-62da54b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T10:06:01.217 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 podman[114883]: 2024-09-21 10:06:01.070589852 +0000 UTC m=+0.015489405 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:06:01.217 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 podman[114883]: 2024-09-21 10:06:01.214691013 +0000 UTC m=+0.159590551 container create 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:06:01.519 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 podman[114883]: 2024-09-21 10:06:01.40862845 +0000 UTC m=+0.353528001 container init 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0) 2024-09-21T10:06:01.519 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 podman[114883]: 2024-09-21 10:06:01.413542053 +0000 UTC m=+0.358441591 container start 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T10:06:01.519 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 ceph-osd[114922]: -- 172.21.15.119:0/72864370 <== mon.2 v2:172.21.15.119:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5593eff43a40 con 0x5593eff21c00 2024-09-21T10:06:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:01 smithi145 ceph-mon[89359]: from='client.34389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:01.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 bash[114883]: 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b 2024-09-21T10:06:01.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:01 smithi119 systemd[1]: Started Ceph osd.0 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:06:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[101432]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[101432]: pgmap v384: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 6.5 KiB/s wr, 102 op/s; 68/444 objects degraded (15.315%) 2024-09-21T10:06:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[101432]: osdmap e502: 8 total, 7 up, 8 in 2024-09-21T10:06:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:02.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[105288]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:02.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[105288]: pgmap v384: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 6.5 KiB/s wr, 102 op/s; 68/444 objects degraded (15.315%) 2024-09-21T10:06:02.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[105288]: osdmap e502: 8 total, 7 up, 8 in 2024-09-21T10:06:02.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:02.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:02.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:02 smithi145 ceph-mon[89359]: from='client.44279 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:02 smithi145 ceph-mon[89359]: pgmap v384: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 70 KiB/s rd, 6.5 KiB/s wr, 102 op/s; 68/444 objects degraded (15.315%) 2024-09-21T10:06:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:02 smithi145 ceph-mon[89359]: osdmap e502: 8 total, 7 up, 8 in 2024-09-21T10:06:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:02.940 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:03.222 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[114899]: 2024-09-21T10:06:03.047+0000 7f438dfce640 -1 Falling back to public interface 2024-09-21T10:06:03.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[101432]: osdmap e503: 8 total, 7 up, 8 in 2024-09-21T10:06:03.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[101432]: pgmap v387: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68/444 objects degraded (15.315%) 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2445001991' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[101432]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[105288]: osdmap e503: 8 total, 7 up, 8 in 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[105288]: pgmap v387: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68/444 objects degraded (15.315%) 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2445001991' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:03.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:03 smithi119 ceph-mon[105288]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:03 smithi145 ceph-mon[89359]: osdmap e503: 8 total, 7 up, 8 in 2024-09-21T10:06:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:03 smithi145 ceph-mon[89359]: pgmap v387: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 68/444 objects degraded (15.315%) 2024-09-21T10:06:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2445001991' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:03 smithi145 ceph-mon[89359]: from='client.44294 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:04 smithi119 ceph-mon[101432]: osdmap e504: 8 total, 7 up, 8 in 2024-09-21T10:06:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:04 smithi119 ceph-mon[101432]: from='client.44300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:04 smithi119 ceph-mon[105288]: osdmap e504: 8 total, 7 up, 8 in 2024-09-21T10:06:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:04 smithi119 ceph-mon[105288]: from='client.44300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:04 smithi145 ceph-mon[89359]: osdmap e504: 8 total, 7 up, 8 in 2024-09-21T10:06:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:04 smithi145 ceph-mon[89359]: from='client.44300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:06:04] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-09-21T10:06:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:05 smithi119 ceph-mon[105288]: osdmap e505: 8 total, 7 up, 8 in 2024-09-21T10:06:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:05 smithi119 ceph-mon[105288]: pgmap v390: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s; 61/396 objects degraded (15.404%) 2024-09-21T10:06:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:05 smithi119 ceph-mon[101432]: osdmap e505: 8 total, 7 up, 8 in 2024-09-21T10:06:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:05 smithi119 ceph-mon[101432]: pgmap v390: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s; 61/396 objects degraded (15.404%) 2024-09-21T10:06:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:05 smithi145 ceph-mon[89359]: osdmap e505: 8 total, 7 up, 8 in 2024-09-21T10:06:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:05 smithi145 ceph-mon[89359]: pgmap v390: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s; 61/396 objects degraded (15.404%) 2024-09-21T10:06:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:05.874 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:06.170 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 61/396 objects degraded (15.404%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: osdmap e506: 8 total, 7 up, 8 in 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/710491441' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[105288]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 61/396 objects degraded (15.404%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:06.171 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: osdmap e506: 8 total, 7 up, 8 in 2024-09-21T10:06:06.172 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/710491441' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:06.172 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.172 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.172 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-mon[101432]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 61/396 objects degraded (15.404%), 23 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: osdmap e506: 8 total, 7 up, 8 in 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/710491441' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:06 smithi145 ceph-mon[89359]: from='client.44312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:06.818 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[114899]: 2024-09-21T10:06:06.695+0000 7f438dfce640 -1 osd.0 496 log_to_monitors true 2024-09-21T10:06:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[101432]: osdmap e507: 8 total, 7 up, 8 in 2024-09-21T10:06:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[101432]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[101432]: pgmap v393: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s; 61/396 objects degraded (15.404%) 2024-09-21T10:06:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[101432]: from='osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:06:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[101432]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:06:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[105288]: osdmap e507: 8 total, 7 up, 8 in 2024-09-21T10:06:07.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[105288]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:07.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[105288]: pgmap v393: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s; 61/396 objects degraded (15.404%) 2024-09-21T10:06:07.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[105288]: from='osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:06:07.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-mon[105288]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:06:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:07 smithi145 ceph-mon[89359]: osdmap e507: 8 total, 7 up, 8 in 2024-09-21T10:06:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:07 smithi145 ceph-mon[89359]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:07 smithi145 ceph-mon[89359]: pgmap v393: 105 pgs: 19 active+undersized, 23 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 80 KiB/s rd, 7.5 KiB/s wr, 116 op/s; 61/396 objects degraded (15.404%) 2024-09-21T10:06:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:07 smithi145 ceph-mon[89359]: from='osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:06:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:07 smithi145 ceph-mon[89359]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-21T10:06:07.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:06:07 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[114899]: 2024-09-21T10:06:07.579+0000 7f4381d70640 -1 osd.0 496 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:06:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:08 smithi145 ceph-mon[89359]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:06:08.686 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:08 smithi145 ceph-mon[89359]: osdmap e508: 8 total, 7 up, 8 in 2024-09-21T10:06:08.686 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:08 smithi145 ceph-mon[89359]: from='osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:08.686 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:08 smithi145 ceph-mon[89359]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:08.687 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:08 smithi145 ceph-mon[89359]: from='osd.0 ' entity='osd.0' 2024-09-21T10:06:08.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[101432]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:06:08.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[101432]: osdmap e508: 8 total, 7 up, 8 in 2024-09-21T10:06:08.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[101432]: from='osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:08.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[101432]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:08.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[101432]: from='osd.0 ' entity='osd.0' 2024-09-21T10:06:08.688 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[105288]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T10:06:08.688 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[105288]: osdmap e508: 8 total, 7 up, 8 in 2024-09-21T10:06:08.688 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[105288]: from='osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:08.688 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[105288]: from='osd.0 ' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:08.688 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:08 smithi119 ceph-mon[105288]: from='osd.0 ' entity='osd.0' 2024-09-21T10:06:08.760 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:09.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:06:09.067+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-21T10:06:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:06:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328] boot 2024-09-21T10:06:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: pgmap v396: 105 pgs: 24 peering, 9 active+undersized, 9 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s; 22/366 objects degraded (6.011%) 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3995993518' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[105288]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328] boot 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T10:06:09.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: pgmap v396: 105 pgs: 24 peering, 9 active+undersized, 9 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s; 22/366 objects degraded (6.011%) 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3995993518' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:06:09.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:06:09.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:09 smithi119 ceph-mon[101432]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-21T10:06:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:06:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: osd.0 [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328] boot 2024-09-21T10:06:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: osdmap e509: 8 total, 8 up, 8 in 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: pgmap v396: 105 pgs: 24 peering, 9 active+undersized, 9 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s; 22/366 objects degraded (6.011%) 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3995993518' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='client.44333 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: osdmap e510: 8 total, 8 up, 8 in 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:09.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:06:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:06:09.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:09 smithi145 ceph-mon[89359]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-21T10:06:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:10 smithi145 ceph-mon[89359]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:06:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:06:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:10 smithi119 ceph-mon[105288]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:10 smithi119 ceph-mon[101432]: from='client.44339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:11 smithi145 ceph-mon[89359]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T10:06:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:11 smithi145 ceph-mon[89359]: pgmap v399: 105 pgs: 24 peering, 9 active+undersized, 9 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s; 22/366 objects degraded (6.011%) 2024-09-21T10:06:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:11 smithi119 ceph-mon[101432]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T10:06:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:11 smithi119 ceph-mon[101432]: pgmap v399: 105 pgs: 24 peering, 9 active+undersized, 9 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s; 22/366 objects degraded (6.011%) 2024-09-21T10:06:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:11 smithi119 ceph-mon[105288]: osdmap e511: 8 total, 8 up, 8 in 2024-09-21T10:06:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:11 smithi119 ceph-mon[105288]: pgmap v399: 105 pgs: 24 peering, 9 active+undersized, 9 active+undersized+degraded, 63 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s; 22/366 objects degraded (6.011%) 2024-09-21T10:06:11.684 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:12.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:12 smithi145 ceph-mon[89359]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T10:06:12.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2430571763' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:12.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:12 smithi145 ceph-mon[89359]: from='client.44351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:12.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:12 smithi145 ceph-mon[89359]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[105288]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T10:06:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2430571763' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[105288]: from='client.44351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[105288]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[101432]: osdmap e512: 8 total, 8 up, 8 in 2024-09-21T10:06:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2430571763' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:12.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[101432]: from='client.44351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:12.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:12 smithi119 ceph-mon[101432]: from='client.34473 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:13.218 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestMirroring.AioSnapshotCreate (33155 ms) 2024-09-21T10:06:13.457 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:13 smithi145 ceph-mon[89359]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T10:06:13.457 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:13 smithi145 ceph-mon[89359]: pgmap v402: 105 pgs: 24 peering, 81 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 63 KiB/s rd, 4.0 KiB/s wr, 87 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T10:06:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:13 smithi119 ceph-mon[101432]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T10:06:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:13 smithi119 ceph-mon[101432]: pgmap v402: 105 pgs: 24 peering, 81 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 63 KiB/s rd, 4.0 KiB/s wr, 87 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T10:06:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:13 smithi119 ceph-mon[105288]: osdmap e513: 8 total, 8 up, 8 in 2024-09-21T10:06:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:13 smithi119 ceph-mon[105288]: pgmap v402: 105 pgs: 24 peering, 81 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 63 KiB/s rd, 4.0 KiB/s wr, 87 op/s; 0 B/s, 2 objects/s recovering 2024-09-21T10:06:14.219 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 51 tests from TestMirroring (87312 ms total) 2024-09-21T10:06:14.220 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:06:14.220 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 34 tests from TestInternal 2024-09-21T10:06:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:14 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/366 objects degraded (6.011%), 9 pgs degraded) 2024-09-21T10:06:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:14 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:06:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:14 smithi145 ceph-mon[89359]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T10:06:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/0c888f67-7971-4c3e-939b-19b94d31f60b"} : dispatch 2024-09-21T10:06:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/0c888f67-7971-4c3e-939b-19b94d31f60b"}]': finished 2024-09-21T10:06:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/366 objects degraded (6.011%), 9 pgs degraded) 2024-09-21T10:06:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:06:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[105288]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T10:06:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/0c888f67-7971-4c3e-939b-19b94d31f60b"} : dispatch 2024-09-21T10:06:14.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/0c888f67-7971-4c3e-939b-19b94d31f60b"}]': finished 2024-09-21T10:06:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/366 objects degraded (6.011%), 9 pgs degraded) 2024-09-21T10:06:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:06:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[101432]: osdmap e514: 8 total, 8 up, 8 in 2024-09-21T10:06:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/16/0c888f67-7971-4c3e-939b-19b94d31f60b"} : dispatch 2024-09-21T10:06:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/620402434' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/16/0c888f67-7971-4c3e-939b-19b94d31f60b"}]': finished 2024-09-21T10:06:14.546 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:06:14] "GET /metrics HTTP/1.1" 200 36596 "" "Prometheus/2.51.0" 2024-09-21T10:06:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:15 smithi145 ceph-mon[89359]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T10:06:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4041160742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:15 smithi145 ceph-mon[89359]: pgmap v405: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:15 smithi145 ceph-mon[89359]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:15 smithi145 ceph-mon[89359]: from='client.44387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[101432]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T10:06:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4041160742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[101432]: pgmap v405: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[101432]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[101432]: from='client.44387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[105288]: osdmap e515: 8 total, 8 up, 8 in 2024-09-21T10:06:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4041160742' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[105288]: pgmap v405: 73 pgs: 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[105288]: from='client.44381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:15 smithi119 ceph-mon[105288]: from='client.44387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:16.239 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.OpenByID 2024-09-21T10:06:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:16 smithi145 ceph-mon[89359]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T10:06:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3516806504' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:16 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:16 smithi119 ceph-mon[101432]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T10:06:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3516806504' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:16 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:16 smithi119 ceph-mon[105288]: osdmap e516: 8 total, 8 up, 8 in 2024-09-21T10:06:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3516806504' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:16 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:17.400 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:17 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:17 smithi145 ceph-mon[89359]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T10:06:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:17 smithi145 ceph-mon[89359]: pgmap v408: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4018902572' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[101432]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[101432]: pgmap v408: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4018902572' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[105288]: osdmap e517: 8 total, 8 up, 8 in 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[105288]: pgmap v408: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4018902572' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:18.273 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.OpenByID (2034 ms) 2024-09-21T10:06:18.273 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.OpenSnapDNE 2024-09-21T10:06:18.297 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.OpenSnapDNE (25 ms) 2024-09-21T10:06:18.297 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.IsExclusiveLockOwner 2024-09-21T10:06:18.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.IsExclusiveLockOwner (25 ms) 2024-09-21T10:06:18.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.ResizeLocksImage 2024-09-21T10:06:18.343 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.ResizeLocksImage (21 ms) 2024-09-21T10:06:18.343 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.ResizeFailsToLockImage 2024-09-21T10:06:18.363 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.ResizeFailsToLockImage (20 ms) 2024-09-21T10:06:18.364 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapCreateLocksImage 2024-09-21T10:06:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:18 smithi145 ceph-mon[89359]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T10:06:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:18 smithi145 ceph-mon[89359]: from='client.44399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:18 smithi145 ceph-mon[89359]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[105288]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T10:06:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[105288]: from='client.44399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[105288]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[101432]: osdmap e518: 8 total, 8 up, 8 in 2024-09-21T10:06:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[101432]: from='client.44399 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[101432]: from='client.34521 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:19 smithi145 ceph-mon[89359]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T10:06:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:19 smithi145 ceph-mon[89359]: pgmap v411: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 62 op/s 2024-09-21T10:06:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:19 smithi119 ceph-mon[101432]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T10:06:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:19 smithi119 ceph-mon[101432]: pgmap v411: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 62 op/s 2024-09-21T10:06:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:19 smithi119 ceph-mon[105288]: osdmap e519: 8 total, 8 up, 8 in 2024-09-21T10:06:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:19 smithi119 ceph-mon[105288]: pgmap v411: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 62 op/s 2024-09-21T10:06:20.262 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:20.265 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapCreateLocksImage (1902 ms) 2024-09-21T10:06:20.265 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapCreateFailsToLockImage 2024-09-21T10:06:20.287 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapCreateFailsToLockImage (21 ms) 2024-09-21T10:06:20.287 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapRollbackLocksImage 2024-09-21T10:06:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:20 smithi145 ceph-mon[89359]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T10:06:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1972582108' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:06:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:06:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:20 smithi119 ceph-mon[101432]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T10:06:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1972582108' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:20 smithi119 ceph-mon[105288]: osdmap e520: 8 total, 8 up, 8 in 2024-09-21T10:06:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1972582108' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:21 smithi145 ceph-mon[89359]: from='client.44417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:21 smithi145 ceph-mon[89359]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T10:06:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:21 smithi145 ceph-mon[89359]: pgmap v414: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 62 op/s 2024-09-21T10:06:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:21 smithi145 ceph-mon[89359]: from='client.44423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[105288]: from='client.44417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[105288]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[105288]: pgmap v414: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 62 op/s 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[105288]: from='client.44423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[101432]: from='client.44417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[101432]: osdmap e521: 8 total, 8 up, 8 in 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[101432]: pgmap v414: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 5.5 KiB/s wr, 62 op/s 2024-09-21T10:06:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:21 smithi119 ceph-mon[101432]: from='client.44423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:22.283 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapRollbackLocksImage (1997 ms) 2024-09-21T10:06:22.283 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapRollbackFailsToLockImage 2024-09-21T10:06:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:22 smithi145 ceph-mon[89359]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T10:06:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:22 smithi119 ceph-mon[105288]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T10:06:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:22 smithi119 ceph-mon[101432]: osdmap e522: 8 total, 8 up, 8 in 2024-09-21T10:06:23.122 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:23 smithi145 ceph-mon[89359]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T10:06:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:23 smithi145 ceph-mon[89359]: pgmap v417: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 1.5 KiB/s wr, 26 op/s 2024-09-21T10:06:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/13000517' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:23 smithi119 ceph-mon[105288]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T10:06:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:23 smithi119 ceph-mon[105288]: pgmap v417: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 1.5 KiB/s wr, 26 op/s 2024-09-21T10:06:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/13000517' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:23 smithi119 ceph-mon[101432]: osdmap e523: 8 total, 8 up, 8 in 2024-09-21T10:06:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:23 smithi119 ceph-mon[101432]: pgmap v417: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 1.5 KiB/s wr, 26 op/s 2024-09-21T10:06:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/13000517' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:24.026 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapRollbackFailsToLockImage (1743 ms) 2024-09-21T10:06:24.026 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapSetReleasesLock 2024-09-21T10:06:24.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[105288]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:24.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[105288]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[105288]: from='client.44441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[105288]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[101432]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[101432]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[101432]: from='client.44441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[101432]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T10:06:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:06:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:24 smithi145 ceph-mon[89359]: from='client.34551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:24 smithi145 ceph-mon[89359]: osdmap e524: 8 total, 8 up, 8 in 2024-09-21T10:06:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:24 smithi145 ceph-mon[89359]: from='client.44441 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:24 smithi145 ceph-mon[89359]: osdmap e525: 8 total, 8 up, 8 in 2024-09-21T10:06:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:24 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16} : dispatch 2024-09-21T10:06:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:06:24] "GET /metrics HTTP/1.1" 200 36532 "" "Prometheus/2.51.0" 2024-09-21T10:06:25.998 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:26.045 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapSetReleasesLock (2019 ms) 2024-09-21T10:06:26.045 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.FlattenLocksImage 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: Upgrade: osd.1 is safe to restart 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: pgmap v420: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 9.7 KiB/s wr, 137 op/s 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:06:26.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1270045860' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: Upgrade: osd.1 is safe to restart 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: pgmap v420: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 9.7 KiB/s wr, 137 op/s 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1270045860' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-21T10:06:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: Upgrade: osd.1 is safe to restart 2024-09-21T10:06:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: pgmap v420: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 97 KiB/s rd, 9.7 KiB/s wr, 137 op/s 2024-09-21T10:06:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: osdmap e526: 8 total, 8 up, 8 in 2024-09-21T10:06:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:26.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-21T10:06:26.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:26.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1270045860' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:27 smithi145 ceph-mon[89359]: Upgrade: Updating osd.1 2024-09-21T10:06:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:27 smithi145 ceph-mon[89359]: Deploying daemon osd.1 on smithi119 2024-09-21T10:06:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:27 smithi145 ceph-mon[89359]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:27 smithi145 ceph-mon[89359]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T10:06:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[101432]: Upgrade: Updating osd.1 2024-09-21T10:06:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[101432]: Deploying daemon osd.1 on smithi119 2024-09-21T10:06:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[101432]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[101432]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T10:06:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[105288]: Upgrade: Updating osd.1 2024-09-21T10:06:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[105288]: Deploying daemon osd.1 on smithi119 2024-09-21T10:06:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[105288]: from='client.34569 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:27 smithi119 ceph-mon[105288]: osdmap e527: 8 total, 8 up, 8 in 2024-09-21T10:06:28.065 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.FlattenLocksImage (2020 ms) 2024-09-21T10:06:28.065 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.FlattenFailsToLockImage 2024-09-21T10:06:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:28 smithi145 ceph-mon[89359]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:28 smithi145 ceph-mon[89359]: pgmap v423: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 8.2 KiB/s wr, 110 op/s 2024-09-21T10:06:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:28 smithi145 ceph-mon[89359]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T10:06:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:28 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:28.343 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[105288]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:28.343 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[105288]: pgmap v423: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 8.2 KiB/s wr, 110 op/s 2024-09-21T10:06:28.343 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[105288]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T10:06:28.343 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:28.343 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[101432]: from='client.34575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:28.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[101432]: pgmap v423: 105 pgs: 105 active+clean; 582 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 8.2 KiB/s wr, 110 op/s 2024-09-21T10:06:28.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[101432]: osdmap e528: 8 total, 8 up, 8 in 2024-09-21T10:06:28.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:28.935 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:28.975 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:28 smithi119 systemd[1]: Stopping Ceph osd.1 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:06:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:29 smithi119 ceph-mon[105288]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T10:06:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/408288294' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:29 smithi119 ceph-mon[105288]: osd.1 marked itself down and dead 2024-09-21T10:06:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:29 smithi119 ceph-mon[101432]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T10:06:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/408288294' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:29.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:29 smithi119 ceph-mon[101432]: osd.1 marked itself down and dead 2024-09-21T10:06:29.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[51253]: 2024-09-21T10:06:28.972+0000 7f4faed6c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:06:29.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[51253]: 2024-09-21T10:06:28.972+0000 7f4faed6c640 -1 osd.1 529 *** Got signal Terminated *** 2024-09-21T10:06:29.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:28 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[51253]: 2024-09-21T10:06:28.972+0000 7f4faed6c640 -1 osd.1 529 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:06:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:29 smithi145 ceph-mon[89359]: osdmap e529: 8 total, 8 up, 8 in 2024-09-21T10:06:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/408288294' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:29 smithi145 ceph-mon[89359]: osd.1 marked itself down and dead 2024-09-21T10:06:29.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:29 smithi119 podman[119623]: 2024-09-21 10:06:29.594241571 +0000 UTC m=+0.718784362 container died b8f7b7d5ac6487bfe40874bbdb2e99ee900bf2d6b62f7d7270155e2209bc7d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True) 2024-09-21T10:06:30.249 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 podman[119623]: 2024-09-21 10:06:30.054115589 +0000 UTC m=+1.178658421 container cleanup b8f7b7d5ac6487bfe40874bbdb2e99ee900bf2d6b62f7d7270155e2209bc7d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-21T10:06:30.249 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 bash[119623]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[101432]: pgmap v426: 105 pgs: 105 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 8.7 KiB/s wr, 154 op/s 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[101432]: from='client.34587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[101432]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[101432]: osdmap e530: 8 total, 7 up, 8 in 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[105288]: pgmap v426: 105 pgs: 105 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 8.7 KiB/s wr, 154 op/s 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[105288]: from='client.34587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[105288]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:06:30.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:30 smithi119 ceph-mon[105288]: osdmap e530: 8 total, 7 up, 8 in 2024-09-21T10:06:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:06:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:06:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:30 smithi145 ceph-mon[89359]: pgmap v426: 105 pgs: 105 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 8.7 KiB/s wr, 154 op/s 2024-09-21T10:06:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:30 smithi145 ceph-mon[89359]: from='client.34587 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:30 smithi145 ceph-mon[89359]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:06:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:30 smithi145 ceph-mon[89359]: osdmap e530: 8 total, 7 up, 8 in 2024-09-21T10:06:30.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 podman[119670]: 2024-09-21 10:06:30.247943271 +0000 UTC m=+0.657237865 container remove b8f7b7d5ac6487bfe40874bbdb2e99ee900bf2d6b62f7d7270155e2209bc7d3e (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=quincy-fbd1a6a) 2024-09-21T10:06:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 podman[119767]: 2024-09-21 10:06:30.566288789 +0000 UTC m=+0.015804789 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:06:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 podman[119767]: 2024-09-21 10:06:30.674238625 +0000 UTC m=+0.123754653 container create 1f01cf37c7b715a06644cb88e07adfd78c9e1161faed6b4086ebe8c0a94b4003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T10:06:31.074 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.FlattenFailsToLockImage (3009 ms) 2024-09-21T10:06:31.074 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklisted 2024-09-21T10:06:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:31 smithi119 ceph-mon[101432]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:31 smithi119 ceph-mon[101432]: osdmap e531: 8 total, 7 up, 8 in 2024-09-21T10:06:31.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:31 smithi119 ceph-mon[105288]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:31.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:31 smithi119 ceph-mon[105288]: osdmap e531: 8 total, 7 up, 8 in 2024-09-21T10:06:31.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 podman[119767]: 2024-09-21 10:06:30.926967297 +0000 UTC m=+0.376483283 container init 1f01cf37c7b715a06644cb88e07adfd78c9e1161faed6b4086ebe8c0a94b4003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T10:06:31.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:30 smithi119 podman[119767]: 2024-09-21 10:06:30.931811096 +0000 UTC m=+0.381327083 container start 1f01cf37c7b715a06644cb88e07adfd78c9e1161faed6b4086ebe8c0a94b4003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-21T10:06:31.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 podman[119767]: 2024-09-21 10:06:31.043456279 +0000 UTC m=+0.492972285 container attach 1f01cf37c7b715a06644cb88e07adfd78c9e1161faed6b4086ebe8c0a94b4003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T10:06:31.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 podman[119767]: 2024-09-21 10:06:31.129019243 +0000 UTC m=+0.578535240 container died 1f01cf37c7b715a06644cb88e07adfd78c9e1161faed6b4086ebe8c0a94b4003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T10:06:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:31 smithi145 ceph-mon[89359]: from='client.44477 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:31 smithi145 ceph-mon[89359]: osdmap e531: 8 total, 7 up, 8 in 2024-09-21T10:06:31.805 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:32.065 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 podman[119829]: 2024-09-21 10:06:31.705599296 +0000 UTC m=+0.570908276 container remove 1f01cf37c7b715a06644cb88e07adfd78c9e1161faed6b4086ebe8c0a94b4003 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-21T10:06:32.065 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-21T10:06:32.065 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 systemd[1]: Stopped Ceph osd.1 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:06:32.065 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.1.service: Consumed 21.105s CPU time. 2024-09-21T10:06:32.065 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:31 smithi119 systemd[1]: Starting Ceph osd.1 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:06:32.073 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.WriteFailsToLockImageBlocklisted (999 ms) 2024-09-21T10:06:32.073 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.WriteFailsToLockImageBlocklistedWatch 2024-09-21T10:06:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[105288]: pgmap v429: 105 pgs: 13 stale+active+clean, 92 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 8.7 KiB/s wr, 154 op/s 2024-09-21T10:06:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[105288]: osdmap e532: 8 total, 7 up, 8 in 2024-09-21T10:06:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2535372813' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/2535372813"} : dispatch 2024-09-21T10:06:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2134213604' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[101432]: pgmap v429: 105 pgs: 13 stale+active+clean, 92 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 8.7 KiB/s wr, 154 op/s 2024-09-21T10:06:32.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[101432]: osdmap e532: 8 total, 7 up, 8 in 2024-09-21T10:06:32.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2535372813' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/2535372813"} : dispatch 2024-09-21T10:06:32.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2134213604' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:32.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 podman[120002]: 2024-09-21 10:06:32.044999837 +0000 UTC m=+0.016107332 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:06:32.368 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 podman[120002]: 2024-09-21 10:06:32.174094583 +0000 UTC m=+0.145202064 container create a83636ec13ee9777a5b6bd182c72f8def935fd8ca45606510188bc0636cf60a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T10:06:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:32 smithi145 ceph-mon[89359]: pgmap v429: 105 pgs: 13 stale+active+clean, 92 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 8.7 KiB/s wr, 154 op/s 2024-09-21T10:06:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:32 smithi145 ceph-mon[89359]: osdmap e532: 8 total, 7 up, 8 in 2024-09-21T10:06:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2535372813' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/2535372813"} : dispatch 2024-09-21T10:06:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2134213604' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:32.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 podman[120002]: 2024-09-21 10:06:32.410283315 +0000 UTC m=+0.381390807 container init a83636ec13ee9777a5b6bd182c72f8def935fd8ca45606510188bc0636cf60a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T10:06:32.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 podman[120002]: 2024-09-21 10:06:32.415018675 +0000 UTC m=+0.386126167 container start a83636ec13ee9777a5b6bd182c72f8def935fd8ca45606510188bc0636cf60a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.license=GPLv2) 2024-09-21T10:06:32.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 podman[120002]: 2024-09-21 10:06:32.510192655 +0000 UTC m=+0.481300145 container attach a83636ec13ee9777a5b6bd182c72f8def935fd8ca45606510188bc0636cf60a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T10:06:32.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:06:32.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 bash[120002]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:06:32.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:06:32.868 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:32 smithi119 bash[120002]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:06:33.309 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:06:33.309 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:06:33.309 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T10:06:33.309 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-21T10:06:33.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='client.44495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:33.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2535372813' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/2535372813"}]': finished 2024-09-21T10:06:33.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: osdmap e533: 8 total, 7 up, 8 in 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3992357764' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2, 3, 1]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 2]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 1]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.4", "id": [0, 6]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.b", "id": [0, 1]} : dispatch 2024-09-21T10:06:33.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.1a", "id": [0, 1]} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='client.44495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2535372813' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/2535372813"}]': finished 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: osdmap e533: 8 total, 7 up, 8 in 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3992357764' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2, 3, 1]} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 2]} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 1]} : dispatch 2024-09-21T10:06:33.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.4", "id": [0, 6]} : dispatch 2024-09-21T10:06:33.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.b", "id": [0, 1]} : dispatch 2024-09-21T10:06:33.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.1a", "id": [0, 1]} : dispatch 2024-09-21T10:06:33.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:33.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='client.44495 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2535372813' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/2535372813"}]': finished 2024-09-21T10:06:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: osdmap e533: 8 total, 7 up, 8 in 2024-09-21T10:06:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3992357764' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"} : dispatch 2024-09-21T10:06:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2, 3, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 2]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.4", "id": [0, 6]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.b", "id": [0, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.1a", "id": [0, 1]} : dispatch 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:33.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T10:06:33.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-21T10:06:33.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T10:06:33.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-21T10:06:33.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T10:06:33.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-21T10:06:33.618 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:06:33.618 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T10:06:33.618 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate[120038]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T10:06:33.618 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 bash[120002]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-21T10:06:33.618 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:33 smithi119 podman[120002]: 2024-09-21 10:06:33.340524434 +0000 UTC m=+1.311631944 container died a83636ec13ee9777a5b6bd182c72f8def935fd8ca45606510188bc0636cf60a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:06:34.086 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.WriteFailsToLockImageBlocklistedWatch (2013 ms) 2024-09-21T10:06:34.086 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.AioWriteRequestsLock 2024-09-21T10:06:34.367 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 podman[120201]: 2024-09-21 10:06:34.048003776 +0000 UTC m=+0.702115271 container remove a83636ec13ee9777a5b6bd182c72f8def935fd8ca45606510188bc0636cf60a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T10:06:34.367 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 podman[120276]: 2024-09-21 10:06:34.171257041 +0000 UTC m=+0.023402888 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:06:34.367 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 podman[120276]: 2024-09-21 10:06:34.291040168 +0000 UTC m=+0.143186000 container create e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:06:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='client.34617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: pgmap v432: 105 pgs: 4 active+undersized, 11 stale+active+clean, 4 active+undersized+degraded, 86 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 3.0 KiB/s wr, 77 op/s; 7/204 objects degraded (3.431%) 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 7/204 objects degraded (3.431%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2, 3, 1]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 2]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 1]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.4", "id": [0, 6]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.b", "id": [0, 1]}]': finished 2024-09-21T10:06:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.1a", "id": [0, 1]}]': finished 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: osdmap e534: 8 total, 7 up, 8 in 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[105288]: osdmap e535: 8 total, 7 up, 8 in 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='client.34617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: pgmap v432: 105 pgs: 4 active+undersized, 11 stale+active+clean, 4 active+undersized+degraded, 86 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 3.0 KiB/s wr, 77 op/s; 7/204 objects degraded (3.431%) 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 7/204 objects degraded (3.431%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"}]': finished 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]': finished 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2, 3, 1]}]': finished 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 2]}]': finished 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]}]': finished 2024-09-21T10:06:34.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 1]}]': finished 2024-09-21T10:06:34.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]': finished 2024-09-21T10:06:34.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.4", "id": [0, 6]}]': finished 2024-09-21T10:06:34.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.b", "id": [0, 1]}]': finished 2024-09-21T10:06:34.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.1a", "id": [0, 1]}]': finished 2024-09-21T10:06:34.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: osdmap e534: 8 total, 7 up, 8 in 2024-09-21T10:06:34.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-mon[101432]: osdmap e535: 8 total, 7 up, 8 in 2024-09-21T10:06:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='client.34617 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: pgmap v432: 105 pgs: 4 active+undersized, 11 stale+active+clean, 4 active+undersized+degraded, 86 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 61 KiB/s rd, 3.0 KiB/s wr, 77 op/s; 7/204 objects degraded (3.431%) 2024-09-21T10:06:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 7/204 objects degraded (3.431%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/3992357764"}]': finished 2024-09-21T10:06:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1", "id": [4, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [4, 2, 3, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 2]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.15", "id": [7, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.4", "id": [0, 6]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.b", "id": [0, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "17.1a", "id": [0, 1]}]': finished 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: osdmap e534: 8 total, 7 up, 8 in 2024-09-21T10:06:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:34 smithi145 ceph-mon[89359]: osdmap e535: 8 total, 7 up, 8 in 2024-09-21T10:06:34.705 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:06:34] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T10:06:34.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 podman[120276]: 2024-09-21 10:06:34.518914156 +0000 UTC m=+0.371060000 container init e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T10:06:34.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 podman[120276]: 2024-09-21 10:06:34.523804261 +0000 UTC m=+0.375950102 container start e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:06:34.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 ceph-osd[120356]: -- 172.21.15.119:0/4257507594 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5607565ffa40 con 0x5607565ddc00 2024-09-21T10:06:34.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 bash[120276]: e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b 2024-09-21T10:06:34.868 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:34 smithi119 systemd[1]: Started Ceph osd.1 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:06:35.050 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.AioWriteRequestsLock (964 ms) 2024-09-21T10:06:35.050 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.AioDiscardRequestsLock 2024-09-21T10:06:35.075 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.AioDiscardRequestsLock (25 ms) 2024-09-21T10:06:35.076 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.CancelAsyncResize 2024-09-21T10:06:35.117 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.CancelAsyncResize (42 ms) 2024-09-21T10:06:35.117 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.MultipleResize 2024-09-21T10:06:35.176 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.MultipleResize (59 ms) 2024-09-21T10:06:35.177 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.Metadata 2024-09-21T10:06:35.222 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.Metadata (45 ms) 2024-09-21T10:06:35.222 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.MetadataConfApply 2024-09-21T10:06:35.249 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.MetadataConfApply (27 ms) 2024-09-21T10:06:35.249 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapshotCopyup 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2926105826' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[105288]: osdmap e536: 8 total, 7 up, 8 in 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2926105826' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:35.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:35.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:35 smithi119 ceph-mon[101432]: osdmap e536: 8 total, 7 up, 8 in 2024-09-21T10:06:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2926105826' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:35 smithi145 ceph-mon[89359]: osdmap e536: 8 total, 7 up, 8 in 2024-09-21T10:06:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[105288]: pgmap v435: 105 pgs: 5 peering, 30 active+undersized, 13 active+undersized+degraded, 57 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 89 KiB/s rd, 7.0 KiB/s wr, 124 op/s; 21/225 objects degraded (9.333%) 2024-09-21T10:06:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[105288]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[105288]: from='client.44528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:36.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:06:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[101432]: pgmap v435: 105 pgs: 5 peering, 30 active+undersized, 13 active+undersized+degraded, 57 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 89 KiB/s rd, 7.0 KiB/s wr, 124 op/s; 21/225 objects degraded (9.333%) 2024-09-21T10:06:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[101432]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[101432]: from='client.44528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:06:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:36 smithi145 ceph-mon[89359]: pgmap v435: 105 pgs: 5 peering, 30 active+undersized, 13 active+undersized+degraded, 57 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 89 KiB/s rd, 7.0 KiB/s wr, 124 op/s; 21/225 objects degraded (9.333%) 2024-09-21T10:06:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:36 smithi145 ceph-mon[89359]: from='client.34635 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:36 smithi145 ceph-mon[89359]: from='client.44528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:36 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:06:36.867 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:36 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[120311]: 2024-09-21T10:06:36.416+0000 7fe2cae63640 -1 Falling back to public interface 2024-09-21T10:06:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:37 smithi119 ceph-mon[105288]: osdmap e537: 8 total, 7 up, 8 in 2024-09-21T10:06:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:37 smithi119 ceph-mon[105288]: pgmap v438: 105 pgs: 5 peering, 30 active+undersized, 13 active+undersized+degraded, 57 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.0 KiB/s wr, 47 op/s; 21/225 objects degraded (9.333%) 2024-09-21T10:06:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:37 smithi119 ceph-mon[101432]: osdmap e537: 8 total, 7 up, 8 in 2024-09-21T10:06:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:37 smithi119 ceph-mon[101432]: pgmap v438: 105 pgs: 5 peering, 30 active+undersized, 13 active+undersized+degraded, 57 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.0 KiB/s wr, 47 op/s; 21/225 objects degraded (9.333%) 2024-09-21T10:06:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:37 smithi145 ceph-mon[89359]: osdmap e537: 8 total, 7 up, 8 in 2024-09-21T10:06:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:37 smithi145 ceph-mon[89359]: pgmap v438: 105 pgs: 5 peering, 30 active+undersized, 13 active+undersized+degraded, 57 active+clean; 583 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 4.0 KiB/s wr, 47 op/s; 21/225 objects degraded (9.333%) 2024-09-21T10:06:37.600 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:38.185 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapshotCopyup (2937 ms) 2024-09-21T10:06:38.185 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapshotCopyupZeros 2024-09-21T10:06:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[105288]: osdmap e538: 8 total, 7 up, 8 in 2024-09-21T10:06:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4199429016' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[105288]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[105288]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[101432]: osdmap e538: 8 total, 7 up, 8 in 2024-09-21T10:06:38.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4199429016' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:38.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[101432]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:38.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:38 smithi119 ceph-mon[101432]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:38 smithi145 ceph-mon[89359]: osdmap e538: 8 total, 7 up, 8 in 2024-09-21T10:06:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4199429016' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:38 smithi145 ceph-mon[89359]: from='client.34656 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:38 smithi145 ceph-mon[89359]: from='client.34662 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: osdmap e539: 8 total, 7 up, 8 in 2024-09-21T10:06:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: pgmap v441: 105 pgs: 8 active+clean+remapped, 28 active+undersized, 14 active+undersized+degraded, 55 active+clean; 586 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 252 KiB/s rd, 30 KiB/s wr, 355 op/s; 31/327 objects degraded (9.480%); 10/327 objects misplaced (3.058%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 21/225 objects degraded (9.333%), 13 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: osdmap e539: 8 total, 7 up, 8 in 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: pgmap v441: 105 pgs: 8 active+clean+remapped, 28 active+undersized, 14 active+undersized+degraded, 55 active+clean; 586 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 252 KiB/s rd, 30 KiB/s wr, 355 op/s; 31/327 objects degraded (9.480%); 10/327 objects misplaced (3.058%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 21/225 objects degraded (9.333%), 13 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: osdmap e539: 8 total, 7 up, 8 in 2024-09-21T10:06:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: pgmap v441: 105 pgs: 8 active+clean+remapped, 28 active+undersized, 14 active+undersized+degraded, 55 active+clean; 586 KiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 252 KiB/s rd, 30 KiB/s wr, 355 op/s; 31/327 objects degraded (9.480%); 10/327 objects misplaced (3.058%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 21/225 objects degraded (9.333%), 13 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:39.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:40.363 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapshotCopyupZeros (2177 ms) 2024-09-21T10:06:40.363 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SnapshotCopyupZerosMigration 2024-09-21T10:06:40.367 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[120311]: 2024-09-21T10:06:40.063+0000 7fe2cae63640 -1 osd.1 529 log_to_monitors true 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[105288]: osdmap e540: 8 total, 7 up, 8 in 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[105288]: from='osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[105288]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[101432]: osdmap e540: 8 total, 7 up, 8 in 2024-09-21T10:06:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[101432]: from='osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:06:40.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-mon[101432]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:06:40.523 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:06:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:06:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:40 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:06:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:40 smithi145 ceph-mon[89359]: osdmap e540: 8 total, 7 up, 8 in 2024-09-21T10:06:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:40 smithi145 ceph-mon[89359]: from='osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:06:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:40 smithi145 ceph-mon[89359]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-21T10:06:40.866 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:06:40 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[120311]: 2024-09-21T10:06:40.609+0000 7fe2bec05640 -1 osd.1 529 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:06:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:06:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: osdmap e541: 8 total, 7 up, 8 in 2024-09-21T10:06:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2622577878' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: pgmap v444: 105 pgs: 8 active+clean+remapped, 28 active+undersized, 14 active+undersized+degraded, 55 active+clean; 586 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 252 KiB/s rd, 30 KiB/s wr, 355 op/s; 31/327 objects degraded (9.480%); 10/327 objects misplaced (3.058%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:41.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='client.44561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:41.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='osd.1 ' entity='osd.1' 2024-09-21T10:06:41.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:41 smithi145 ceph-mon[89359]: from='client.44567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: osdmap e541: 8 total, 7 up, 8 in 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2622577878' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: pgmap v444: 105 pgs: 8 active+clean+remapped, 28 active+undersized, 14 active+undersized+degraded, 55 active+clean; 586 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 252 KiB/s rd, 30 KiB/s wr, 355 op/s; 31/327 objects degraded (9.480%); 10/327 objects misplaced (3.058%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='client.44561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='osd.1 ' entity='osd.1' 2024-09-21T10:06:41.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[105288]: from='client.44567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: osdmap e541: 8 total, 7 up, 8 in 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='osd.1 ' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2622577878' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: pgmap v444: 105 pgs: 8 active+clean+remapped, 28 active+undersized, 14 active+undersized+degraded, 55 active+clean; 586 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 252 KiB/s rd, 30 KiB/s wr, 355 op/s; 31/327 objects degraded (9.480%); 10/327 objects misplaced (3.058%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='client.44561 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='osd.1 ' entity='osd.1' 2024-09-21T10:06:41.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:41 smithi119 ceph-mon[101432]: from='client.44567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:42.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[105288]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:06:42.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[105288]: osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805] boot 2024-09-21T10:06:42.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[105288]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T10:06:42.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:06:42.164 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[101432]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:06:42.164 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[101432]: osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805] boot 2024-09-21T10:06:42.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[101432]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T10:06:42.165 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:06:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:42 smithi145 ceph-mon[89359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:06:42.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:42 smithi145 ceph-mon[89359]: osd.1 [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805] boot 2024-09-21T10:06:42.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:42 smithi145 ceph-mon[89359]: osdmap e542: 8 total, 8 up, 8 in 2024-09-21T10:06:42.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-21T10:06:42.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:06:42.290+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T10:06:43.415 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: pgmap v446: 105 pgs: 5 peering, 3 remapped+peering, 5 active+clean+remapped, 26 active+undersized, 11 active+undersized+degraded, 55 active+clean; 586 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 221 KiB/s rd, 26 KiB/s wr, 311 op/s; 27/330 objects degraded (8.182%); 9/330 objects misplaced (2.727%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T10:06:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4091622022' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: pgmap v446: 105 pgs: 5 peering, 3 remapped+peering, 5 active+clean+remapped, 26 active+undersized, 11 active+undersized+degraded, 55 active+clean; 586 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 221 KiB/s rd, 26 KiB/s wr, 311 op/s; 27/330 objects degraded (8.182%); 9/330 objects misplaced (2.727%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:43.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4091622022' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: pgmap v446: 105 pgs: 5 peering, 3 remapped+peering, 5 active+clean+remapped, 26 active+undersized, 11 active+undersized+degraded, 55 active+clean; 586 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 221 KiB/s rd, 26 KiB/s wr, 311 op/s; 27/330 objects degraded (8.182%); 9/330 objects misplaced (2.727%); 0 B/s, 0 objects/s recovering 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: osdmap e543: 8 total, 8 up, 8 in 2024-09-21T10:06:43.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4091622022' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:43.637 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SnapshotCopyupZerosMigration (3275 ms) 2024-09-21T10:06:43.637 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.ResizeCopyup 2024-09-21T10:06:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:06:44] "GET /metrics HTTP/1.1" 200 36577 "" "Prometheus/2.51.0" 2024-09-21T10:06:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[105288]: from='client.44579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[105288]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T10:06:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[105288]: from='client.34701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 27/330 objects degraded (8.182%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[101432]: from='client.44579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[101432]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T10:06:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[101432]: from='client.34701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 27/330 objects degraded (8.182%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:44 smithi145 ceph-mon[89359]: from='client.44579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:44 smithi145 ceph-mon[89359]: osdmap e544: 8 total, 8 up, 8 in 2024-09-21T10:06:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:44 smithi145 ceph-mon[89359]: from='client.34701 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 27/330 objects degraded (8.182%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:06:45.681 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.ResizeCopyup (2044 ms) 2024-09-21T10:06:45.681 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.DiscardCopyup 2024-09-21T10:06:45.689 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:06:45.689 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.DiscardCopyup (7 ms) 2024-09-21T10:06:45.689 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.ImageOptions 2024-09-21T10:06:45.696 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.ImageOptions (8 ms) 2024-09-21T10:06:45.696 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.WriteFullCopyup 2024-09-21T10:06:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:45 smithi145 ceph-mon[89359]: pgmap v449: 105 pgs: 7 peering, 3 remapped+peering, 3 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 79 active+clean; 587 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.4 KiB/s wr, 104 op/s; 11/360 objects degraded (3.056%); 7/360 objects misplaced (1.944%); 271 B/s, 5 keys/s, 2 objects/s recovering 2024-09-21T10:06:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:45 smithi145 ceph-mon[89359]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T10:06:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:45 smithi119 ceph-mon[101432]: pgmap v449: 105 pgs: 7 peering, 3 remapped+peering, 3 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 79 active+clean; 587 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.4 KiB/s wr, 104 op/s; 11/360 objects degraded (3.056%); 7/360 objects misplaced (1.944%); 271 B/s, 5 keys/s, 2 objects/s recovering 2024-09-21T10:06:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:45 smithi119 ceph-mon[101432]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T10:06:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:45 smithi119 ceph-mon[105288]: pgmap v449: 105 pgs: 7 peering, 3 remapped+peering, 3 active+clean+remapped, 10 active+undersized, 3 active+undersized+degraded, 79 active+clean; 587 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.4 KiB/s wr, 104 op/s; 11/360 objects degraded (3.056%); 7/360 objects misplaced (1.944%); 271 B/s, 5 keys/s, 2 objects/s recovering 2024-09-21T10:06:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:45 smithi119 ceph-mon[105288]: osdmap e545: 8 total, 8 up, 8 in 2024-09-21T10:06:46.283 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:46 smithi145 ceph-mon[89359]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T10:06:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/752276890' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:46 smithi119 ceph-mon[101432]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T10:06:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/752276890' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:46 smithi119 ceph-mon[105288]: osdmap e546: 8 total, 8 up, 8 in 2024-09-21T10:06:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/752276890' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:47 smithi145 ceph-mon[89359]: from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:47 smithi145 ceph-mon[89359]: pgmap v452: 105 pgs: 7 peering, 2 remapped+peering, 96 active+clean; 591 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 124 KiB/s rd, 11 KiB/s wr, 166 op/s; 478 B/s, 25 keys/s, 6 objects/s recovering 2024-09-21T10:06:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:47 smithi145 ceph-mon[89359]: from='client.34719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:47 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/360 objects degraded (3.056%), 3 pgs degraded) 2024-09-21T10:06:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:47 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:06:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:47 smithi145 ceph-mon[89359]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T10:06:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[105288]: from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[105288]: pgmap v452: 105 pgs: 7 peering, 2 remapped+peering, 96 active+clean; 591 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 124 KiB/s rd, 11 KiB/s wr, 166 op/s; 478 B/s, 25 keys/s, 6 objects/s recovering 2024-09-21T10:06:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[105288]: from='client.34719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/360 objects degraded (3.056%), 3 pgs degraded) 2024-09-21T10:06:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:06:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[105288]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T10:06:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[101432]: from='client.34713 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[101432]: pgmap v452: 105 pgs: 7 peering, 2 remapped+peering, 96 active+clean; 591 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 124 KiB/s rd, 11 KiB/s wr, 166 op/s; 478 B/s, 25 keys/s, 6 objects/s recovering 2024-09-21T10:06:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[101432]: from='client.34719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/360 objects degraded (3.056%), 3 pgs degraded) 2024-09-21T10:06:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:06:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:47 smithi119 ceph-mon[101432]: osdmap e547: 8 total, 8 up, 8 in 2024-09-21T10:06:49.035 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.WriteFullCopyup (3339 ms) 2024-09-21T10:06:49.035 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.DiffIterateCloneOverwrite 2024-09-21T10:06:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:48 smithi145 ceph-mon[89359]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T10:06:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:48 smithi119 ceph-mon[105288]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T10:06:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:48 smithi119 ceph-mon[101432]: osdmap e548: 8 total, 8 up, 8 in 2024-09-21T10:06:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:06:49.168 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:49 smithi145 ceph-mon[89359]: pgmap v455: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 237 op/s; 192 B/s, 26 keys/s, 4 objects/s recovering 2024-09-21T10:06:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:49 smithi145 ceph-mon[89359]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T10:06:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1314183750' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:49 smithi145 ceph-mon[89359]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[105288]: pgmap v455: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 237 op/s; 192 B/s, 26 keys/s, 4 objects/s recovering 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[105288]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1314183750' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[105288]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[101432]: pgmap v455: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.2 MiB/s rd, 3.0 MiB/s wr, 237 op/s; 192 B/s, 26 keys/s, 4 objects/s recovering 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[101432]: osdmap e549: 8 total, 8 up, 8 in 2024-09-21T10:06:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1314183750' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:50.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:49 smithi119 ceph-mon[101432]: osdmap e550: 8 total, 8 up, 8 in 2024-09-21T10:06:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:06:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:06:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:06:51.060 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.DiffIterateCloneOverwrite (2025 ms) 2024-09-21T10:06:51.060 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.TestCoR 2024-09-21T10:06:51.068 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING due to disabled rbd_copy_on_read 2024-09-21T10:06:51.069 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.TestCoR (8 ms) 2024-09-21T10:06:51.069 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.FlattenNoEmptyObjects 2024-09-21T10:06:51.285 INFO:tasks.workunit.client.0.smithi119.stdout:made parent image "image392": 4194304K (1024 * 4K) 2024-09-21T10:06:51.285 INFO:tasks.workunit.client.0.smithi119.stdout:generated random write map: 2024-09-21T10:06:51.285 INFO:tasks.workunit.client.0.smithi119.stdout: [37 , 153663 ] 2024-09-21T10:06:51.285 INFO:tasks.workunit.client.0.smithi119.stdout: [44 , 181285 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [48 , 196839 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [92 , 380372 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [93 , 381198 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [104 , 426842 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [131 , 539282 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [152 , 623489 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [156 , 640458 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [165 , 676513 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [173 , 710736 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [174 , 713648 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [175 , 716983 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [180 , 737702 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [208 , 853912 ] 2024-09-21T10:06:51.286 INFO:tasks.workunit.client.0.smithi119.stdout: [214 , 879569 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [235 , 965440 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [248 , 1017487 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [266 , 1092909 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [283 , 1161543 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [290 , 1189485 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [306 , 1256911 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [308 , 1262699 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [309 , 1268255 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [329 , 1347619 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [332 , 1362029 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [349 , 1432170 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [359 , 1473384 ] 2024-09-21T10:06:51.287 INFO:tasks.workunit.client.0.smithi119.stdout: [365 , 1496913 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [397 , 1628318 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [412 , 1688355 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [423 , 1733811 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [425 , 1742274 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [445 , 1823520 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [459 , 1881525 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [470 , 1926648 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [496 , 2033487 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [503 , 2061155 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [504 , 2065453 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [517 , 2120152 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [522 , 2138227 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [530 , 2171657 ] 2024-09-21T10:06:51.288 INFO:tasks.workunit.client.0.smithi119.stdout: [547 , 2240906 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [549 , 2249098 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [561 , 2301202 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [563 , 2307337 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [564 , 2313415 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [584 , 2395208 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [603 , 2471235 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [625 , 2560381 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [627 , 2570961 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [628 , 2575816 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [656 , 2688547 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [662 , 2713622 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [673 , 2758722 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [677 , 2774917 ] 2024-09-21T10:06:51.289 INFO:tasks.workunit.client.0.smithi119.stdout: [681 , 2792203 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [683 , 2800408 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [689 , 2823904 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [701 , 2873133 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [703 , 2879824 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [708 , 2903424 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [709 , 2905055 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [728 , 2982931 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [741 , 3035251 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [747 , 3061472 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [791 , 3243296 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [800 , 3277751 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [814 , 3334190 ] 2024-09-21T10:06:51.290 INFO:tasks.workunit.client.0.smithi119.stdout: [817 , 3349496 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [830 , 3401369 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [843 , 3454086 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [844 , 3458497 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [852 , 3491358 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [866 , 3547759 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [881 , 3609818 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [902 , 3697071 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [914 , 3745544 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [919 , 3766886 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [934 , 3826755 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [944 , 3869604 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [947 , 3881542 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [950 , 3893252 ] 2024-09-21T10:06:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: [974 , 3993003 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [981 , 4020418 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [991 , 4060946 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [992 , 4066672 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1000 , 4098422 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1005 , 4116497 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1006 , 4121370 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1010 , 4139434 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1018 , 4171949 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1021 , 4183849 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: [1023 , 4191835 ] 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout:write data based on random map 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: write object-37 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: write object-44 2024-09-21T10:06:51.292 INFO:tasks.workunit.client.0.smithi119.stdout: write object-48 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-92 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-93 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-104 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-131 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-152 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-156 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-165 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-173 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-174 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-175 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-180 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-208 2024-09-21T10:06:51.293 INFO:tasks.workunit.client.0.smithi119.stdout: write object-214 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-235 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-248 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-266 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-283 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-290 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-306 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-308 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-309 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-329 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-332 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-349 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-359 2024-09-21T10:06:51.294 INFO:tasks.workunit.client.0.smithi119.stdout: write object-365 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-397 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-412 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-423 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-425 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-445 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-459 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-470 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-496 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-503 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-504 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-517 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-522 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-530 2024-09-21T10:06:51.295 INFO:tasks.workunit.client.0.smithi119.stdout: write object-547 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-549 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-561 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-563 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-564 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-584 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-603 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-625 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-627 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-628 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-656 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-662 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-673 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-677 2024-09-21T10:06:51.296 INFO:tasks.workunit.client.0.smithi119.stdout: write object-681 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-683 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-689 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-701 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-703 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-708 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-709 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-728 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-741 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-747 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-791 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-800 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-814 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-817 2024-09-21T10:06:51.297 INFO:tasks.workunit.client.0.smithi119.stdout: write object-830 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-843 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-844 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-852 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-866 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-881 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-902 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-914 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-919 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-934 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-944 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-947 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-950 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-974 2024-09-21T10:06:51.298 INFO:tasks.workunit.client.0.smithi119.stdout: write object-981 2024-09-21T10:06:51.299 INFO:tasks.workunit.client.0.smithi119.stdout: write object-991 2024-09-21T10:06:51.299 INFO:tasks.workunit.client.0.smithi119.stdout: write object-992 2024-09-21T10:06:51.299 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1000 2024-09-21T10:06:51.299 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1005 2024-09-21T10:06:51.299 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1006 2024-09-21T10:06:51.299 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1010 2024-09-21T10:06:51.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:51 smithi145 ceph-mon[89359]: from='client.44615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:51.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:51 smithi145 ceph-mon[89359]: from='client.44621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:51.299 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:51 smithi145 ceph-mon[89359]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T10:06:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:51 smithi119 ceph-mon[105288]: from='client.44615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:51 smithi119 ceph-mon[105288]: from='client.44621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:51 smithi119 ceph-mon[105288]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T10:06:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:51 smithi119 ceph-mon[101432]: from='client.44615 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:51 smithi119 ceph-mon[101432]: from='client.44621 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:51 smithi119 ceph-mon[101432]: osdmap e551: 8 total, 8 up, 8 in 2024-09-21T10:06:52.012 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1018 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1021 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout: write object-1023 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout:verify written data by reading 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout: read object-37 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout:made snapshot "image392@parent_snap" and protect it 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout:made and opened clone "image393" 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout:flattening clone: "image393" 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout:check whether child image has the same set of objects as parent 2024-09-21T10:06:52.201 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.FlattenNoEmptyObjects (1133 ms) 2024-09-21T10:06:52.202 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.PoolMetadataConfApply 2024-09-21T10:06:52.273 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.PoolMetadataConfApply (71 ms) 2024-09-21T10:06:52.273 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.Sparsify 2024-09-21T10:06:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:52 smithi145 ceph-mon[89359]: pgmap v459: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 246 op/s; 8 B/s, 7 keys/s, 1 objects/s recovering 2024-09-21T10:06:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:52 smithi145 ceph-mon[89359]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T10:06:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3960274199' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:52.295 INFO:tasks.workunit.client.0.smithi119.stdout:sparsify_supported=1 2024-09-21T10:06:52.295 INFO:tasks.workunit.client.0.smithi119.stdout:sparse_read_supported=1 2024-09-21T10:06:52.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.Sparsify (50 ms) 2024-09-21T10:06:52.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.SparsifyClone 2024-09-21T10:06:52.337 INFO:tasks.workunit.client.0.smithi119.stdout:sparsify_supported=1 2024-09-21T10:06:52.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:06:52] ENGINE Bus STOPPING 2024-09-21T10:06:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-mon[101432]: pgmap v459: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 246 op/s; 8 B/s, 7 keys/s, 1 objects/s recovering 2024-09-21T10:06:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-mon[101432]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T10:06:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3960274199' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-mon[105288]: pgmap v459: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 246 op/s; 8 B/s, 7 keys/s, 1 objects/s recovering 2024-09-21T10:06:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-mon[105288]: osdmap e552: 8 total, 8 up, 8 in 2024-09-21T10:06:52.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3960274199' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:52.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:06:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:06:52.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:06:52] ENGINE Bus STOPPED 2024-09-21T10:06:52.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:06:52] ENGINE Bus STARTING 2024-09-21T10:06:52.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:06:52] ENGINE Serving on http://:::9283 2024-09-21T10:06:52.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:06:52] ENGINE Bus STARTED 2024-09-21T10:06:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:53 smithi145 ceph-mon[89359]: from='client.44633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:53 smithi145 ceph-mon[89359]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T10:06:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:53 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' 2024-09-21T10:06:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[105288]: from='client.44633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[105288]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[101432]: from='client.44633 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[101432]: osdmap e553: 8 total, 8 up, 8 in 2024-09-21T10:06:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' 2024-09-21T10:06:53.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:06:54.242 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.SparsifyClone (1920 ms) 2024-09-21T10:06:54.242 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestInternal.MissingDataPool 2024-09-21T10:06:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:54 smithi145 ceph-mon[89359]: from='client.44639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:54 smithi145 ceph-mon[89359]: pgmap v462: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 74 KiB/s rd, 11 KiB/s wr, 102 op/s 2024-09-21T10:06:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:54 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:54 smithi145 ceph-mon[89359]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T10:06:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[101432]: from='client.44639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[101432]: pgmap v462: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 74 KiB/s rd, 11 KiB/s wr, 102 op/s 2024-09-21T10:06:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[101432]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T10:06:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[105288]: from='client.44639 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[105288]: pgmap v462: 105 pgs: 105 active+clean; 13 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 74 KiB/s rd, 11 KiB/s wr, 102 op/s 2024-09-21T10:06:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:06:54.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-mon[105288]: osdmap e554: 8 total, 8 up, 8 in 2024-09-21T10:06:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:06:54] "GET /metrics HTTP/1.1" 200 36593 "" "Prometheus/2.51.0" 2024-09-21T10:06:54.916 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:55.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[101432]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T10:06:55.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[101432]: pgmap v465: 105 pgs: 105 active+clean; 5.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 554 KiB/s rd, 175 KiB/s wr, 878 op/s 2024-09-21T10:06:55.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/349351175' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:55.235 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[101432]: from='client.34767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:55.343 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestInternal.MissingDataPool (1101 ms) 2024-09-21T10:06:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:55 smithi145 ceph-mon[89359]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T10:06:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:55 smithi145 ceph-mon[89359]: pgmap v465: 105 pgs: 105 active+clean; 5.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 554 KiB/s rd, 175 KiB/s wr, 878 op/s 2024-09-21T10:06:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/349351175' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:55 smithi145 ceph-mon[89359]: from='client.34767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[105288]: osdmap e555: 8 total, 8 up, 8 in 2024-09-21T10:06:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[105288]: pgmap v465: 105 pgs: 105 active+clean; 5.0 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 554 KiB/s rd, 175 KiB/s wr, 878 op/s 2024-09-21T10:06:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/349351175' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:55 smithi119 ceph-mon[105288]: from='client.34767 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:56.249 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 34 tests from TestInternal (39105 ms total) 2024-09-21T10:06:56.249 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:06:56.249 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 17 tests from TestImageWatcher 2024-09-21T10:06:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:56 smithi145 ceph-mon[89359]: from='client.44657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:56 smithi145 ceph-mon[89359]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T10:06:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:56 smithi119 ceph-mon[105288]: from='client.44657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:56 smithi119 ceph-mon[105288]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T10:06:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:56 smithi119 ceph-mon[101432]: from='client.44657 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:56 smithi119 ceph-mon[101432]: osdmap e556: 8 total, 8 up, 8 in 2024-09-21T10:06:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:57 smithi145 ceph-mon[89359]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T10:06:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:57 smithi145 ceph-mon[89359]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:57 smithi119 ceph-mon[101432]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T10:06:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:57 smithi119 ceph-mon[101432]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:57.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:06:57.305+0000 7fc0be6c7640 -1 mgr.server reply reply (11) Resource temporarily unavailable 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:06:57.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:06:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:06:57.305+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy 32 pgs have unknown state; cannot draw any conclusionsunsafe to stop osd(s) at this time (0 PGs are or would become offline) 2024-09-21T10:06:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:57 smithi119 ceph-mon[105288]: osdmap e557: 8 total, 8 up, 8 in 2024-09-21T10:06:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:57 smithi119 ceph-mon[105288]: pgmap v468: 73 pgs: 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:57.782 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:06:58.255 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyHeaderUpdate 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3239338259' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: Upgrade: 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:06:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/885601730' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:58.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:58 smithi145 ceph-mon[89359]: from='client.44687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3239338259' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: Upgrade: 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/885601730' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[105288]: from='client.44687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: osdmap e558: 8 total, 8 up, 8 in 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3239338259' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: Upgrade: 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/885601730' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:06:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='client.34797 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:58.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:58 smithi119 ceph-mon[101432]: from='client.44687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:06:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:59 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:59 smithi145 ceph-mon[89359]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T10:06:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:06:59 smithi145 ceph-mon[89359]: pgmap v471: 105 pgs: 11 unknown, 94 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:59 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:59 smithi119 ceph-mon[101432]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T10:06:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:06:59 smithi119 ceph-mon[101432]: pgmap v471: 105 pgs: 11 unknown, 94 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:06:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:59 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:06:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:59 smithi119 ceph-mon[105288]: osdmap e559: 8 total, 8 up, 8 in 2024-09-21T10:06:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:06:59 smithi119 ceph-mon[105288]: pgmap v471: 105 pgs: 11 unknown, 94 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:00.300 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyHeaderUpdate (2045 ms) 2024-09-21T10:07:00.300 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyFlatten 2024-09-21T10:07:00.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyFlatten (22 ms) 2024-09-21T10:07:00.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyResize 2024-09-21T10:07:00.342 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyResize (20 ms) 2024-09-21T10:07:00.342 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyRebuildObjectMap 2024-09-21T10:07:00.365 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyRebuildObjectMap (22 ms) 2024-09-21T10:07:00.365 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyUpdateFeatures 2024-09-21T10:07:00.387 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyUpdateFeatures (23 ms) 2024-09-21T10:07:00.387 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapCreate 2024-09-21T10:07:00.411 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapCreate (24 ms) 2024-09-21T10:07:00.411 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapCreateError 2024-09-21T10:07:00.429 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapCreateError (18 ms) 2024-09-21T10:07:00.429 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapRename 2024-09-21T10:07:00.450 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapRename (21 ms) 2024-09-21T10:07:00.450 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapRenameError 2024-09-21T10:07:00.473 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapRenameError (22 ms) 2024-09-21T10:07:00.473 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapRemove 2024-09-21T10:07:00.492 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapRemove (20 ms) 2024-09-21T10:07:00.492 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapProtect 2024-09-21T10:07:00.513 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapProtect (21 ms) 2024-09-21T10:07:00.513 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifySnapUnprotect 2024-09-21T10:07:00.535 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifySnapUnprotect (22 ms) 2024-09-21T10:07:00.535 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyRename 2024-09-21T10:07:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:00 smithi145 ceph-mon[89359]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T10:07:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:07:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:07:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:00.555 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyRename (20 ms) 2024-09-21T10:07:00.555 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyAsyncTimedOut 2024-09-21T10:07:00.574 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyAsyncTimedOut (19 ms) 2024-09-21T10:07:00.574 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyAsyncError 2024-09-21T10:07:00.593 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyAsyncError (19 ms) 2024-09-21T10:07:00.593 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyAsyncCompleteError 2024-09-21T10:07:00.613 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyAsyncCompleteError (19 ms) 2024-09-21T10:07:00.616 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestImageWatcher.NotifyAsyncRequestTimedOut 2024-09-21T10:07:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:00 smithi119 ceph-mon[101432]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T10:07:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:00 smithi119 ceph-mon[105288]: osdmap e560: 8 total, 8 up, 8 in 2024-09-21T10:07:00.631 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestImageWatcher.NotifyAsyncRequestTimedOut (18 ms) 2024-09-21T10:07:00.701 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:01.285 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 17 tests from TestImageWatcher (2376 ms total) 2024-09-21T10:07:01.286 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:07:01.286 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 161 tests from TestLibRBD 2024-09-21T10:07:01.322 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CreateAndStat 2024-09-21T10:07:01.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[105288]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/965859694' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[105288]: pgmap v474: 105 pgs: 5 unknown, 100 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[105288]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[105288]: from='client.44705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[101432]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/965859694' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[101432]: pgmap v474: 105 pgs: 5 unknown, 100 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[101432]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:01.389 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:01 smithi119 ceph-mon[101432]: from='client.44705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:01 smithi145 ceph-mon[89359]: osdmap e561: 8 total, 8 up, 8 in 2024-09-21T10:07:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/965859694' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:01 smithi145 ceph-mon[89359]: pgmap v474: 105 pgs: 5 unknown, 100 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:01 smithi145 ceph-mon[89359]: from='client.34815 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:01 smithi145 ceph-mon[89359]: from='client.44705 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:02 smithi145 ceph-mon[89359]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T10:07:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:02 smithi119 ceph-mon[101432]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T10:07:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:02 smithi119 ceph-mon[105288]: osdmap e562: 8 total, 8 up, 8 in 2024-09-21T10:07:03.297 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:03 smithi145 ceph-mon[89359]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T10:07:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2528352744' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:03 smithi145 ceph-mon[89359]: pgmap v477: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1233763880' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:03.553 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[101432]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T10:07:03.553 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2528352744' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[101432]: pgmap v477: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1233763880' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[105288]: osdmap e563: 8 total, 8 up, 8 in 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2528352744' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-15","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[105288]: pgmap v477: 105 pgs: 32 unknown, 73 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:03.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1233763880' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:03.555 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2528352744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[105288]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[105288]: from='client.44735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[105288]: from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2528352744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[101432]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[101432]: from='client.44735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-mon[101432]: from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:04 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2528352744' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:04 smithi145 ceph-mon[89359]: osdmap e564: 8 total, 8 up, 8 in 2024-09-21T10:07:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:04 smithi145 ceph-mon[89359]: from='client.44735 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:04 smithi145 ceph-mon[89359]: from='client.44741 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:07:04] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-21T10:07:05.334 INFO:tasks.workunit.client.0.smithi119.stdout:image has size 2097152 and order 22 2024-09-21T10:07:05.334 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CreateAndStat (4012 ms) 2024-09-21T10:07:05.335 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CreateWithSameDataPool 2024-09-21T10:07:05.335 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.353 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CreateWithSameDataPool (19 ms) 2024-09-21T10:07:05.353 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CreateAndStatPP 2024-09-21T10:07:05.353 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.372 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CreateAndStatPP (19 ms) 2024-09-21T10:07:05.372 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.GetId 2024-09-21T10:07:05.372 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.397 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.GetId (25 ms) 2024-09-21T10:07:05.398 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.GetIdPP 2024-09-21T10:07:05.398 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.424 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.GetIdPP (27 ms) 2024-09-21T10:07:05.424 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefix 2024-09-21T10:07:05.425 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.439 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.GetBlockNamePrefix (15 ms) 2024-09-21T10:07:05.440 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.GetBlockNamePrefixPP 2024-09-21T10:07:05.440 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.454 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.GetBlockNamePrefixPP (15 ms) 2024-09-21T10:07:05.454 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestGetCreateTimestamp 2024-09-21T10:07:05.455 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.469 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestGetCreateTimestamp (15 ms) 2024-09-21T10:07:05.469 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.GetCreateTimestampPP 2024-09-21T10:07:05.470 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.485 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.GetCreateTimestampPP (16 ms) 2024-09-21T10:07:05.485 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.OpenAio 2024-09-21T10:07:05.485 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.501 INFO:tasks.workunit.client.0.smithi119.stdout:image has size 2097152 and order 22 2024-09-21T10:07:05.501 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.OpenAio (15 ms) 2024-09-21T10:07:05.501 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.OpenAioFail 2024-09-21T10:07:05.503 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.OpenAioFail (3 ms) 2024-09-21T10:07:05.503 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.OpenAioPP 2024-09-21T10:07:05.503 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.526 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.OpenAioPP (23 ms) 2024-09-21T10:07:05.526 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.OpenAioFailPP 2024-09-21T10:07:05.529 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.OpenAioFailPP (3 ms) 2024-09-21T10:07:05.529 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ResizeAndStat 2024-09-21T10:07:05.529 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.560 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ResizeAndStat (31 ms) 2024-09-21T10:07:05.560 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ResizeAndStatPP 2024-09-21T10:07:05.561 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.587 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ResizeAndStatPP (27 ms) 2024-09-21T10:07:05.587 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResize 2024-09-21T10:07:05.587 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:05 smithi119 ceph-mon[105288]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T10:07:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:05 smithi119 ceph-mon[105288]: pgmap v480: 105 pgs: 6 unknown, 99 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:05 smithi119 ceph-mon[101432]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T10:07:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:05 smithi119 ceph-mon[101432]: pgmap v480: 105 pgs: 6 unknown, 99 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:05.621 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.UpdateWatchAndResize (34 ms) 2024-09-21T10:07:05.621 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.UpdateWatchAndResizePP 2024-09-21T10:07:05.621 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:05.649 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.UpdateWatchAndResizePP (28 ms) 2024-09-21T10:07:05.649 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCreateLsDelete 2024-09-21T10:07:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:05 smithi145 ceph-mon[89359]: osdmap e565: 8 total, 8 up, 8 in 2024-09-21T10:07:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:05 smithi145 ceph-mon[89359]: pgmap v480: 105 pgs: 6 unknown, 99 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:06.399 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[105288]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T10:07:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3333170363' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[101432]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T10:07:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3333170363' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:06 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:06 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:06 smithi145 ceph-mon[89359]: osdmap e566: 8 total, 8 up, 8 in 2024-09-21T10:07:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3333170363' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:07.328 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[105288]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1186314432' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[105288]: from='client.34875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[105288]: pgmap v483: 137 pgs: 32 unknown, 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[105288]: from='client.34881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[101432]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1186314432' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[101432]: from='client.34875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[101432]: pgmap v483: 137 pgs: 32 unknown, 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:07.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:07 smithi119 ceph-mon[101432]: from='client.34881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:07 smithi145 ceph-mon[89359]: osdmap e567: 8 total, 8 up, 8 in 2024-09-21T10:07:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1186314432' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:07 smithi145 ceph-mon[89359]: from='client.34875 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:07 smithi145 ceph-mon[89359]: pgmap v483: 137 pgs: 32 unknown, 105 active+clean; 582 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:07:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:07 smithi145 ceph-mon[89359]: from='client.34881 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:08 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:08 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:08 smithi119 ceph-mon[105288]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T10:07:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:08 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:08 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:08 smithi119 ceph-mon[101432]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T10:07:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:08 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:08 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:08 smithi145 ceph-mon[89359]: osdmap e568: 8 total, 8 up, 8 in 2024-09-21T10:07:09.258 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:09.346 INFO:tasks.workunit.client.0.smithi119.stdout:image: image18 2024-09-21T10:07:09.346 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image18 2024-09-21T10:07:09.346 INFO:tasks.workunit.client.0.smithi119.stdout:found image18 2024-09-21T10:07:09.346 INFO:tasks.workunit.client.0.smithi119.stdout:erased image18 2024-09-21T10:07:09.346 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:09.418 INFO:tasks.workunit.client.0.smithi119.stdout:image: image18 2024-09-21T10:07:09.418 INFO:tasks.workunit.client.0.smithi119.stdout:image: image19 2024-09-21T10:07:09.418 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image18 2024-09-21T10:07:09.418 INFO:tasks.workunit.client.0.smithi119.stdout:found image18 2024-09-21T10:07:09.418 INFO:tasks.workunit.client.0.smithi119.stdout:erased image18 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:found image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:erased image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:image: image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:found image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:erased image19 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCreateLsDelete (3769 ms) 2024-09-21T10:07:09.419 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCreateLsDeletePP 2024-09-21T10:07:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:09 smithi119 ceph-mon[101432]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T10:07:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:09 smithi119 ceph-mon[101432]: pgmap v486: 137 pgs: 3 creating+peering, 4 unknown, 130 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 16 KiB/s wr, 197 op/s 2024-09-21T10:07:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3910037467' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:09 smithi119 ceph-mon[105288]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T10:07:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:09 smithi119 ceph-mon[105288]: pgmap v486: 137 pgs: 3 creating+peering, 4 unknown, 130 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 16 KiB/s wr, 197 op/s 2024-09-21T10:07:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3910037467' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:09 smithi145 ceph-mon[89359]: osdmap e569: 8 total, 8 up, 8 in 2024-09-21T10:07:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:09 smithi145 ceph-mon[89359]: pgmap v486: 137 pgs: 3 creating+peering, 4 unknown, 130 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 144 KiB/s rd, 16 KiB/s wr, 197 op/s 2024-09-21T10:07:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3910037467' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:07:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:07:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:10 smithi145 ceph-mon[89359]: from='client.44777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:10 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:10 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:10 smithi145 ceph-mon[89359]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T10:07:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:10 smithi145 ceph-mon[89359]: from='client.34899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[101432]: from='client.44777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[101432]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[101432]: from='client.34899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[105288]: from='client.44777 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[105288]: osdmap e570: 8 total, 8 up, 8 in 2024-09-21T10:07:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:10 smithi119 ceph-mon[105288]: from='client.34899 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:11.363 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[101432]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T10:07:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2368917368' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[101432]: pgmap v489: 169 pgs: 32 unknown, 3 creating+peering, 134 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 182 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:07:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[105288]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T10:07:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2368917368' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:11 smithi119 ceph-mon[105288]: pgmap v489: 169 pgs: 32 unknown, 3 creating+peering, 134 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 182 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:07:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:11 smithi145 ceph-mon[89359]: osdmap e571: 8 total, 8 up, 8 in 2024-09-21T10:07:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2368917368' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:11 smithi145 ceph-mon[89359]: pgmap v489: 169 pgs: 32 unknown, 3 creating+peering, 134 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 182 KiB/s rd, 21 KiB/s wr, 252 op/s 2024-09-21T10:07:12.129 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:07:12.322+0000 7fc0be6c7640 -1 mgr.server reply reply (11) Resource temporarily unavailable 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:07:12.322+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy 32 pgs have unknown state; cannot draw any conclusionsunsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[105288]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/229505463' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:12.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:12.420 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:12.420 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:12.420 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[101432]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T10:07:12.420 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/229505463' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:12.420 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:12.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:12 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:12.696 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:12 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:12.696 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:12 smithi145 ceph-mon[89359]: osdmap e572: 8 total, 8 up, 8 in 2024-09-21T10:07:12.696 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/229505463' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:12.696 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:13.380 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 1 2024-09-21T10:07:13.380 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:13.380 INFO:tasks.workunit.client.0.smithi119.stdout:image: image20 2024-09-21T10:07:13.380 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image20 2024-09-21T10:07:13.380 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:13.389 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 2 2024-09-21T10:07:13.389 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:13.389 INFO:tasks.workunit.client.0.smithi119.stdout:image: image20 2024-09-21T10:07:13.389 INFO:tasks.workunit.client.0.smithi119.stdout:image: image21 2024-09-21T10:07:13.389 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image20 2024-09-21T10:07:13.390 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image21 2024-09-21T10:07:13.441 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 1 2024-09-21T10:07:13.441 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:13.441 INFO:tasks.workunit.client.0.smithi119.stdout:image: image21 2024-09-21T10:07:13.441 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image21 2024-09-21T10:07:13.442 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCreateLsDeletePP (4023 ms) 2024-09-21T10:07:13.442 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCopy 2024-09-21T10:07:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[105288]: from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[105288]: Upgrade: 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:07:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[105288]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T10:07:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[105288]: from='client.34923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[105288]: pgmap v492: 169 pgs: 29 unknown, 140 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 5.0 KiB/s wr, 57 op/s 2024-09-21T10:07:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[101432]: from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[101432]: Upgrade: 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:07:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[101432]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T10:07:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[101432]: from='client.34923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:13 smithi119 ceph-mon[101432]: pgmap v492: 169 pgs: 29 unknown, 140 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 5.0 KiB/s wr, 57 op/s 2024-09-21T10:07:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:13 smithi145 ceph-mon[89359]: from='client.44801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:13 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:13 smithi145 ceph-mon[89359]: Upgrade: 32 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:07:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:13 smithi145 ceph-mon[89359]: osdmap e573: 8 total, 8 up, 8 in 2024-09-21T10:07:13.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:13 smithi145 ceph-mon[89359]: from='client.34923 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:13.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:13 smithi145 ceph-mon[89359]: pgmap v492: 169 pgs: 29 unknown, 140 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 5.0 KiB/s wr, 57 op/s 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:07:14] "GET /metrics HTTP/1.1" 200 36524 "" "Prometheus/2.51.0" 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[101432]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[101432]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3349452402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[105288]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T10:07:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[105288]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T10:07:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3349452402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:14 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:14 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:14 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:14 smithi145 ceph-mon[89359]: osdmap e574: 8 total, 8 up, 8 in 2024-09-21T10:07:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:14 smithi145 ceph-mon[89359]: osdmap e575: 8 total, 8 up, 8 in 2024-09-21T10:07:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3349452402' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:14 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:15.001 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:15.032 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[105288]: pgmap v495: 201 pgs: 35 unknown, 166 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s 2024-09-21T10:07:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2863858074' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[105288]: from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:15.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[105288]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T10:07:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[101432]: pgmap v495: 201 pgs: 35 unknown, 166 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s 2024-09-21T10:07:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2863858074' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[101432]: from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:15.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:15 smithi119 ceph-mon[101432]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T10:07:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:15 smithi145 ceph-mon[89359]: pgmap v495: 201 pgs: 35 unknown, 166 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 43 KiB/s rd, 4.7 KiB/s wr, 63 op/s 2024-09-21T10:07:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2863858074' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:15.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:15 smithi145 ceph-mon[89359]: from='client.34941 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:15.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:15 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:15.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:15 smithi145 ceph-mon[89359]: osdmap e576: 8 total, 8 up, 8 in 2024-09-21T10:07:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:16 smithi145 ceph-mon[89359]: from='client.44831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:16 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:16 smithi145 ceph-mon[89359]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T10:07:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:16 smithi119 ceph-mon[105288]: from='client.44831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:16 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:16 smithi119 ceph-mon[105288]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T10:07:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:16 smithi119 ceph-mon[101432]: from='client.44831 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:16.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:16 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:16.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:16 smithi119 ceph-mon[101432]: osdmap e577: 8 total, 8 up, 8 in 2024-09-21T10:07:17.379 INFO:tasks.workunit.client.0.smithi119.stdout:image: image22 2024-09-21T10:07:17.379 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image22 2024-09-21T10:07:17.379 INFO:tasks.workunit.client.0.smithi119.stdout:found image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:erased image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:image: image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:image: image23 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:found image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:erased image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image23 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:found image23 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:erased image23 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:image: image22 2024-09-21T10:07:17.380 INFO:tasks.workunit.client.0.smithi119.stdout:image: image23 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:image: image24 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image22 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:found image22 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:erased image22 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image23 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:found image23 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:erased image23 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image24 2024-09-21T10:07:17.381 INFO:tasks.workunit.client.0.smithi119.stdout:found image24 2024-09-21T10:07:17.382 INFO:tasks.workunit.client.0.smithi119.stdout:erased image24 2024-09-21T10:07:17.382 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCopy (3938 ms) 2024-09-21T10:07:17.382 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCopyPP 2024-09-21T10:07:17.870 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:18 smithi145 ceph-mon[89359]: pgmap v498: 201 pgs: 24 unknown, 177 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 4.7 KiB/s wr, 66 op/s 2024-09-21T10:07:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:18 smithi145 ceph-mon[89359]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T10:07:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:18 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:18 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/437541890' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[105288]: pgmap v498: 201 pgs: 24 unknown, 177 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 4.7 KiB/s wr, 66 op/s 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[105288]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/437541890' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[101432]: pgmap v498: 201 pgs: 24 unknown, 177 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 44 KiB/s rd, 4.7 KiB/s wr, 66 op/s 2024-09-21T10:07:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[101432]: osdmap e578: 8 total, 8 up, 8 in 2024-09-21T10:07:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/437541890' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:19.035 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:19 smithi145 ceph-mon[89359]: from='client.44849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:19 smithi145 ceph-mon[89359]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T10:07:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1792964785' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1792964785' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:19 smithi145 ceph-mon[89359]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[105288]: from='client.44849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[105288]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1792964785' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1792964785' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[105288]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[101432]: from='client.44849 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[101432]: osdmap e579: 8 total, 8 up, 8 in 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1792964785' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-19","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1792964785' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:19.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:19 smithi119 ceph-mon[101432]: osdmap e580: 8 total, 8 up, 8 in 2024-09-21T10:07:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:20 smithi145 ceph-mon[89359]: from='client.44855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:20 smithi145 ceph-mon[89359]: pgmap v501: 233 pgs: 20 creating+peering, 12 unknown, 201 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 42 KiB/s rd, 3.0 KiB/s wr, 54 op/s 2024-09-21T10:07:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:20 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:07:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:07:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:20 smithi119 ceph-mon[101432]: from='client.44855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:20 smithi119 ceph-mon[101432]: pgmap v501: 233 pgs: 20 creating+peering, 12 unknown, 201 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 42 KiB/s rd, 3.0 KiB/s wr, 54 op/s 2024-09-21T10:07:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:20 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:20 smithi119 ceph-mon[105288]: from='client.44855 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:20 smithi119 ceph-mon[105288]: pgmap v501: 233 pgs: 20 creating+peering, 12 unknown, 201 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 42 KiB/s rd, 3.0 KiB/s wr, 54 op/s 2024-09-21T10:07:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:20 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:20.722 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:21.279 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 1 2024-09-21T10:07:21.279 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:21.279 INFO:tasks.workunit.client.0.smithi119.stdout:image: image25 2024-09-21T10:07:21.279 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image25 2024-09-21T10:07:21.296 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 2 2024-09-21T10:07:21.296 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:21.296 INFO:tasks.workunit.client.0.smithi119.stdout:image: image25 2024-09-21T10:07:21.296 INFO:tasks.workunit.client.0.smithi119.stdout:image: image26 2024-09-21T10:07:21.296 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image25 2024-09-21T10:07:21.296 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image26 2024-09-21T10:07:21.338 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:07:21.338 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 3 2024-09-21T10:07:21.339 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:07:21.339 INFO:tasks.workunit.client.0.smithi119.stdout:image: image25 2024-09-21T10:07:21.339 INFO:tasks.workunit.client.0.smithi119.stdout:image: image26 2024-09-21T10:07:21.339 INFO:tasks.workunit.client.0.smithi119.stdout:image: image27 2024-09-21T10:07:21.339 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image25 2024-09-21T10:07:21.340 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image26 2024-09-21T10:07:21.340 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image27 2024-09-21T10:07:21.365 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCopyPP (3986 ms) 2024-09-21T10:07:21.365 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestDeepCopy 2024-09-21T10:07:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:21 smithi119 ceph-mon[101432]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T10:07:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1568028426' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:21 smithi119 ceph-mon[105288]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T10:07:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1568028426' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:21 smithi145 ceph-mon[89359]: osdmap e581: 8 total, 8 up, 8 in 2024-09-21T10:07:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1568028426' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[101432]: pgmap v504: 233 pgs: 20 creating+peering, 7 unknown, 206 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 350 KiB/s rd, 22 KiB/s wr, 409 op/s 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[101432]: from='client.44867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[101432]: from='client.44873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[101432]: osdmap e582: 8 total, 8 up, 8 in 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[105288]: pgmap v504: 233 pgs: 20 creating+peering, 7 unknown, 206 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 350 KiB/s rd, 22 KiB/s wr, 409 op/s 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[105288]: from='client.44867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[105288]: from='client.44873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:22 smithi119 ceph-mon[105288]: osdmap e582: 8 total, 8 up, 8 in 2024-09-21T10:07:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:22 smithi145 ceph-mon[89359]: pgmap v504: 233 pgs: 20 creating+peering, 7 unknown, 206 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 350 KiB/s rd, 22 KiB/s wr, 409 op/s 2024-09-21T10:07:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:22 smithi145 ceph-mon[89359]: from='client.44867 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:22 smithi145 ceph-mon[89359]: from='client.44873 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:22 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:22 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:22 smithi145 ceph-mon[89359]: osdmap e582: 8 total, 8 up, 8 in 2024-09-21T10:07:23.070 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:23 smithi119 ceph-mon[101432]: osdmap e583: 8 total, 8 up, 8 in 2024-09-21T10:07:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4092685399' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:23 smithi119 ceph-mon[105288]: osdmap e583: 8 total, 8 up, 8 in 2024-09-21T10:07:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4092685399' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:23 smithi145 ceph-mon[89359]: osdmap e583: 8 total, 8 up, 8 in 2024-09-21T10:07:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4092685399' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:23.597 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[105288]: pgmap v507: 265 pgs: 23 creating+peering, 29 unknown, 213 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 360 KiB/s rd, 21 KiB/s wr, 413 op/s 2024-09-21T10:07:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[105288]: osdmap e584: 8 total, 8 up, 8 in 2024-09-21T10:07:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4128053658' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[105288]: osdmap e585: 8 total, 8 up, 8 in 2024-09-21T10:07:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[101432]: pgmap v507: 265 pgs: 23 creating+peering, 29 unknown, 213 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 360 KiB/s rd, 21 KiB/s wr, 413 op/s 2024-09-21T10:07:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[101432]: osdmap e584: 8 total, 8 up, 8 in 2024-09-21T10:07:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4128053658' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:24.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-mon[101432]: osdmap e585: 8 total, 8 up, 8 in 2024-09-21T10:07:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:24 smithi145 ceph-mon[89359]: pgmap v507: 265 pgs: 23 creating+peering, 29 unknown, 213 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 360 KiB/s rd, 21 KiB/s wr, 413 op/s 2024-09-21T10:07:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:24 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:24 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:24 smithi145 ceph-mon[89359]: osdmap e584: 8 total, 8 up, 8 in 2024-09-21T10:07:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4128053658' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:24 smithi145 ceph-mon[89359]: osdmap e585: 8 total, 8 up, 8 in 2024-09-21T10:07:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:07:24] "GET /metrics HTTP/1.1" 200 43735 "" "Prometheus/2.51.0" 2024-09-21T10:07:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:25 smithi119 ceph-mon[101432]: from='client.44891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:25 smithi119 ceph-mon[101432]: from='client.35013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:25 smithi119 ceph-mon[101432]: osdmap e586: 8 total, 8 up, 8 in 2024-09-21T10:07:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:25 smithi119 ceph-mon[105288]: from='client.44891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:25 smithi119 ceph-mon[105288]: from='client.35013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:25 smithi119 ceph-mon[105288]: osdmap e586: 8 total, 8 up, 8 in 2024-09-21T10:07:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:25 smithi145 ceph-mon[89359]: from='client.44891 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:25 smithi145 ceph-mon[89359]: from='client.35013 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:25 smithi145 ceph-mon[89359]: osdmap e586: 8 total, 8 up, 8 in 2024-09-21T10:07:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:26 smithi119 ceph-mon[101432]: pgmap v510: 265 pgs: 3 creating+peering, 5 unknown, 257 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 KiB/s rd, 22 KiB/s wr, 396 op/s 2024-09-21T10:07:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:26 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:26 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:26 smithi119 ceph-mon[105288]: pgmap v510: 265 pgs: 3 creating+peering, 5 unknown, 257 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 KiB/s rd, 22 KiB/s wr, 396 op/s 2024-09-21T10:07:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:26 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:26 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:26.460 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:26 smithi145 ceph-mon[89359]: pgmap v510: 265 pgs: 3 creating+peering, 5 unknown, 257 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 341 KiB/s rd, 22 KiB/s wr, 396 op/s 2024-09-21T10:07:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:26 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:26 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:27.341 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:27 smithi119 ceph-mon[105288]: osdmap e587: 8 total, 8 up, 8 in 2024-09-21T10:07:27.341 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4247610250' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:27.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:27 smithi119 ceph-mon[101432]: osdmap e587: 8 total, 8 up, 8 in 2024-09-21T10:07:27.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4247610250' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:27 smithi145 ceph-mon[89359]: osdmap e587: 8 total, 8 up, 8 in 2024-09-21T10:07:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4247610250' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:27.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:27 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:07:27.338+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T10:07:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[101432]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[101432]: pgmap v513: 265 pgs: 3 creating+peering, 262 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 398 KiB/s rd, 28 KiB/s wr, 475 op/s 2024-09-21T10:07:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[101432]: from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[101432]: osdmap e588: 8 total, 8 up, 8 in 2024-09-21T10:07:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[105288]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[105288]: pgmap v513: 265 pgs: 3 creating+peering, 262 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 398 KiB/s rd, 28 KiB/s wr, 475 op/s 2024-09-21T10:07:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[105288]: from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[105288]: osdmap e588: 8 total, 8 up, 8 in 2024-09-21T10:07:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:28 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:28 smithi145 ceph-mon[89359]: from='client.44909 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:28 smithi145 ceph-mon[89359]: pgmap v513: 265 pgs: 3 creating+peering, 262 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 398 KiB/s rd, 28 KiB/s wr, 475 op/s 2024-09-21T10:07:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:28 smithi145 ceph-mon[89359]: from='client.35031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:28 smithi145 ceph-mon[89359]: osdmap e588: 8 total, 8 up, 8 in 2024-09-21T10:07:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:28 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:29.098 INFO:tasks.workunit.client.0.smithi119.stdout:image: image28 2024-09-21T10:07:29.098 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image28 2024-09-21T10:07:29.098 INFO:tasks.workunit.client.0.smithi119.stdout:found image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:erased image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:image: image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:image: image29 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:found image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:erased image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image29 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:found image29 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:erased image29 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:image: image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:image: image29 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:image: image30 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image28 2024-09-21T10:07:29.099 INFO:tasks.workunit.client.0.smithi119.stdout:found image28 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:erased image28 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image29 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:found image29 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:erased image29 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image30 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:found image30 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:erased image30 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:image: image28 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:image: image29 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:image: image30 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:image: image31 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image28 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:found image28 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:erased image28 2024-09-21T10:07:29.100 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image29 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:found image29 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:erased image29 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image30 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:found image30 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:erased image30 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image31 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:found image31 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:erased image31 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:image: image28 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:image: image29 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:image: image30 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:image: image31 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:image: image32 2024-09-21T10:07:29.101 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image28 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:found image28 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:erased image28 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image29 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:found image29 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:erased image29 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image30 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:found image30 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:erased image30 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image31 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:found image31 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:erased image31 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image32 2024-09-21T10:07:29.102 INFO:tasks.workunit.client.0.smithi119.stdout:found image32 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:erased image32 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:image: image28 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:image: image29 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:image: image30 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:image: image31 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:image: image32 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:image: image33 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image28 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:found image28 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:erased image28 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image29 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:found image29 2024-09-21T10:07:29.103 INFO:tasks.workunit.client.0.smithi119.stdout:erased image29 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image30 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:found image30 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:erased image30 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image31 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:found image31 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:erased image31 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image32 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:found image32 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:erased image32 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image33 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:found image33 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:erased image33 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestDeepCopy (7733 ms) 2024-09-21T10:07:29.104 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestDeepCopyPP 2024-09-21T10:07:29.326 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[101432]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[101432]: osdmap e589: 8 total, 8 up, 8 in 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2499282632' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[101432]: osdmap e590: 8 total, 8 up, 8 in 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[105288]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T10:07:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[105288]: osdmap e589: 8 total, 8 up, 8 in 2024-09-21T10:07:29.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2499282632' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:29.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:29 smithi119 ceph-mon[105288]: osdmap e590: 8 total, 8 up, 8 in 2024-09-21T10:07:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:29 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:29 smithi145 ceph-mon[89359]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-21T10:07:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:29 smithi145 ceph-mon[89359]: osdmap e589: 8 total, 8 up, 8 in 2024-09-21T10:07:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2499282632' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:29 smithi145 ceph-mon[89359]: osdmap e590: 8 total, 8 up, 8 in 2024-09-21T10:07:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:30 smithi119 ceph-mon[105288]: pgmap v516: 265 pgs: 265 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 219 KiB/s rd, 15 KiB/s wr, 272 op/s 2024-09-21T10:07:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:30 smithi119 ceph-mon[105288]: from='client.44927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:30 smithi119 ceph-mon[105288]: from='client.44939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:30 smithi119 ceph-mon[101432]: pgmap v516: 265 pgs: 265 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 219 KiB/s rd, 15 KiB/s wr, 272 op/s 2024-09-21T10:07:30.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:30 smithi119 ceph-mon[101432]: from='client.44927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:30 smithi119 ceph-mon[101432]: from='client.44939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:07:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:07:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:30 smithi145 ceph-mon[89359]: pgmap v516: 265 pgs: 265 active+clean; 583 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 219 KiB/s rd, 15 KiB/s wr, 272 op/s 2024-09-21T10:07:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:30 smithi145 ceph-mon[89359]: from='client.44927 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:30 smithi145 ceph-mon[89359]: from='client.44939 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:31.120 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[101432]: osdmap e591: 8 total, 8 up, 8 in 2024-09-21T10:07:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3328034512' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[101432]: pgmap v519: 297 pgs: 2 creating+peering, 30 unknown, 265 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 412 KiB/s rd, 23 KiB/s wr, 486 op/s 2024-09-21T10:07:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[105288]: osdmap e591: 8 total, 8 up, 8 in 2024-09-21T10:07:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3328034512' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:31.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:31.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:31 smithi119 ceph-mon[105288]: pgmap v519: 297 pgs: 2 creating+peering, 30 unknown, 265 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 412 KiB/s rd, 23 KiB/s wr, 486 op/s 2024-09-21T10:07:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:31 smithi145 ceph-mon[89359]: osdmap e591: 8 total, 8 up, 8 in 2024-09-21T10:07:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:31 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3328034512' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:31 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:31 smithi145 ceph-mon[89359]: pgmap v519: 297 pgs: 2 creating+peering, 30 unknown, 265 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 412 KiB/s rd, 23 KiB/s wr, 486 op/s 2024-09-21T10:07:32.185 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[101432]: osdmap e592: 8 total, 8 up, 8 in 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/817497035' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[105288]: osdmap e592: 8 total, 8 up, 8 in 2024-09-21T10:07:32.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/817497035' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:32 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:07:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:32 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:32 smithi145 ceph-mon[89359]: osdmap e592: 8 total, 8 up, 8 in 2024-09-21T10:07:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/817497035' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[101432]: osdmap e593: 8 total, 8 up, 8 in 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[101432]: from='client.44951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[101432]: from='client.44957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[101432]: pgmap v522: 297 pgs: 2 creating+peering, 22 unknown, 273 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 300 KiB/s rd, 16 KiB/s wr, 349 op/s 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[105288]: osdmap e593: 8 total, 8 up, 8 in 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[105288]: from='client.44951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[105288]: from='client.44957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[105288]: pgmap v522: 297 pgs: 2 creating+peering, 22 unknown, 273 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 300 KiB/s rd, 16 KiB/s wr, 349 op/s 2024-09-21T10:07:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:33.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:33.370 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 1 2024-09-21T10:07:33.370 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:33.370 INFO:tasks.workunit.client.0.smithi119.stdout:image: image34 2024-09-21T10:07:33.371 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image34 2024-09-21T10:07:33.397 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 2 2024-09-21T10:07:33.397 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:33.397 INFO:tasks.workunit.client.0.smithi119.stdout:image: image34 2024-09-21T10:07:33.397 INFO:tasks.workunit.client.0.smithi119.stdout:image: image35 2024-09-21T10:07:33.397 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image34 2024-09-21T10:07:33.397 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image35 2024-09-21T10:07:33.445 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:07:33.445 INFO:tasks.workunit.client.0.smithi119.stdout:num images is: 3 2024-09-21T10:07:33.445 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:07:33.445 INFO:tasks.workunit.client.0.smithi119.stdout:image: image34 2024-09-21T10:07:33.445 INFO:tasks.workunit.client.0.smithi119.stdout:image: image35 2024-09-21T10:07:33.446 INFO:tasks.workunit.client.0.smithi119.stdout:image: image36 2024-09-21T10:07:33.446 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image34 2024-09-21T10:07:33.446 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image35 2024-09-21T10:07:33.446 INFO:tasks.workunit.client.0.smithi119.stdout:expected = image36 2024-09-21T10:07:33.475 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestDeepCopyPP (4377 ms) 2024-09-21T10:07:33.475 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnap 2024-09-21T10:07:33.475 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:33 smithi145 ceph-mon[89359]: osdmap e593: 8 total, 8 up, 8 in 2024-09-21T10:07:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:33 smithi145 ceph-mon[89359]: from='client.44951 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:33 smithi145 ceph-mon[89359]: from='client.44957 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:33 smithi145 ceph-mon[89359]: pgmap v522: 297 pgs: 2 creating+peering, 22 unknown, 273 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 300 KiB/s rd, 16 KiB/s wr, 349 op/s 2024-09-21T10:07:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:07:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[101432]: osdmap e594: 8 total, 8 up, 8 in 2024-09-21T10:07:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[101432]: osdmap e595: 8 total, 8 up, 8 in 2024-09-21T10:07:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:07:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[105288]: osdmap e594: 8 total, 8 up, 8 in 2024-09-21T10:07:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-mon[105288]: osdmap e595: 8 total, 8 up, 8 in 2024-09-21T10:07:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:34 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:07:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:34 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:07:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:34 smithi145 ceph-mon[89359]: osdmap e594: 8 total, 8 up, 8 in 2024-09-21T10:07:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:34 smithi145 ceph-mon[89359]: osdmap e595: 8 total, 8 up, 8 in 2024-09-21T10:07:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:07:34] "GET /metrics HTTP/1.1" 200 50987 "" "Prometheus/2.51.0" 2024-09-21T10:07:35.043 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[101432]: pgmap v525: 297 pgs: 2 creating+peering, 295 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 22 KiB/s wr, 377 op/s 2024-09-21T10:07:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1768335469' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[101432]: osdmap e596: 8 total, 8 up, 8 in 2024-09-21T10:07:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[101432]: from='client.35085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[105288]: pgmap v525: 297 pgs: 2 creating+peering, 295 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 22 KiB/s wr, 377 op/s 2024-09-21T10:07:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1768335469' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:35.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[105288]: osdmap e596: 8 total, 8 up, 8 in 2024-09-21T10:07:35.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:35 smithi119 ceph-mon[105288]: from='client.35085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:35 smithi145 ceph-mon[89359]: pgmap v525: 297 pgs: 2 creating+peering, 295 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 320 KiB/s rd, 22 KiB/s wr, 377 op/s 2024-09-21T10:07:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1768335469' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:35 smithi145 ceph-mon[89359]: osdmap e596: 8 total, 8 up, 8 in 2024-09-21T10:07:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:35 smithi145 ceph-mon[89359]: from='client.35085 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:37.058 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:07:37.058 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:37.058 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:07:37.058 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1 with size 2097152 2024-09-21T10:07:37.058 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 2 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1 with size 2097152 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2 with size 4194304 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:37.059 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:07:37.060 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2 with size 4194304 2024-09-21T10:07:37.060 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:07:37.060 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:07:37.060 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnap (3582 ms) 2024-09-21T10:07:37.060 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestGetSnapShotTimeStamp 2024-09-21T10:07:37.060 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:37 smithi145 ceph-mon[89359]: from='client.35091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:37 smithi145 ceph-mon[89359]: osdmap e597: 8 total, 8 up, 8 in 2024-09-21T10:07:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:37 smithi119 ceph-mon[105288]: from='client.35091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:37 smithi119 ceph-mon[105288]: osdmap e597: 8 total, 8 up, 8 in 2024-09-21T10:07:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:37 smithi119 ceph-mon[101432]: from='client.35091 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:37 smithi119 ceph-mon[101432]: osdmap e597: 8 total, 8 up, 8 in 2024-09-21T10:07:37.896 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:38 smithi145 ceph-mon[89359]: pgmap v528: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 22 KiB/s wr, 380 op/s 2024-09-21T10:07:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:38 smithi145 ceph-mon[89359]: osdmap e598: 8 total, 8 up, 8 in 2024-09-21T10:07:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/44015631' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:38 smithi119 ceph-mon[105288]: pgmap v528: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 22 KiB/s wr, 380 op/s 2024-09-21T10:07:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:38 smithi119 ceph-mon[105288]: osdmap e598: 8 total, 8 up, 8 in 2024-09-21T10:07:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/44015631' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:38 smithi119 ceph-mon[101432]: pgmap v528: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 321 KiB/s rd, 22 KiB/s wr, 380 op/s 2024-09-21T10:07:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:38 smithi119 ceph-mon[101432]: osdmap e598: 8 total, 8 up, 8 in 2024-09-21T10:07:38.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/44015631' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:39.069 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestGetSnapShotTimeStamp (2012 ms) 2024-09-21T10:07:39.069 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCreateLsDeleteSnapPP 2024-09-21T10:07:39.069 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:39 smithi119 ceph-mon[101432]: from='client.35103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:39 smithi119 ceph-mon[101432]: osdmap e599: 8 total, 8 up, 8 in 2024-09-21T10:07:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:39 smithi119 ceph-mon[105288]: from='client.35103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:39 smithi119 ceph-mon[105288]: osdmap e599: 8 total, 8 up, 8 in 2024-09-21T10:07:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:39 smithi145 ceph-mon[89359]: from='client.35103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:39 smithi145 ceph-mon[89359]: osdmap e599: 8 total, 8 up, 8 in 2024-09-21T10:07:40.079 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:07:40.079 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:40.079 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:07:40.079 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1 with size 2097152 2024-09-21T10:07:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:40 smithi119 ceph-mon[105288]: from='client.35109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:40 smithi119 ceph-mon[105288]: pgmap v531: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:07:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:40 smithi119 ceph-mon[105288]: osdmap e600: 8 total, 8 up, 8 in 2024-09-21T10:07:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:40 smithi119 ceph-mon[101432]: from='client.35109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:40 smithi119 ceph-mon[101432]: pgmap v531: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:07:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:40 smithi119 ceph-mon[101432]: osdmap e600: 8 total, 8 up, 8 in 2024-09-21T10:07:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:40 smithi145 ceph-mon[89359]: from='client.35109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:40 smithi145 ceph-mon[89359]: pgmap v531: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 8.2 KiB/s wr, 144 op/s 2024-09-21T10:07:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:40 smithi145 ceph-mon[89359]: osdmap e600: 8 total, 8 up, 8 in 2024-09-21T10:07:40.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:07:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:07:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:40.750 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:41.083 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 2 2024-09-21T10:07:41.083 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:41.083 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:07:41.084 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:07:41.084 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1 with size 2097152 2024-09-21T10:07:41.084 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2 with size 4194304 2024-09-21T10:07:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:41 smithi119 ceph-mon[105288]: osdmap e601: 8 total, 8 up, 8 in 2024-09-21T10:07:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1275102226' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:41 smithi119 ceph-mon[101432]: osdmap e601: 8 total, 8 up, 8 in 2024-09-21T10:07:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1275102226' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:41 smithi145 ceph-mon[89359]: osdmap e601: 8 total, 8 up, 8 in 2024-09-21T10:07:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1275102226' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:42.080 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:07:42.080 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:42.081 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:07:42.081 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2 with size 4194304 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[101432]: pgmap v534: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 7.7 KiB/s wr, 138 op/s 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[101432]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[101432]: from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[101432]: osdmap e602: 8 total, 8 up, 8 in 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[105288]: pgmap v534: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 7.7 KiB/s wr, 138 op/s 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[105288]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[105288]: from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:42 smithi119 ceph-mon[105288]: osdmap e602: 8 total, 8 up, 8 in 2024-09-21T10:07:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:42 smithi145 ceph-mon[89359]: pgmap v534: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 7.7 KiB/s wr, 138 op/s 2024-09-21T10:07:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:42 smithi145 ceph-mon[89359]: from='client.35121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:42 smithi145 ceph-mon[89359]: from='client.35127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:42 smithi145 ceph-mon[89359]: osdmap e602: 8 total, 8 up, 8 in 2024-09-21T10:07:43.088 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:07:43.088 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:07:43.094 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCreateLsDeleteSnapPP (4025 ms) 2024-09-21T10:07:43.094 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestGetNameIdSnapPP 2024-09-21T10:07:43.095 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:43 smithi145 ceph-mon[89359]: osdmap e603: 8 total, 8 up, 8 in 2024-09-21T10:07:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:43 smithi119 ceph-mon[101432]: osdmap e603: 8 total, 8 up, 8 in 2024-09-21T10:07:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:43 smithi119 ceph-mon[105288]: osdmap e603: 8 total, 8 up, 8 in 2024-09-21T10:07:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:43.677 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:44 smithi145 ceph-mon[89359]: pgmap v537: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:07:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:44 smithi145 ceph-mon[89359]: osdmap e604: 8 total, 8 up, 8 in 2024-09-21T10:07:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1588017576' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:44 smithi145 ceph-mon[89359]: osdmap e605: 8 total, 8 up, 8 in 2024-09-21T10:07:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[105288]: pgmap v537: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:07:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[105288]: osdmap e604: 8 total, 8 up, 8 in 2024-09-21T10:07:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1588017576' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[105288]: osdmap e605: 8 total, 8 up, 8 in 2024-09-21T10:07:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[101432]: pgmap v537: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:07:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[101432]: osdmap e604: 8 total, 8 up, 8 in 2024-09-21T10:07:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1588017576' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-mon[101432]: osdmap e605: 8 total, 8 up, 8 in 2024-09-21T10:07:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:07:44] "GET /metrics HTTP/1.1" 200 50987 "" "Prometheus/2.51.0" 2024-09-21T10:07:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:45 smithi119 ceph-mon[101432]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:45 smithi119 ceph-mon[101432]: from='client.35145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:45 smithi119 ceph-mon[101432]: osdmap e606: 8 total, 8 up, 8 in 2024-09-21T10:07:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:45 smithi119 ceph-mon[105288]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:45 smithi119 ceph-mon[105288]: from='client.35145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:45 smithi119 ceph-mon[105288]: osdmap e606: 8 total, 8 up, 8 in 2024-09-21T10:07:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:45 smithi145 ceph-mon[89359]: from='client.45023 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:45 smithi145 ceph-mon[89359]: from='client.35145 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:45 smithi145 ceph-mon[89359]: osdmap e606: 8 total, 8 up, 8 in 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[105288]: pgmap v540: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[105288]: osdmap e607: 8 total, 8 up, 8 in 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[101432]: pgmap v540: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:46.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:46 smithi119 ceph-mon[101432]: osdmap e607: 8 total, 8 up, 8 in 2024-09-21T10:07:46.454 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:46 smithi145 ceph-mon[89359]: pgmap v540: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 6.2 KiB/s wr, 82 op/s 2024-09-21T10:07:46.454 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:46 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:46.454 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:46 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:46.455 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:46 smithi145 ceph-mon[89359]: osdmap e607: 8 total, 8 up, 8 in 2024-09-21T10:07:46.567 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3451253794' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[105288]: osdmap e608: 8 total, 8 up, 8 in 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3451253794' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:47 smithi119 ceph-mon[101432]: osdmap e608: 8 total, 8 up, 8 in 2024-09-21T10:07:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3451253794' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:47 smithi145 ceph-mon[89359]: osdmap e608: 8 total, 8 up, 8 in 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: pgmap v543: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 6.5 KiB/s wr, 84 op/s 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='client.35157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='client.45047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:48.149 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[105288]: osdmap e609: 8 total, 8 up, 8 in 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: pgmap v543: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 6.5 KiB/s wr, 84 op/s 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='client.35157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='client.45047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:48.150 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:48.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:48.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:48.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:48 smithi119 ceph-mon[101432]: osdmap e609: 8 total, 8 up, 8 in 2024-09-21T10:07:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: pgmap v543: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 60 KiB/s rd, 6.5 KiB/s wr, 84 op/s 2024-09-21T10:07:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='client.35157 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='client.45047 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:07:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:48 smithi145 ceph-mon[89359]: osdmap e609: 8 total, 8 up, 8 in 2024-09-21T10:07:49.044 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:07:49.045 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:07:49.049 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestGetNameIdSnapPP (5955 ms) 2024-09-21T10:07:49.049 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCreateLsRenameSnapPP 2024-09-21T10:07:49.049 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:49.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:49.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[105288]: Upgrade: osd.2 is safe to restart 2024-09-21T10:07:49.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:49.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[105288]: osdmap e610: 8 total, 8 up, 8 in 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[101432]: Upgrade: osd.2 is safe to restart 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:49.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:49 smithi119 ceph-mon[101432]: osdmap e610: 8 total, 8 up, 8 in 2024-09-21T10:07:49.459 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:49 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-21T10:07:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:49 smithi145 ceph-mon[89359]: Upgrade: osd.2 is safe to restart 2024-09-21T10:07:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:49 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:49 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-21T10:07:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:49 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:07:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:49 smithi145 ceph-mon[89359]: osdmap e610: 8 total, 8 up, 8 in 2024-09-21T10:07:50.050 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:07:50.050 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:50.051 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:07:50.051 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1 with size 2097152 2024-09-21T10:07:50.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[101432]: pgmap v546: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 5.2 KiB/s wr, 62 op/s 2024-09-21T10:07:50.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[101432]: Upgrade: Updating osd.2 2024-09-21T10:07:50.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[101432]: Deploying daemon osd.2 on smithi119 2024-09-21T10:07:50.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2111164453' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:50.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[101432]: osdmap e611: 8 total, 8 up, 8 in 2024-09-21T10:07:50.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[105288]: pgmap v546: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 5.2 KiB/s wr, 62 op/s 2024-09-21T10:07:50.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[105288]: Upgrade: Updating osd.2 2024-09-21T10:07:50.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[105288]: Deploying daemon osd.2 on smithi119 2024-09-21T10:07:50.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2111164453' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:50.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:50 smithi119 ceph-mon[105288]: osdmap e611: 8 total, 8 up, 8 in 2024-09-21T10:07:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:07:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:07:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:07:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:50 smithi145 ceph-mon[89359]: pgmap v546: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 5.2 KiB/s wr, 62 op/s 2024-09-21T10:07:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:50 smithi145 ceph-mon[89359]: Upgrade: Updating osd.2 2024-09-21T10:07:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:50 smithi145 ceph-mon[89359]: Deploying daemon osd.2 on smithi119 2024-09-21T10:07:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2111164453' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:50.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:50 smithi145 ceph-mon[89359]: osdmap e611: 8 total, 8 up, 8 in 2024-09-21T10:07:51.051 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 2 2024-09-21T10:07:51.051 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:51.051 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1 2024-09-21T10:07:51.051 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:07:51.051 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1 with size 2097152 2024-09-21T10:07:51.051 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2 with size 4194304 2024-09-21T10:07:51.054 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 2 2024-09-21T10:07:51.054 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:07:51.054 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap1-rename 2024-09-21T10:07:51.054 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2 2024-09-21T10:07:51.054 INFO:tasks.workunit.client.0.smithi119.stdout:found snap1-rename with size 2097152 2024-09-21T10:07:51.054 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2 with size 4194304 2024-09-21T10:07:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[105288]: from='client.35175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[105288]: from='client.45065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[105288]: pgmap v549: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 5.5 KiB/s wr, 63 op/s 2024-09-21T10:07:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[105288]: osdmap e612: 8 total, 8 up, 8 in 2024-09-21T10:07:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[101432]: from='client.35175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[101432]: from='client.45065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[101432]: pgmap v549: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 5.5 KiB/s wr, 63 op/s 2024-09-21T10:07:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:51 smithi119 ceph-mon[101432]: osdmap e612: 8 total, 8 up, 8 in 2024-09-21T10:07:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:51 smithi145 ceph-mon[89359]: from='client.35175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:51 smithi145 ceph-mon[89359]: from='client.45065 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:51 smithi145 ceph-mon[89359]: pgmap v549: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 45 KiB/s rd, 5.5 KiB/s wr, 63 op/s 2024-09-21T10:07:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:51 smithi145 ceph-mon[89359]: osdmap e612: 8 total, 8 up, 8 in 2024-09-21T10:07:52.051 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:07:52.051 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:07:52.051 INFO:tasks.workunit.client.0.smithi119.stdout:snap: snap2-rename 2024-09-21T10:07:52.051 INFO:tasks.workunit.client.0.smithi119.stdout:found snap2-rename with size 4194304 2024-09-21T10:07:52.109 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:51 smithi119 systemd[1]: Stopping Ceph osd.2 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:07:52.345 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:52.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[57200]: 2024-09-21T10:07:52.106+0000 7f0777991640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:07:52.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[57200]: 2024-09-21T10:07:52.106+0000 7f0777991640 -1 osd.2 613 *** Got signal Terminated *** 2024-09-21T10:07:52.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[57200]: 2024-09-21T10:07:52.106+0000 7f0777991640 -1 osd.2 613 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:07:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:53 smithi119 ceph-mon[101432]: osdmap e613: 8 total, 8 up, 8 in 2024-09-21T10:07:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/238352462' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:53 smithi119 ceph-mon[101432]: osd.2 marked itself down and dead 2024-09-21T10:07:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:53 smithi119 ceph-mon[105288]: osdmap e613: 8 total, 8 up, 8 in 2024-09-21T10:07:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/238352462' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:53 smithi119 ceph-mon[105288]: osd.2 marked itself down and dead 2024-09-21T10:07:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:53 smithi145 ceph-mon[89359]: osdmap e613: 8 total, 8 up, 8 in 2024-09-21T10:07:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/238352462' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:53 smithi145 ceph-mon[89359]: osd.2 marked itself down and dead 2024-09-21T10:07:53.858 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:53 smithi119 podman[127069]: 2024-09-21 10:07:53.59860589 +0000 UTC m=+1.613951596 container died 87104f5801c3a1959f8094f26836e49feab6e5ea564dafcfa1d22831c8cbf639 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:07:54.067 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:07:54.067 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:07:54.071 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCreateLsRenameSnapPP (5022 ms) 2024-09-21T10:07:54.071 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ConcurrentCreatesUnvalidatedPool 2024-09-21T10:07:54.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 podman[127069]: 2024-09-21 10:07:54.022046028 +0000 UTC m=+2.037391790 container cleanup 87104f5801c3a1959f8094f26836e49feab6e5ea564dafcfa1d22831c8cbf639 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T10:07:54.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 bash[127069]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2 2024-09-21T10:07:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[105288]: from='client.35193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[105288]: pgmap v552: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-21T10:07:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[105288]: from='client.45083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[105288]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[105288]: osdmap e614: 8 total, 7 up, 8 in 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[105288]: osdmap e615: 8 total, 7 up, 8 in 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[101432]: from='client.35193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[101432]: pgmap v552: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[101432]: from='client.45083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[101432]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[101432]: osdmap e614: 8 total, 7 up, 8 in 2024-09-21T10:07:54.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-mon[101432]: osdmap e615: 8 total, 7 up, 8 in 2024-09-21T10:07:54.542 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 podman[127140]: 2024-09-21 10:07:54.248922786 +0000 UTC m=+0.644349096 container remove 87104f5801c3a1959f8094f26836e49feab6e5ea564dafcfa1d22831c8cbf639 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T10:07:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:54 smithi145 ceph-mon[89359]: from='client.35193 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:54 smithi145 ceph-mon[89359]: pgmap v552: 297 pgs: 297 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-21T10:07:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:54 smithi145 ceph-mon[89359]: from='client.45083 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:54 smithi145 ceph-mon[89359]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:07:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:54 smithi145 ceph-mon[89359]: osdmap e614: 8 total, 7 up, 8 in 2024-09-21T10:07:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:54 smithi145 ceph-mon[89359]: osdmap e615: 8 total, 7 up, 8 in 2024-09-21T10:07:54.848 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:07:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:07:54] "GET /metrics HTTP/1.1" 200 51001 "" "Prometheus/2.51.0" 2024-09-21T10:07:54.848 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 podman[127249]: 2024-09-21 10:07:54.595084439 +0000 UTC m=+0.015827509 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:54.849 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 podman[127249]: 2024-09-21 10:07:54.716765734 +0000 UTC m=+0.137508812 container create e18514f2b63e08e980b19c682a965f7ae13cb7fdd88974ff7be9cf721bfa178a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T10:07:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1020792717' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1020792717' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:55.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 podman[127249]: 2024-09-21 10:07:54.944301717 +0000 UTC m=+0.365044786 container init e18514f2b63e08e980b19c682a965f7ae13cb7fdd88974ff7be9cf721bfa178a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-21T10:07:55.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:54 smithi119 podman[127249]: 2024-09-21 10:07:54.949428088 +0000 UTC m=+0.370171148 container start e18514f2b63e08e980b19c682a965f7ae13cb7fdd88974ff7be9cf721bfa178a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.schema-version=1.0) 2024-09-21T10:07:55.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:55 smithi119 podman[127249]: 2024-09-21 10:07:55.035774245 +0000 UTC m=+0.456517322 container attach e18514f2b63e08e980b19c682a965f7ae13cb7fdd88974ff7be9cf721bfa178a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T10:07:55.244 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:55.516 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:55 smithi119 podman[127249]: 2024-09-21 10:07:55.158020778 +0000 UTC m=+0.578763852 container died e18514f2b63e08e980b19c682a965f7ae13cb7fdd88974ff7be9cf721bfa178a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T10:07:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1020792717' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:56.067 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:56.067 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:56.067 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:56.067 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[105288]: pgmap v555: 297 pgs: 72 peering, 9 stale+active+clean, 216 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[105288]: osdmap e616: 8 total, 7 up, 8 in 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3766460040' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[101432]: pgmap v555: 297 pgs: 72 peering, 9 stale+active+clean, 216 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[101432]: osdmap e616: 8 total, 7 up, 8 in 2024-09-21T10:07:56.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3766460040' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:56.118 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:55 smithi119 podman[127338]: 2024-09-21 10:07:55.849406158 +0000 UTC m=+0.685902526 container remove e18514f2b63e08e980b19c682a965f7ae13cb7fdd88974ff7be9cf721bfa178a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:07:56.118 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:55 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-21T10:07:56.118 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:55 smithi119 systemd[1]: Stopped Ceph osd.2 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:07:56.118 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:55 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.2.service: Consumed 24.001s CPU time. 2024-09-21T10:07:56.118 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 systemd[1]: Starting Ceph osd.2 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:07:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:56 smithi145 ceph-mon[89359]: pgmap v555: 297 pgs: 72 peering, 9 stale+active+clean, 216 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.5 KiB/s wr, 17 op/s 2024-09-21T10:07:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:56 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:07:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:56 smithi145 ceph-mon[89359]: osdmap e616: 8 total, 7 up, 8 in 2024-09-21T10:07:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3766460040' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-22","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:56.556 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 podman[127473]: 2024-09-21 10:07:56.17956376 +0000 UTC m=+0.015468466 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:56.556 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 podman[127473]: 2024-09-21 10:07:56.327255863 +0000 UTC m=+0.163160558 container create 33611232807e0768322a1a503146fe2e310b11fb239e44695dcc64015f356b24 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-21T10:07:56.867 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 podman[127473]: 2024-09-21 10:07:56.554936878 +0000 UTC m=+0.390841590 container init 33611232807e0768322a1a503146fe2e310b11fb239e44695dcc64015f356b24 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T10:07:56.867 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 podman[127473]: 2024-09-21 10:07:56.559753133 +0000 UTC m=+0.395657832 container start 33611232807e0768322a1a503146fe2e310b11fb239e44695dcc64015f356b24 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:07:56.867 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 podman[127473]: 2024-09-21 10:07:56.646381762 +0000 UTC m=+0.482286459 container attach 33611232807e0768322a1a503146fe2e310b11fb239e44695dcc64015f356b24 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:07:56.867 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:56.868 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 bash[127473]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:56.868 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:56.868 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:56 smithi119 bash[127473]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:07:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[105288]: from='client.45101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[105288]: from='client.45107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3766460040' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[105288]: osdmap e617: 8 total, 7 up, 8 in 2024-09-21T10:07:57.368 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:07:57.368 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:07:57.368 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:07:57.368 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:07:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[101432]: from='client.45101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[101432]: from='client.45107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:57.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3766460040' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:57.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-mon[101432]: osdmap e617: 8 total, 7 up, 8 in 2024-09-21T10:07:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:57 smithi145 ceph-mon[89359]: from='client.45101 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:57 smithi145 ceph-mon[89359]: from='client.45107 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:07:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3766460040' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:07:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:57 smithi145 ceph-mon[89359]: osdmap e617: 8 total, 7 up, 8 in 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate[127488]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T10:07:57.761 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 bash[127473]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-21T10:07:57.762 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:57 smithi119 podman[127473]: 2024-09-21 10:07:57.477453077 +0000 UTC m=+1.313357782 container died 33611232807e0768322a1a503146fe2e310b11fb239e44695dcc64015f356b24 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate, RELEASE=main-62da54b, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T10:07:58.112 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:07:58.317 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ConcurrentCreatesUnvalidatedPool (4246 ms) 2024-09-21T10:07:58.317 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RemoveFullTry 2024-09-21T10:07:58.357 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 podman[127649]: 2024-09-21 10:07:58.134434087 +0000 UTC m=+0.646962066 container remove 33611232807e0768322a1a503146fe2e310b11fb239e44695dcc64015f356b24 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-activate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=main-62da54b) 2024-09-21T10:07:58.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[101432]: pgmap v558: 329 pgs: 8 creating+peering, 12 active+undersized, 24 unknown, 72 peering, 6 stale+active+clean, 7 active+undersized+degraded, 200 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 KiB/s wr, 20 op/s; 13/477 objects degraded (2.725%) 2024-09-21T10:07:58.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 13/477 objects degraded (2.725%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:58.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[101432]: osdmap e618: 8 total, 7 up, 8 in 2024-09-21T10:07:58.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/643916210' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:58.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[105288]: pgmap v558: 329 pgs: 8 creating+peering, 12 active+undersized, 24 unknown, 72 peering, 6 stale+active+clean, 7 active+undersized+degraded, 200 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 KiB/s wr, 20 op/s; 13/477 objects degraded (2.725%) 2024-09-21T10:07:58.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 13/477 objects degraded (2.725%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:58.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[105288]: osdmap e618: 8 total, 7 up, 8 in 2024-09-21T10:07:58.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/643916210' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:58 smithi145 ceph-mon[89359]: pgmap v558: 329 pgs: 8 creating+peering, 12 active+undersized, 24 unknown, 72 peering, 6 stale+active+clean, 7 active+undersized+degraded, 200 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.5 KiB/s wr, 20 op/s; 13/477 objects degraded (2.725%) 2024-09-21T10:07:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:58 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 13/477 objects degraded (2.725%), 7 pgs degraded (PG_DEGRADED) 2024-09-21T10:07:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:58 smithi145 ceph-mon[89359]: osdmap e618: 8 total, 7 up, 8 in 2024-09-21T10:07:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:58 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/643916210' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:07:58.617 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 podman[127779]: 2024-09-21 10:07:58.258845613 +0000 UTC m=+0.017431868 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:07:58.617 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 podman[127779]: 2024-09-21 10:07:58.391423389 +0000 UTC m=+0.150009644 container create 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=main-62da54b, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:07:59.070 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 podman[127779]: 2024-09-21 10:07:58.696650225 +0000 UTC m=+0.455236488 container init 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, GIT_CLEAN=True) 2024-09-21T10:07:59.070 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 podman[127779]: 2024-09-21 10:07:58.701389932 +0000 UTC m=+0.459976189 container start 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:07:59.070 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 ceph-osd[127837]: -- 172.21.15.119:0/3377952831 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ce8f3e7a40 con 0x55ce901b6400 2024-09-21T10:07:59.070 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 bash[127779]: 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 2024-09-21T10:07:59.070 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:07:58 smithi119 systemd[1]: Started Ceph osd.2 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[105288]: osdmap e619: 8 total, 7 up, 8 in 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[105288]: osdmap e620: 8 total, 7 up, 8 in 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2183326645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[101432]: osdmap e619: 8 total, 7 up, 8 in 2024-09-21T10:07:59.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:59.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:59.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:59.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[101432]: osdmap e620: 8 total, 7 up, 8 in 2024-09-21T10:07:59.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:07:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2183326645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:07:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:59 smithi145 ceph-mon[89359]: osdmap e619: 8 total, 7 up, 8 in 2024-09-21T10:07:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:07:59.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:07:59.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:59 smithi145 ceph-mon[89359]: osdmap e620: 8 total, 7 up, 8 in 2024-09-21T10:07:59.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:07:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2183326645' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:00.327 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: from='client.45119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:00.327 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: from='client.54730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:00.327 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: pgmap v561: 329 pgs: 8 creating+peering, 73 active+undersized, 46 active+undersized+degraded, 202 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 85 KiB/s rd, 6.7 KiB/s wr, 119 op/s; 72/486 objects degraded (14.815%) 2024-09-21T10:08:00.327 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2183326645' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: osdmap e621: 8 total, 7 up, 8 in 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: from='client.45119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: from='client.54730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: pgmap v561: 329 pgs: 8 creating+peering, 73 active+undersized, 46 active+undersized+degraded, 202 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 85 KiB/s rd, 6.7 KiB/s wr, 119 op/s; 72/486 objects degraded (14.815%) 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2183326645' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: osdmap e621: 8 total, 7 up, 8 in 2024-09-21T10:08:00.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:08:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: from='client.45119 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: from='client.54730 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: pgmap v561: 329 pgs: 8 creating+peering, 73 active+undersized, 46 active+undersized+degraded, 202 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 85 KiB/s rd, 6.7 KiB/s wr, 119 op/s; 72/486 objects degraded (14.815%) 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2183326645' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: osdmap e621: 8 total, 7 up, 8 in 2024-09-21T10:08:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:00.617 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:08:00 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[127814]: 2024-09-21T10:08:00.324+0000 7f91299f7640 -1 Falling back to public interface 2024-09-21T10:08:01.040 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:01.325 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/34975565' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:01.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:01.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:01 smithi119 ceph-mon[105288]: osdmap e622: 8 total, 7 up, 8 in 2024-09-21T10:08:01.326 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/34975565' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:01.326 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:01.326 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:01 smithi119 ceph-mon[101432]: osdmap e622: 8 total, 7 up, 8 in 2024-09-21T10:08:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/34975565' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-23","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:01 smithi145 ceph-mon[89359]: osdmap e622: 8 total, 7 up, 8 in 2024-09-21T10:08:02.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:02 smithi119 ceph-mon[101432]: pgmap v564: 361 pgs: 29 unknown, 78 active+undersized, 46 active+undersized+degraded, 208 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 9.0 KiB/s wr, 157 op/s; 72/489 objects degraded (14.724%) 2024-09-21T10:08:02.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:02 smithi119 ceph-mon[101432]: from='client.45146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:02 smithi119 ceph-mon[101432]: osdmap e623: 8 total, 7 up, 8 in 2024-09-21T10:08:02.345 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:02 smithi119 ceph-mon[105288]: pgmap v564: 361 pgs: 29 unknown, 78 active+undersized, 46 active+undersized+degraded, 208 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 9.0 KiB/s wr, 157 op/s; 72/489 objects degraded (14.724%) 2024-09-21T10:08:02.345 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:02 smithi119 ceph-mon[105288]: from='client.45146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.345 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:02 smithi119 ceph-mon[105288]: osdmap e623: 8 total, 7 up, 8 in 2024-09-21T10:08:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:02 smithi145 ceph-mon[89359]: pgmap v564: 361 pgs: 29 unknown, 78 active+undersized, 46 active+undersized+degraded, 208 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 110 KiB/s rd, 9.0 KiB/s wr, 157 op/s; 72/489 objects degraded (14.724%) 2024-09-21T10:08:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:02 smithi145 ceph-mon[89359]: from='client.45146 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:02 smithi145 ceph-mon[89359]: osdmap e623: 8 total, 7 up, 8 in 2024-09-21T10:08:03.059 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:03] ENGINE Bus STOPPING 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: osdmap e624: 8 total, 7 up, 8 in 2024-09-21T10:08:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: osdmap e624: 8 total, 7 up, 8 in 2024-09-21T10:08:03.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: from='client.35268 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: osdmap e624: 8 total, 7 up, 8 in 2024-09-21T10:08:03.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:03.867 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[127814]: 2024-09-21T10:08:03.800+0000 7f91299f7640 -1 osd.2 613 log_to_monitors true 2024-09-21T10:08:03.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:08:03.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:03] ENGINE Bus STOPPED 2024-09-21T10:08:03.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:03] ENGINE Bus STARTING 2024-09-21T10:08:03.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:03] ENGINE Serving on http://:::9283 2024-09-21T10:08:03.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:03 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:03] ENGINE Bus STARTED 2024-09-21T10:08:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:04.203 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: pgmap v567: 361 pgs: 23 unknown, 80 active+undersized, 47 active+undersized+degraded, 211 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.5 KiB/s wr, 40 op/s; 73/492 objects degraded (14.837%) 2024-09-21T10:08:04.203 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 73/492 objects degraded (14.837%), 47 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:04.203 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:04.203 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:04.204 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:04.204 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:04.204 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3257098263' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:04.204 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:08:04.204 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:04.204 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:08:04.205 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: osdmap e625: 8 total, 7 up, 8 in 2024-09-21T10:08:04.205 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[105288]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:08:04.205 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: pgmap v567: 361 pgs: 23 unknown, 80 active+undersized, 47 active+undersized+degraded, 211 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.5 KiB/s wr, 40 op/s; 73/492 objects degraded (14.837%) 2024-09-21T10:08:04.205 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 73/492 objects degraded (14.837%), 47 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:04.205 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:04.206 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:04.206 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:04.206 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:04.206 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3257098263' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:04.206 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:08:04.206 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:04.207 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:08:04.207 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: osdmap e625: 8 total, 7 up, 8 in 2024-09-21T10:08:04.207 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-mon[101432]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: pgmap v567: 361 pgs: 23 unknown, 80 active+undersized, 47 active+undersized+degraded, 211 active+clean; 584 KiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.5 KiB/s wr, 40 op/s; 73/492 objects degraded (14.837%) 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 73/492 objects degraded (14.837%), 47 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-23", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3257098263' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-21T10:08:04.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-23", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:04.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T10:08:04.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: osdmap e625: 8 total, 7 up, 8 in 2024-09-21T10:08:04.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:04 smithi145 ceph-mon[89359]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:08:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:08:04] "GET /metrics HTTP/1.1" 200 55808 "" "Prometheus/2.51.0" 2024-09-21T10:08:04.867 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:08:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[127814]: 2024-09-21T10:08:04.493+0000 7f911d799640 -1 osd.2 613 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:08:05.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:05.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[101432]: from='client.45164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[101432]: from='client.35289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[101432]: pgmap v570: 361 pgs: 92 active+undersized, 44 active+undersized+degraded, 225 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s; 75/504 objects degraded (14.881%) 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[101432]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[105288]: from='client.45164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[105288]: from='client.35289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[105288]: pgmap v570: 361 pgs: 92 active+undersized, 44 active+undersized+degraded, 225 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s; 75/504 objects degraded (14.881%) 2024-09-21T10:08:05.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:05 smithi119 ceph-mon[105288]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' 2024-09-21T10:08:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:05 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:05 smithi145 ceph-mon[89359]: from='client.45164 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:05 smithi145 ceph-mon[89359]: from='client.35289 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:05 smithi145 ceph-mon[89359]: pgmap v570: 361 pgs: 92 active+undersized, 44 active+undersized+degraded, 225 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 1023 B/s wr, 1 op/s; 75/504 objects degraded (14.881%) 2024-09-21T10:08:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:05 smithi145 ceph-mon[89359]: from='osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048]' entity='osd.2' 2024-09-21T10:08:06.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[101432]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:06.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[101432]: osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048] boot 2024-09-21T10:08:06.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[101432]: osdmap e626: 8 total, 8 up, 8 in 2024-09-21T10:08:06.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:08:06.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[105288]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:06.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[105288]: osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048] boot 2024-09-21T10:08:06.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[105288]: osdmap e626: 8 total, 8 up, 8 in 2024-09-21T10:08:06.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:08:06.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:06 smithi145 ceph-mon[89359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:06.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:06 smithi145 ceph-mon[89359]: osd.2 [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048] boot 2024-09-21T10:08:06.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:06 smithi145 ceph-mon[89359]: osdmap e626: 8 total, 8 up, 8 in 2024-09-21T10:08:06.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-21T10:08:06.917 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:07.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:08:07.054+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (43 PGs are or would become offline) 2024-09-21T10:08:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: pgmap v572: 361 pgs: 92 active+undersized, 44 active+undersized+degraded, 225 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 230 B/s rd, 922 B/s wr, 1 op/s; 75/504 objects degraded (14.881%) 2024-09-21T10:08:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: osdmap e627: 8 total, 8 up, 8 in 2024-09-21T10:08:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3776952368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='client.35301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:07.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:07 smithi145 ceph-mon[89359]: Upgrade: unsafe to stop osd(s) at this time (43 PGs are or would become offline) 2024-09-21T10:08:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: pgmap v572: 361 pgs: 92 active+undersized, 44 active+undersized+degraded, 225 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 230 B/s rd, 922 B/s wr, 1 op/s; 75/504 objects degraded (14.881%) 2024-09-21T10:08:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: osdmap e627: 8 total, 8 up, 8 in 2024-09-21T10:08:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3776952368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='client.35301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:07.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[101432]: Upgrade: unsafe to stop osd(s) at this time (43 PGs are or would become offline) 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: pgmap v572: 361 pgs: 92 active+undersized, 44 active+undersized+degraded, 225 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 230 B/s rd, 922 B/s wr, 1 op/s; 75/504 objects degraded (14.881%) 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: osdmap e627: 8 total, 8 up, 8 in 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3776952368' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='client.35301 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:07.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:07.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:07.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:07 smithi119 ceph-mon[105288]: Upgrade: unsafe to stop osd(s) at this time (43 PGs are or would become offline) 2024-09-21T10:08:08.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:08 smithi145 ceph-mon[89359]: from='client.45191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:08 smithi119 ceph-mon[101432]: from='client.45191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:08 smithi119 ceph-mon[105288]: from='client.45191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:09.784 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:09.784 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:09 smithi119 ceph-mon[101432]: pgmap v574: 361 pgs: 39 active+undersized, 18 active+undersized+degraded, 304 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.0 KiB/s rd, 943 KiB/s wr, 12 op/s; 32/513 objects degraded (6.238%); 10 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:08:09.784 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:09 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 32/513 objects degraded (6.238%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:09.784 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/77221049' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:09.785 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:09 smithi119 ceph-mon[105288]: pgmap v574: 361 pgs: 39 active+undersized, 18 active+undersized+degraded, 304 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.0 KiB/s rd, 943 KiB/s wr, 12 op/s; 32/513 objects degraded (6.238%); 10 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:08:09.785 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:09 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 32/513 objects degraded (6.238%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:09.785 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/77221049' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:09.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:09 smithi145 ceph-mon[89359]: pgmap v574: 361 pgs: 39 active+undersized, 18 active+undersized+degraded, 304 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.0 KiB/s rd, 943 KiB/s wr, 12 op/s; 32/513 objects degraded (6.238%); 10 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:08:09.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:09 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 32/513 objects degraded (6.238%), 18 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:09.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/77221049' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:10.505 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:08:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:08:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:10.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:10 smithi145 ceph-mon[89359]: from='client.35319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:10.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:10 smithi145 ceph-mon[89359]: from='client.35325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:10 smithi119 ceph-mon[105288]: from='client.35319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:10 smithi119 ceph-mon[105288]: from='client.35325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:10 smithi119 ceph-mon[101432]: from='client.35319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:10 smithi119 ceph-mon[101432]: from='client.35325 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:11 smithi145 ceph-mon[89359]: pgmap v575: 361 pgs: 361 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 796 KiB/s wr, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:11 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/513 objects degraded (6.238%), 18 pgs degraded) 2024-09-21T10:08:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:11 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:08:11.823 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:11 smithi119 ceph-mon[105288]: pgmap v575: 361 pgs: 361 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 796 KiB/s wr, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:11.823 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:11 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/513 objects degraded (6.238%), 18 pgs degraded) 2024-09-21T10:08:11.824 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:11 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:08:11.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:11 smithi119 ceph-mon[101432]: pgmap v575: 361 pgs: 361 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.7 KiB/s rd, 796 KiB/s wr, 9 op/s; 8 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:11.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:11 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/513 objects degraded (6.238%), 18 pgs degraded) 2024-09-21T10:08:11.824 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:11 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:08:12.645 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3778707543' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3778707543' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3778707543' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:13 smithi145 ceph-mon[89359]: pgmap v576: 361 pgs: 361 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 641 KiB/s wr, 7 op/s; 7 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:13 smithi145 ceph-mon[89359]: from='client.45221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:13 smithi145 ceph-mon[89359]: from='client.45227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:13 smithi119 ceph-mon[101432]: pgmap v576: 361 pgs: 361 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 641 KiB/s wr, 7 op/s; 7 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:13 smithi119 ceph-mon[101432]: from='client.45221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:13 smithi119 ceph-mon[101432]: from='client.45227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:13.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:13 smithi119 ceph-mon[105288]: pgmap v576: 361 pgs: 361 active+clean; 5.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.6 KiB/s rd, 641 KiB/s wr, 7 op/s; 7 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:13.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:13 smithi119 ceph-mon[105288]: from='client.45221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:13.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:13 smithi119 ceph-mon[105288]: from='client.45227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:08:14] "GET /metrics HTTP/1.1" 200 55808 "" "Prometheus/2.51.0" 2024-09-21T10:08:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:15 smithi145 ceph-mon[89359]: osdmap e628: 8 total, 8 up, 8 in 2024-09-21T10:08:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:15 smithi119 ceph-mon[105288]: osdmap e628: 8 total, 8 up, 8 in 2024-09-21T10:08:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:15 smithi119 ceph-mon[101432]: osdmap e628: 8 total, 8 up, 8 in 2024-09-21T10:08:15.503 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:16 smithi119 ceph-mon[105288]: pgmap v578: 361 pgs: 361 active+clean; 12 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.4 MiB/s wr, 9 op/s; 7 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1279289808' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:16 smithi119 ceph-mon[101432]: pgmap v578: 361 pgs: 361 active+clean; 12 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.4 MiB/s wr, 9 op/s; 7 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1279289808' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:16 smithi145 ceph-mon[89359]: pgmap v578: 361 pgs: 361 active+clean; 12 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 1.4 MiB/s wr, 9 op/s; 7 B/s, 2 keys/s, 1 objects/s recovering 2024-09-21T10:08:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1279289808' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:17 smithi119 ceph-mon[105288]: from='client.35355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:17 smithi119 ceph-mon[105288]: from='client.35361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:17 smithi119 ceph-mon[101432]: from='client.35355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:17 smithi119 ceph-mon[101432]: from='client.35361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:17 smithi145 ceph-mon[89359]: from='client.35355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:17 smithi145 ceph-mon[89359]: from='client.35361 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:18.357 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:18.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[101432]: pgmap v579: 361 pgs: 361 active+clean; 12 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.1 MiB/s wr, 7 op/s; 5 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:08:18.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:18.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:18.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:18.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[105288]: pgmap v579: 361 pgs: 361 active+clean; 12 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.1 MiB/s wr, 7 op/s; 5 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:08:18.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:18.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:18.358 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:18 smithi145 ceph-mon[89359]: pgmap v579: 361 pgs: 361 active+clean; 12 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.1 MiB/s wr, 7 op/s; 5 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:08:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:19.246 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RemoveFullTry (20928 ms) 2024-09-21T10:08:19.246 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RemoveFullTryDataPool 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2568213545' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-23' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[101432]: osdmap e629: 8 total, 8 up, 8 in 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2568213545' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-23' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:08:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:19.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:19 smithi119 ceph-mon[105288]: osdmap e629: 8 total, 8 up, 8 in 2024-09-21T10:08:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2568213545' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:19 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-23' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:08:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:19 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:19 smithi145 ceph-mon[89359]: osdmap e629: 8 total, 8 up, 8 in 2024-09-21T10:08:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:20 smithi119 ceph-mon[101432]: from='client.35373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:20 smithi119 ceph-mon[101432]: pgmap v580: 361 pgs: 361 active+clean; 15 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 922 KiB/s wr, 2 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:20 smithi119 ceph-mon[101432]: from='client.35379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:20 smithi119 ceph-mon[105288]: from='client.35373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:20.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:20 smithi119 ceph-mon[105288]: pgmap v580: 361 pgs: 361 active+clean; 15 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 922 KiB/s wr, 2 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:20.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:20 smithi119 ceph-mon[105288]: from='client.35379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:20 smithi145 ceph-mon[89359]: from='client.35373 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:20 smithi145 ceph-mon[89359]: pgmap v580: 361 pgs: 361 active+clean; 15 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 922 KiB/s wr, 2 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:20 smithi145 ceph-mon[89359]: from='client.35379 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:08:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:08:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:21.233 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:21 smithi119 ceph-mon[105288]: osdmap e630: 8 total, 8 up, 8 in 2024-09-21T10:08:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2192069303' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2432865665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:21 smithi119 ceph-mon[101432]: osdmap e630: 8 total, 8 up, 8 in 2024-09-21T10:08:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2192069303' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2432865665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:21 smithi145 ceph-mon[89359]: osdmap e630: 8 total, 8 up, 8 in 2024-09-21T10:08:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2192069303' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2432865665' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:22.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:08:22.070+0000 7fc0be6c7640 -1 mgr.server reply reply (11) Resource temporarily unavailable 28 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:08:22.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:08:22.070+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy 28 pgs have unknown state; cannot draw any conclusionsunsafe to stop osd(s) at this time (0 PGs are or would become offline) 2024-09-21T10:08:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[105288]: pgmap v583: 393 pgs: 4 creating+peering, 28 unknown, 361 active+clean; 16 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-21T10:08:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:08:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2192069303' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[105288]: osdmap e631: 8 total, 8 up, 8 in 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[101432]: pgmap v583: 393 pgs: 4 creating+peering, 28 unknown, 361 active+clean; 16 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2192069303' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[101432]: osdmap e631: 8 total, 8 up, 8 in 2024-09-21T10:08:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:22 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:22 smithi145 ceph-mon[89359]: pgmap v583: 393 pgs: 4 creating+peering, 28 unknown, 361 active+clean; 16 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 794 B/s rd, 1.1 MiB/s wr, 2 op/s 2024-09-21T10:08:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:22 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:08:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2192069303' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:22 smithi145 ceph-mon[89359]: osdmap e631: 8 total, 8 up, 8 in 2024-09-21T10:08:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:22 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: from='client.45281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: from='client.45293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: Upgrade: 28 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:08:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: osdmap e632: 8 total, 8 up, 8 in 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3064351439' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: from='client.45281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: from='client.45293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: Upgrade: 28 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: osdmap e632: 8 total, 8 up, 8 in 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3064351439' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: from='client.45281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: from='client.45293 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: Upgrade: 28 pgs have unknown state; cannot draw any conclusions 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: osdmap e632: 8 total, 8 up, 8 in 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3064351439' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:23.425 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:24.087 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:24 smithi145 ceph-mon[89359]: pgmap v586: 425 pgs: 5 creating+peering, 54 unknown, 366 active+clean; 16 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 256 KiB/s wr, 2 op/s 2024-09-21T10:08:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:24 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:24 smithi145 ceph-mon[89359]: osdmap e633: 8 total, 8 up, 8 in 2024-09-21T10:08:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4229223736' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[101432]: pgmap v586: 425 pgs: 5 creating+peering, 54 unknown, 366 active+clean; 16 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 256 KiB/s wr, 2 op/s 2024-09-21T10:08:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[101432]: osdmap e633: 8 total, 8 up, 8 in 2024-09-21T10:08:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4229223736' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[105288]: pgmap v586: 425 pgs: 5 creating+peering, 54 unknown, 366 active+clean; 16 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 256 KiB/s wr, 2 op/s 2024-09-21T10:08:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[105288]: osdmap e633: 8 total, 8 up, 8 in 2024-09-21T10:08:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rbd","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4229223736' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:08:24] "GET /metrics HTTP/1.1" 200 55839 "" "Prometheus/2.51.0" 2024-09-21T10:08:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:25 smithi119 ceph-mon[101432]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:25 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:25 smithi119 ceph-mon[101432]: osdmap e634: 8 total, 8 up, 8 in 2024-09-21T10:08:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:25 smithi119 ceph-mon[105288]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:25 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:25 smithi119 ceph-mon[105288]: osdmap e634: 8 total, 8 up, 8 in 2024-09-21T10:08:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:25 smithi145 ceph-mon[89359]: from='client.45305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:25 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-24","app": "rbd","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:25 smithi145 ceph-mon[89359]: osdmap e634: 8 total, 8 up, 8 in 2024-09-21T10:08:26.133 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[101432]: from='client.35427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[101432]: pgmap v589: 425 pgs: 5 creating+peering, 5 unknown, 415 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.7 KiB/s wr, 39 op/s 2024-09-21T10:08:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:08:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[101432]: osdmap e635: 8 total, 8 up, 8 in 2024-09-21T10:08:26.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:26] ENGINE Bus STOPPING 2024-09-21T10:08:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[105288]: from='client.35427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[105288]: pgmap v589: 425 pgs: 5 creating+peering, 5 unknown, 415 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.7 KiB/s wr, 39 op/s 2024-09-21T10:08:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:08:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-mon[105288]: osdmap e635: 8 total, 8 up, 8 in 2024-09-21T10:08:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:26 smithi145 ceph-mon[89359]: from='client.35427 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:26 smithi145 ceph-mon[89359]: pgmap v589: 425 pgs: 5 creating+peering, 5 unknown, 415 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 1.7 KiB/s wr, 39 op/s 2024-09-21T10:08:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:26 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:08:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:26 smithi145 ceph-mon[89359]: osdmap e635: 8 total, 8 up, 8 in 2024-09-21T10:08:26.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:08:26.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:26] ENGINE Bus STOPPED 2024-09-21T10:08:26.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:26] ENGINE Bus STARTING 2024-09-21T10:08:26.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:26] ENGINE Serving on http://:::9283 2024-09-21T10:08:26.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:26 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:26] ENGINE Bus STARTED 2024-09-21T10:08:26.954 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: osdmap e636: 8 total, 8 up, 8 in 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: pgmap v592: 425 pgs: 1 creating+peering, 424 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 KiB/s wr, 36 op/s 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2770161962' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[101432]: from='client.45323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: osdmap e636: 8 total, 8 up, 8 in 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: pgmap v592: 425 pgs: 1 creating+peering, 424 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 KiB/s wr, 36 op/s 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2770161962' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:27.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:27 smithi119 ceph-mon[105288]: from='client.45323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: osdmap e636: 8 total, 8 up, 8 in 2024-09-21T10:08:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: pgmap v592: 425 pgs: 1 creating+peering, 424 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 KiB/s wr, 36 op/s 2024-09-21T10:08:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2770161962' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:27 smithi145 ceph-mon[89359]: from='client.45323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:28 smithi145 ceph-mon[89359]: osdmap e637: 8 total, 8 up, 8 in 2024-09-21T10:08:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:28 smithi145 ceph-mon[89359]: from='client.35445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:28 smithi119 ceph-mon[105288]: osdmap e637: 8 total, 8 up, 8 in 2024-09-21T10:08:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:28 smithi119 ceph-mon[105288]: from='client.35445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:28 smithi119 ceph-mon[101432]: osdmap e637: 8 total, 8 up, 8 in 2024-09-21T10:08:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:28 smithi119 ceph-mon[101432]: from='client.35445 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: osdmap e638: 8 total, 8 up, 8 in 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: pgmap v595: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-23' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:08:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:29.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:29 smithi145 ceph-mon[89359]: osdmap e639: 8 total, 8 up, 8 in 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: osdmap e638: 8 total, 8 up, 8 in 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: pgmap v595: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-23' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[101432]: osdmap e639: 8 total, 8 up, 8 in 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: osdmap e638: 8 total, 8 up, 8 in 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-25", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: pgmap v595: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-23' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-25", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:29.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:29 smithi119 ceph-mon[105288]: osdmap e639: 8 total, 8 up, 8 in 2024-09-21T10:08:29.826 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1922853410' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:30 smithi145 ceph-mon[89359]: from='client.45341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:08:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:08:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1922853410' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:30 smithi119 ceph-mon[101432]: from='client.45341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:30.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1922853410' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:30.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:30 smithi119 ceph-mon[105288]: from='client.45341 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:31.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:31 smithi119 ceph-mon[105288]: from='client.45347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:31.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:31 smithi119 ceph-mon[105288]: pgmap v597: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-21T10:08:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:31 smithi119 ceph-mon[101432]: from='client.45347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:31 smithi119 ceph-mon[101432]: pgmap v597: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-21T10:08:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:31 smithi145 ceph-mon[89359]: from='client.45347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:31 smithi145 ceph-mon[89359]: pgmap v597: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.9 KiB/s rd, 2.1 KiB/s wr, 10 op/s 2024-09-21T10:08:32.692 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3225855070' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:32.692 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3225855070' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:32.692 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3225855070' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: pgmap v598: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='client.45359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='client.45365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.6", "id": [7, 0]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.16", "id": [7, 4]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.d", "id": [7, 0]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.9", "id": [0, 4]} : dispatch 2024-09-21T10:08:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.f", "id": [0, 2]} : dispatch 2024-09-21T10:08:33.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.13", "id": [6, 1]} : dispatch 2024-09-21T10:08:33.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.1a", "id": [6, 4]} : dispatch 2024-09-21T10:08:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: pgmap v598: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T10:08:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='client.45359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='client.45365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.6", "id": [7, 0]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.16", "id": [7, 4]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.d", "id": [7, 0]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.9", "id": [0, 4]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.f", "id": [0, 2]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.13", "id": [6, 1]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.1a", "id": [6, 4]} : dispatch 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: pgmap v598: 425 pgs: 425 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-09-21T10:08:33.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='client.45359 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='client.45365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.6", "id": [7, 0]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.16", "id": [7, 4]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.d", "id": [7, 0]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.9", "id": [0, 4]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.f", "id": [0, 2]} : dispatch 2024-09-21T10:08:33.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.13", "id": [6, 1]} : dispatch 2024-09-21T10:08:33.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.1a", "id": [6, 4]} : dispatch 2024-09-21T10:08:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.6", "id": [7, 0]}]': finished 2024-09-21T10:08:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.16", "id": [7, 4]}]': finished 2024-09-21T10:08:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.d", "id": [7, 0]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.9", "id": [0, 4]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.f", "id": [0, 2]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.13", "id": [6, 1]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.1a", "id": [6, 4]}]': finished 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: osdmap e640: 8 total, 8 up, 8 in 2024-09-21T10:08:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:34 smithi145 ceph-mon[89359]: osdmap e641: 8 total, 8 up, 8 in 2024-09-21T10:08:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:08:34] "GET /metrics HTTP/1.1" 200 60624 "" "Prometheus/2.51.0" 2024-09-21T10:08:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.6", "id": [7, 0]}]': finished 2024-09-21T10:08:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.16", "id": [7, 4]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.d", "id": [7, 0]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.9", "id": [0, 4]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.f", "id": [0, 2]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.13", "id": [6, 1]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.1a", "id": [6, 4]}]': finished 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: osdmap e640: 8 total, 8 up, 8 in 2024-09-21T10:08:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[105288]: osdmap e641: 8 total, 8 up, 8 in 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.6", "id": [7, 0]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.16", "id": [7, 4]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.18", "id": [7, 5]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "20.1c", "id": [7, 5]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.d", "id": [7, 0]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.1a", "id": [7, 2]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.9", "id": [0, 4]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "25.f", "id": [0, 2]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.13", "id": [6, 1]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "28.1a", "id": [6, 4]}]': finished 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: osdmap e640: 8 total, 8 up, 8 in 2024-09-21T10:08:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:34 smithi119 ceph-mon[101432]: osdmap e641: 8 total, 8 up, 8 in 2024-09-21T10:08:35.570 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:35 smithi145 ceph-mon[89359]: pgmap v601: 425 pgs: 425 active+clean; 5.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 854 KiB/s wr, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/232794558' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:35 smithi119 ceph-mon[105288]: pgmap v601: 425 pgs: 425 active+clean; 5.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 854 KiB/s wr, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/232794558' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:35 smithi119 ceph-mon[101432]: pgmap v601: 425 pgs: 425 active+clean; 5.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.8 KiB/s rd, 854 KiB/s wr, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/232794558' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:36 smithi145 ceph-mon[89359]: from='client.45377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:36 smithi145 ceph-mon[89359]: from='client.35499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:36 smithi119 ceph-mon[105288]: from='client.45377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:36 smithi119 ceph-mon[105288]: from='client.35499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:36 smithi119 ceph-mon[101432]: from='client.45377 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:36 smithi119 ceph-mon[101432]: from='client.35499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:37.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[101432]: pgmap v602: 425 pgs: 425 active+clean; 5.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 688 KiB/s wr, 8 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:37.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:37.736 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:37.736 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[101432]: Upgrade: osd.3 is safe to restart 2024-09-21T10:08:37.736 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[105288]: pgmap v602: 425 pgs: 425 active+clean; 5.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 688 KiB/s wr, 8 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:37.736 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:37.736 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:37.736 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:37 smithi119 ceph-mon[105288]: Upgrade: osd.3 is safe to restart 2024-09-21T10:08:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:37 smithi145 ceph-mon[89359]: pgmap v602: 425 pgs: 425 active+clean; 5.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 4.7 KiB/s rd, 688 KiB/s wr, 8 op/s; 4 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:37 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16} : dispatch 2024-09-21T10:08:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:37 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-21T10:08:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:37 smithi145 ceph-mon[89359]: Upgrade: osd.3 is safe to restart 2024-09-21T10:08:38.420 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4159227420' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:08:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:38.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4159227420' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:38.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:38.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:08:38.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4159227420' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-21T10:08:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:40.006 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: from='client.45395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:40.006 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: pgmap v603: 425 pgs: 425 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 6 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:40.006 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: Upgrade: Updating osd.3 2024-09-21T10:08:40.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: Deploying daemon osd.3 on smithi119 2024-09-21T10:08:40.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: from='client.35517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:40.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-25' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:08:40.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:40.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: osdmap e642: 8 total, 8 up, 8 in 2024-09-21T10:08:40.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-24", "field": "max_objects", "val": "6"} : dispatch 2024-09-21T10:08:40.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: from='client.45395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: pgmap v603: 425 pgs: 425 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 6 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: Upgrade: Updating osd.3 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: Deploying daemon osd.3 on smithi119 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: from='client.35517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-25' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: osdmap e642: 8 total, 8 up, 8 in 2024-09-21T10:08:40.011 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-24", "field": "max_objects", "val": "6"} : dispatch 2024-09-21T10:08:40.048 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: from='client.45395 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: pgmap v603: 425 pgs: 425 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.9 KiB/s rd, 1.3 MiB/s wr, 4 op/s; 6 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: Upgrade: Updating osd.3 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: Deploying daemon osd.3 on smithi119 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: from='client.35517 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-25' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: osdmap e642: 8 total, 8 up, 8 in 2024-09-21T10:08:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-24", "field": "max_objects", "val": "6"} : dispatch 2024-09-21T10:08:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:08:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:08:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:41.070 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:41.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-24", "field": "max_objects", "val": "6"}]': finished 2024-09-21T10:08:41.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-mon[105288]: osdmap e643: 8 total, 8 up, 8 in 2024-09-21T10:08:41.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2959341688' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:41.198 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-24", "field": "max_objects", "val": "6"}]': finished 2024-09-21T10:08:41.198 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-mon[101432]: osdmap e643: 8 total, 8 up, 8 in 2024-09-21T10:08:41.198 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2959341688' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:41.312 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-24", "field": "max_objects", "val": "6"}]': finished 2024-09-21T10:08:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:41 smithi145 ceph-mon[89359]: osdmap e643: 8 total, 8 up, 8 in 2024-09-21T10:08:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2959341688' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:41.821 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:41 smithi119 systemd[1]: Stopping Ceph osd.3 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:08:42.072 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[63343]: 2024-09-21T10:08:41.818+0000 7f397b971640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:08:42.073 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[63343]: 2024-09-21T10:08:41.818+0000 7f397b971640 -1 osd.3 643 *** Got signal Terminated *** 2024-09-21T10:08:42.073 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[63343]: 2024-09-21T10:08:41.818+0000 7f397b971640 -1 osd.3 643 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:08:42.081 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:42 smithi119 ceph-mon[105288]: pgmap v606: 425 pgs: 425 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 635 B/s rd, 794 KiB/s wr, 2 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:42 smithi119 ceph-mon[105288]: osd.3 marked itself down and dead 2024-09-21T10:08:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:42 smithi119 ceph-mon[101432]: pgmap v606: 425 pgs: 425 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 635 B/s rd, 794 KiB/s wr, 2 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:42 smithi119 ceph-mon[101432]: osd.3 marked itself down and dead 2024-09-21T10:08:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:42 smithi145 ceph-mon[89359]: pgmap v606: 425 pgs: 425 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 635 B/s rd, 794 KiB/s wr, 2 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:08:42.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:42 smithi145 ceph-mon[89359]: osd.3 marked itself down and dead 2024-09-21T10:08:43.086 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:42 smithi119 podman[133413]: 2024-09-21 10:08:42.630146783 +0000 UTC m=+0.930441589 container died 700faab8401d03c7358499dabc93ef77b9f606f5ffad24f7baa96b9780b72cdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T10:08:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[105288]: from='client.35529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[105288]: from='client.45419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[105288]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[105288]: osdmap e644: 8 total, 7 up, 8 in 2024-09-21T10:08:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[105288]: osd.3 marked itself dead as of e644 2024-09-21T10:08:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[105288]: osdmap e645: 8 total, 7 up, 8 in 2024-09-21T10:08:43.368 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:43 smithi119 podman[133413]: 2024-09-21 10:08:43.192081459 +0000 UTC m=+1.492376254 container cleanup 700faab8401d03c7358499dabc93ef77b9f606f5ffad24f7baa96b9780b72cdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T10:08:43.368 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:43 smithi119 bash[133413]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3 2024-09-21T10:08:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[101432]: from='client.35529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[101432]: from='client.45419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[101432]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[101432]: osdmap e644: 8 total, 7 up, 8 in 2024-09-21T10:08:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[101432]: osd.3 marked itself dead as of e644 2024-09-21T10:08:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:43 smithi119 ceph-mon[101432]: osdmap e645: 8 total, 7 up, 8 in 2024-09-21T10:08:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:43 smithi145 ceph-mon[89359]: from='client.35529 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:43 smithi145 ceph-mon[89359]: from='client.45419 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:43 smithi145 ceph-mon[89359]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:08:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:43 smithi145 ceph-mon[89359]: osdmap e644: 8 total, 7 up, 8 in 2024-09-21T10:08:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:43 smithi145 ceph-mon[89359]: osd.3 marked itself dead as of e644 2024-09-21T10:08:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:43 smithi145 ceph-mon[89359]: osdmap e645: 8 total, 7 up, 8 in 2024-09-21T10:08:43.862 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:43 smithi119 podman[133433]: 2024-09-21 10:08:43.494202042 +0000 UTC m=+0.858416702 container remove 700faab8401d03c7358499dabc93ef77b9f606f5ffad24f7baa96b9780b72cdb (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T10:08:44.112 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[101432]: pgmap v608: 425 pgs: 22 peering, 54 stale+active+clean, 349 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 853 KiB/s wr, 1 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3695840681' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-24' is full (reached quota's max_objects: 6) 2024-09-21T10:08:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[101432]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:08:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[101432]: osdmap e646: 8 total, 7 up, 8 in 2024-09-21T10:08:44.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[105288]: pgmap v608: 425 pgs: 22 peering, 54 stale+active+clean, 349 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 853 KiB/s wr, 1 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:44.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:44.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3695840681' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:44.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-24' is full (reached quota's max_objects: 6) 2024-09-21T10:08:44.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[105288]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:08:44.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-mon[105288]: osdmap e646: 8 total, 7 up, 8 in 2024-09-21T10:08:44.118 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:43 smithi119 podman[133548]: 2024-09-21 10:08:43.763629518 +0000 UTC m=+0.016080033 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:44.119 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:43 smithi119 podman[133548]: 2024-09-21 10:08:43.869988317 +0000 UTC m=+0.122438819 container create 395e519f3f0c0c0159049c6b568b2a6d3e7911c7cff91ae32713454fb2fffa15 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T10:08:44.119 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:44 smithi119 podman[133548]: 2024-09-21 10:08:44.106007803 +0000 UTC m=+0.358458312 container init 395e519f3f0c0c0159049c6b568b2a6d3e7911c7cff91ae32713454fb2fffa15 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-21T10:08:44.119 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:44 smithi119 podman[133548]: 2024-09-21 10:08:44.11063764 +0000 UTC m=+0.363088141 container start 395e519f3f0c0c0159049c6b568b2a6d3e7911c7cff91ae32713454fb2fffa15 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=main-62da54b) 2024-09-21T10:08:44.183 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RemoveFullTryDataPool (24937 ms) 2024-09-21T10:08:44.183 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespace 2024-09-21T10:08:44.255 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:44.542 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:44 smithi119 podman[133548]: 2024-09-21 10:08:44.214750943 +0000 UTC m=+0.467201452 container attach 395e519f3f0c0c0159049c6b568b2a6d3e7911c7cff91ae32713454fb2fffa15 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T10:08:44.542 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:44 smithi119 podman[133548]: 2024-09-21 10:08:44.320866284 +0000 UTC m=+0.573316787 container died 395e519f3f0c0c0159049c6b568b2a6d3e7911c7cff91ae32713454fb2fffa15 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T10:08:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:44 smithi145 ceph-mon[89359]: pgmap v608: 425 pgs: 22 peering, 54 stale+active+clean, 349 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 853 KiB/s wr, 1 op/s; 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:44 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:08:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3695840681' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:44 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-24' is full (reached quota's max_objects: 6) 2024-09-21T10:08:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:44 smithi145 ceph-mon[89359]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:08:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:44 smithi145 ceph-mon[89359]: osdmap e646: 8 total, 7 up, 8 in 2024-09-21T10:08:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:08:44] "GET /metrics HTTP/1.1" 200 60624 "" "Prometheus/2.51.0" 2024-09-21T10:08:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[105288]: from='client.45431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[105288]: pgmap v611: 425 pgs: 94 active+undersized, 22 peering, 13 stale+active+clean, 27 active+undersized+degraded, 269 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 2.5 KiB/s wr, 4 op/s; 49/552 objects degraded (8.877%) 2024-09-21T10:08:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[105288]: from='client.35559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 49/552 objects degraded (8.877%), 27 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[105288]: osdmap e647: 8 total, 7 up, 8 in 2024-09-21T10:08:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2115629175' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[101432]: from='client.45431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[101432]: pgmap v611: 425 pgs: 94 active+undersized, 22 peering, 13 stale+active+clean, 27 active+undersized+degraded, 269 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 2.5 KiB/s wr, 4 op/s; 49/552 objects degraded (8.877%) 2024-09-21T10:08:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[101432]: from='client.35559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 49/552 objects degraded (8.877%), 27 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[101432]: osdmap e647: 8 total, 7 up, 8 in 2024-09-21T10:08:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2115629175' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:45.368 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 podman[133647]: 2024-09-21 10:08:45.049116833 +0000 UTC m=+0.717205744 container remove 395e519f3f0c0c0159049c6b568b2a6d3e7911c7cff91ae32713454fb2fffa15 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T10:08:45.369 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-21T10:08:45.369 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 systemd[1]: Stopped Ceph osd.3 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:08:45.369 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.3.service: Consumed 25.447s CPU time. 2024-09-21T10:08:45.369 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 systemd[1]: Starting Ceph osd.3 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:08:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:45 smithi145 ceph-mon[89359]: from='client.45431 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:45 smithi145 ceph-mon[89359]: pgmap v611: 425 pgs: 94 active+undersized, 22 peering, 13 stale+active+clean, 27 active+undersized+degraded, 269 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 2.5 KiB/s wr, 4 op/s; 49/552 objects degraded (8.877%) 2024-09-21T10:08:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:45 smithi145 ceph-mon[89359]: from='client.35559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:45 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 49/552 objects degraded (8.877%), 27 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:45 smithi145 ceph-mon[89359]: osdmap e647: 8 total, 7 up, 8 in 2024-09-21T10:08:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2115629175' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-26","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:08:45.759 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 podman[133763]: 2024-09-21 10:08:45.495244922 +0000 UTC m=+0.107449376 container create a9c790ca4d28f71a8cd41cf7bb5afd58f6c2aa82c9f9326e013b3e404638abb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T10:08:45.759 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 podman[133763]: 2024-09-21 10:08:45.40368573 +0000 UTC m=+0.015890195 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:46.012 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 podman[133763]: 2024-09-21 10:08:45.873056693 +0000 UTC m=+0.485261157 container init a9c790ca4d28f71a8cd41cf7bb5afd58f6c2aa82c9f9326e013b3e404638abb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T10:08:46.012 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 podman[133763]: 2024-09-21 10:08:45.878081101 +0000 UTC m=+0.490285552 container start a9c790ca4d28f71a8cd41cf7bb5afd58f6c2aa82c9f9326e013b3e404638abb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-62da54b) 2024-09-21T10:08:46.012 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:45 smithi119 podman[133763]: 2024-09-21 10:08:45.93110046 +0000 UTC m=+0.543304939 container attach a9c790ca4d28f71a8cd41cf7bb5afd58f6c2aa82c9f9326e013b3e404638abb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T10:08:46.127 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:08:46.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:46] ENGINE Bus STOPPING 2024-09-21T10:08:46.367 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:46.367 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:46.367 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:46.367 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T10:08:46.779 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:08:46.779 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:46] ENGINE Bus STOPPED 2024-09-21T10:08:46.779 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:46] ENGINE Bus STARTING 2024-09-21T10:08:46.779 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:46] ENGINE Serving on http://:::9283 2024-09-21T10:08:46.779 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:08:46] ENGINE Bus STARTED 2024-09-21T10:08:46.779 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:46.779 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:46.779 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:46.780 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:46.780 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:46.780 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T10:08:47.095 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate[133777]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T10:08:47.096 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 bash[133763]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-21T10:08:47.096 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:46 smithi119 podman[133763]: 2024-09-21 10:08:46.805376908 +0000 UTC m=+1.417581370 container died a9c790ca4d28f71a8cd41cf7bb5afd58f6c2aa82c9f9326e013b3e404638abb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:08:47.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2115629175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:47.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[101432]: osdmap e648: 8 total, 7 up, 8 in 2024-09-21T10:08:47.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:47.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:47.096 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/815671183' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:47.127 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2115629175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[105288]: osdmap e648: 8 total, 7 up, 8 in 2024-09-21T10:08:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/815671183' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2115629175' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:08:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:47 smithi145 ceph-mon[89359]: osdmap e648: 8 total, 7 up, 8 in 2024-09-21T10:08:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:08:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:08:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/815671183' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:47.764 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 podman[133957]: 2024-09-21 10:08:47.452567083 +0000 UTC m=+0.637761655 container remove a9c790ca4d28f71a8cd41cf7bb5afd58f6c2aa82c9f9326e013b3e404638abb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=main-62da54b, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T10:08:47.765 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 podman[134071]: 2024-09-21 10:08:47.645058461 +0000 UTC m=+0.101366650 container create 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T10:08:47.765 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 podman[134071]: 2024-09-21 10:08:47.559405797 +0000 UTC m=+0.015714000 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:48.118 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 podman[134071]: 2024-09-21 10:08:47.839260632 +0000 UTC m=+0.295568833 container init 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b) 2024-09-21T10:08:48.118 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 podman[134071]: 2024-09-21 10:08:47.844240255 +0000 UTC m=+0.300548445 container start 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:08:48.118 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 ceph-osd[134109]: -- 172.21.15.119:0/3274072881 <== mon.2 v2:172.21.15.119:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562247fe3a40 con 0x562248db2000 2024-09-21T10:08:48.118 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 bash[134071]: 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 2024-09-21T10:08:48.118 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:47 smithi119 systemd[1]: Started Ceph osd.3 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: pgmap v614: 457 pgs: 6 creating+peering, 26 unknown, 118 active+undersized, 22 peering, 40 active+undersized+degraded, 245 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s wr, 4 op/s; 66/552 objects degraded (11.957%) 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: osdmap e649: 8 total, 7 up, 8 in 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: pgmap v614: 457 pgs: 6 creating+peering, 26 unknown, 118 active+undersized, 22 peering, 40 active+undersized+degraded, 245 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s wr, 4 op/s; 66/552 objects degraded (11.957%) 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: osdmap e649: 8 total, 7 up, 8 in 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: pgmap v614: 457 pgs: 6 creating+peering, 26 unknown, 118 active+undersized, 22 peering, 40 active+undersized+degraded, 245 active+clean; 11 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.0 KiB/s wr, 4 op/s; 66/552 objects degraded (11.957%) 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: osdmap e649: 8 total, 7 up, 8 in 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:49.388 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:49 smithi145 ceph-mon[89359]: from='client.45455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:49.388 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:49 smithi145 ceph-mon[89359]: from='client.45461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:49.388 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:49 smithi145 ceph-mon[89359]: osdmap e650: 8 total, 7 up, 8 in 2024-09-21T10:08:49.388 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:49.388 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:49 smithi145 ceph-mon[89359]: pgmap v617: 457 pgs: 6 creating+peering, 4 unknown, 144 active+undersized, 44 active+undersized+degraded, 259 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s wr, 7 op/s; 72/555 objects degraded (12.973%) 2024-09-21T10:08:49.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[105288]: from='client.45455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:49.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[105288]: from='client.45461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:49.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[105288]: osdmap e650: 8 total, 7 up, 8 in 2024-09-21T10:08:49.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:49.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[105288]: pgmap v617: 457 pgs: 6 creating+peering, 4 unknown, 144 active+undersized, 44 active+undersized+degraded, 259 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s wr, 7 op/s; 72/555 objects degraded (12.973%) 2024-09-21T10:08:49.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[101432]: from='client.45455 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:49.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[101432]: from='client.45461 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:49.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[101432]: osdmap e650: 8 total, 7 up, 8 in 2024-09-21T10:08:49.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-26", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:08:49.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-mon[101432]: pgmap v617: 457 pgs: 6 creating+peering, 4 unknown, 144 active+undersized, 44 active+undersized+degraded, 259 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.5 KiB/s wr, 7 op/s; 72/555 objects degraded (12.973%) 2024-09-21T10:08:49.824 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:49 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:08:49.478+0000 7f5503617640 -1 Falling back to public interface 2024-09-21T10:08:49.990 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:50.409 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:50 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-21T10:08:50.410 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:50.410 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:50 smithi145 ceph-mon[89359]: osdmap e651: 8 total, 7 up, 8 in 2024-09-21T10:08:50.410 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/797272883' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:50.410 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:50 smithi145 ceph-mon[89359]: from='client.45476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:50.410 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:08:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:08:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:08:50.410 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[101432]: osdmap e651: 8 total, 7 up, 8 in 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/797272883' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[101432]: from='client.45476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-26", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[105288]: osdmap e651: 8 total, 7 up, 8 in 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/797272883' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:50.411 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:50 smithi119 ceph-mon[105288]: from='client.45476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:51.407 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:51 smithi119 ceph-mon[101432]: from='client.45482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:51.407 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:51 smithi119 ceph-mon[101432]: pgmap v619: 457 pgs: 147 active+undersized, 45 active+undersized+degraded, 265 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 3.6 KiB/s wr, 27 op/s; 73/558 objects degraded (13.082%) 2024-09-21T10:08:51.407 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:51 smithi119 ceph-mon[105288]: from='client.45482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:51.407 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:51 smithi119 ceph-mon[105288]: pgmap v619: 457 pgs: 147 active+undersized, 45 active+undersized+degraded, 265 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 3.6 KiB/s wr, 27 op/s; 73/558 objects degraded (13.082%) 2024-09-21T10:08:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:51 smithi145 ceph-mon[89359]: from='client.45482 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:51 smithi145 ceph-mon[89359]: pgmap v619: 457 pgs: 147 active+undersized, 45 active+undersized+degraded, 265 active+clean; 584 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 3.6 KiB/s wr, 27 op/s; 73/558 objects degraded (13.082%) 2024-09-21T10:08:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:52 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 73/558 objects degraded (13.082%), 45 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:52.565 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 73/558 objects degraded (13.082%), 45 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:52.565 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:52.565 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:52.565 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 73/558 objects degraded (13.082%), 45 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:52.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:52.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:52.932 INFO:teuthology.orchestra.run.smithi119.stdout:true 2024-09-21T10:08:53.175 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:52 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:08:52.987+0000 7f5503617640 -1 osd.3 644 log_to_monitors true 2024-09-21T10:08:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: pgmap v620: 457 pgs: 147 active+undersized, 45 active+undersized+degraded, 265 active+clean; 4.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 685 KiB/s wr, 20 op/s; 73/561 objects degraded (13.012%) 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3504917403' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='client.45494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:53.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:53 smithi145 ceph-mon[89359]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:08:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: pgmap v620: 457 pgs: 147 active+undersized, 45 active+undersized+degraded, 265 active+clean; 4.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 685 KiB/s wr, 20 op/s; 73/561 objects degraded (13.012%) 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3504917403' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='client.45494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[101432]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: pgmap v620: 457 pgs: 147 active+undersized, 45 active+undersized+degraded, 265 active+clean; 4.6 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 685 KiB/s wr, 20 op/s; 73/561 objects degraded (13.012%) 2024-09-21T10:08:53.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3504917403' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:53.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='client.45494 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:53.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-mon[105288]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-21T10:08:53.968 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:08:53 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:08:53.866+0000 7f54f73b9640 -1 osd.3 644 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:08:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: from='client.45503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:08:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: osdmap e652: 8 total, 7 up, 8 in 2024-09-21T10:08:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-25' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352] boot 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: osdmap e653: 8 total, 8 up, 8 in 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: from='client.45503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:54.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: osdmap e652: 8 total, 7 up, 8 in 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-25' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352] boot 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: osdmap e653: 8 total, 8 up, 8 in 2024-09-21T10:08:54.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:08:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: from='client.45503 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T10:08:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: osdmap e652: 8 total, 7 up, 8 in 2024-09-21T10:08:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]} : dispatch 2024-09-21T10:08:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: from='osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352]' entity='osd.3' 2024-09-21T10:08:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-25' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:08:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:08:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:08:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: osd.3 [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352] boot 2024-09-21T10:08:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: osdmap e653: 8 total, 8 up, 8 in 2024-09-21T10:08:54.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-21T10:08:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:08:54] "GET /metrics HTTP/1.1" 200 63162 "" "Prometheus/2.51.0" 2024-09-21T10:08:55.526 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T10:08:55.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:08:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:08:55.601+0000 7fc0be6c7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (50 PGs are or would become offline) 2024-09-21T10:08:56.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: pgmap v623: 457 pgs: 15 peering, 136 active+undersized, 41 active+undersized+degraded, 265 active+clean; 5.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.3 KiB/s rd, 854 KiB/s wr, 11 op/s; 64/564 objects degraded (11.348%) 2024-09-21T10:08:56.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: osdmap e654: 8 total, 8 up, 8 in 2024-09-21T10:08:56.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:56.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1423627804' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: pgmap v623: 457 pgs: 15 peering, 136 active+undersized, 41 active+undersized+degraded, 265 active+clean; 5.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.3 KiB/s rd, 854 KiB/s wr, 11 op/s; 64/564 objects degraded (11.348%) 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: osdmap e654: 8 total, 8 up, 8 in 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.240 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1423627804' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T10:08:56.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch upgrade pause' 2024-09-21T10:08:56.518 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: pgmap v623: 457 pgs: 15 peering, 136 active+undersized, 41 active+undersized+degraded, 265 active+clean; 5.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.3 KiB/s rd, 854 KiB/s wr, 11 op/s; 64/564 objects degraded (11.348%) 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: osdmap e654: 8 total, 8 up, 8 in 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1423627804' entity='client.admin' cmd={"prefix": "osd versions"} : dispatch 2024-09-21T10:08:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:08:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:08:56.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16} : dispatch 2024-09-21T10:08:57.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:57 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T10:08:57.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:57 smithi119 ceph-mon[105288]: Upgrade: unsafe to stop osd(s) at this time (50 PGs are or would become offline) 2024-09-21T10:08:57.363 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:57 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T10:08:57.363 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:57 smithi119 ceph-mon[101432]: Upgrade: unsafe to stop osd(s) at this time (50 PGs are or would become offline) 2024-09-21T10:08:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:57 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-21T10:08:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:57 smithi145 ceph-mon[89359]: Upgrade: unsafe to stop osd(s) at this time (50 PGs are or would become offline) 2024-09-21T10:08:58.292 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:58 smithi119 ceph-mon[101432]: pgmap v625: 457 pgs: 15 peering, 122 active+undersized, 30 active+undersized+degraded, 290 active+clean; 5.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 854 KiB/s wr, 4 op/s; 50/564 objects degraded (8.865%); 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:58.292 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:58 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 50/564 objects degraded (8.865%), 30 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:58.292 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:58 smithi119 ceph-mon[105288]: pgmap v625: 457 pgs: 15 peering, 122 active+undersized, 30 active+undersized+degraded, 290 active+clean; 5.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 854 KiB/s wr, 4 op/s; 50/564 objects degraded (8.865%); 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:58.292 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:58 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 50/564 objects degraded (8.865%), 30 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:58 smithi145 ceph-mon[89359]: pgmap v625: 457 pgs: 15 peering, 122 active+undersized, 30 active+undersized+degraded, 290 active+clean; 5.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 854 KiB/s wr, 4 op/s; 50/564 objects degraded (8.865%); 3 B/s, 0 keys/s, 0 objects/s recovering 2024-09-21T10:08:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:58 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 50/564 objects degraded (8.865%), 30 pgs degraded (PG_DEGRADED) 2024-09-21T10:08:58.585 INFO:teuthology.orchestra.run.smithi119.stdout:Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:59.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:08:59.524 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:08:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[101432]: pgmap v626: 457 pgs: 15 peering, 442 active+clean; 7.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 513 KiB/s wr, 5 op/s; 12 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[101432]: from='client.35631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[101432]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/564 objects degraded (8.865%), 30 pgs degraded) 2024-09-21T10:08:59.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[105288]: pgmap v626: 457 pgs: 15 peering, 442 active+clean; 7.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 513 KiB/s wr, 5 op/s; 12 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:08:59.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[105288]: from='client.35631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:08:59.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[105288]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:08:59.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:08:59.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:08:59 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/564 objects degraded (8.865%), 30 pgs degraded) 2024-09-21T10:09:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:59 smithi145 ceph-mon[89359]: pgmap v626: 457 pgs: 15 peering, 442 active+clean; 7.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.8 KiB/s rd, 513 KiB/s wr, 5 op/s; 12 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:09:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:59 smithi145 ceph-mon[89359]: from='client.35631 -' entity='client.admin' cmd=[{"prefix": "orch upgrade pause", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:59 smithi145 ceph-mon[89359]: Upgrade: Paused upgrade to quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:09:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:08:59 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 50/564 objects degraded (8.865%), 30 pgs degraded) 2024-09-21T10:09:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:09:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:09:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:alertmanager.a smithi119 *:9093,9094 running (8m) 9s ago 13m 18.9M - 0.27.0 11f11916f8cd e4cf1f6d8355 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:grafana.a smithi145 *:3000 running (8m) 8s ago 12m 81.7M - 9.4.7 954c08fa6188 e033bb16a7cd 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi119.oikglq smithi119 running (11m) 9s ago 11m 20.7M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4ff2238155bf 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi145.tjomfr smithi145 running (11m) 8s ago 11m 16.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b1338cf49871 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi145 *:8443,9283,8765 running (7m) 8s ago 17m 458M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 1346375dd747 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi119 *:8443,9283,8765 running (8m) 9s ago 18m 532M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d7bcb26ddb62 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (7m) 9s ago 18m 69.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 030d68226d6c 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi145 running (4m) 8s ago 17m 46.6M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 550baecb68e0 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi119 running (6m) 9s ago 17m 58.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d747bb0bb662 2024-09-21T10:09:01.365 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.a smithi119 *:9100 running (8m) 9s ago 12m 9.81M - 1.7.0 72c9c2088986 05912f78c070 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.b smithi145 *:9100 running (8m) 8s ago 12m 11.0M - 1.7.0 72c9c2088986 d906bafb8d86 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (3m) 9s ago 16m 75.6M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 55bebfd490f2 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (2m) 9s ago 16m 74.7M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e679cf598e41 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (63s) 9s ago 15m 64.4M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 3019e7bd2405 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (13s) 9s ago 15m 14.4M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 52358544d24e 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi145 running (15m) 8s ago 15m 126M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 18086eb220ec 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi145 running (14m) 8s ago 14m 114M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 90ab6d4f8571 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi145 running (14m) 8s ago 14m 129M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 772774e9be4a 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi145 running (13m) 8s ago 13m 122M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1a18c8e91654 2024-09-21T10:09:01.366 INFO:teuthology.orchestra.run.smithi119.stdout:prometheus.a smithi145 *:9095 running (7m) 8s ago 13m 58.6M - 2.51.0 1d3b7f56885b d2de6cb75d75 2024-09-21T10:09:01.587 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:01 smithi119 ceph-mon[101432]: pgmap v627: 457 pgs: 457 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 863 KiB/s wr, 5 op/s; 10 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:09:01.850 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:01 smithi119 ceph-mon[105288]: pgmap v627: 457 pgs: 457 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 863 KiB/s wr, 5 op/s; 10 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:09:02.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph orch ps' 2024-09-21T10:09:02.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:01 smithi145 ceph-mon[89359]: pgmap v627: 457 pgs: 457 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 863 KiB/s wr, 5 op/s; 10 B/s, 2 keys/s, 0 objects/s recovering 2024-09-21T10:09:02.311 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:02.706 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:02 smithi119 ceph-mon[101432]: from='client.35637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:02.707 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:02 smithi119 ceph-mon[105288]: from='client.35637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:02 smithi145 ceph-mon[89359]: from='client.35637 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:03.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:03 smithi119 ceph-mon[101432]: pgmap v628: 457 pgs: 457 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 605 B/s rd, 727 KiB/s wr, 1 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:09:03.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:03.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:03.794 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:03 smithi119 ceph-mon[105288]: pgmap v628: 457 pgs: 457 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 605 B/s rd, 727 KiB/s wr, 1 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:09:03.794 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:03.794 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:03 smithi145 ceph-mon[89359]: pgmap v628: 457 pgs: 457 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 605 B/s rd, 727 KiB/s wr, 1 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:09:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:alertmanager.a smithi119 *:9093,9094 running (8m) 12s ago 13m 18.9M - 0.27.0 11f11916f8cd e4cf1f6d8355 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:grafana.a smithi145 *:3000 running (8m) 11s ago 12m 81.7M - 9.4.7 954c08fa6188 e033bb16a7cd 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi119.oikglq smithi119 running (11m) 12s ago 11m 20.7M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 4ff2238155bf 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:mds.foo.smithi145.tjomfr smithi145 running (11m) 11s ago 11m 16.5M - 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be b1338cf49871 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.x smithi145 *:8443,9283,8765 running (7m) 11s ago 17m 458M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 1346375dd747 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:mgr.y smithi119 *:8443,9283,8765 running (8m) 12s ago 18m 532M - 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d7bcb26ddb62 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a smithi119 running (7m) 12s ago 18m 69.2M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 030d68226d6c 2024-09-21T10:09:04.192 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b smithi145 running (4m) 11s ago 17m 46.6M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 550baecb68e0 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c smithi119 running (6m) 12s ago 17m 58.1M 2048M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 d747bb0bb662 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.a smithi119 *:9100 running (8m) 12s ago 12m 9.81M - 1.7.0 72c9c2088986 05912f78c070 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:node-exporter.b smithi145 *:9100 running (8m) 11s ago 12m 11.0M - 1.7.0 72c9c2088986 d906bafb8d86 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 smithi119 running (3m) 12s ago 16m 75.6M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 55bebfd490f2 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 smithi119 running (2m) 12s ago 16m 74.7M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 e679cf598e41 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 smithi119 running (65s) 12s ago 15m 64.4M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 3019e7bd2405 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 smithi119 running (16s) 12s ago 15m 14.4M 2480M 19.3.0-5092-g62da54b4 a8ebb3d62dd1 52358544d24e 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 smithi145 running (15m) 11s ago 15m 126M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 18086eb220ec 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 smithi145 running (14m) 11s ago 14m 114M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 90ab6d4f8571 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 smithi145 running (14m) 11s ago 14m 129M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 772774e9be4a 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 smithi145 running (13m) 11s ago 13m 122M 2480M 17.2.7-1649-gfbd1a6ab fc3ec4a4f6be 1a18c8e91654 2024-09-21T10:09:04.193 INFO:teuthology.orchestra.run.smithi119.stdout:prometheus.a smithi145 *:9095 running (8m) 11s ago 13m 58.6M - 2.51.0 1d3b7f56885b d2de6cb75d75 2024-09-21T10:09:04.285 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespace (20102 ms) 2024-09-21T10:09:04.286 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RemoveFullTryNamespaceDataPool 2024-09-21T10:09:04.836 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-26' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:09:04.836 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-mon[101432]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:04.837 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-mon[101432]: osdmap e655: 8 total, 8 up, 8 in 2024-09-21T10:09:04.837 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:09:04] "GET /metrics HTTP/1.1" 200 63200 "" "Prometheus/2.51.0" 2024-09-21T10:09:04.837 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-26' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:09:04.837 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-mon[105288]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:04.837 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:04 smithi119 ceph-mon[105288]: osdmap e655: 8 total, 8 up, 8 in 2024-09-21T10:09:04.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -e sha1=62da54b4b83a96ca9976b7e1d0c8272a564a2208 -- bash -c 'ceph versions' 2024-09-21T10:09:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:04 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-26' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:09:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:04 smithi145 ceph-mon[89359]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:04 smithi145 ceph-mon[89359]: osdmap e655: 8 total, 8 up, 8 in 2024-09-21T10:09:05.119 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:09:06.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[101432]: from='client.35643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[101432]: pgmap v630: 457 pgs: 457 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 MiB/s wr, 2 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:09:06.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[101432]: osdmap e656: 8 total, 8 up, 8 in 2024-09-21T10:09:06.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2353820414' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:06.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[105288]: from='client.35643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[105288]: pgmap v630: 457 pgs: 457 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 MiB/s wr, 2 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:09:06.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[105288]: osdmap e656: 8 total, 8 up, 8 in 2024-09-21T10:09:06.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2353820414' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:06 smithi145 ceph-mon[89359]: from='client.35643 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:09:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:06 smithi145 ceph-mon[89359]: pgmap v630: 457 pgs: 457 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 MiB/s wr, 2 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-09-21T10:09:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:06 smithi145 ceph-mon[89359]: osdmap e656: 8 total, 8 up, 8 in 2024-09-21T10:09:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2353820414' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-27","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": { 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 3 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": { 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 2 2024-09-21T10:09:07.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": { 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 4, 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 4 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": { 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 2 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "overall": { 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 17.2.7-1649-gfbd1a6ab (fbd1a6ab0448f011c60f6376195d5c078b0cf20b) quincy (stable)": 6, 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph version 19.3.0-5092-g62da54b4 (62da54b4b83a96ca9976b7e1d0c8272a564a2208) squid (dev)": 9 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:09:07.109 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:09:07.110 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2353820414' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:07.110 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:07 smithi119 ceph-mon[101432]: osdmap e657: 8 total, 8 up, 8 in 2024-09-21T10:09:07.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2353820414' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:07.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:07 smithi119 ceph-mon[105288]: osdmap e657: 8 total, 8 up, 8 in 2024-09-21T10:09:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2353820414' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:07 smithi145 ceph-mon[89359]: osdmap e657: 8 total, 8 up, 8 in 2024-09-21T10:09:07.800 DEBUG:teuthology.parallel:result is None 2024-09-21T10:09:08.122 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:08.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:09:08] ENGINE Bus STOPPING 2024-09-21T10:09:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[105288]: pgmap v633: 489 pgs: 5 creating+peering, 27 unknown, 457 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 853 KiB/s wr, 2 op/s 2024-09-21T10:09:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[105288]: osdmap e658: 8 total, 8 up, 8 in 2024-09-21T10:09:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3706575139' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4219484153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[101432]: pgmap v633: 489 pgs: 5 creating+peering, 27 unknown, 457 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 853 KiB/s wr, 2 op/s 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[101432]: osdmap e658: 8 total, 8 up, 8 in 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3706575139' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4219484153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:08.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:08 smithi145 ceph-mon[89359]: pgmap v633: 489 pgs: 5 creating+peering, 27 unknown, 457 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 853 KiB/s wr, 2 op/s 2024-09-21T10:09:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:08 smithi145 ceph-mon[89359]: osdmap e658: 8 total, 8 up, 8 in 2024-09-21T10:09:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:08 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3706575139' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-28","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:09:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:08 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4219484153' entity='client.admin' cmd={"prefix": "versions"} : dispatch 2024-09-21T10:09:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:08 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:08.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:09:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:09:08.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:09:08] ENGINE Bus STOPPED 2024-09-21T10:09:08.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:09:08] ENGINE Bus STARTING 2024-09-21T10:09:08.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:09:08] ENGINE Serving on http://:::9283 2024-09-21T10:09:08.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:09:08] ENGINE Bus STARTED 2024-09-21T10:09:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3706575139' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[101432]: osdmap e659: 8 total, 8 up, 8 in 2024-09-21T10:09:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:09:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:09:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3706575139' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[105288]: osdmap e659: 8 total, 8 up, 8 in 2024-09-21T10:09:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:09:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:09:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3706575139' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:09:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:09 smithi145 ceph-mon[89359]: osdmap e659: 8 total, 8 up, 8 in 2024-09-21T10:09:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:09:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:09:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:10 smithi119 ceph-mon[101432]: pgmap v636: 521 pgs: 8 creating+peering, 12 unknown, 501 active+clean; 4.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-09-21T10:09:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:10 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:09:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:10 smithi119 ceph-mon[101432]: osdmap e660: 8 total, 8 up, 8 in 2024-09-21T10:09:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:10 smithi119 ceph-mon[105288]: pgmap v636: 521 pgs: 8 creating+peering, 12 unknown, 501 active+clean; 4.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-09-21T10:09:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:10 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:09:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:10 smithi119 ceph-mon[105288]: osdmap e660: 8 total, 8 up, 8 in 2024-09-21T10:09:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:10 smithi145 ceph-mon[89359]: pgmap v636: 521 pgs: 8 creating+peering, 12 unknown, 501 active+clean; 4.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.2 KiB/s wr, 39 op/s 2024-09-21T10:09:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:10 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:09:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:10 smithi145 ceph-mon[89359]: osdmap e660: 8 total, 8 up, 8 in 2024-09-21T10:09:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:09:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:09:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: osdmap e661: 8 total, 8 up, 8 in 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: pgmap v639: 521 pgs: 3 creating+peering, 518 active+clean; 584 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 KiB/s wr, 40 op/s 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: osdmap e661: 8 total, 8 up, 8 in 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: pgmap v639: 521 pgs: 3 creating+peering, 518 active+clean; 584 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 KiB/s wr, 40 op/s 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: osdmap e661: 8 total, 8 up, 8 in 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-28", "field": "max_bytes", "val": "10485760"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: pgmap v639: 521 pgs: 3 creating+peering, 518 active+clean; 584 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 KiB/s wr, 40 op/s 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:11.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:09:11.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:09:11.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:09:11.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:09:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:12 smithi119 ceph-mon[101432]: osdmap e662: 8 total, 8 up, 8 in 2024-09-21T10:09:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:09:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:12 smithi119 ceph-mon[105288]: osdmap e662: 8 total, 8 up, 8 in 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-28", "field": "max_bytes", "val": "10485760"}]': finished 2024-09-21T10:09:12.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:12 smithi145 ceph-mon[89359]: osdmap e662: 8 total, 8 up, 8 in 2024-09-21T10:09:13.457 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:13 smithi145 ceph-mon[89359]: pgmap v641: 521 pgs: 521 active+clean; 584 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 KiB/s wr, 35 op/s 2024-09-21T10:09:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:13 smithi119 ceph-mon[101432]: pgmap v641: 521 pgs: 521 active+clean; 584 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 KiB/s wr, 35 op/s 2024-09-21T10:09:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:13 smithi119 ceph-mon[105288]: pgmap v641: 521 pgs: 521 active+clean; 584 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 KiB/s wr, 35 op/s 2024-09-21T10:09:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-26' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:09:14.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-mon[105288]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:09:14.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-mon[105288]: osdmap e663: 8 total, 8 up, 8 in 2024-09-21T10:09:14.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-26' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:09:14.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-mon[101432]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:09:14.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-mon[101432]: osdmap e663: 8 total, 8 up, 8 in 2024-09-21T10:09:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:14 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-26' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:09:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:14 smithi145 ceph-mon[89359]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:09:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:14 smithi145 ceph-mon[89359]: osdmap e663: 8 total, 8 up, 8 in 2024-09-21T10:09:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:09:14] "GET /metrics HTTP/1.1" 200 63200 "" "Prometheus/2.51.0" 2024-09-21T10:09:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:15 smithi145 ceph-mon[89359]: pgmap v643: 521 pgs: 521 active+clean; 3.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 569 KiB/s wr, 4 op/s 2024-09-21T10:09:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:15 smithi119 ceph-mon[101432]: pgmap v643: 521 pgs: 521 active+clean; 3.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 569 KiB/s wr, 4 op/s 2024-09-21T10:09:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:15 smithi119 ceph-mon[105288]: pgmap v643: 521 pgs: 521 active+clean; 3.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 569 KiB/s wr, 4 op/s 2024-09-21T10:09:17.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:17 smithi119 ceph-mon[101432]: pgmap v644: 521 pgs: 521 active+clean; 3.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 482 KiB/s wr, 9 op/s 2024-09-21T10:09:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:17 smithi119 ceph-mon[105288]: pgmap v644: 521 pgs: 521 active+clean; 3.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 482 KiB/s wr, 9 op/s 2024-09-21T10:09:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:17 smithi145 ceph-mon[89359]: pgmap v644: 521 pgs: 521 active+clean; 3.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.3 KiB/s rd, 482 KiB/s wr, 9 op/s 2024-09-21T10:09:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:18.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:18.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:20 smithi119 ceph-mon[101432]: pgmap v645: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-21T10:09:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:20 smithi119 ceph-mon[101432]: osdmap e664: 8 total, 8 up, 8 in 2024-09-21T10:09:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:20 smithi119 ceph-mon[105288]: pgmap v645: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-21T10:09:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:20 smithi119 ceph-mon[105288]: osdmap e664: 8 total, 8 up, 8 in 2024-09-21T10:09:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:09:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:09:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:20 smithi145 ceph-mon[89359]: pgmap v645: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.1 KiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-21T10:09:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:20 smithi145 ceph-mon[89359]: osdmap e664: 8 total, 8 up, 8 in 2024-09-21T10:09:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:22 smithi119 ceph-mon[101432]: pgmap v647: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2024-09-21T10:09:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:22 smithi119 ceph-mon[105288]: pgmap v647: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2024-09-21T10:09:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:22 smithi145 ceph-mon[89359]: pgmap v647: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 1.0 MiB/s wr, 8 op/s 2024-09-21T10:09:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:24 smithi119 ceph-mon[101432]: pgmap v648: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 606 KiB/s wr, 7 op/s 2024-09-21T10:09:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:24 smithi119 ceph-mon[105288]: pgmap v648: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 606 KiB/s wr, 7 op/s 2024-09-21T10:09:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:24 smithi145 ceph-mon[89359]: pgmap v648: 521 pgs: 521 active+clean; 8.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.4 KiB/s rd, 606 KiB/s wr, 7 op/s 2024-09-21T10:09:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:09:24] "GET /metrics HTTP/1.1" 200 67996 "" "Prometheus/2.51.0" 2024-09-21T10:09:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:26 smithi119 ceph-mon[101432]: pgmap v649: 521 pgs: 521 active+clean; 14 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-21T10:09:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:26 smithi119 ceph-mon[105288]: pgmap v649: 521 pgs: 521 active+clean; 14 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-21T10:09:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:26 smithi145 ceph-mon[89359]: pgmap v649: 521 pgs: 521 active+clean; 14 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.0 MiB/s wr, 6 op/s 2024-09-21T10:09:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:28 smithi119 ceph-mon[101432]: pgmap v650: 521 pgs: 521 active+clean; 14 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1024 KiB/s wr, 3 op/s 2024-09-21T10:09:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:28 smithi119 ceph-mon[105288]: pgmap v650: 521 pgs: 521 active+clean; 14 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1024 KiB/s wr, 3 op/s 2024-09-21T10:09:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:28 smithi145 ceph-mon[89359]: pgmap v650: 521 pgs: 521 active+clean; 14 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1024 KiB/s wr, 3 op/s 2024-09-21T10:09:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:29 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-28' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:09:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:29 smithi119 ceph-mon[101432]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:29 smithi119 ceph-mon[101432]: osdmap e665: 8 total, 8 up, 8 in 2024-09-21T10:09:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:29 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-28' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:09:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:29 smithi119 ceph-mon[105288]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:29.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:29 smithi119 ceph-mon[105288]: osdmap e665: 8 total, 8 up, 8 in 2024-09-21T10:09:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:29 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-28' is full (reached quota's max_bytes: 10 MiB) 2024-09-21T10:09:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:29 smithi145 ceph-mon[89359]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:29.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:29 smithi145 ceph-mon[89359]: osdmap e665: 8 total, 8 up, 8 in 2024-09-21T10:09:30.089 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:30 smithi119 ceph-mon[105288]: pgmap v651: 521 pgs: 521 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 819 KiB/s wr, 3 op/s 2024-09-21T10:09:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-27", "field": "max_objects", "val": "7"} : dispatch 2024-09-21T10:09:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:30 smithi119 ceph-mon[101432]: pgmap v651: 521 pgs: 521 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 819 KiB/s wr, 3 op/s 2024-09-21T10:09:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-27", "field": "max_objects", "val": "7"} : dispatch 2024-09-21T10:09:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:30 smithi145 ceph-mon[89359]: pgmap v651: 521 pgs: 521 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 819 KiB/s wr, 3 op/s 2024-09-21T10:09:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-27", "field": "max_objects", "val": "7"} : dispatch 2024-09-21T10:09:30.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:09:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:09:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:31.116 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:31 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-27", "field": "max_objects", "val": "7"}]': finished 2024-09-21T10:09:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:31 smithi119 ceph-mon[101432]: osdmap e666: 8 total, 8 up, 8 in 2024-09-21T10:09:31.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:31 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-27", "field": "max_objects", "val": "7"}]': finished 2024-09-21T10:09:31.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:31 smithi119 ceph-mon[105288]: osdmap e666: 8 total, 8 up, 8 in 2024-09-21T10:09:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:31 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "test-librbd-smithi119-84733-27", "field": "max_objects", "val": "7"}]': finished 2024-09-21T10:09:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:31 smithi145 ceph-mon[89359]: osdmap e666: 8 total, 8 up, 8 in 2024-09-21T10:09:32.127 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:32 smithi119 ceph-mon[101432]: pgmap v654: 521 pgs: 521 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 4 op/s 2024-09-21T10:09:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:32 smithi119 ceph-mon[105288]: pgmap v654: 521 pgs: 521 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 4 op/s 2024-09-21T10:09:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:32 smithi145 ceph-mon[89359]: pgmap v654: 521 pgs: 521 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 4 op/s 2024-09-21T10:09:33.135 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:34.145 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:34.208 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RemoveFullTryNamespaceDataPool (29923 ms) 2024-09-21T10:09:34.209 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestIO 2024-09-21T10:09:34.209 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: pgmap v655: 521 pgs: 521 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 640 KiB/s wr, 4 op/s 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]} : dispatch 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.c", "id": [6, 0]} : dispatch 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-27' is full (reached quota's max_objects: 7) 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: Health check update: 3 pool(s) full (POOL_FULL) 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]}]': finished 2024-09-21T10:09:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.c", "id": [6, 0]}]': finished 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[101432]: osdmap e667: 8 total, 8 up, 8 in 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: pgmap v655: 521 pgs: 521 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 640 KiB/s wr, 4 op/s 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]} : dispatch 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.c", "id": [6, 0]} : dispatch 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-27' is full (reached quota's max_objects: 7) 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: Health check update: 3 pool(s) full (POOL_FULL) 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]}]': finished 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.c", "id": [6, 0]}]': finished 2024-09-21T10:09:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-mon[105288]: osdmap e667: 8 total, 8 up, 8 in 2024-09-21T10:09:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: pgmap v655: 521 pgs: 521 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 640 KiB/s wr, 4 op/s 2024-09-21T10:09:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]} : dispatch 2024-09-21T10:09:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.c", "id": [6, 0]} : dispatch 2024-09-21T10:09:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-27' is full (reached quota's max_objects: 7) 2024-09-21T10:09:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: Health check update: 3 pool(s) full (POOL_FULL) 2024-09-21T10:09:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "22.16", "id": [7, 0]}]': finished 2024-09-21T10:09:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "29.c", "id": [6, 0]}]': finished 2024-09-21T10:09:34.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:34 smithi145 ceph-mon[89359]: osdmap e667: 8 total, 8 up, 8 in 2024-09-21T10:09:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:09:34] "GET /metrics HTTP/1.1" 200 67998 "" "Prometheus/2.51.0" 2024-09-21T10:09:35.751 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.752 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:35.753 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:35.754 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.755 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.756 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:35.757 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:35.758 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 512~512 = 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 1536~512 = 0 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.759 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.760 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:35.761 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.762 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.763 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.764 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.765 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.766 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.767 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:35.768 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:35.769 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:35.769 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:35.769 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:35.769 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:35.769 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:35.769 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:36 smithi119 ceph-mon[105288]: pgmap v657: 521 pgs: 1 peering, 520 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 344 KiB/s wr, 9 op/s 2024-09-21T10:09:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:36 smithi119 ceph-mon[105288]: osdmap e668: 8 total, 8 up, 8 in 2024-09-21T10:09:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:36 smithi119 ceph-mon[101432]: pgmap v657: 521 pgs: 1 peering, 520 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 344 KiB/s wr, 9 op/s 2024-09-21T10:09:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:36 smithi119 ceph-mon[101432]: osdmap e668: 8 total, 8 up, 8 in 2024-09-21T10:09:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:36 smithi145 ceph-mon[89359]: pgmap v657: 521 pgs: 1 peering, 520 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.0 KiB/s rd, 344 KiB/s wr, 9 op/s 2024-09-21T10:09:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:36 smithi145 ceph-mon[89359]: osdmap e668: 8 total, 8 up, 8 in 2024-09-21T10:09:37.042 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.043 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.044 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.045 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.046 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.047 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.048 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.049 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.050 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:37.051 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestIO (2834 ms) 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS1 2024-09-21T10:09:37.052 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:38 smithi119 ceph-mon[105288]: pgmap v659: 521 pgs: 1 peering, 520 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 239 KiB/s rd, 136 KiB/s wr, 454 op/s 2024-09-21T10:09:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:38 smithi119 ceph-mon[101432]: pgmap v659: 521 pgs: 1 peering, 520 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 239 KiB/s rd, 136 KiB/s wr, 454 op/s 2024-09-21T10:09:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:38 smithi145 ceph-mon[89359]: pgmap v659: 521 pgs: 1 peering, 520 active+clean; 19 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 239 KiB/s rd, 136 KiB/s wr, 454 op/s 2024-09-21T10:09:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:40 smithi119 ceph-mon[105288]: pgmap v660: 521 pgs: 1 peering, 520 active+clean; 2.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 199 KiB/s rd, 111 KiB/s wr, 376 op/s 2024-09-21T10:09:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:40 smithi119 ceph-mon[101432]: pgmap v660: 521 pgs: 1 peering, 520 active+clean; 2.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 199 KiB/s rd, 111 KiB/s wr, 376 op/s 2024-09-21T10:09:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:40 smithi145 ceph-mon[89359]: pgmap v660: 521 pgs: 1 peering, 520 active+clean; 2.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 199 KiB/s rd, 111 KiB/s wr, 376 op/s 2024-09-21T10:09:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:09:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:09:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:42.290 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:42.290 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:42.290 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:42.290 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:42.291 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.292 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:42.294 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.295 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 512~512 = 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 1536~512 = 0 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:42.297 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.298 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.299 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.301 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.302 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.303 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.304 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:42.306 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:42.306 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:42.306 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:42.306 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:42.346 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:42 smithi145 ceph-mon[89359]: pgmap v661: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 548 KiB/s rd, 267 KiB/s wr, 1.09k op/s; 0 B/s, 9 keys/s, 0 objects/s recovering 2024-09-21T10:09:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:42 smithi119 ceph-mon[101432]: pgmap v661: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 548 KiB/s rd, 267 KiB/s wr, 1.09k op/s; 0 B/s, 9 keys/s, 0 objects/s recovering 2024-09-21T10:09:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:42 smithi119 ceph-mon[105288]: pgmap v661: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 548 KiB/s rd, 267 KiB/s wr, 1.09k op/s; 0 B/s, 9 keys/s, 0 objects/s recovering 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[105288]: pgmap v662: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 516 KiB/s rd, 251 KiB/s wr, 1.02k op/s; 0 B/s, 9 keys/s, 0 objects/s recovering 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-28' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[105288]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[105288]: osdmap e669: 8 total, 8 up, 8 in 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[101432]: pgmap v662: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 516 KiB/s rd, 251 KiB/s wr, 1.02k op/s; 0 B/s, 9 keys/s, 0 objects/s recovering 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-28' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:09:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[101432]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-mon[101432]: osdmap e669: 8 total, 8 up, 8 in 2024-09-21T10:09:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:44 smithi145 ceph-mon[89359]: pgmap v662: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 516 KiB/s rd, 251 KiB/s wr, 1.02k op/s; 0 B/s, 9 keys/s, 0 objects/s recovering 2024-09-21T10:09:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:44 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-28' no longer out of quota; removing NO_QUOTA flag 2024-09-21T10:09:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:44 smithi145 ceph-mon[89359]: Health check update: 2 pool(s) full (POOL_FULL) 2024-09-21T10:09:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:44 smithi145 ceph-mon[89359]: osdmap e669: 8 total, 8 up, 8 in 2024-09-21T10:09:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:09:44] "GET /metrics HTTP/1.1" 200 67998 "" "Prometheus/2.51.0" 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.955 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.956 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.957 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.958 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.959 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.960 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.961 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.962 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:45.963 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 4 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:read: 4 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS1 (8912 ms) 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestEncryptionLUKS2 2024-09-21T10:09:45.964 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:46 smithi119 ceph-mon[105288]: pgmap v664: 521 pgs: 521 active+clean; 6.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 1.81k op/s; 0 B/s, 8 keys/s, 0 objects/s recovering 2024-09-21T10:09:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:46 smithi119 ceph-mon[101432]: pgmap v664: 521 pgs: 521 active+clean; 6.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 1.81k op/s; 0 B/s, 8 keys/s, 0 objects/s recovering 2024-09-21T10:09:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:46 smithi145 ceph-mon[89359]: pgmap v664: 521 pgs: 521 active+clean; 6.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 1.81k op/s; 0 B/s, 8 keys/s, 0 objects/s recovering 2024-09-21T10:09:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:48 smithi119 ceph-mon[105288]: pgmap v665: 521 pgs: 521 active+clean; 6.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 956 KiB/s wr, 1.43k op/s; 0 B/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:48 smithi119 ceph-mon[101432]: pgmap v665: 521 pgs: 521 active+clean; 6.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 956 KiB/s wr, 1.43k op/s; 0 B/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:48 smithi145 ceph-mon[89359]: pgmap v665: 521 pgs: 521 active+clean; 6.8 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 956 KiB/s wr, 1.43k op/s; 0 B/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:09:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:09:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:50 smithi119 ceph-mon[105288]: pgmap v666: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 955 KiB/s wr, 1.42k op/s; 0 B/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:50.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:50 smithi119 ceph-mon[101432]: pgmap v666: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 955 KiB/s wr, 1.42k op/s; 0 B/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:09:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:09:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:09:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:50 smithi145 ceph-mon[89359]: pgmap v666: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.1 MiB/s rd, 955 KiB/s wr, 1.42k op/s; 0 B/s, 7 keys/s, 0 objects/s recovering 2024-09-21T10:09:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:52 smithi119 ceph-mon[105288]: pgmap v667: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 874 KiB/s rd, 830 KiB/s wr, 854 op/s 2024-09-21T10:09:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:52 smithi119 ceph-mon[101432]: pgmap v667: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 874 KiB/s rd, 830 KiB/s wr, 854 op/s 2024-09-21T10:09:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:52 smithi145 ceph-mon[89359]: pgmap v667: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 874 KiB/s rd, 830 KiB/s wr, 854 op/s 2024-09-21T10:09:53.383 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:53.383 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:53.383 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:53.383 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:53.383 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:53.383 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.384 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.385 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.386 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.387 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:53.388 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:53.389 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:53.390 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 512~512 = 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 1536~512 = 0 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.391 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.392 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.393 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.394 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.395 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.396 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:53.397 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.398 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.399 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.400 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:53.401 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:54 smithi119 ceph-mon[105288]: pgmap v668: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 874 KiB/s rd, 830 KiB/s wr, 854 op/s 2024-09-21T10:09:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:54 smithi119 ceph-mon[101432]: pgmap v668: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 874 KiB/s rd, 830 KiB/s wr, 854 op/s 2024-09-21T10:09:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:54 smithi145 ceph-mon[89359]: pgmap v668: 521 pgs: 521 active+clean; 11 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 874 KiB/s rd, 830 KiB/s wr, 854 op/s 2024-09-21T10:09:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:09:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:09:54] "GET /metrics HTTP/1.1" 200 68004 "" "Prometheus/2.51.0" 2024-09-21T10:09:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:56 smithi119 ceph-mon[101432]: pgmap v669: 521 pgs: 521 active+clean; 23 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 1.27k op/s 2024-09-21T10:09:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:56 smithi119 ceph-mon[105288]: pgmap v669: 521 pgs: 521 active+clean; 23 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 1.27k op/s 2024-09-21T10:09:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:56 smithi145 ceph-mon[89359]: pgmap v669: 521 pgs: 521 active+clean; 23 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 1.27k op/s 2024-09-21T10:09:57.255 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.255 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.255 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.255 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.256 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.257 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.258 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.259 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.260 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.261 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.262 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.263 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 4 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:read: 4 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestEncryptionLUKS2 (11300 ms) 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestIOWithIOHint 2024-09-21T10:09:57.264 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:09:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:09:58 smithi119 ceph-mon[101432]: pgmap v670: 521 pgs: 521 active+clean; 27 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 403 op/s 2024-09-21T10:09:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:09:58 smithi119 ceph-mon[105288]: pgmap v670: 521 pgs: 521 active+clean; 27 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 403 op/s 2024-09-21T10:09:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:09:58 smithi145 ceph-mon[89359]: pgmap v670: 521 pgs: 521 active+clean; 27 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 403 op/s 2024-09-21T10:09:59.201 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:59.202 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:09:59.203 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.204 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:59.205 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.206 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:59.207 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 512~512 = 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 1536~512 = 0 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:09:59.208 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.209 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.210 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:59.211 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:59.212 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.213 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.214 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.215 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:09:59.216 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:09:59.217 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:09:59.217 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:09:59.217 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:09:59.217 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:09:59.217 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[105288]: pgmap v671: 521 pgs: 521 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 952 op/s 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[105288]: Health detail: HEALTH_WARN 2 pool(s) full 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[105288]: [WRN] POOL_FULL: 2 pool(s) full 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-24' is full (running out of quota) 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[105288]: pool 'test-librbd-smithi119-84733-27' is full (running out of quota) 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[101432]: pgmap v671: 521 pgs: 521 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 952 op/s 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN 2 pool(s) full 2024-09-21T10:10:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[101432]: [WRN] POOL_FULL: 2 pool(s) full 2024-09-21T10:10:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-24' is full (running out of quota) 2024-09-21T10:10:00.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:00 smithi119 ceph-mon[101432]: pool 'test-librbd-smithi119-84733-27' is full (running out of quota) 2024-09-21T10:10:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:10:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:10:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:00 smithi145 ceph-mon[89359]: pgmap v671: 521 pgs: 521 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 952 op/s 2024-09-21T10:10:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:00 smithi145 ceph-mon[89359]: Health detail: HEALTH_WARN 2 pool(s) full 2024-09-21T10:10:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:00 smithi145 ceph-mon[89359]: [WRN] POOL_FULL: 2 pool(s) full 2024-09-21T10:10:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:00 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-24' is full (running out of quota) 2024-09-21T10:10:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:00 smithi145 ceph-mon[89359]: pool 'test-librbd-smithi119-84733-27' is full (running out of quota) 2024-09-21T10:10:01.023 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.023 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.023 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.023 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.023 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.023 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.024 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.025 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.026 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.027 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestIOWithIOHint (3768 ms) 2024-09-21T10:10:01.032 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestDataPoolIO 2024-09-21T10:10:02.118 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:02 smithi145 ceph-mon[89359]: pgmap v672: 521 pgs: 521 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 948 op/s 2024-09-21T10:10:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:02 smithi145 ceph-mon[89359]: osdmap e670: 8 total, 8 up, 8 in 2024-09-21T10:10:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2138939683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:02 smithi119 ceph-mon[101432]: pgmap v672: 521 pgs: 521 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 948 op/s 2024-09-21T10:10:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:02 smithi119 ceph-mon[101432]: osdmap e670: 8 total, 8 up, 8 in 2024-09-21T10:10:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2138939683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:02 smithi119 ceph-mon[105288]: pgmap v672: 521 pgs: 521 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 948 op/s 2024-09-21T10:10:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:02 smithi119 ceph-mon[105288]: osdmap e670: 8 total, 8 up, 8 in 2024-09-21T10:10:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2138939683' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-29","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2138939683' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:03 smithi145 ceph-mon[89359]: osdmap e671: 8 total, 8 up, 8 in 2024-09-21T10:10:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:03 smithi145 ceph-mon[89359]: pgmap v675: 553 pgs: 30 unknown, 523 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 863 KiB/s wr, 827 op/s 2024-09-21T10:10:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2138939683' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[105288]: osdmap e671: 8 total, 8 up, 8 in 2024-09-21T10:10:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[105288]: pgmap v675: 553 pgs: 30 unknown, 523 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 863 KiB/s wr, 827 op/s 2024-09-21T10:10:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2138939683' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[101432]: osdmap e671: 8 total, 8 up, 8 in 2024-09-21T10:10:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[101432]: pgmap v675: 553 pgs: 30 unknown, 523 active+clean; 31 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 MiB/s rd, 863 KiB/s wr, 827 op/s 2024-09-21T10:10:03.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:04.196 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:04.196 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:04.196 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:04.196 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:04.197 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.198 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:04.198 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:04.198 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:04.198 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:04.198 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.198 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:04.199 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:04.200 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:04.200 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.200 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:04.200 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:04.200 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:04.200 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.201 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:04.202 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.202 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:04.202 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.202 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:04.202 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.202 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:04.203 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.203 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:04.203 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.203 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:04.203 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.203 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:04.204 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.204 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:04.204 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:04.204 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:04.204 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.204 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 512~512 = 512 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 1536~512 = 0 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.205 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.206 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:04.206 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.206 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:04.206 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestDataPoolIO (3172 ms) 2024-09-21T10:10:04.206 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatch 2024-09-21T10:10:04.206 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.222 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatch (27 ms) 2024-09-21T10:10:04.222 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatch 2024-09-21T10:10:04.222 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.248 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatch (26 ms) 2024-09-21T10:10:04.248 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccess 2024-09-21T10:10:04.248 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.271 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccess (23 ms) 2024-09-21T10:10:04.271 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccess 2024-09-21T10:10:04.271 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.298 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccess (26 ms) 2024-09-21T10:10:04.298 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned 2024-09-21T10:10:04.298 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.372 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnaligned (75 ms) 2024-09-21T10:10:04.373 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned 2024-09-21T10:10:04.373 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.431 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnaligned (59 ms) 2024-09-21T10:10:04.431 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLarge 2024-09-21T10:10:04.431 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.509 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLarge (78 ms) 2024-09-21T10:10:04.509 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLarge 2024-09-21T10:10:04.510 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.540 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:04 smithi119 ceph-mon[105288]: osdmap e672: 8 total, 8 up, 8 in 2024-09-21T10:10:04.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:04 smithi119 ceph-mon[101432]: osdmap e672: 8 total, 8 up, 8 in 2024-09-21T10:10:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:04 smithi145 ceph-mon[89359]: osdmap e672: 8 total, 8 up, 8 in 2024-09-21T10:10:04.581 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLarge (71 ms) 2024-09-21T10:10:04.581 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess 2024-09-21T10:10:04.581 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.824 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccess (244 ms) 2024-09-21T10:10:04.824 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess 2024-09-21T10:10:04.824 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:10:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:10:04] "GET /metrics HTTP/1.1" 200 70368 "" "Prometheus/2.51.0" 2024-09-21T10:10:05.092 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccess (267 ms) 2024-09-21T10:10:05.092 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers 2024-09-21T10:10:05.092 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.114 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVIovecLenDiffers (23 ms) 2024-09-21T10:10:05.115 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVMismatch 2024-09-21T10:10:05.115 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.139 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVMismatch (25 ms) 2024-09-21T10:10:05.139 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteVSuccess 2024-09-21T10:10:05.139 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.164 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteVSuccess (25 ms) 2024-09-21T10:10:05.165 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestScatterGatherIO 2024-09-21T10:10:05.165 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.191 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestScatterGatherIO (26 ms) 2024-09-21T10:10:05.192 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestEmptyDiscard 2024-09-21T10:10:05.192 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.209 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 0~1048576 = 0 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 0~4194304 = 0 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:aio discard: 3145728~1048576 = 0 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestEmptyDiscard (19 ms) 2024-09-21T10:10:05.210 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestFUA 2024-09-21T10:10:05.211 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.309 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.310 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.311 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.312 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.313 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.313 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.313 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.313 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.313 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.313 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.314 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.314 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.314 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.314 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.314 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.314 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestFUA (99 ms) 2024-09-21T10:10:05.315 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestIOPP 2024-09-21T10:10:05.315 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:05.326 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.326 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.327 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.327 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.327 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:05.327 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.327 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.327 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.328 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.328 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.328 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.328 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.328 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.328 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.329 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.330 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.330 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.330 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.330 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.330 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:05.330 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.341 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.341 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:05.341 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:10:05.342 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:10:05.342 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:10:05.342 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:10:05.342 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:10:05.342 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:start compare and write 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:compare and wrote: 512 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:10:05.343 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.344 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:10:05.345 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.345 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:10:05.345 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.346 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:10:05.347 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.347 INFO:tasks.workunit.client.0.smithi119.stdout:started aio compare and write 2024-09-21T10:10:05.347 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.348 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:05.348 INFO:tasks.workunit.client.0.smithi119.stdout:finished aio compare and write 2024-09-21T10:10:05.348 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.349 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.350 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.351 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:discard: 512~512 2024-09-21T10:10:05.352 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:05 smithi145 ceph-mon[89359]: osdmap e673: 8 total, 8 up, 8 in 2024-09-21T10:10:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:05 smithi145 ceph-mon[89359]: pgmap v678: 553 pgs: 5 unknown, 548 active+clean; 33 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 706 KiB/s rd, 312 KiB/s wr, 1.43k op/s 2024-09-21T10:10:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:05 smithi119 ceph-mon[101432]: osdmap e673: 8 total, 8 up, 8 in 2024-09-21T10:10:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:05 smithi119 ceph-mon[101432]: pgmap v678: 553 pgs: 5 unknown, 548 active+clean; 33 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 706 KiB/s rd, 312 KiB/s wr, 1.43k op/s 2024-09-21T10:10:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:05 smithi119 ceph-mon[105288]: osdmap e673: 8 total, 8 up, 8 in 2024-09-21T10:10:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:05 smithi119 ceph-mon[105288]: pgmap v678: 553 pgs: 5 unknown, 548 active+clean; 33 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 706 KiB/s rd, 312 KiB/s wr, 1.43k op/s 2024-09-21T10:10:06.607 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.608 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.609 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.610 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.611 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.612 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.613 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.614 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.614 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.614 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.614 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.614 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.614 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.615 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.615 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.616 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.616 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.617 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.617 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.617 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.622 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.623 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.623 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.623 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.623 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.642 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.642 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.642 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.643 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.643 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.643 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.643 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.669 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.669 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.670 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.670 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.670 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.670 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.670 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.713 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.713 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.713 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.713 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.713 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.714 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.714 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.739 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.739 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.739 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.739 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.739 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.739 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.740 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.783 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.783 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.783 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.783 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.783 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.783 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.784 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.825 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.825 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.825 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.825 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.826 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.826 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.826 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.892 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.949 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:06.950 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:06.950 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:06.950 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.016 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.016 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.016 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.016 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.016 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.016 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.017 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.075 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.075 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.075 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.076 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.076 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.076 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.076 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.156 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.219 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.219 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.220 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.319 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.319 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.320 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.320 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.320 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.320 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.320 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.409 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.538 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.538 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.539 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.539 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.539 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.539 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.539 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:07.623 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:07.624 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestIOPP (2314 ms) 2024-09-21T10:10:07.624 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP 2024-09-21T10:10:07.624 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.645 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteCompareTooSmallPP (23 ms) 2024-09-21T10:10:07.645 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP 2024-09-21T10:10:07.645 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.666 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteCompareTooSmallPP (21 ms) 2024-09-21T10:10:07.666 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP 2024-09-21T10:10:07.666 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.689 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteWriteTooSmallPP (23 ms) 2024-09-21T10:10:07.689 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP 2024-09-21T10:10:07.689 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.712 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteWriteTooSmallPP (23 ms) 2024-09-21T10:10:07.712 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchPP 2024-09-21T10:10:07.712 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.737 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchPP (25 ms) 2024-09-21T10:10:07.737 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchPP 2024-09-21T10:10:07.737 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.757 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.761 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchPP (24 ms) 2024-09-21T10:10:07.761 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-21T10:10:07.761 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteMismatchBufferlistGreaterLenPP (27 ms) 2024-09-21T10:10:07.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP 2024-09-21T10:10:07.788 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.810 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.813 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteMismatchBufferlistGreaterLenPP (25 ms) 2024-09-21T10:10:07.813 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessPP 2024-09-21T10:10:07.813 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.841 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessPP (28 ms) 2024-09-21T10:10:07.841 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessPP 2024-09-21T10:10:07.841 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.864 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:07 smithi119 ceph-mon[105288]: pgmap v679: 553 pgs: 553 active+clean; 37 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 989 KiB/s wr, 1.09k op/s 2024-09-21T10:10:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:07 smithi119 ceph-mon[101432]: pgmap v679: 553 pgs: 553 active+clean; 37 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 989 KiB/s wr, 1.09k op/s 2024-09-21T10:10:07.867 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessPP (26 ms) 2024-09-21T10:10:07.867 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-21T10:10:07.867 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.896 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteSuccessBufferlistGreaterLenPP (29 ms) 2024-09-21T10:10:07.896 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP 2024-09-21T10:10:07.896 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:07.922 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:07.926 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteSuccessBufferlistGreaterLenPP (29 ms) 2024-09-21T10:10:07.926 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP 2024-09-21T10:10:07.926 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.001 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitUnalignedPP (76 ms) 2024-09-21T10:10:08.001 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP 2024-09-21T10:10:08.001 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:07 smithi145 ceph-mon[89359]: pgmap v679: 553 pgs: 553 active+clean; 37 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 989 KiB/s wr, 1.09k op/s 2024-09-21T10:10:08.050 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.069 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitUnalignedPP (68 ms) 2024-09-21T10:10:08.070 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteTooLargePP 2024-09-21T10:10:08.070 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.136 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteTooLargePP (67 ms) 2024-09-21T10:10:08.137 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteTooLargePP 2024-09-21T10:10:08.137 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.183 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.203 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteTooLargePP (67 ms) 2024-09-21T10:10:08.203 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP 2024-09-21T10:10:08.203 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.486 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestCompareAndWriteStripeUnitSuccessPP (282 ms) 2024-09-21T10:10:08.486 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP 2024-09-21T10:10:08.486 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.726 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.744 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestAioCompareAndWriteStripeUnitSuccessPP (259 ms) 2024-09-21T10:10:08.744 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestIOPPWithIOHint 2024-09-21T10:10:08.744 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:08.769 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.770 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:08.770 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:08.770 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.770 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:08.770 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.770 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:08.771 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:08.771 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.771 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:08.771 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.772 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:08.772 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:08.772 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.772 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:08.772 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.774 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:08.774 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:08.774 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.774 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:10:08.774 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:08.775 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:08.775 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:10:08.775 INFO:tasks.workunit.client.0.smithi119.stdout:read: 512 2024-09-21T10:10:08.775 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.775 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:08.775 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:08.776 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:08.776 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:08.776 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.776 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:08.776 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:08.776 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:08.777 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:08.778 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:08.778 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:08.778 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:08.778 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:10:08.778 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:finished read 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 0 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.715 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 16384 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 49152 2024-09-21T10:10:09.716 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 65536 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:10:09.717 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 98304 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 114688 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.718 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 147456 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 163840 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.719 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 196608 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 212992 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.720 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.721 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.722 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.724 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.724 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.725 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.725 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.725 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.725 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.725 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.732 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.733 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.733 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.733 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.733 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.733 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.733 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.754 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.754 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.755 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.755 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.755 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.755 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.755 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.776 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.805 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.805 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.806 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.806 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.806 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.806 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.806 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:09.831 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.832 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.832 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.873 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.874 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.874 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.874 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.874 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.874 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.874 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.912 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:09.975 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:09.975 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:09.976 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:09.976 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:09.976 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:09.976 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:09.976 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.023 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.024 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:10.024 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.024 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.024 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.088 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.089 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.089 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.089 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.089 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.089 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.089 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.152 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.226 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:10.305 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.306 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.306 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:10 smithi119 ceph-mon[105288]: pgmap v680: 553 pgs: 553 active+clean; 103 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 2.51k op/s 2024-09-21T10:10:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:10 smithi119 ceph-mon[105288]: osdmap e674: 8 total, 8 up, 8 in 2024-09-21T10:10:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:10 smithi119 ceph-mon[101432]: pgmap v680: 553 pgs: 553 active+clean; 103 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 2.51k op/s 2024-09-21T10:10:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:10 smithi119 ceph-mon[101432]: osdmap e674: 8 total, 8 up, 8 in 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.395 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.479 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:10 smithi145 ceph-mon[89359]: pgmap v680: 553 pgs: 553 active+clean; 103 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 12 MiB/s wr, 2.51k op/s 2024-09-21T10:10:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:10 smithi145 ceph-mon[89359]: osdmap e674: 8 total, 8 up, 8 in 2024-09-21T10:10:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:10:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:10:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:10.586 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.586 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.586 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.587 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.587 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.587 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.587 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:10:10.676 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:10:10.676 INFO:tasks.workunit.client.0.smithi119.stdout:finished writesame 2024-09-21T10:10:10.676 INFO:tasks.workunit.client.0.smithi119.stdout:to verify the data 2024-09-21T10:10:10.676 INFO:tasks.workunit.client.0.smithi119.stdout:verified 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:started writesame 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:expected fail, finished writesame 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestIOPPWithIOHint (1932 ms) 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestIOToSnapshot 2024-09-21T10:10:10.677 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:11.334 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:11.334 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:11.365 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:10:12.109 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:10:12.109 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:10:12.109 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:snap: orig 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:found orig with size 2097152 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:write test data! 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 80 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 2 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:snap: orig 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:snap: written 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:found orig with size 2097152 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:found written with size 2097152 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.110 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.111 INFO:tasks.workunit.client.0.smithi119.stdout:write to snapshot returned -30 2024-09-21T10:10:12.111 INFO:tasks.workunit.client.0.smithi119.stdout:Read-only file system 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 80 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:opening testimg@orig 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:read: 80 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:write to snapshot returned -30 2024-09-21T10:10:12.131 INFO:tasks.workunit.client.0.smithi119.stdout:Read-only file system 2024-09-21T10:10:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:12 smithi119 ceph-mon[101432]: pgmap v682: 553 pgs: 553 active+clean; 123 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 13 MiB/s wr, 2.24k op/s 2024-09-21T10:10:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:12 smithi119 ceph-mon[101432]: osdmap e675: 8 total, 8 up, 8 in 2024-09-21T10:10:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:12 smithi119 ceph-mon[105288]: pgmap v682: 553 pgs: 553 active+clean; 123 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 13 MiB/s wr, 2.24k op/s 2024-09-21T10:10:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:12 smithi119 ceph-mon[105288]: osdmap e675: 8 total, 8 up, 8 in 2024-09-21T10:10:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:12 smithi145 ceph-mon[89359]: pgmap v682: 553 pgs: 553 active+clean; 123 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 13 MiB/s wr, 2.24k op/s 2024-09-21T10:10:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:12 smithi145 ceph-mon[89359]: osdmap e675: 8 total, 8 up, 8 in 2024-09-21T10:10:13.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:13 smithi119 ceph-mon[101432]: osdmap e676: 8 total, 8 up, 8 in 2024-09-21T10:10:13.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:13 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:13.350 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:13 smithi119 ceph-mon[105288]: osdmap e676: 8 total, 8 up, 8 in 2024-09-21T10:10:13.350 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:13 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:13.457 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:13 smithi145 ceph-mon[89359]: osdmap e676: 8 total, 8 up, 8 in 2024-09-21T10:10:13.457 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:13 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:14.114 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 2 2024-09-21T10:10:14.114 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:10:14.114 INFO:tasks.workunit.client.0.smithi119.stdout:snap: orig 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:snap: written 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:found orig with size 2097152 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:found written with size 2097152 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 1 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:snap: orig 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:found orig with size 2097152 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestIOToSnapshot (3438 ms) 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestSnapshotDeletedIo 2024-09-21T10:10:14.115 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:14 smithi119 ceph-mon[101432]: pgmap v685: 553 pgs: 553 active+clean; 123 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 16 MiB/s wr, 1.78k op/s 2024-09-21T10:10:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:14 smithi119 ceph-mon[101432]: osdmap e677: 8 total, 8 up, 8 in 2024-09-21T10:10:14.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:14 smithi119 ceph-mon[105288]: pgmap v685: 553 pgs: 553 active+clean; 123 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 16 MiB/s wr, 1.78k op/s 2024-09-21T10:10:14.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:14 smithi119 ceph-mon[105288]: osdmap e677: 8 total, 8 up, 8 in 2024-09-21T10:10:14.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:14 smithi145 ceph-mon[89359]: pgmap v685: 553 pgs: 553 active+clean; 123 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 16 MiB/s wr, 1.78k op/s 2024-09-21T10:10:14.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:14 smithi145 ceph-mon[89359]: osdmap e677: 8 total, 8 up, 8 in 2024-09-21T10:10:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:10:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:10:14] "GET /metrics HTTP/1.1" 200 70368 "" "Prometheus/2.51.0" 2024-09-21T10:10:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:15 smithi145 ceph-mon[89359]: osdmap e678: 8 total, 8 up, 8 in 2024-09-21T10:10:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:15 smithi119 ceph-mon[101432]: osdmap e678: 8 total, 8 up, 8 in 2024-09-21T10:10:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:15 smithi119 ceph-mon[105288]: osdmap e678: 8 total, 8 up, 8 in 2024-09-21T10:10:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.126 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestSnapshotDeletedIo (2012 ms) 2024-09-21T10:10:16.126 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestClone 2024-09-21T10:10:16.126 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: pgmap v688: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 4.5 MiB/s wr, 2.15k op/s 2024-09-21T10:10:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: osdmap e679: 8 total, 8 up, 8 in 2024-09-21T10:10:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: pgmap v688: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 4.5 MiB/s wr, 2.15k op/s 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: osdmap e679: 8 total, 8 up, 8 in 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: pgmap v688: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.0 MiB/s rd, 4.5 MiB/s wr, 2.15k op/s 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: osdmap e679: 8 total, 8 up, 8 in 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:10:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:17 smithi119 ceph-mon[105288]: osdmap e680: 8 total, 8 up, 8 in 2024-09-21T10:10:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:17 smithi119 ceph-mon[101432]: osdmap e680: 8 total, 8 up, 8 in 2024-09-21T10:10:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:17 smithi145 ceph-mon[89359]: osdmap e680: 8 total, 8 up, 8 in 2024-09-21T10:10:18.140 INFO:tasks.workunit.client.0.smithi119.stdout:made parent image "parent" 2024-09-21T10:10:18.140 INFO:tasks.workunit.client.0.smithi119.stdout:parent has no parent info 2024-09-21T10:10:18.140 INFO:tasks.workunit.client.0.smithi119.stdout:made snapshot "parent@parent_snap" 2024-09-21T10:10:18.140 INFO:tasks.workunit.client.0.smithi119.stdout:can't unprotect an unprotected snap 2024-09-21T10:10:18.141 INFO:tasks.workunit.client.0.smithi119.stdout:can't protect a protected snap 2024-09-21T10:10:18.141 INFO:tasks.workunit.client.0.smithi119.stdout:made and opened clone "child" 2024-09-21T10:10:18.141 INFO:tasks.workunit.client.0.smithi119.stdout:read: 8 2024-09-21T10:10:18.141 INFO:tasks.workunit.client.0.smithi119.stdout:read: 8 2024-09-21T10:10:18.141 INFO:tasks.workunit.client.0.smithi119.stdout:read: 8 2024-09-21T10:10:18.141 INFO:tasks.workunit.client.0.smithi119.stdout:sizes and overlaps are good between parent and child 2024-09-21T10:10:18.142 INFO:tasks.workunit.client.0.smithi119.stdout:child image successfully cloned all image-meta pairs 2024-09-21T10:10:18.142 INFO:tasks.workunit.client.0.smithi119.stdout:sized down clone, changed overlap 2024-09-21T10:10:18.142 INFO:tasks.workunit.client.0.smithi119.stdout:parent info: size 4194304 obj_size 4194304 parent_pool 18446744073709551615 2024-09-21T10:10:18.142 INFO:tasks.workunit.client.0.smithi119.stdout:sized up clone, changed size but not overlap or parent's size 2024-09-21T10:10:18.142 INFO:tasks.workunit.client.0.smithi119.stdout:can't remove parent while child still exists 2024-09-21T10:10:18.143 INFO:tasks.workunit.client.0.smithi119.stdout:can't remove parent while still protected 2024-09-21T10:10:18.143 INFO:tasks.workunit.client.0.smithi119.stdout:removed parent snap after unprotecting 2024-09-21T10:10:18.143 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestClone (2013 ms) 2024-09-21T10:10:18.143 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestClone2 2024-09-21T10:10:18.143 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:18 smithi119 ceph-mon[105288]: pgmap v691: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 4.5 MiB/s wr, 2.21k op/s 2024-09-21T10:10:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:18 smithi119 ceph-mon[105288]: osdmap e681: 8 total, 8 up, 8 in 2024-09-21T10:10:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:18 smithi119 ceph-mon[101432]: pgmap v691: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 4.5 MiB/s wr, 2.21k op/s 2024-09-21T10:10:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:18 smithi119 ceph-mon[101432]: osdmap e681: 8 total, 8 up, 8 in 2024-09-21T10:10:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:18 smithi145 ceph-mon[89359]: pgmap v691: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.1 MiB/s rd, 4.5 MiB/s wr, 2.21k op/s 2024-09-21T10:10:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:18 smithi145 ceph-mon[89359]: osdmap e681: 8 total, 8 up, 8 in 2024-09-21T10:10:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:made parent image "parent" 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:parent has no parent info 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:made snapshot "parent@parent_snap" 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:made and opened clone "child" 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:child image successfully cloned all image-meta pairs 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestClone2 (1045 ms) 2024-09-21T10:10:19.185 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ListChildren 2024-09-21T10:10:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:19 smithi145 ceph-mon[89359]: osdmap e682: 8 total, 8 up, 8 in 2024-09-21T10:10:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:19 smithi145 ceph-mon[89359]: osdmap e683: 8 total, 8 up, 8 in 2024-09-21T10:10:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:19 smithi119 ceph-mon[101432]: osdmap e682: 8 total, 8 up, 8 in 2024-09-21T10:10:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:19 smithi119 ceph-mon[101432]: osdmap e683: 8 total, 8 up, 8 in 2024-09-21T10:10:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:19 smithi119 ceph-mon[105288]: osdmap e682: 8 total, 8 up, 8 in 2024-09-21T10:10:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:19 smithi119 ceph-mon[105288]: osdmap e683: 8 total, 8 up, 8 in 2024-09-21T10:10:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:20 smithi145 ceph-mon[89359]: pgmap v694: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 453 KiB/s rd, 33 KiB/s wr, 550 op/s 2024-09-21T10:10:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:10:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:10:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:20 smithi119 ceph-mon[101432]: pgmap v694: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 453 KiB/s rd, 33 KiB/s wr, 550 op/s 2024-09-21T10:10:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:20 smithi119 ceph-mon[105288]: pgmap v694: 553 pgs: 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 453 KiB/s rd, 33 KiB/s wr, 550 op/s 2024-09-21T10:10:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:21 smithi145 ceph-mon[89359]: osdmap e684: 8 total, 8 up, 8 in 2024-09-21T10:10:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/399415760' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:21 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:21 smithi145 ceph-mon[89359]: pgmap v697: 585 pgs: 3 creating+peering, 29 unknown, 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 721 KiB/s rd, 51 KiB/s wr, 862 op/s 2024-09-21T10:10:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[101432]: osdmap e684: 8 total, 8 up, 8 in 2024-09-21T10:10:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/399415760' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[101432]: pgmap v697: 585 pgs: 3 creating+peering, 29 unknown, 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 721 KiB/s rd, 51 KiB/s wr, 862 op/s 2024-09-21T10:10:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[105288]: osdmap e684: 8 total, 8 up, 8 in 2024-09-21T10:10:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/399415760' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:21 smithi119 ceph-mon[105288]: pgmap v697: 585 pgs: 3 creating+peering, 29 unknown, 553 active+clean; 131 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 721 KiB/s rd, 51 KiB/s wr, 862 op/s 2024-09-21T10:10:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:22 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:10:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:22 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:22 smithi145 ceph-mon[89359]: osdmap e685: 8 total, 8 up, 8 in 2024-09-21T10:10:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:22 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:10:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:22 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:22 smithi119 ceph-mon[101432]: osdmap e685: 8 total, 8 up, 8 in 2024-09-21T10:10:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:22 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:10:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:22 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:22 smithi119 ceph-mon[105288]: osdmap e685: 8 total, 8 up, 8 in 2024-09-21T10:10:23.164 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:23 smithi145 ceph-mon[89359]: osdmap e686: 8 total, 8 up, 8 in 2024-09-21T10:10:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3762297148' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:23 smithi145 ceph-mon[89359]: pgmap v700: 617 pgs: 5 creating+peering, 52 unknown, 560 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 317 KiB/s rd, 22 KiB/s wr, 376 op/s 2024-09-21T10:10:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:23 smithi119 ceph-mon[101432]: osdmap e686: 8 total, 8 up, 8 in 2024-09-21T10:10:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3762297148' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:23 smithi119 ceph-mon[101432]: pgmap v700: 617 pgs: 5 creating+peering, 52 unknown, 560 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 317 KiB/s rd, 22 KiB/s wr, 376 op/s 2024-09-21T10:10:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:23 smithi119 ceph-mon[105288]: osdmap e686: 8 total, 8 up, 8 in 2024-09-21T10:10:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3762297148' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-31","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:23 smithi119 ceph-mon[105288]: pgmap v700: 617 pgs: 5 creating+peering, 52 unknown, 560 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 317 KiB/s rd, 22 KiB/s wr, 376 op/s 2024-09-21T10:10:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3762297148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-mon[101432]: osdmap e687: 8 total, 8 up, 8 in 2024-09-21T10:10:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-mon[101432]: osdmap e688: 8 total, 8 up, 8 in 2024-09-21T10:10:24.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3762297148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:24.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-mon[105288]: osdmap e687: 8 total, 8 up, 8 in 2024-09-21T10:10:24.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-mon[105288]: osdmap e688: 8 total, 8 up, 8 in 2024-09-21T10:10:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3762297148' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:24 smithi145 ceph-mon[89359]: osdmap e687: 8 total, 8 up, 8 in 2024-09-21T10:10:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:24 smithi145 ceph-mon[89359]: osdmap e688: 8 total, 8 up, 8 in 2024-09-21T10:10:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:10:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:10:24] "GET /metrics HTTP/1.1" 200 70418 "" "Prometheus/2.51.0" 2024-09-21T10:10:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:26 smithi119 ceph-mon[101432]: pgmap v703: 617 pgs: 5 creating+peering, 2 unknown, 610 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 6.0 KiB/s wr, 88 op/s 2024-09-21T10:10:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:26 smithi119 ceph-mon[101432]: osdmap e689: 8 total, 8 up, 8 in 2024-09-21T10:10:26.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:26 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:10:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:26 smithi119 ceph-mon[105288]: pgmap v703: 617 pgs: 5 creating+peering, 2 unknown, 610 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 6.0 KiB/s wr, 88 op/s 2024-09-21T10:10:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:26 smithi119 ceph-mon[105288]: osdmap e689: 8 total, 8 up, 8 in 2024-09-21T10:10:26.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:26 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:10:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:26 smithi145 ceph-mon[89359]: pgmap v703: 617 pgs: 5 creating+peering, 2 unknown, 610 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 72 KiB/s rd, 6.0 KiB/s wr, 88 op/s 2024-09-21T10:10:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:26 smithi145 ceph-mon[89359]: osdmap e689: 8 total, 8 up, 8 in 2024-09-21T10:10:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:26 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:10:27.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:27 smithi119 ceph-mon[101432]: osdmap e690: 8 total, 8 up, 8 in 2024-09-21T10:10:27.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:27 smithi119 ceph-mon[105288]: osdmap e690: 8 total, 8 up, 8 in 2024-09-21T10:10:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:27 smithi145 ceph-mon[89359]: osdmap e690: 8 total, 8 up, 8 in 2024-09-21T10:10:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:28 smithi119 ceph-mon[105288]: pgmap v706: 617 pgs: 2 creating+peering, 615 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 6.2 KiB/s wr, 90 op/s 2024-09-21T10:10:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:28 smithi119 ceph-mon[105288]: osdmap e691: 8 total, 8 up, 8 in 2024-09-21T10:10:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:28 smithi119 ceph-mon[101432]: pgmap v706: 617 pgs: 2 creating+peering, 615 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 6.2 KiB/s wr, 90 op/s 2024-09-21T10:10:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:28 smithi119 ceph-mon[101432]: osdmap e691: 8 total, 8 up, 8 in 2024-09-21T10:10:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:28 smithi145 ceph-mon[89359]: pgmap v706: 617 pgs: 2 creating+peering, 615 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 6.2 KiB/s wr, 90 op/s 2024-09-21T10:10:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:28 smithi145 ceph-mon[89359]: osdmap e691: 8 total, 8 up, 8 in 2024-09-21T10:10:28.575 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.575 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.575 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.575 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.575 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.575 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 1 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:child: image107 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107/880bfe15295b 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.576 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 2 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:child: image107 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107/880bfe15295b 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.577 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:28.578 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:child: image107 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:child: image109 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107/880bfe15295b 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109/880b5a622f19 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.579 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:28.580 INFO:tasks.workunit.client.0.smithi119.stdout:child: image107 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:child: image109 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107/880bfe15295b 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109/880b5a622f19 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.581 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.582 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 4 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 4 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:child: image107 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:child: image109 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:child: image110 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107/880bfe15295b 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109/880b5a622f19 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110/880b5394dc3d 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.583 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:28.584 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:28.585 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:28.585 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:28.585 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.585 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:28.585 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:28.585 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.180 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image107 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 4 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 4 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:child: image107 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:child: image109 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:child: image110 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image107/880bfe15295b 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.181 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109/880b5a622f19 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110/880b5394dc3d 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:29.182 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image109 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:child: image109 2024-09-21T10:10:29.183 INFO:tasks.workunit.client.0.smithi119.stdout:child: image110 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image109/880b5a622f19 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110/880b5394dc3d 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.184 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 2 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:child: image110 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-31/image110/880b5394dc3d 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.185 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 1 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:child: image108 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-30/image108/880b9363403d 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 0 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ListChildren (9996 ms) 2024-09-21T10:10:29.186 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ListChildrenTiered 2024-09-21T10:10:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:29 smithi119 ceph-mon[101432]: osdmap e692: 8 total, 8 up, 8 in 2024-09-21T10:10:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:29 smithi119 ceph-mon[105288]: osdmap e692: 8 total, 8 up, 8 in 2024-09-21T10:10:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:29 smithi145 ceph-mon[89359]: osdmap e692: 8 total, 8 up, 8 in 2024-09-21T10:10:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:30 smithi119 ceph-mon[101432]: pgmap v709: 617 pgs: 617 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 168 KiB/s rd, 5.7 KiB/s wr, 208 op/s 2024-09-21T10:10:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:30 smithi119 ceph-mon[101432]: osdmap e693: 8 total, 8 up, 8 in 2024-09-21T10:10:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:30 smithi119 ceph-mon[105288]: pgmap v709: 617 pgs: 617 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 168 KiB/s rd, 5.7 KiB/s wr, 208 op/s 2024-09-21T10:10:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:30 smithi119 ceph-mon[105288]: osdmap e693: 8 total, 8 up, 8 in 2024-09-21T10:10:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:10:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:10:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:30 smithi145 ceph-mon[89359]: pgmap v709: 617 pgs: 617 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 168 KiB/s rd, 5.7 KiB/s wr, 208 op/s 2024-09-21T10:10:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:30 smithi145 ceph-mon[89359]: osdmap e693: 8 total, 8 up, 8 in 2024-09-21T10:10:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:31 smithi145 ceph-mon[89359]: osdmap e694: 8 total, 8 up, 8 in 2024-09-21T10:10:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:31 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1274955362' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:31 smithi119 ceph-mon[101432]: osdmap e694: 8 total, 8 up, 8 in 2024-09-21T10:10:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:31 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1274955362' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:31 smithi119 ceph-mon[105288]: osdmap e694: 8 total, 8 up, 8 in 2024-09-21T10:10:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:31 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1274955362' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-32","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:32.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:10:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=sqlstore.transactions t=2024-09-21T10:10:31.648151268Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T10:10:32.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:10:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=cleanup t=2024-09-21T10:10:31.651761047Z level=info msg="Completed cleanup jobs" duration=14.011743ms 2024-09-21T10:10:32.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[105288]: pgmap v712: 649 pgs: 5 creating+peering, 27 unknown, 617 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 382 KiB/s rd, 9.0 KiB/s wr, 466 op/s 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1274955362' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[105288]: osdmap e695: 8 total, 8 up, 8 in 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[101432]: pgmap v712: 649 pgs: 5 creating+peering, 27 unknown, 617 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 382 KiB/s rd, 9.0 KiB/s wr, 466 op/s 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1274955362' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:32 smithi119 ceph-mon[101432]: osdmap e695: 8 total, 8 up, 8 in 2024-09-21T10:10:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:32 smithi145 ceph-mon[89359]: pgmap v712: 649 pgs: 5 creating+peering, 27 unknown, 617 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 382 KiB/s rd, 9.0 KiB/s wr, 466 op/s 2024-09-21T10:10:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:32 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:10:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1274955362' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:32 smithi145 ceph-mon[89359]: osdmap e695: 8 total, 8 up, 8 in 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: osdmap e696: 8 total, 8 up, 8 in 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3723935596' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: pgmap v715: 681 pgs: 11 creating+peering, 47 unknown, 623 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 308 KiB/s rd, 4.2 KiB/s wr, 370 op/s 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[101432]: osdmap e697: 8 total, 8 up, 8 in 2024-09-21T10:10:33.536 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: osdmap e696: 8 total, 8 up, 8 in 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3723935596' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: pgmap v715: 681 pgs: 11 creating+peering, 47 unknown, 623 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 308 KiB/s rd, 4.2 KiB/s wr, 370 op/s 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:33.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:33 smithi119 ceph-mon[105288]: osdmap e697: 8 total, 8 up, 8 in 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: osdmap e696: 8 total, 8 up, 8 in 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3723935596' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: pgmap v715: 681 pgs: 11 creating+peering, 47 unknown, 623 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 308 KiB/s rd, 4.2 KiB/s wr, 370 op/s 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:33 smithi145 ceph-mon[89359]: osdmap e697: 8 total, 8 up, 8 in 2024-09-21T10:10:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:10:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:10:34] "GET /metrics HTTP/1.1" 200 79949 "" "Prometheus/2.51.0" 2024-09-21T10:10:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:35 smithi119 ceph-mon[105288]: osdmap e698: 8 total, 8 up, 8 in 2024-09-21T10:10:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2966737122' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:35 smithi119 ceph-mon[101432]: osdmap e698: 8 total, 8 up, 8 in 2024-09-21T10:10:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2966737122' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:35 smithi145 ceph-mon[89359]: osdmap e698: 8 total, 8 up, 8 in 2024-09-21T10:10:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2966737122' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-34","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[105288]: pgmap v718: 713 pgs: 13 creating+peering, 35 unknown, 665 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 213 KiB/s rd, 7.0 KiB/s wr, 262 op/s 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2966737122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[105288]: osdmap e699: 8 total, 8 up, 8 in 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34", "force_nonempty":""} : dispatch 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34", "force_nonempty":""}]': finished 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[105288]: osdmap e700: 8 total, 8 up, 8 in 2024-09-21T10:10:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[101432]: pgmap v718: 713 pgs: 13 creating+peering, 35 unknown, 665 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 213 KiB/s rd, 7.0 KiB/s wr, 262 op/s 2024-09-21T10:10:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2966737122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[101432]: osdmap e699: 8 total, 8 up, 8 in 2024-09-21T10:10:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34", "force_nonempty":""} : dispatch 2024-09-21T10:10:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34", "force_nonempty":""}]': finished 2024-09-21T10:10:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:36 smithi119 ceph-mon[101432]: osdmap e700: 8 total, 8 up, 8 in 2024-09-21T10:10:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:36 smithi145 ceph-mon[89359]: pgmap v718: 713 pgs: 13 creating+peering, 35 unknown, 665 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 213 KiB/s rd, 7.0 KiB/s wr, 262 op/s 2024-09-21T10:10:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2966737122' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:10:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:36 smithi145 ceph-mon[89359]: osdmap e699: 8 total, 8 up, 8 in 2024-09-21T10:10:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34", "force_nonempty":""} : dispatch 2024-09-21T10:10:36.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34", "force_nonempty":""}]': finished 2024-09-21T10:10:36.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:36 smithi145 ceph-mon[89359]: osdmap e700: 8 total, 8 up, 8 in 2024-09-21T10:10:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi119-84733-34", "mode":"writeback"} : dispatch 2024-09-21T10:10:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[105288]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-21T10:10:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi119-84733-34", "mode":"writeback"}]': finished 2024-09-21T10:10:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[105288]: osdmap e701: 8 total, 8 up, 8 in 2024-09-21T10:10:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi119-84733-32", "overlaypool":"test-librbd-smithi119-84733-34"} : dispatch 2024-09-21T10:10:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi119-84733-34", "mode":"writeback"} : dispatch 2024-09-21T10:10:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[101432]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-21T10:10:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi119-84733-34", "mode":"writeback"}]': finished 2024-09-21T10:10:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[101432]: osdmap e701: 8 total, 8 up, 8 in 2024-09-21T10:10:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi119-84733-32", "overlaypool":"test-librbd-smithi119-84733-34"} : dispatch 2024-09-21T10:10:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi119-84733-34", "mode":"writeback"} : dispatch 2024-09-21T10:10:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:37 smithi145 ceph-mon[89359]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-21T10:10:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-librbd-smithi119-84733-34", "mode":"writeback"}]': finished 2024-09-21T10:10:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:37 smithi145 ceph-mon[89359]: osdmap e701: 8 total, 8 up, 8 in 2024-09-21T10:10:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi119-84733-32", "overlaypool":"test-librbd-smithi119-84733-34"} : dispatch 2024-09-21T10:10:38.089 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[105288]: pgmap v721: 713 pgs: 8 creating+peering, 29 unknown, 676 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 118 KiB/s rd, 6.2 KiB/s wr, 150 op/s 2024-09-21T10:10:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:10:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi119-84733-32", "overlaypool":"test-librbd-smithi119-84733-34"}]': finished 2024-09-21T10:10:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[105288]: osdmap e702: 8 total, 8 up, 8 in 2024-09-21T10:10:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[101432]: pgmap v721: 713 pgs: 8 creating+peering, 29 unknown, 676 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 118 KiB/s rd, 6.2 KiB/s wr, 150 op/s 2024-09-21T10:10:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:10:38.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi119-84733-32", "overlaypool":"test-librbd-smithi119-84733-34"}]': finished 2024-09-21T10:10:38.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:38 smithi119 ceph-mon[101432]: osdmap e702: 8 total, 8 up, 8 in 2024-09-21T10:10:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:38 smithi145 ceph-mon[89359]: pgmap v721: 713 pgs: 8 creating+peering, 29 unknown, 676 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 118 KiB/s rd, 6.2 KiB/s wr, 150 op/s 2024-09-21T10:10:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:38 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:10:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-librbd-smithi119-84733-32", "overlaypool":"test-librbd-smithi119-84733-34"}]': finished 2024-09-21T10:10:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:38 smithi145 ceph-mon[89359]: osdmap e702: 8 total, 8 up, 8 in 2024-09-21T10:10:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:40 smithi119 ceph-mon[105288]: pgmap v724: 713 pgs: 2 creating+peering, 711 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:10:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:40 smithi119 ceph-mon[105288]: osdmap e703: 8 total, 8 up, 8 in 2024-09-21T10:10:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:40 smithi119 ceph-mon[101432]: pgmap v724: 713 pgs: 2 creating+peering, 711 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:10:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:40 smithi119 ceph-mon[101432]: osdmap e703: 8 total, 8 up, 8 in 2024-09-21T10:10:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:10:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:10:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:40 smithi145 ceph-mon[89359]: pgmap v724: 713 pgs: 2 creating+peering, 711 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:10:40.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:40 smithi145 ceph-mon[89359]: osdmap e703: 8 total, 8 up, 8 in 2024-09-21T10:10:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:41 smithi119 ceph-mon[105288]: osdmap e704: 8 total, 8 up, 8 in 2024-09-21T10:10:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:41 smithi119 ceph-mon[101432]: osdmap e704: 8 total, 8 up, 8 in 2024-09-21T10:10:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:41 smithi145 ceph-mon[89359]: osdmap e704: 8 total, 8 up, 8 in 2024-09-21T10:10:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:42 smithi119 ceph-mon[101432]: pgmap v727: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:10:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:42 smithi119 ceph-mon[101432]: osdmap e705: 8 total, 8 up, 8 in 2024-09-21T10:10:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:42 smithi119 ceph-mon[105288]: pgmap v727: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:10:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:42 smithi119 ceph-mon[105288]: osdmap e705: 8 total, 8 up, 8 in 2024-09-21T10:10:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:42 smithi145 ceph-mon[89359]: pgmap v727: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:10:42.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:42 smithi145 ceph-mon[89359]: osdmap e705: 8 total, 8 up, 8 in 2024-09-21T10:10:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:43 smithi119 ceph-mon[101432]: osdmap e706: 8 total, 8 up, 8 in 2024-09-21T10:10:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:43 smithi119 ceph-mon[105288]: osdmap e706: 8 total, 8 up, 8 in 2024-09-21T10:10:43.464 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:43 smithi145 ceph-mon[89359]: osdmap e706: 8 total, 8 up, 8 in 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 1 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:child: image112 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112/880bd22c09ae 2024-09-21T10:10:43.606 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 2 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:43.607 INFO:tasks.workunit.client.0.smithi119.stdout:child: image112 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112/880bd22c09ae 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.608 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:child: image112 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:child: image114 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112/880bd22c09ae 2024-09-21T10:10:43.609 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114/880b30ff7af8 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.610 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:child: image112 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:child: image114 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112/880bd22c09ae 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114/880b30ff7af8 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.611 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:43.612 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 4 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 4 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:child: image112 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:child: image114 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:child: image115 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112/880bd22c09ae 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.613 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114/880b30ff7af8 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115/880bf46afab6 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.614 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:43.615 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:44.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:44 smithi119 ceph-mon[101432]: pgmap v730: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.2 KiB/s wr, 34 op/s 2024-09-21T10:10:44.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:44 smithi119 ceph-mon[101432]: osdmap e707: 8 total, 8 up, 8 in 2024-09-21T10:10:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:44 smithi119 ceph-mon[105288]: pgmap v730: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.2 KiB/s wr, 34 op/s 2024-09-21T10:10:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:44 smithi119 ceph-mon[105288]: osdmap e707: 8 total, 8 up, 8 in 2024-09-21T10:10:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:44 smithi145 ceph-mon[89359]: pgmap v730: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.2 KiB/s wr, 34 op/s 2024-09-21T10:10:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:44 smithi145 ceph-mon[89359]: osdmap e707: 8 total, 8 up, 8 in 2024-09-21T10:10:44.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:10:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:10:44] "GET /metrics HTTP/1.1" 200 79949 "" "Prometheus/2.51.0" 2024-09-21T10:10:45.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:45 smithi145 ceph-mon[89359]: osdmap e708: 8 total, 8 up, 8 in 2024-09-21T10:10:45.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi119-84733-32"} : dispatch 2024-09-21T10:10:45.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:45 smithi145 ceph-mon[89359]: pgmap v733: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 236 KiB/s rd, 8.7 KiB/s wr, 289 op/s 2024-09-21T10:10:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:45 smithi119 ceph-mon[105288]: osdmap e708: 8 total, 8 up, 8 in 2024-09-21T10:10:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi119-84733-32"} : dispatch 2024-09-21T10:10:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:45 smithi119 ceph-mon[105288]: pgmap v733: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 236 KiB/s rd, 8.7 KiB/s wr, 289 op/s 2024-09-21T10:10:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:45 smithi119 ceph-mon[101432]: osdmap e708: 8 total, 8 up, 8 in 2024-09-21T10:10:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi119-84733-32"} : dispatch 2024-09-21T10:10:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:45 smithi119 ceph-mon[101432]: pgmap v733: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 236 KiB/s rd, 8.7 KiB/s wr, 289 op/s 2024-09-21T10:10:46.161 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image112 2024-09-21T10:10:46.161 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:46.161 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.161 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.161 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.161 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 4 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 4 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:child: image112 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:child: image114 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:child: image115 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image112/880bd22c09ae 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114/880b30ff7af8 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115/880bf46afab6 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.162 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.163 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image114 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:child: image114 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:child: image115 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image114/880b30ff7af8 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115/880bf46afab6 2024-09-21T10:10:46.164 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 2 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 2 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:46.165 INFO:tasks.workunit.client.0.smithi119.stdout:child: image115 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-33/image115/880bf46afab6 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:trying to find test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:checking test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 1 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 1 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:child: image113 2024-09-21T10:10:46.166 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-32/image113/880b673cafd1 2024-09-21T10:10:46.167 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:10:46.167 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 0 2024-09-21T10:10:46.167 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:10:46.167 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ListChildrenTiered (16981 ms) 2024-09-21T10:10:46.167 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.LockingPP 2024-09-21T10:10:46.167 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.225 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.LockingPP (63 ms) 2024-09-21T10:10:46.225 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.FlushAio 2024-09-21T10:10:46.225 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.482 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.FlushAio (258 ms) 2024-09-21T10:10:46.482 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.FlushAioPP 2024-09-21T10:10:46.482 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi119-84733-32"}]': finished 2024-09-21T10:10:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:46 smithi145 ceph-mon[89359]: osdmap e709: 8 total, 8 up, 8 in 2024-09-21T10:10:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34"} : dispatch 2024-09-21T10:10:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi119-84733-32"}]': finished 2024-09-21T10:10:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:46 smithi119 ceph-mon[105288]: osdmap e709: 8 total, 8 up, 8 in 2024-09-21T10:10:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34"} : dispatch 2024-09-21T10:10:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-librbd-smithi119-84733-32"}]': finished 2024-09-21T10:10:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:46 smithi119 ceph-mon[101432]: osdmap e709: 8 total, 8 up, 8 in 2024-09-21T10:10:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd tier remove", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34"} : dispatch 2024-09-21T10:10:46.684 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.FlushAioPP (201 ms) 2024-09-21T10:10:46.684 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ZeroLengthWrite 2024-09-21T10:10:46.684 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.707 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ZeroLengthWrite (24 ms) 2024-09-21T10:10:46.707 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ZeroLengthDiscard 2024-09-21T10:10:46.707 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.733 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ZeroLengthDiscard (26 ms) 2024-09-21T10:10:46.733 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ZeroLengthRead 2024-09-21T10:10:46.733 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.748 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ZeroLengthRead (15 ms) 2024-09-21T10:10:46.749 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.LargeCacheRead 2024-09-21T10:10:46.749 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.787 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.LargeCacheRead (39 ms) 2024-09-21T10:10:46.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestPendingAio 2024-09-21T10:10:46.788 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:46.986 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestPendingAio (199 ms) 2024-09-21T10:10:46.987 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyup 2024-09-21T10:10:46.987 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:47.273 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:47 smithi145 ceph-mon[89359]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-21T10:10:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34"}]': finished 2024-09-21T10:10:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:47 smithi145 ceph-mon[89359]: osdmap e710: 8 total, 8 up, 8 in 2024-09-21T10:10:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:47 smithi145 ceph-mon[89359]: pgmap v736: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 8.2 KiB/s wr, 271 op/s 2024-09-21T10:10:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[105288]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-21T10:10:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34"}]': finished 2024-09-21T10:10:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[105288]: osdmap e710: 8 total, 8 up, 8 in 2024-09-21T10:10:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[105288]: pgmap v736: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 8.2 KiB/s wr, 271 op/s 2024-09-21T10:10:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[101432]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-21T10:10:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-librbd-smithi119-84733-32", "tierpool":"test-librbd-smithi119-84733-34"}]': finished 2024-09-21T10:10:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[101432]: osdmap e710: 8 total, 8 up, 8 in 2024-09-21T10:10:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:47 smithi119 ceph-mon[101432]: pgmap v736: 713 pgs: 713 active+clean; 131 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 223 KiB/s rd, 8.2 KiB/s wr, 271 op/s 2024-09-21T10:10:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:48 smithi145 ceph-mon[89359]: osdmap e711: 8 total, 8 up, 8 in 2024-09-21T10:10:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[105288]: osdmap e711: 8 total, 8 up, 8 in 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[101432]: osdmap e711: 8 total, 8 up, 8 in 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:10:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:10:49.127 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyup (2141 ms) 2024-09-21T10:10:49.128 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CompareAndWriteCopyupMismatch 2024-09-21T10:10:49.128 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:49 smithi145 ceph-mon[89359]: osdmap e712: 8 total, 8 up, 8 in 2024-09-21T10:10:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:49 smithi145 ceph-mon[89359]: pgmap v739: 713 pgs: 713 active+clean; 135 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 609 KiB/s rd, 89 KiB/s wr, 822 op/s 2024-09-21T10:10:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:49 smithi145 ceph-mon[89359]: osdmap e713: 8 total, 8 up, 8 in 2024-09-21T10:10:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:49 smithi119 ceph-mon[101432]: osdmap e712: 8 total, 8 up, 8 in 2024-09-21T10:10:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:49 smithi119 ceph-mon[101432]: pgmap v739: 713 pgs: 713 active+clean; 135 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 609 KiB/s rd, 89 KiB/s wr, 822 op/s 2024-09-21T10:10:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:49 smithi119 ceph-mon[101432]: osdmap e713: 8 total, 8 up, 8 in 2024-09-21T10:10:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:49 smithi119 ceph-mon[105288]: osdmap e712: 8 total, 8 up, 8 in 2024-09-21T10:10:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:49 smithi119 ceph-mon[105288]: pgmap v739: 713 pgs: 713 active+clean; 135 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 609 KiB/s rd, 89 KiB/s wr, 822 op/s 2024-09-21T10:10:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:49 smithi119 ceph-mon[105288]: osdmap e713: 8 total, 8 up, 8 in 2024-09-21T10:10:50.179 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:10:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:10:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:10:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:51 smithi119 ceph-mon[101432]: osdmap e714: 8 total, 8 up, 8 in 2024-09-21T10:10:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:51 smithi119 ceph-mon[105288]: osdmap e714: 8 total, 8 up, 8 in 2024-09-21T10:10:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:51 smithi145 ceph-mon[89359]: osdmap e714: 8 total, 8 up, 8 in 2024-09-21T10:10:52.138 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CompareAndWriteCopyupMismatch (3010 ms) 2024-09-21T10:10:52.138 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Flatten 2024-09-21T10:10:52.138 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:52 smithi119 ceph-mon[101432]: pgmap v742: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 737 KiB/s wr, 1.02k op/s 2024-09-21T10:10:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:52 smithi119 ceph-mon[101432]: osdmap e715: 8 total, 8 up, 8 in 2024-09-21T10:10:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:52 smithi119 ceph-mon[105288]: pgmap v742: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 737 KiB/s wr, 1.02k op/s 2024-09-21T10:10:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:52 smithi119 ceph-mon[105288]: osdmap e715: 8 total, 8 up, 8 in 2024-09-21T10:10:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:52 smithi145 ceph-mon[89359]: pgmap v742: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 1.1 MiB/s rd, 737 KiB/s wr, 1.02k op/s 2024-09-21T10:10:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:52 smithi145 ceph-mon[89359]: osdmap e715: 8 total, 8 up, 8 in 2024-09-21T10:10:53.207 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Flatten (1070 ms) 2024-09-21T10:10:53.207 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Sparsify 2024-09-21T10:10:53.208 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:53.238 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 32768 2024-09-21T10:10:53.239 INFO:tasks.workunit.client.0.smithi119.stdout:read: 32768 2024-09-21T10:10:53.239 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Sparsify (31 ms) 2024-09-21T10:10:53.239 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SparsifyPP 2024-09-21T10:10:53.239 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:53.269 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SparsifyPP (31 ms) 2024-09-21T10:10:53.269 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapshotLimit 2024-09-21T10:10:53.269 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:53 smithi119 ceph-mon[101432]: osdmap e716: 8 total, 8 up, 8 in 2024-09-21T10:10:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:53 smithi119 ceph-mon[105288]: osdmap e716: 8 total, 8 up, 8 in 2024-09-21T10:10:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:53 smithi145 ceph-mon[89359]: osdmap e716: 8 total, 8 up, 8 in 2024-09-21T10:10:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-mon[105288]: pgmap v745: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 572 KiB/s rd, 649 KiB/s wr, 219 op/s 2024-09-21T10:10:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-mon[105288]: osdmap e717: 8 total, 8 up, 8 in 2024-09-21T10:10:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-mon[105288]: osdmap e718: 8 total, 8 up, 8 in 2024-09-21T10:10:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-mon[101432]: pgmap v745: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 572 KiB/s rd, 649 KiB/s wr, 219 op/s 2024-09-21T10:10:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-mon[101432]: osdmap e717: 8 total, 8 up, 8 in 2024-09-21T10:10:54.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-mon[101432]: osdmap e718: 8 total, 8 up, 8 in 2024-09-21T10:10:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:54 smithi145 ceph-mon[89359]: pgmap v745: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 572 KiB/s rd, 649 KiB/s wr, 219 op/s 2024-09-21T10:10:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:54 smithi145 ceph-mon[89359]: osdmap e717: 8 total, 8 up, 8 in 2024-09-21T10:10:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:54 smithi145 ceph-mon[89359]: osdmap e718: 8 total, 8 up, 8 in 2024-09-21T10:10:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:10:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:10:54] "GET /metrics HTTP/1.1" 200 82401 "" "Prometheus/2.51.0" 2024-09-21T10:10:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:56 smithi145 ceph-mon[89359]: pgmap v748: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 217 KiB/s rd, 27 KiB/s wr, 284 op/s 2024-09-21T10:10:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:56 smithi145 ceph-mon[89359]: osdmap e719: 8 total, 8 up, 8 in 2024-09-21T10:10:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:56 smithi119 ceph-mon[105288]: pgmap v748: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 217 KiB/s rd, 27 KiB/s wr, 284 op/s 2024-09-21T10:10:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:56 smithi119 ceph-mon[105288]: osdmap e719: 8 total, 8 up, 8 in 2024-09-21T10:10:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:56 smithi119 ceph-mon[101432]: pgmap v748: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 217 KiB/s rd, 27 KiB/s wr, 284 op/s 2024-09-21T10:10:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:56 smithi119 ceph-mon[101432]: osdmap e719: 8 total, 8 up, 8 in 2024-09-21T10:10:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:57 smithi145 ceph-mon[89359]: osdmap e720: 8 total, 8 up, 8 in 2024-09-21T10:10:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:57 smithi145 ceph-mon[89359]: pgmap v751: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 272 KiB/s rd, 46 KiB/s wr, 344 op/s 2024-09-21T10:10:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:57 smithi119 ceph-mon[105288]: osdmap e720: 8 total, 8 up, 8 in 2024-09-21T10:10:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:57 smithi119 ceph-mon[105288]: pgmap v751: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 272 KiB/s rd, 46 KiB/s wr, 344 op/s 2024-09-21T10:10:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:57 smithi119 ceph-mon[101432]: osdmap e720: 8 total, 8 up, 8 in 2024-09-21T10:10:57.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:57 smithi119 ceph-mon[101432]: pgmap v751: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 272 KiB/s rd, 46 KiB/s wr, 344 op/s 2024-09-21T10:10:58.178 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapshotLimit (4909 ms) 2024-09-21T10:10:58.179 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapshotLimitPP 2024-09-21T10:10:58.179 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:10:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:58 smithi145 ceph-mon[89359]: osdmap e721: 8 total, 8 up, 8 in 2024-09-21T10:10:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:58 smithi119 ceph-mon[101432]: osdmap e721: 8 total, 8 up, 8 in 2024-09-21T10:10:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:58 smithi119 ceph-mon[105288]: osdmap e721: 8 total, 8 up, 8 in 2024-09-21T10:10:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:59 smithi145 ceph-mon[89359]: osdmap e722: 8 total, 8 up, 8 in 2024-09-21T10:10:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:10:59 smithi145 ceph-mon[89359]: pgmap v754: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 20 KiB/s wr, 76 op/s 2024-09-21T10:10:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:59 smithi119 ceph-mon[101432]: osdmap e722: 8 total, 8 up, 8 in 2024-09-21T10:10:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:10:59 smithi119 ceph-mon[101432]: pgmap v754: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 20 KiB/s wr, 76 op/s 2024-09-21T10:10:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:59 smithi119 ceph-mon[105288]: osdmap e722: 8 total, 8 up, 8 in 2024-09-21T10:10:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:10:59 smithi119 ceph-mon[105288]: pgmap v754: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 66 KiB/s rd, 20 KiB/s wr, 76 op/s 2024-09-21T10:11:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:00 smithi145 ceph-mon[89359]: osdmap e723: 8 total, 8 up, 8 in 2024-09-21T10:11:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:11:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:11:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:00 smithi119 ceph-mon[105288]: osdmap e723: 8 total, 8 up, 8 in 2024-09-21T10:11:00.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:00 smithi119 ceph-mon[101432]: osdmap e723: 8 total, 8 up, 8 in 2024-09-21T10:11:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:01 smithi145 ceph-mon[89359]: osdmap e724: 8 total, 8 up, 8 in 2024-09-21T10:11:01.552 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:01 smithi145 ceph-mon[89359]: pgmap v757: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 22 op/s 2024-09-21T10:11:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:01 smithi119 ceph-mon[101432]: osdmap e724: 8 total, 8 up, 8 in 2024-09-21T10:11:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:01 smithi119 ceph-mon[101432]: pgmap v757: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 22 op/s 2024-09-21T10:11:01.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:01 smithi119 ceph-mon[105288]: osdmap e724: 8 total, 8 up, 8 in 2024-09-21T10:11:01.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:01 smithi119 ceph-mon[105288]: pgmap v757: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 22 op/s 2024-09-21T10:11:02.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:02 smithi119 ceph-mon[105288]: osdmap e725: 8 total, 8 up, 8 in 2024-09-21T10:11:02.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:02 smithi119 ceph-mon[101432]: osdmap e725: 8 total, 8 up, 8 in 2024-09-21T10:11:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:02 smithi145 ceph-mon[89359]: osdmap e725: 8 total, 8 up, 8 in 2024-09-21T10:11:03.218 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapshotLimitPP (5039 ms) 2024-09-21T10:11:03.218 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RebuildObjectMapViaLockOwner 2024-09-21T10:11:03.218 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:03.266 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:03.267 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RebuildObjectMapViaLockOwner (49 ms) 2024-09-21T10:11:03.267 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RenameViaLockOwner 2024-09-21T10:11:03.267 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:03.334 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RenameViaLockOwner (68 ms) 2024-09-21T10:11:03.334 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapCreateViaLockOwner 2024-09-21T10:11:03.334 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:03 smithi145 ceph-mon[89359]: osdmap e726: 8 total, 8 up, 8 in 2024-09-21T10:11:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:03 smithi145 ceph-mon[89359]: pgmap v760: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-21T10:11:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:03 smithi119 ceph-mon[101432]: osdmap e726: 8 total, 8 up, 8 in 2024-09-21T10:11:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:03 smithi119 ceph-mon[101432]: pgmap v760: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-21T10:11:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:03 smithi119 ceph-mon[105288]: osdmap e726: 8 total, 8 up, 8 in 2024-09-21T10:11:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:03 smithi119 ceph-mon[105288]: pgmap v760: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 6 op/s 2024-09-21T10:11:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:04.124 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapCreateViaLockOwner (790 ms) 2024-09-21T10:11:04.124 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapRemoveViaLockOwner 2024-09-21T10:11:04.124 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:04.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:04 smithi119 ceph-mon[101432]: osdmap e727: 8 total, 8 up, 8 in 2024-09-21T10:11:04.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:04 smithi119 ceph-mon[101432]: osdmap e728: 8 total, 8 up, 8 in 2024-09-21T10:11:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:04 smithi119 ceph-mon[105288]: osdmap e727: 8 total, 8 up, 8 in 2024-09-21T10:11:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:04 smithi119 ceph-mon[105288]: osdmap e728: 8 total, 8 up, 8 in 2024-09-21T10:11:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:04 smithi145 ceph-mon[89359]: osdmap e727: 8 total, 8 up, 8 in 2024-09-21T10:11:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:04 smithi145 ceph-mon[89359]: osdmap e728: 8 total, 8 up, 8 in 2024-09-21T10:11:04.869 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:11:04] "GET /metrics HTTP/1.1" 200 82402 "" "Prometheus/2.51.0" 2024-09-21T10:11:06.129 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapRemoveViaLockOwner (2005 ms) 2024-09-21T10:11:06.130 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.UpdateFeaturesViaLockOwner 2024-09-21T10:11:06.130 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:06.180 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.UpdateFeaturesViaLockOwner (50 ms) 2024-09-21T10:11:06.180 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.EnableJournalingViaLockOwner 2024-09-21T10:11:06.180 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:11:06.180 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.EnableJournalingViaLockOwner (0 ms) 2024-09-21T10:11:06.180 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapRemove2 2024-09-21T10:11:06.180 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:06 smithi119 ceph-mon[101432]: pgmap v763: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 7.5 KiB/s wr, 117 op/s 2024-09-21T10:11:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:06 smithi119 ceph-mon[101432]: osdmap e729: 8 total, 8 up, 8 in 2024-09-21T10:11:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:06 smithi119 ceph-mon[105288]: pgmap v763: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 7.5 KiB/s wr, 117 op/s 2024-09-21T10:11:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:06 smithi119 ceph-mon[105288]: osdmap e729: 8 total, 8 up, 8 in 2024-09-21T10:11:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:06 smithi145 ceph-mon[89359]: pgmap v763: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 88 KiB/s rd, 7.5 KiB/s wr, 117 op/s 2024-09-21T10:11:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:06 smithi145 ceph-mon[89359]: osdmap e729: 8 total, 8 up, 8 in 2024-09-21T10:11:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:07 smithi119 ceph-mon[105288]: osdmap e730: 8 total, 8 up, 8 in 2024-09-21T10:11:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:07 smithi119 ceph-mon[101432]: osdmap e730: 8 total, 8 up, 8 in 2024-09-21T10:11:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:07 smithi145 ceph-mon[89359]: osdmap e730: 8 total, 8 up, 8 in 2024-09-21T10:11:08.133 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:11:08.133 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapRemove2 (1954 ms) 2024-09-21T10:11:08.133 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapRenameViaLockOwner 2024-09-21T10:11:08.133 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:08 smithi119 ceph-mon[105288]: pgmap v766: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 10 KiB/s wr, 152 op/s 2024-09-21T10:11:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:08 smithi119 ceph-mon[105288]: osdmap e731: 8 total, 8 up, 8 in 2024-09-21T10:11:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:08 smithi119 ceph-mon[101432]: pgmap v766: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 10 KiB/s wr, 152 op/s 2024-09-21T10:11:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:08 smithi119 ceph-mon[101432]: osdmap e731: 8 total, 8 up, 8 in 2024-09-21T10:11:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:08 smithi145 ceph-mon[89359]: pgmap v766: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 111 KiB/s rd, 10 KiB/s wr, 152 op/s 2024-09-21T10:11:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:08 smithi145 ceph-mon[89359]: osdmap e731: 8 total, 8 up, 8 in 2024-09-21T10:11:09.134 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapRenameViaLockOwner (1000 ms) 2024-09-21T10:11:09.134 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapProtectViaLockOwner 2024-09-21T10:11:09.134 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:09 smithi145 ceph-mon[89359]: osdmap e732: 8 total, 8 up, 8 in 2024-09-21T10:11:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:09 smithi145 ceph-mon[89359]: osdmap e733: 8 total, 8 up, 8 in 2024-09-21T10:11:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:09 smithi119 ceph-mon[101432]: osdmap e732: 8 total, 8 up, 8 in 2024-09-21T10:11:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:09 smithi119 ceph-mon[101432]: osdmap e733: 8 total, 8 up, 8 in 2024-09-21T10:11:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:09 smithi119 ceph-mon[105288]: osdmap e732: 8 total, 8 up, 8 in 2024-09-21T10:11:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:09 smithi119 ceph-mon[105288]: osdmap e733: 8 total, 8 up, 8 in 2024-09-21T10:11:10.174 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapProtectViaLockOwner (1041 ms) 2024-09-21T10:11:10.174 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapUnprotectViaLockOwner 2024-09-21T10:11:10.175 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:10 smithi145 ceph-mon[89359]: pgmap v769: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 125 KiB/s rd, 11 KiB/s wr, 175 op/s 2024-09-21T10:11:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:11:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:11:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:10 smithi119 ceph-mon[101432]: pgmap v769: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 125 KiB/s rd, 11 KiB/s wr, 175 op/s 2024-09-21T10:11:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:10 smithi119 ceph-mon[105288]: pgmap v769: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 125 KiB/s rd, 11 KiB/s wr, 175 op/s 2024-09-21T10:11:11.238 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapUnprotectViaLockOwner (1063 ms) 2024-09-21T10:11:11.238 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.FlattenViaLockOwner 2024-09-21T10:11:11.238 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:11 smithi145 ceph-mon[89359]: osdmap e734: 8 total, 8 up, 8 in 2024-09-21T10:11:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:11 smithi145 ceph-mon[89359]: pgmap v772: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 133 KiB/s rd, 10 KiB/s wr, 181 op/s 2024-09-21T10:11:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:11 smithi119 ceph-mon[105288]: osdmap e734: 8 total, 8 up, 8 in 2024-09-21T10:11:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:11 smithi119 ceph-mon[105288]: pgmap v772: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 133 KiB/s rd, 10 KiB/s wr, 181 op/s 2024-09-21T10:11:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:11 smithi119 ceph-mon[101432]: osdmap e734: 8 total, 8 up, 8 in 2024-09-21T10:11:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:11 smithi119 ceph-mon[101432]: pgmap v772: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 133 KiB/s rd, 10 KiB/s wr, 181 op/s 2024-09-21T10:11:12.255 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.FlattenViaLockOwner (1018 ms) 2024-09-21T10:11:12.257 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ResizeViaLockOwner 2024-09-21T10:11:12.258 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:12.291 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ResizeViaLockOwner (36 ms) 2024-09-21T10:11:12.291 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SparsifyViaLockOwner 2024-09-21T10:11:12.291 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:12.325 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SparsifyViaLockOwner (33 ms) 2024-09-21T10:11:12.325 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ObjectMapConsistentSnap 2024-09-21T10:11:12.325 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:12.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:12 smithi145 ceph-mon[89359]: osdmap e735: 8 total, 8 up, 8 in 2024-09-21T10:11:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:12 smithi119 ceph-mon[101432]: osdmap e735: 8 total, 8 up, 8 in 2024-09-21T10:11:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:12 smithi119 ceph-mon[105288]: osdmap e735: 8 total, 8 up, 8 in 2024-09-21T10:11:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:13 smithi145 ceph-mon[89359]: osdmap e736: 8 total, 8 up, 8 in 2024-09-21T10:11:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:13 smithi145 ceph-mon[89359]: pgmap v775: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 2.0 KiB/s wr, 40 op/s 2024-09-21T10:11:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:13 smithi119 ceph-mon[101432]: osdmap e736: 8 total, 8 up, 8 in 2024-09-21T10:11:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:13 smithi119 ceph-mon[101432]: pgmap v775: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 2.0 KiB/s wr, 40 op/s 2024-09-21T10:11:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:13 smithi119 ceph-mon[105288]: osdmap e736: 8 total, 8 up, 8 in 2024-09-21T10:11:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:13 smithi119 ceph-mon[105288]: pgmap v775: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 31 KiB/s rd, 2.0 KiB/s wr, 40 op/s 2024-09-21T10:11:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:14 smithi119 ceph-mon[101432]: osdmap e737: 8 total, 8 up, 8 in 2024-09-21T10:11:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:14 smithi119 ceph-mon[101432]: osdmap e738: 8 total, 8 up, 8 in 2024-09-21T10:11:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:14 smithi119 ceph-mon[105288]: osdmap e737: 8 total, 8 up, 8 in 2024-09-21T10:11:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:14 smithi119 ceph-mon[105288]: osdmap e738: 8 total, 8 up, 8 in 2024-09-21T10:11:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:14 smithi145 ceph-mon[89359]: osdmap e737: 8 total, 8 up, 8 in 2024-09-21T10:11:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:14 smithi145 ceph-mon[89359]: osdmap e738: 8 total, 8 up, 8 in 2024-09-21T10:11:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:11:14] "GET /metrics HTTP/1.1" 200 82402 "" "Prometheus/2.51.0" 2024-09-21T10:11:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:16 smithi119 ceph-mon[105288]: pgmap v778: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 201 KiB/s rd, 11 KiB/s wr, 265 op/s 2024-09-21T10:11:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:16 smithi119 ceph-mon[105288]: osdmap e739: 8 total, 8 up, 8 in 2024-09-21T10:11:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:16 smithi119 ceph-mon[101432]: pgmap v778: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 201 KiB/s rd, 11 KiB/s wr, 265 op/s 2024-09-21T10:11:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:16 smithi119 ceph-mon[101432]: osdmap e739: 8 total, 8 up, 8 in 2024-09-21T10:11:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:16 smithi145 ceph-mon[89359]: pgmap v778: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 201 KiB/s rd, 11 KiB/s wr, 265 op/s 2024-09-21T10:11:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:16 smithi145 ceph-mon[89359]: osdmap e739: 8 total, 8 up, 8 in 2024-09-21T10:11:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:17 smithi119 ceph-mon[105288]: osdmap e740: 8 total, 8 up, 8 in 2024-09-21T10:11:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:17 smithi119 ceph-mon[101432]: osdmap e740: 8 total, 8 up, 8 in 2024-09-21T10:11:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:17.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:17.428 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:17 smithi145 ceph-mon[89359]: osdmap e740: 8 total, 8 up, 8 in 2024-09-21T10:11:17.428 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:17.428 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:18 smithi119 ceph-mon[101432]: pgmap v781: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 231 KiB/s rd, 14 KiB/s wr, 306 op/s 2024-09-21T10:11:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:18 smithi119 ceph-mon[101432]: osdmap e741: 8 total, 8 up, 8 in 2024-09-21T10:11:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:18 smithi119 ceph-mon[105288]: pgmap v781: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 231 KiB/s rd, 14 KiB/s wr, 306 op/s 2024-09-21T10:11:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:18 smithi119 ceph-mon[105288]: osdmap e741: 8 total, 8 up, 8 in 2024-09-21T10:11:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:18.432 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:18 smithi145 ceph-mon[89359]: pgmap v781: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 231 KiB/s rd, 14 KiB/s wr, 306 op/s 2024-09-21T10:11:18.432 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:18 smithi145 ceph-mon[89359]: osdmap e741: 8 total, 8 up, 8 in 2024-09-21T10:11:18.432 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:19 smithi145 ceph-mon[89359]: osdmap e742: 8 total, 8 up, 8 in 2024-09-21T10:11:19.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:19 smithi119 ceph-mon[101432]: osdmap e742: 8 total, 8 up, 8 in 2024-09-21T10:11:19.476 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:19 smithi119 ceph-mon[105288]: osdmap e742: 8 total, 8 up, 8 in 2024-09-21T10:11:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:20 smithi145 ceph-mon[89359]: pgmap v784: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 KiB/s wr, 48 op/s 2024-09-21T10:11:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:20 smithi145 ceph-mon[89359]: osdmap e743: 8 total, 8 up, 8 in 2024-09-21T10:11:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:11:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:11:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[101432]: pgmap v784: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 KiB/s wr, 48 op/s 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[101432]: osdmap e743: 8 total, 8 up, 8 in 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[105288]: pgmap v784: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 3.7 KiB/s wr, 48 op/s 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[105288]: osdmap e743: 8 total, 8 up, 8 in 2024-09-21T10:11:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:20.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: osdmap e744: 8 total, 8 up, 8 in 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:21 smithi145 ceph-mon[89359]: pgmap v787: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 3.5 KiB/s wr, 89 op/s 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: osdmap e744: 8 total, 8 up, 8 in 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[101432]: pgmap v787: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 3.5 KiB/s wr, 89 op/s 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: osdmap e744: 8 total, 8 up, 8 in 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:21.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:21 smithi119 ceph-mon[105288]: pgmap v787: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 3.5 KiB/s wr, 89 op/s 2024-09-21T10:11:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:22 smithi145 ceph-mon[89359]: osdmap e745: 8 total, 8 up, 8 in 2024-09-21T10:11:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:22 smithi119 ceph-mon[105288]: osdmap e745: 8 total, 8 up, 8 in 2024-09-21T10:11:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:22 smithi119 ceph-mon[101432]: osdmap e745: 8 total, 8 up, 8 in 2024-09-21T10:11:22.703 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ObjectMapConsistentSnap (10378 ms) 2024-09-21T10:11:22.703 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Metadata 2024-09-21T10:11:22.703 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:23.263 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Metadata (561 ms) 2024-09-21T10:11:23.263 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.MetadataPP 2024-09-21T10:11:23.263 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:23 smithi145 ceph-mon[89359]: osdmap e746: 8 total, 8 up, 8 in 2024-09-21T10:11:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:23 smithi145 ceph-mon[89359]: pgmap v790: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-21T10:11:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:23 smithi119 ceph-mon[105288]: osdmap e746: 8 total, 8 up, 8 in 2024-09-21T10:11:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:23 smithi119 ceph-mon[105288]: pgmap v790: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-21T10:11:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:23 smithi119 ceph-mon[101432]: osdmap e746: 8 total, 8 up, 8 in 2024-09-21T10:11:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:23 smithi119 ceph-mon[101432]: pgmap v790: 713 pgs: 713 active+clean; 139 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 2.5 KiB/s wr, 82 op/s 2024-09-21T10:11:24.260 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.MetadataPP (997 ms) 2024-09-21T10:11:24.260 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.UpdateFeatures 2024-09-21T10:11:24.260 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:24.350 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.UpdateFeatures (89 ms) 2024-09-21T10:11:24.350 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.FeaturesBitmaskString 2024-09-21T10:11:24.350 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.FeaturesBitmaskString (0 ms) 2024-09-21T10:11:24.350 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RebuildObjectMap 2024-09-21T10:11:24.351 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:24.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:24 smithi119 ceph-mon[105288]: osdmap e747: 8 total, 8 up, 8 in 2024-09-21T10:11:24.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:24 smithi119 ceph-mon[101432]: osdmap e747: 8 total, 8 up, 8 in 2024-09-21T10:11:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:24 smithi145 ceph-mon[89359]: osdmap e747: 8 total, 8 up, 8 in 2024-09-21T10:11:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:11:24] "GET /metrics HTTP/1.1" 200 82400 "" "Prometheus/2.51.0" 2024-09-21T10:11:25.234 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:25.234 INFO:tasks.workunit.client.0.smithi119.stdout:25.00% done 2024-09-21T10:11:25.234 INFO:tasks.workunit.client.0.smithi119.stdout:50.00% done 2024-09-21T10:11:25.234 INFO:tasks.workunit.client.0.smithi119.stdout:75.00% done 2024-09-21T10:11:25.234 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RebuildObjectMap (885 ms) 2024-09-21T10:11:25.235 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.RebuildNewObjectMap 2024-09-21T10:11:25.262 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:25.262 INFO:tasks.workunit.client.0.smithi119.stdout:25.00% done 2024-09-21T10:11:25.263 INFO:tasks.workunit.client.0.smithi119.stdout:50.00% done 2024-09-21T10:11:25.263 INFO:tasks.workunit.client.0.smithi119.stdout:75.00% done 2024-09-21T10:11:25.263 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.RebuildNewObjectMap (28 ms) 2024-09-21T10:11:25.263 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CheckObjectMap 2024-09-21T10:11:25.263 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:25 smithi145 ceph-mon[89359]: osdmap e748: 8 total, 8 up, 8 in 2024-09-21T10:11:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:25 smithi145 ceph-mon[89359]: pgmap v793: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 385 KiB/s rd, 186 KiB/s wr, 652 op/s 2024-09-21T10:11:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:25 smithi119 ceph-mon[101432]: osdmap e748: 8 total, 8 up, 8 in 2024-09-21T10:11:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:25 smithi119 ceph-mon[101432]: pgmap v793: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 385 KiB/s rd, 186 KiB/s wr, 652 op/s 2024-09-21T10:11:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:25 smithi119 ceph-mon[105288]: osdmap e748: 8 total, 8 up, 8 in 2024-09-21T10:11:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:25 smithi119 ceph-mon[105288]: pgmap v793: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 385 KiB/s rd, 186 KiB/s wr, 652 op/s 2024-09-21T10:11:26.252 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:26.252 INFO:tasks.workunit.client.0.smithi119.stdout:25.00% done 2024-09-21T10:11:26.252 INFO:tasks.workunit.client.0.smithi119.stdout:50.00% done 2024-09-21T10:11:26.253 INFO:tasks.workunit.client.0.smithi119.stdout:75.00% done 2024-09-21T10:11:26.253 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CheckObjectMap (990 ms) 2024-09-21T10:11:26.253 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.BlockingAIO 2024-09-21T10:11:26.253 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:26.277 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.BlockingAIO (25 ms) 2024-09-21T10:11:26.277 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ExclusiveLockTransition 2024-09-21T10:11:26.277 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:26.394 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ExclusiveLockTransition (117 ms) 2024-09-21T10:11:26.395 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ExclusiveLockReadTransition 2024-09-21T10:11:26.395 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:11:26.395 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ExclusiveLockReadTransition (0 ms) 2024-09-21T10:11:26.395 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CacheMayCopyOnWrite 2024-09-21T10:11:26.395 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:26 smithi145 ceph-mon[89359]: osdmap e749: 8 total, 8 up, 8 in 2024-09-21T10:11:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:26 smithi119 ceph-mon[101432]: osdmap e749: 8 total, 8 up, 8 in 2024-09-21T10:11:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:26 smithi119 ceph-mon[105288]: osdmap e749: 8 total, 8 up, 8 in 2024-09-21T10:11:27.265 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CacheMayCopyOnWrite (871 ms) 2024-09-21T10:11:27.266 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot 2024-09-21T10:11:27.266 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:27 smithi145 ceph-mon[89359]: osdmap e750: 8 total, 8 up, 8 in 2024-09-21T10:11:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:27 smithi145 ceph-mon[89359]: pgmap v796: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 494 KiB/s rd, 203 KiB/s wr, 808 op/s 2024-09-21T10:11:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:27 smithi119 ceph-mon[105288]: osdmap e750: 8 total, 8 up, 8 in 2024-09-21T10:11:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:27 smithi119 ceph-mon[105288]: pgmap v796: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 494 KiB/s rd, 203 KiB/s wr, 808 op/s 2024-09-21T10:11:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:27 smithi119 ceph-mon[101432]: osdmap e750: 8 total, 8 up, 8 in 2024-09-21T10:11:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:27 smithi119 ceph-mon[101432]: pgmap v796: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 494 KiB/s rd, 203 KiB/s wr, 808 op/s 2024-09-21T10:11:28.230 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.FlushEmptyOpsOnExternalSnapshot (965 ms) 2024-09-21T10:11:28.231 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestImageOptions 2024-09-21T10:11:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:28 smithi145 ceph-mon[89359]: osdmap e751: 8 total, 8 up, 8 in 2024-09-21T10:11:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:28 smithi119 ceph-mon[105288]: osdmap e751: 8 total, 8 up, 8 in 2024-09-21T10:11:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:28 smithi119 ceph-mon[101432]: osdmap e751: 8 total, 8 up, 8 in 2024-09-21T10:11:29.324 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:29.324 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:11:29.324 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestImageOptions (1093 ms) 2024-09-21T10:11:29.324 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestImageOptionsPP 2024-09-21T10:11:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:29 smithi145 ceph-mon[89359]: osdmap e752: 8 total, 8 up, 8 in 2024-09-21T10:11:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:29 smithi145 ceph-mon[89359]: pgmap v799: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 469 KiB/s rd, 115 KiB/s wr, 631 op/s 2024-09-21T10:11:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:29 smithi119 ceph-mon[105288]: osdmap e752: 8 total, 8 up, 8 in 2024-09-21T10:11:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:29 smithi119 ceph-mon[105288]: pgmap v799: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 469 KiB/s rd, 115 KiB/s wr, 631 op/s 2024-09-21T10:11:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:29 smithi119 ceph-mon[101432]: osdmap e752: 8 total, 8 up, 8 in 2024-09-21T10:11:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:29 smithi119 ceph-mon[101432]: pgmap v799: 713 pgs: 713 active+clean; 139 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 469 KiB/s rd, 115 KiB/s wr, 631 op/s 2024-09-21T10:11:30.311 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:30.311 INFO:tasks.workunit.client.0.smithi119.stdout:100.00% done 2024-09-21T10:11:30.311 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestImageOptionsPP (986 ms) 2024-09-21T10:11:30.311 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.EventSocketPipe 2024-09-21T10:11:30.312 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.EventSocketPipe (1 ms) 2024-09-21T10:11:30.312 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.EventSocketEventfd 2024-09-21T10:11:30.312 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.EventSocketEventfd (0 ms) 2024-09-21T10:11:30.312 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ImagePollIO 2024-09-21T10:11:30.312 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:wrote: 512 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:11:30.351 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:11:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:started write 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:write completion cb called! 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 0 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:finished write 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:11:30.353 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:created completion 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:started read 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:read completion cb called! 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:return value is: 512 2024-09-21T10:11:30.354 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ImagePollIO (40 ms) 2024-09-21T10:11:30.355 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Mirror 2024-09-21T10:11:30.355 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:30.363 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:30 smithi145 ceph-mon[89359]: osdmap e753: 8 total, 8 up, 8 in 2024-09-21T10:11:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:11:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:11:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:30 smithi119 ceph-mon[101432]: osdmap e753: 8 total, 8 up, 8 in 2024-09-21T10:11:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:30 smithi119 ceph-mon[105288]: osdmap e753: 8 total, 8 up, 8 in 2024-09-21T10:11:31.373 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Mirror (1022 ms) 2024-09-21T10:11:31.373 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.MirrorPeerAttributes 2024-09-21T10:11:31.390 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.MirrorPeerAttributes (17 ms) 2024-09-21T10:11:31.390 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.CreateWithMirrorEnabled 2024-09-21T10:11:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:31 smithi145 ceph-mon[89359]: osdmap e754: 8 total, 8 up, 8 in 2024-09-21T10:11:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:31 smithi145 ceph-mon[89359]: pgmap v802: 713 pgs: 713 active+clean; 140 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 501 KiB/s rd, 258 KiB/s wr, 651 op/s 2024-09-21T10:11:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:31 smithi119 ceph-mon[105288]: osdmap e754: 8 total, 8 up, 8 in 2024-09-21T10:11:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:31 smithi119 ceph-mon[105288]: pgmap v802: 713 pgs: 713 active+clean; 140 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 501 KiB/s rd, 258 KiB/s wr, 651 op/s 2024-09-21T10:11:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:31 smithi119 ceph-mon[101432]: osdmap e754: 8 total, 8 up, 8 in 2024-09-21T10:11:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:31 smithi119 ceph-mon[101432]: pgmap v802: 713 pgs: 713 active+clean; 140 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 501 KiB/s rd, 258 KiB/s wr, 651 op/s 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: osdmap e755: 8 total, 8 up, 8 in 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b124b5f8-ba08-44f3-9c79-8a22b0e15ee7"} : dispatch 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b124b5f8-ba08-44f3-9c79-8a22b0e15ee7"}]': finished 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/f10b6c4b-46c3-44e7-8977-18ca4b375f2c"} : dispatch 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/f10b6c4b-46c3-44e7-8977-18ca4b375f2c"}]': finished 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/1dc594cd-d34e-4782-8839-372c3bc2b82e"} : dispatch 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/1dc594cd-d34e-4782-8839-372c3bc2b82e"}]': finished 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"}]': finished 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: osdmap e755: 8 total, 8 up, 8 in 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b124b5f8-ba08-44f3-9c79-8a22b0e15ee7"} : dispatch 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b124b5f8-ba08-44f3-9c79-8a22b0e15ee7"}]': finished 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/f10b6c4b-46c3-44e7-8977-18ca4b375f2c"} : dispatch 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/f10b6c4b-46c3-44e7-8977-18ca4b375f2c"}]': finished 2024-09-21T10:11:32.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/1dc594cd-d34e-4782-8839-372c3bc2b82e"} : dispatch 2024-09-21T10:11:32.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/1dc594cd-d34e-4782-8839-372c3bc2b82e"}]': finished 2024-09-21T10:11:32.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' 2024-09-21T10:11:32.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"}]': finished 2024-09-21T10:11:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: osdmap e755: 8 total, 8 up, 8 in 2024-09-21T10:11:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"} : dispatch 2024-09-21T10:11:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/uuid4"}]': finished 2024-09-21T10:11:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b124b5f8-ba08-44f3-9c79-8a22b0e15ee7"} : dispatch 2024-09-21T10:11:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/b124b5f8-ba08-44f3-9c79-8a22b0e15ee7"}]': finished 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/f10b6c4b-46c3-44e7-8977-18ca4b375f2c"} : dispatch 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/f10b6c4b-46c3-44e7-8977-18ca4b375f2c"}]': finished 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/1dc594cd-d34e-4782-8839-372c3bc2b82e"} : dispatch 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/1dc594cd-d34e-4782-8839-372c3bc2b82e"}]': finished 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key get", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd={"prefix": "config-key rm", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"} : dispatch 2024-09-21T10:11:32.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/551556084' entity='client.admin' cmd='[{"prefix": "config-key rm", "key": "rbd/mirror/peer/19/62d2df77-b6aa-4346-9411-ce51f0da9775"}]': finished 2024-09-21T10:11:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:33 smithi145 ceph-mon[89359]: osdmap e756: 8 total, 8 up, 8 in 2024-09-21T10:11:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:33 smithi145 ceph-mon[89359]: pgmap v805: 713 pgs: 713 active+clean; 140 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 141 KiB/s rd, 160 KiB/s wr, 176 op/s 2024-09-21T10:11:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:33 smithi119 ceph-mon[101432]: osdmap e756: 8 total, 8 up, 8 in 2024-09-21T10:11:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:33 smithi119 ceph-mon[101432]: pgmap v805: 713 pgs: 713 active+clean; 140 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 141 KiB/s rd, 160 KiB/s wr, 176 op/s 2024-09-21T10:11:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:33 smithi119 ceph-mon[105288]: osdmap e756: 8 total, 8 up, 8 in 2024-09-21T10:11:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:33 smithi119 ceph-mon[105288]: pgmap v805: 713 pgs: 713 active+clean; 140 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 141 KiB/s rd, 160 KiB/s wr, 176 op/s 2024-09-21T10:11:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: osdmap e757: 8 total, 8 up, 8 in 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.e", "id": [2, 4]} : dispatch 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1f", "id": [2, 4]} : dispatch 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]} : dispatch 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.18", "id": [6, 3]} : dispatch 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]} : dispatch 2024-09-21T10:11:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.e", "id": [0, 3]} : dispatch 2024-09-21T10:11:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]} : dispatch 2024-09-21T10:11:34.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.14", "id": [7, 1]} : dispatch 2024-09-21T10:11:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:11:34] "GET /metrics HTTP/1.1" 200 82404 "" "Prometheus/2.51.0" 2024-09-21T10:11:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: osdmap e757: 8 total, 8 up, 8 in 2024-09-21T10:11:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.e", "id": [2, 4]} : dispatch 2024-09-21T10:11:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1f", "id": [2, 4]} : dispatch 2024-09-21T10:11:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]} : dispatch 2024-09-21T10:11:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.18", "id": [6, 3]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.e", "id": [0, 3]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.14", "id": [7, 1]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: osdmap e757: 8 total, 8 up, 8 in 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.e", "id": [2, 4]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1f", "id": [2, 4]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.18", "id": [6, 3]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.e", "id": [0, 3]} : dispatch 2024-09-21T10:11:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]} : dispatch 2024-09-21T10:11:34.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.14", "id": [7, 1]} : dispatch 2024-09-21T10:11:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.e", "id": [2, 4]}]': finished 2024-09-21T10:11:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1f", "id": [2, 4]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.18", "id": [6, 3]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.e", "id": [0, 3]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.14", "id": [7, 1]}]': finished 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: osdmap e758: 8 total, 8 up, 8 in 2024-09-21T10:11:35.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:35 smithi145 ceph-mon[89359]: pgmap v808: 713 pgs: 713 active+clean; 147 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 179 KiB/s rd, 1.9 MiB/s wr, 252 op/s 2024-09-21T10:11:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.e", "id": [2, 4]}]': finished 2024-09-21T10:11:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1f", "id": [2, 4]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.18", "id": [6, 3]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.e", "id": [0, 3]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.14", "id": [7, 1]}]': finished 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: osdmap e758: 8 total, 8 up, 8 in 2024-09-21T10:11:35.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[101432]: pgmap v808: 713 pgs: 713 active+clean; 147 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 179 KiB/s rd, 1.9 MiB/s wr, 252 op/s 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.e", "id": [2, 4]}]': finished 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "33.1f", "id": [2, 4]}]': finished 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "35.17", "id": [4, 2]}]': finished 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "36.18", "id": [6, 3]}]': finished 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.3", "id": [0, 3]}]': finished 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "37.e", "id": [0, 3]}]': finished 2024-09-21T10:11:35.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.1", "id": [0, 1]}]': finished 2024-09-21T10:11:35.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "38.14", "id": [7, 1]}]': finished 2024-09-21T10:11:35.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: osdmap e758: 8 total, 8 up, 8 in 2024-09-21T10:11:35.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:35 smithi119 ceph-mon[105288]: pgmap v808: 713 pgs: 713 active+clean; 147 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 179 KiB/s rd, 1.9 MiB/s wr, 252 op/s 2024-09-21T10:11:36.454 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.CreateWithMirrorEnabled (5064 ms) 2024-09-21T10:11:36.455 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot 2024-09-21T10:11:36.455 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:36 smithi145 ceph-mon[89359]: osdmap e759: 8 total, 8 up, 8 in 2024-09-21T10:11:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:36 smithi119 ceph-mon[101432]: osdmap e759: 8 total, 8 up, 8 in 2024-09-21T10:11:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:36 smithi119 ceph-mon[105288]: osdmap e759: 8 total, 8 up, 8 in 2024-09-21T10:11:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:37 smithi145 ceph-mon[89359]: osdmap e760: 8 total, 8 up, 8 in 2024-09-21T10:11:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:37 smithi145 ceph-mon[89359]: pgmap v811: 713 pgs: 2 peering, 711 active+clean; 147 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 251 KiB/s rd, 1.9 MiB/s wr, 340 op/s 2024-09-21T10:11:37.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:37 smithi119 ceph-mon[105288]: osdmap e760: 8 total, 8 up, 8 in 2024-09-21T10:11:37.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:37 smithi119 ceph-mon[105288]: pgmap v811: 713 pgs: 2 peering, 711 active+clean; 147 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 251 KiB/s rd, 1.9 MiB/s wr, 340 op/s 2024-09-21T10:11:37.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:37 smithi119 ceph-mon[101432]: osdmap e760: 8 total, 8 up, 8 in 2024-09-21T10:11:37.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:37 smithi119 ceph-mon[101432]: pgmap v811: 713 pgs: 2 peering, 711 active+clean; 147 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 251 KiB/s rd, 1.9 MiB/s wr, 340 op/s 2024-09-21T10:11:38.459 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.FlushCacheWithCopyupOnExternalSnapshot (2005 ms) 2024-09-21T10:11:38.459 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ExclusiveLock 2024-09-21T10:11:38.460 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:38.504 INFO:tasks.workunit.client.0.smithi119.stdout:1: acquiring exclusive lock 2024-09-21T10:11:38.504 INFO:tasks.workunit.client.0.smithi119.stdout:2: acquiring exclusive lock 2024-09-21T10:11:38.508 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock acquired 2024-09-21T10:11:38.536 INFO:tasks.workunit.client.0.smithi119.stdout:2: releasing exclusive lock 2024-09-21T10:11:38.539 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock released 2024-09-21T10:11:38.539 INFO:tasks.workunit.client.0.smithi119.stdout:2: acquiring exclusive lock 2024-09-21T10:11:38.542 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock acquired 2024-09-21T10:11:38.549 INFO:tasks.workunit.client.0.smithi119.stdout:2: releasing exclusive lock 2024-09-21T10:11:38.551 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock released 2024-09-21T10:11:38.551 INFO:tasks.workunit.client.0.smithi119.stdout:2: acquiring exclusive lock 2024-09-21T10:11:38.555 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock acquired 2024-09-21T10:11:38.582 INFO:tasks.workunit.client.0.smithi119.stdout:1: releasing exclusive lock 2024-09-21T10:11:38.585 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock released 2024-09-21T10:11:38.585 INFO:tasks.workunit.client.0.smithi119.stdout:1: acquiring exclusive lock 2024-09-21T10:11:38.589 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock acquired 2024-09-21T10:11:38.613 INFO:tasks.workunit.client.0.smithi119.stdout:1: releasing exclusive lock 2024-09-21T10:11:38.616 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock released 2024-09-21T10:11:38.616 INFO:tasks.workunit.client.0.smithi119.stdout:1: acquiring exclusive lock 2024-09-21T10:11:38.618 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock acquired 2024-09-21T10:11:38.621 INFO:tasks.workunit.client.0.smithi119.stdout:2: releasing exclusive lock 2024-09-21T10:11:38.624 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock released 2024-09-21T10:11:38.624 INFO:tasks.workunit.client.0.smithi119.stdout:2: acquiring exclusive lock 2024-09-21T10:11:38.627 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock acquired 2024-09-21T10:11:38.671 INFO:tasks.workunit.client.0.smithi119.stdout:1: releasing exclusive lock 2024-09-21T10:11:38.673 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock released 2024-09-21T10:11:38.673 INFO:tasks.workunit.client.0.smithi119.stdout:1: acquiring exclusive lock 2024-09-21T10:11:38.677 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock acquired 2024-09-21T10:11:38.711 INFO:tasks.workunit.client.0.smithi119.stdout:1: releasing exclusive lock 2024-09-21T10:11:38.714 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock released 2024-09-21T10:11:38.714 INFO:tasks.workunit.client.0.smithi119.stdout:1: acquiring exclusive lock 2024-09-21T10:11:38.717 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock acquired 2024-09-21T10:11:38.723 INFO:tasks.workunit.client.0.smithi119.stdout:2: releasing exclusive lock 2024-09-21T10:11:38.726 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock released 2024-09-21T10:11:38.726 INFO:tasks.workunit.client.0.smithi119.stdout:2: acquiring exclusive lock 2024-09-21T10:11:38.729 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock acquired 2024-09-21T10:11:38.771 INFO:tasks.workunit.client.0.smithi119.stdout:1: releasing exclusive lock 2024-09-21T10:11:38.774 INFO:tasks.workunit.client.0.smithi119.stdout:1: exclusive lock released 2024-09-21T10:11:38.776 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock acquired 2024-09-21T10:11:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:38 smithi145 ceph-mon[89359]: osdmap e761: 8 total, 8 up, 8 in 2024-09-21T10:11:38.818 INFO:tasks.workunit.client.0.smithi119.stdout:2: releasing exclusive lock 2024-09-21T10:11:38.821 INFO:tasks.workunit.client.0.smithi119.stdout:2: exclusive lock released 2024-09-21T10:11:38.835 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ExclusiveLock (376 ms) 2024-09-21T10:11:38.835 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.BreakLock 2024-09-21T10:11:38.849 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:38 smithi119 ceph-mon[105288]: osdmap e761: 8 total, 8 up, 8 in 2024-09-21T10:11:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:38 smithi119 ceph-mon[101432]: osdmap e761: 8 total, 8 up, 8 in 2024-09-21T10:11:39.475 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.BreakLock (640 ms) 2024-09-21T10:11:39.475 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.DiscardAfterWrite 2024-09-21T10:11:39.476 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.497 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.DiscardAfterWrite (22 ms) 2024-09-21T10:11:39.497 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.DefaultFeatures 2024-09-21T10:11:39.497 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.DefaultFeatures (0 ms) 2024-09-21T10:11:39.498 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurge 2024-09-21T10:11:39.498 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.554 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:39.554 INFO:tasks.workunit.client.0.smithi119.stdout:1.56% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:3.12% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:4.69% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:6.25% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:7.81% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:9.38% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:10.94% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:12.50% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:14.06% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:15.62% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:17.19% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:18.75% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:20.31% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:21.88% done 2024-09-21T10:11:39.555 INFO:tasks.workunit.client.0.smithi119.stdout:23.44% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:25.00% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:26.56% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:28.12% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:29.69% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:31.25% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:32.81% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:34.38% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:35.94% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:37.50% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:39.06% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:40.62% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:42.19% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:43.75% done 2024-09-21T10:11:39.556 INFO:tasks.workunit.client.0.smithi119.stdout:45.31% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:46.88% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:48.44% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:50.00% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:51.56% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:53.12% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:54.69% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:56.25% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:57.81% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:59.38% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:60.94% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:62.50% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:64.06% done 2024-09-21T10:11:39.557 INFO:tasks.workunit.client.0.smithi119.stdout:65.62% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:67.19% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:68.75% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:70.31% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:71.88% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:73.44% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:75.00% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:76.56% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:78.12% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:79.69% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:81.25% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:82.81% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:84.38% done 2024-09-21T10:11:39.558 INFO:tasks.workunit.client.0.smithi119.stdout:85.94% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:87.50% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:89.06% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:90.62% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:92.19% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:93.75% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:95.31% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:96.88% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:98.44% done 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurge (57 ms) 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay 2024-09-21T10:11:39.559 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.607 INFO:tasks.workunit.client.0.smithi119.stdout:0.00% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:1.56% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:3.12% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:4.69% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:6.25% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:7.81% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:9.38% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:10.94% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:12.50% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:14.06% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:15.62% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:17.19% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:18.75% done 2024-09-21T10:11:39.608 INFO:tasks.workunit.client.0.smithi119.stdout:20.31% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:21.88% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:23.44% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:25.00% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:26.56% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:28.12% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:29.69% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:31.25% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:32.81% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:34.38% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:35.94% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:37.50% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:39.06% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:40.62% done 2024-09-21T10:11:39.609 INFO:tasks.workunit.client.0.smithi119.stdout:42.19% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:43.75% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:45.31% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:46.88% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:48.44% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:50.00% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:51.56% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:53.12% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:54.69% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:56.25% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:57.81% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:59.38% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:60.94% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:62.50% done 2024-09-21T10:11:39.610 INFO:tasks.workunit.client.0.smithi119.stdout:64.06% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:65.62% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:67.19% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:68.75% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:70.31% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:71.88% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:73.44% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:75.00% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:76.56% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:78.12% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:79.69% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:81.25% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:82.81% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:84.38% done 2024-09-21T10:11:39.611 INFO:tasks.workunit.client.0.smithi119.stdout:85.94% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:87.50% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:89.06% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:90.62% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:92.19% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:93.75% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:95.31% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:96.88% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:98.44% done 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestTrashMoveAndPurgeNonExpiredDelay (53 ms) 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestTrashPurge 2024-09-21T10:11:39.612 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.615 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.721 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestTrashPurge (114 ms) 2024-09-21T10:11:39.721 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestTrashMoveAndRestore 2024-09-21T10:11:39.721 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.766 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestTrashMoveAndRestore (45 ms) 2024-09-21T10:11:39.766 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestListWatchers 2024-09-21T10:11:39.766 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestListWatchers (22 ms) 2024-09-21T10:11:39.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestSetSnapById 2024-09-21T10:11:39.788 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:39 smithi145 ceph-mon[89359]: osdmap e762: 8 total, 8 up, 8 in 2024-09-21T10:11:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:39 smithi145 ceph-mon[89359]: pgmap v814: 713 pgs: 2 peering, 711 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 225 KiB/s rd, 138 KiB/s wr, 209 op/s 2024-09-21T10:11:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/1139856442"} : dispatch 2024-09-21T10:11:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:39 smithi119 ceph-mon[105288]: osdmap e762: 8 total, 8 up, 8 in 2024-09-21T10:11:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:39 smithi119 ceph-mon[105288]: pgmap v814: 713 pgs: 2 peering, 711 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 225 KiB/s rd, 138 KiB/s wr, 209 op/s 2024-09-21T10:11:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/1139856442"} : dispatch 2024-09-21T10:11:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:39 smithi119 ceph-mon[101432]: osdmap e762: 8 total, 8 up, 8 in 2024-09-21T10:11:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:39 smithi119 ceph-mon[101432]: pgmap v814: 713 pgs: 2 peering, 711 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 225 KiB/s rd, 138 KiB/s wr, 209 op/s 2024-09-21T10:11:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd={"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/1139856442"} : dispatch 2024-09-21T10:11:40.488 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestSetSnapById (699 ms) 2024-09-21T10:11:40.488 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Namespaces 2024-09-21T10:11:40.508 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Namespaces (21 ms) 2024-09-21T10:11:40.508 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.NamespacesPP 2024-09-21T10:11:40.537 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:40.542 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:40 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:11:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/1139856442"}]': finished 2024-09-21T10:11:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:40 smithi145 ceph-mon[89359]: osdmap e763: 8 total, 8 up, 8 in 2024-09-21T10:11:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:11:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:11:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:40.617 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.NamespacesPP (109 ms) 2024-09-21T10:11:40.617 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Migration 2024-09-21T10:11:40.617 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:40.617 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:40 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:11:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/1139856442"}]': finished 2024-09-21T10:11:40.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:40 smithi119 ceph-mon[105288]: osdmap e763: 8 total, 8 up, 8 in 2024-09-21T10:11:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:40 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:11:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' cmd='[{"prefix": "osd blocklist", "blocklistop": "add", "addr": "172.21.15.119:0/1139856442"}]': finished 2024-09-21T10:11:40.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:40 smithi119 ceph-mon[101432]: osdmap e763: 8 total, 8 up, 8 in 2024-09-21T10:11:41.002 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Migration (385 ms) 2024-09-21T10:11:41.002 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.MigrationPP 2024-09-21T10:11:41.003 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.003 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.387 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.MigrationPP (385 ms) 2024-09-21T10:11:41.388 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestGetAccessTimestamp 2024-09-21T10:11:41.388 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.403 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestGetAccessTimestamp (16 ms) 2024-09-21T10:11:41.403 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.TestGetModifyTimestamp 2024-09-21T10:11:41.403 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.419 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.TestGetModifyTimestamp (16 ms) 2024-09-21T10:11:41.419 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ZeroOverlapFlatten 2024-09-21T10:11:41.419 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.544 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ZeroOverlapFlatten (125 ms) 2024-09-21T10:11:41.545 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.PoolMetadata 2024-09-21T10:11:41.571 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.PoolMetadata (27 ms) 2024-09-21T10:11:41.571 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.PoolMetadataPP 2024-09-21T10:11:41.587 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.PoolMetadataPP (16 ms) 2024-09-21T10:11:41.588 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.Config 2024-09-21T10:11:41.591 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.626 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.Config (39 ms) 2024-09-21T10:11:41.626 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ConfigPP 2024-09-21T10:11:41.630 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:41.662 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ConfigPP (36 ms) 2024-09-21T10:11:41.662 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.PoolStatsPP 2024-09-21T10:11:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:41 smithi145 ceph-mon[89359]: osdmap e764: 8 total, 8 up, 8 in 2024-09-21T10:11:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:41 smithi145 ceph-mon[89359]: pgmap v817: 713 pgs: 713 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 264 KiB/s rd, 140 KiB/s wr, 285 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:11:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:41 smithi119 ceph-mon[101432]: osdmap e764: 8 total, 8 up, 8 in 2024-09-21T10:11:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:41 smithi119 ceph-mon[101432]: pgmap v817: 713 pgs: 713 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 264 KiB/s rd, 140 KiB/s wr, 285 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:11:41.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:41 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:11:41] ENGINE Bus STOPPING 2024-09-21T10:11:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:41 smithi119 ceph-mon[105288]: osdmap e764: 8 total, 8 up, 8 in 2024-09-21T10:11:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:41 smithi119 ceph-mon[105288]: pgmap v817: 713 pgs: 713 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 264 KiB/s rd, 140 KiB/s wr, 285 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:11:42.367 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:11:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:11:42.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:11:42] ENGINE Bus STOPPED 2024-09-21T10:11:42.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:11:42] ENGINE Bus STARTING 2024-09-21T10:11:42.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:11:42] ENGINE Serving on http://:::9283 2024-09-21T10:11:42.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:11:42] ENGINE Bus STARTED 2024-09-21T10:11:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:11:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: osdmap e765: 8 total, 8 up, 8 in 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:11:42.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:42 smithi145 ceph-mon[89359]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:11:42.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:11:42.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: osdmap e765: 8 total, 8 up, 8 in 2024-09-21T10:11:42.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[105288]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:11:42.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: osdmap e765: 8 total, 8 up, 8 in 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1458181801' entity='client.admin' 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "orch prometheus get-credentials"} : dispatch 2024-09-21T10:11:42.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:42 smithi119 ceph-mon[101432]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T10:11:43.578 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[105288]: osdmap e766: 8 total, 8 up, 8 in 2024-09-21T10:11:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3507342748' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[105288]: pgmap v820: 745 pgs: 32 unknown, 713 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 4.7 KiB/s wr, 163 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:11:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[101432]: osdmap e766: 8 total, 8 up, 8 in 2024-09-21T10:11:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3507342748' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:43.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:43 smithi119 ceph-mon[101432]: pgmap v820: 745 pgs: 32 unknown, 713 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 4.7 KiB/s wr, 163 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:11:43.931 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:43 smithi145 ceph-mon[89359]: osdmap e766: 8 total, 8 up, 8 in 2024-09-21T10:11:43.931 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3507342748' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:43.931 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:43 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:43.931 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:43 smithi145 ceph-mon[89359]: pgmap v820: 745 pgs: 32 unknown, 713 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 112 KiB/s rd, 4.7 KiB/s wr, 163 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:11:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:11:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[105288]: osdmap e767: 8 total, 8 up, 8 in 2024-09-21T10:11:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[105288]: osdmap e768: 8 total, 8 up, 8 in 2024-09-21T10:11:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:11:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[101432]: osdmap e767: 8 total, 8 up, 8 in 2024-09-21T10:11:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-mon[101432]: osdmap e768: 8 total, 8 up, 8 in 2024-09-21T10:11:44.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:11:44] "GET /metrics HTTP/1.1" 200 82404 "" "Prometheus/2.51.0" 2024-09-21T10:11:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:44 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:11:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:44 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:44 smithi145 ceph-mon[89359]: osdmap e767: 8 total, 8 up, 8 in 2024-09-21T10:11:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:44 smithi145 ceph-mon[89359]: osdmap e768: 8 total, 8 up, 8 in 2024-09-21T10:11:45.130 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:45.139 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:45.147 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:46.171 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.PoolStatsPP (4509 ms) 2024-09-21T10:11:46.172 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ImageSpec 2024-09-21T10:11:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:46 smithi119 ceph-mon[105288]: pgmap v823: 745 pgs: 22 creating+peering, 7 unknown, 716 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 649 KiB/s rd, 50 KiB/s wr, 873 op/s 2024-09-21T10:11:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:46 smithi119 ceph-mon[105288]: osdmap e769: 8 total, 8 up, 8 in 2024-09-21T10:11:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:46 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:11:46.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:46 smithi119 ceph-mon[101432]: pgmap v823: 745 pgs: 22 creating+peering, 7 unknown, 716 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 649 KiB/s rd, 50 KiB/s wr, 873 op/s 2024-09-21T10:11:46.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:46 smithi119 ceph-mon[101432]: osdmap e769: 8 total, 8 up, 8 in 2024-09-21T10:11:46.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:46 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:11:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:46 smithi145 ceph-mon[89359]: pgmap v823: 745 pgs: 22 creating+peering, 7 unknown, 716 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 649 KiB/s rd, 50 KiB/s wr, 873 op/s 2024-09-21T10:11:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:46 smithi145 ceph-mon[89359]: osdmap e769: 8 total, 8 up, 8 in 2024-09-21T10:11:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:46 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:11:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:47 smithi145 ceph-mon[89359]: osdmap e770: 8 total, 8 up, 8 in 2024-09-21T10:11:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:47 smithi119 ceph-mon[105288]: osdmap e770: 8 total, 8 up, 8 in 2024-09-21T10:11:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:47 smithi119 ceph-mon[101432]: osdmap e770: 8 total, 8 up, 8 in 2024-09-21T10:11:48.151 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:48 smithi145 ceph-mon[89359]: pgmap v826: 745 pgs: 22 creating+peering, 4 unknown, 719 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 822 KiB/s rd, 58 KiB/s wr, 1.09k op/s 2024-09-21T10:11:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:48 smithi145 ceph-mon[89359]: osdmap e771: 8 total, 8 up, 8 in 2024-09-21T10:11:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1595782499' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:48 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[105288]: pgmap v826: 745 pgs: 22 creating+peering, 4 unknown, 719 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 822 KiB/s rd, 58 KiB/s wr, 1.09k op/s 2024-09-21T10:11:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[105288]: osdmap e771: 8 total, 8 up, 8 in 2024-09-21T10:11:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1595782499' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[101432]: pgmap v826: 745 pgs: 22 creating+peering, 4 unknown, 719 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 822 KiB/s rd, 58 KiB/s wr, 1.09k op/s 2024-09-21T10:11:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[101432]: osdmap e771: 8 total, 8 up, 8 in 2024-09-21T10:11:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1595782499' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:48.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:11:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:49 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:49 smithi145 ceph-mon[89359]: osdmap e772: 8 total, 8 up, 8 in 2024-09-21T10:11:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:49 smithi145 ceph-mon[89359]: pgmap v829: 777 pgs: 8 creating+peering, 5 unknown, 764 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 183 KiB/s rd, 11 KiB/s wr, 236 op/s 2024-09-21T10:11:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:49 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:49 smithi119 ceph-mon[105288]: osdmap e772: 8 total, 8 up, 8 in 2024-09-21T10:11:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:49 smithi119 ceph-mon[105288]: pgmap v829: 777 pgs: 8 creating+peering, 5 unknown, 764 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 183 KiB/s rd, 11 KiB/s wr, 236 op/s 2024-09-21T10:11:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:49 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:49 smithi119 ceph-mon[101432]: osdmap e772: 8 total, 8 up, 8 in 2024-09-21T10:11:49.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:49 smithi119 ceph-mon[101432]: pgmap v829: 777 pgs: 8 creating+peering, 5 unknown, 764 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 183 KiB/s rd, 11 KiB/s wr, 236 op/s 2024-09-21T10:11:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:50 smithi145 ceph-mon[89359]: osdmap e773: 8 total, 8 up, 8 in 2024-09-21T10:11:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:11:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:11:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:11:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:50 smithi119 ceph-mon[105288]: osdmap e773: 8 total, 8 up, 8 in 2024-09-21T10:11:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:50 smithi119 ceph-mon[101432]: osdmap e773: 8 total, 8 up, 8 in 2024-09-21T10:11:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:51 smithi145 ceph-mon[89359]: osdmap e774: 8 total, 8 up, 8 in 2024-09-21T10:11:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:51 smithi145 ceph-mon[89359]: pgmap v832: 777 pgs: 8 creating+peering, 769 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 4.2 KiB/s wr, 48 op/s 2024-09-21T10:11:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:51 smithi119 ceph-mon[105288]: osdmap e774: 8 total, 8 up, 8 in 2024-09-21T10:11:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:51 smithi119 ceph-mon[105288]: pgmap v832: 777 pgs: 8 creating+peering, 769 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 4.2 KiB/s wr, 48 op/s 2024-09-21T10:11:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:51 smithi119 ceph-mon[101432]: osdmap e774: 8 total, 8 up, 8 in 2024-09-21T10:11:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:51 smithi119 ceph-mon[101432]: pgmap v832: 777 pgs: 8 creating+peering, 769 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 36 KiB/s rd, 4.2 KiB/s wr, 48 op/s 2024-09-21T10:11:52.308 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ImageSpec (6136 ms) 2024-09-21T10:11:52.308 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.SnapRemoveWithChildMissing 2024-09-21T10:11:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:52 smithi145 ceph-mon[89359]: osdmap e775: 8 total, 8 up, 8 in 2024-09-21T10:11:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:52 smithi119 ceph-mon[105288]: osdmap e775: 8 total, 8 up, 8 in 2024-09-21T10:11:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:52 smithi119 ceph-mon[101432]: osdmap e775: 8 total, 8 up, 8 in 2024-09-21T10:11:53.426 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:53 smithi145 ceph-mon[89359]: osdmap e776: 8 total, 8 up, 8 in 2024-09-21T10:11:53.426 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:53 smithi145 ceph-mon[89359]: pgmap v835: 777 pgs: 777 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 78 KiB/s rd, 3.7 KiB/s wr, 95 op/s 2024-09-21T10:11:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:53 smithi119 ceph-mon[105288]: osdmap e776: 8 total, 8 up, 8 in 2024-09-21T10:11:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:53 smithi119 ceph-mon[105288]: pgmap v835: 777 pgs: 777 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 78 KiB/s rd, 3.7 KiB/s wr, 95 op/s 2024-09-21T10:11:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:53 smithi119 ceph-mon[101432]: osdmap e776: 8 total, 8 up, 8 in 2024-09-21T10:11:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:53 smithi119 ceph-mon[101432]: pgmap v835: 777 pgs: 777 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 78 KiB/s rd, 3.7 KiB/s wr, 95 op/s 2024-09-21T10:11:54.114 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:11:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:54 smithi145 ceph-mon[89359]: osdmap e777: 8 total, 8 up, 8 in 2024-09-21T10:11:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/277230704' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/277230704' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:54 smithi145 ceph-mon[89359]: osdmap e778: 8 total, 8 up, 8 in 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[101432]: osdmap e777: 8 total, 8 up, 8 in 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/277230704' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/277230704' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[101432]: osdmap e778: 8 total, 8 up, 8 in 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[105288]: osdmap e777: 8 total, 8 up, 8 in 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/277230704' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-37","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:11:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/277230704' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:11:54.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-mon[105288]: osdmap e778: 8 total, 8 up, 8 in 2024-09-21T10:11:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:11:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:11:54] "GET /metrics HTTP/1.1" 200 87159 "" "Prometheus/2.51.0" 2024-09-21T10:11:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:56 smithi119 ceph-mon[101432]: pgmap v838: 809 pgs: 22 creating+peering, 8 unknown, 779 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 138 KiB/s rd, 6.5 KiB/s wr, 174 op/s 2024-09-21T10:11:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:56 smithi119 ceph-mon[101432]: osdmap e779: 8 total, 8 up, 8 in 2024-09-21T10:11:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:56 smithi119 ceph-mon[105288]: pgmap v838: 809 pgs: 22 creating+peering, 8 unknown, 779 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 138 KiB/s rd, 6.5 KiB/s wr, 174 op/s 2024-09-21T10:11:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:56 smithi119 ceph-mon[105288]: osdmap e779: 8 total, 8 up, 8 in 2024-09-21T10:11:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:56 smithi145 ceph-mon[89359]: pgmap v838: 809 pgs: 22 creating+peering, 8 unknown, 779 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 138 KiB/s rd, 6.5 KiB/s wr, 174 op/s 2024-09-21T10:11:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:56 smithi145 ceph-mon[89359]: osdmap e779: 8 total, 8 up, 8 in 2024-09-21T10:11:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:57 smithi119 ceph-mon[105288]: osdmap e780: 8 total, 8 up, 8 in 2024-09-21T10:11:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:57 smithi119 ceph-mon[101432]: osdmap e780: 8 total, 8 up, 8 in 2024-09-21T10:11:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:57 smithi145 ceph-mon[89359]: osdmap e780: 8 total, 8 up, 8 in 2024-09-21T10:11:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:58 smithi119 ceph-mon[105288]: pgmap v841: 809 pgs: 22 creating+peering, 5 unknown, 782 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s rd, 4.0 KiB/s wr, 109 op/s 2024-09-21T10:11:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:58 smithi119 ceph-mon[105288]: osdmap e781: 8 total, 8 up, 8 in 2024-09-21T10:11:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:58 smithi119 ceph-mon[101432]: pgmap v841: 809 pgs: 22 creating+peering, 5 unknown, 782 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s rd, 4.0 KiB/s wr, 109 op/s 2024-09-21T10:11:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:58 smithi119 ceph-mon[101432]: osdmap e781: 8 total, 8 up, 8 in 2024-09-21T10:11:58.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:58 smithi145 ceph-mon[89359]: pgmap v841: 809 pgs: 22 creating+peering, 5 unknown, 782 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 87 KiB/s rd, 4.0 KiB/s wr, 109 op/s 2024-09-21T10:11:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:58 smithi145 ceph-mon[89359]: osdmap e781: 8 total, 8 up, 8 in 2024-09-21T10:11:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:11:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:59 smithi145 ceph-mon[89359]: osdmap e782: 8 total, 8 up, 8 in 2024-09-21T10:11:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:11:59 smithi145 ceph-mon[89359]: osdmap e783: 8 total, 8 up, 8 in 2024-09-21T10:11:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:59 smithi119 ceph-mon[105288]: osdmap e782: 8 total, 8 up, 8 in 2024-09-21T10:11:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:11:59 smithi119 ceph-mon[105288]: osdmap e783: 8 total, 8 up, 8 in 2024-09-21T10:11:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:59 smithi119 ceph-mon[101432]: osdmap e782: 8 total, 8 up, 8 in 2024-09-21T10:11:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:11:59 smithi119 ceph-mon[101432]: osdmap e783: 8 total, 8 up, 8 in 2024-09-21T10:12:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:00 smithi145 ceph-mon[89359]: pgmap v844: 809 pgs: 809 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 5.7 KiB/s wr, 73 op/s 2024-09-21T10:12:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:12:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:12:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:00 smithi119 ceph-mon[105288]: pgmap v844: 809 pgs: 809 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 5.7 KiB/s wr, 73 op/s 2024-09-21T10:12:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:00 smithi119 ceph-mon[101432]: pgmap v844: 809 pgs: 809 active+clean; 148 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 5.7 KiB/s wr, 73 op/s 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 3 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 3 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:child: image212 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:child: image214 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:child: image213 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-15/image212/880bf264fb9e 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:12:01.183 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-37/image213/880bd41a5ff9 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:found child test-librbd-smithi119-84733-15/image214/880b8e0c5aa2 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:num children is: 0 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:num snaps is: 0 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:expected: 0 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.SnapRemoveWithChildMissing (8876 ms) 2024-09-21T10:12:01.184 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.QuiesceWatch 2024-09-21T10:12:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:01 smithi145 ceph-mon[89359]: osdmap e784: 8 total, 8 up, 8 in 2024-09-21T10:12:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:01 smithi145 ceph-mon[89359]: pgmap v847: 777 pgs: 777 active+clean; 2.6 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:12:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:01 smithi119 ceph-mon[105288]: osdmap e784: 8 total, 8 up, 8 in 2024-09-21T10:12:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:01 smithi119 ceph-mon[105288]: pgmap v847: 777 pgs: 777 active+clean; 2.6 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:12:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:01 smithi119 ceph-mon[101432]: osdmap e784: 8 total, 8 up, 8 in 2024-09-21T10:12:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:01 smithi119 ceph-mon[101432]: pgmap v847: 777 pgs: 777 active+clean; 2.6 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:12:02.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:02 smithi119 ceph-mon[105288]: osdmap e785: 8 total, 8 up, 8 in 2024-09-21T10:12:02.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:02 smithi119 ceph-mon[101432]: osdmap e785: 8 total, 8 up, 8 in 2024-09-21T10:12:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:02 smithi145 ceph-mon[89359]: osdmap e785: 8 total, 8 up, 8 in 2024-09-21T10:12:03.183 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[101432]: osdmap e786: 8 total, 8 up, 8 in 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3278207320' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[101432]: pgmap v850: 809 pgs: 1 creating+peering, 31 unknown, 777 active+clean; 2.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[105288]: osdmap e786: 8 total, 8 up, 8 in 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3278207320' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[105288]: pgmap v850: 809 pgs: 1 creating+peering, 31 unknown, 777 active+clean; 2.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:03.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:03 smithi145 ceph-mon[89359]: osdmap e786: 8 total, 8 up, 8 in 2024-09-21T10:12:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3278207320' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-38","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:12:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:03 smithi145 ceph-mon[89359]: pgmap v850: 809 pgs: 1 creating+peering, 31 unknown, 777 active+clean; 2.6 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3278207320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[105288]: osdmap e787: 8 total, 8 up, 8 in 2024-09-21T10:12:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[105288]: osdmap e788: 8 total, 8 up, 8 in 2024-09-21T10:12:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3278207320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[101432]: osdmap e787: 8 total, 8 up, 8 in 2024-09-21T10:12:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-mon[101432]: osdmap e788: 8 total, 8 up, 8 in 2024-09-21T10:12:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:04 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:12:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3278207320' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:12:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:04 smithi145 ceph-mon[89359]: osdmap e787: 8 total, 8 up, 8 in 2024-09-21T10:12:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:04 smithi145 ceph-mon[89359]: osdmap e788: 8 total, 8 up, 8 in 2024-09-21T10:12:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:12:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:12:04] "GET /metrics HTTP/1.1" 200 89539 "" "Prometheus/2.51.0" 2024-09-21T10:12:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:06 smithi119 ceph-mon[105288]: pgmap v853: 809 pgs: 1 creating+peering, 5 unknown, 803 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 6.2 KiB/s wr, 139 op/s 2024-09-21T10:12:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:06 smithi119 ceph-mon[105288]: osdmap e789: 8 total, 8 up, 8 in 2024-09-21T10:12:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:06 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:06 smithi119 ceph-mon[101432]: pgmap v853: 809 pgs: 1 creating+peering, 5 unknown, 803 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 6.2 KiB/s wr, 139 op/s 2024-09-21T10:12:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:06 smithi119 ceph-mon[101432]: osdmap e789: 8 total, 8 up, 8 in 2024-09-21T10:12:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:06 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:06 smithi145 ceph-mon[89359]: pgmap v853: 809 pgs: 1 creating+peering, 5 unknown, 803 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 109 KiB/s rd, 6.2 KiB/s wr, 139 op/s 2024-09-21T10:12:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:06 smithi145 ceph-mon[89359]: osdmap e789: 8 total, 8 up, 8 in 2024-09-21T10:12:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:06 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:12:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:07 smithi145 ceph-mon[89359]: osdmap e790: 8 total, 8 up, 8 in 2024-09-21T10:12:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:07 smithi119 ceph-mon[105288]: osdmap e790: 8 total, 8 up, 8 in 2024-09-21T10:12:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:07 smithi119 ceph-mon[101432]: osdmap e790: 8 total, 8 up, 8 in 2024-09-21T10:12:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:08 smithi145 ceph-mon[89359]: pgmap v856: 809 pgs: 1 creating+peering, 808 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.5 KiB/s wr, 50 op/s 2024-09-21T10:12:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:08 smithi145 ceph-mon[89359]: osdmap e791: 8 total, 8 up, 8 in 2024-09-21T10:12:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:08 smithi119 ceph-mon[105288]: pgmap v856: 809 pgs: 1 creating+peering, 808 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.5 KiB/s wr, 50 op/s 2024-09-21T10:12:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:08 smithi119 ceph-mon[105288]: osdmap e791: 8 total, 8 up, 8 in 2024-09-21T10:12:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:08 smithi119 ceph-mon[101432]: pgmap v856: 809 pgs: 1 creating+peering, 808 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 1.5 KiB/s wr, 50 op/s 2024-09-21T10:12:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:08 smithi119 ceph-mon[101432]: osdmap e791: 8 total, 8 up, 8 in 2024-09-21T10:12:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:09 smithi145 ceph-mon[89359]: osdmap e792: 8 total, 8 up, 8 in 2024-09-21T10:12:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:09 smithi145 ceph-mon[89359]: osdmap e793: 8 total, 8 up, 8 in 2024-09-21T10:12:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:09 smithi119 ceph-mon[105288]: osdmap e792: 8 total, 8 up, 8 in 2024-09-21T10:12:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:09 smithi119 ceph-mon[105288]: osdmap e793: 8 total, 8 up, 8 in 2024-09-21T10:12:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:09 smithi119 ceph-mon[101432]: osdmap e792: 8 total, 8 up, 8 in 2024-09-21T10:12:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:09 smithi119 ceph-mon[101432]: osdmap e793: 8 total, 8 up, 8 in 2024-09-21T10:12:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:10 smithi145 ceph-mon[89359]: pgmap v859: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 20 op/s 2024-09-21T10:12:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:12:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:12:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:10 smithi119 ceph-mon[105288]: pgmap v859: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 20 op/s 2024-09-21T10:12:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:10 smithi119 ceph-mon[101432]: pgmap v859: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.2 KiB/s wr, 20 op/s 2024-09-21T10:12:11.265 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.QuiesceWatch (10081 ms) 2024-09-21T10:12:11.265 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.QuiesceWatchPP 2024-09-21T10:12:11.265 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:11 smithi145 ceph-mon[89359]: osdmap e794: 8 total, 8 up, 8 in 2024-09-21T10:12:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:11 smithi145 ceph-mon[89359]: pgmap v862: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 5.7 KiB/s wr, 67 op/s 2024-09-21T10:12:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:11 smithi119 ceph-mon[105288]: osdmap e794: 8 total, 8 up, 8 in 2024-09-21T10:12:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:11 smithi119 ceph-mon[105288]: pgmap v862: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 5.7 KiB/s wr, 67 op/s 2024-09-21T10:12:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:11 smithi119 ceph-mon[101432]: osdmap e794: 8 total, 8 up, 8 in 2024-09-21T10:12:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:11 smithi119 ceph-mon[101432]: pgmap v862: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 48 KiB/s rd, 5.7 KiB/s wr, 67 op/s 2024-09-21T10:12:12.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:12 smithi145 ceph-mon[89359]: osdmap e795: 8 total, 8 up, 8 in 2024-09-21T10:12:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:12 smithi119 ceph-mon[101432]: osdmap e795: 8 total, 8 up, 8 in 2024-09-21T10:12:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:12 smithi119 ceph-mon[105288]: osdmap e795: 8 total, 8 up, 8 in 2024-09-21T10:12:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:13 smithi145 ceph-mon[89359]: osdmap e796: 8 total, 8 up, 8 in 2024-09-21T10:12:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:13 smithi145 ceph-mon[89359]: pgmap v865: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 5.0 KiB/s wr, 67 op/s 2024-09-21T10:12:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:13 smithi119 ceph-mon[105288]: osdmap e796: 8 total, 8 up, 8 in 2024-09-21T10:12:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:13 smithi119 ceph-mon[105288]: pgmap v865: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 5.0 KiB/s wr, 67 op/s 2024-09-21T10:12:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:13 smithi119 ceph-mon[101432]: osdmap e796: 8 total, 8 up, 8 in 2024-09-21T10:12:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:13 smithi119 ceph-mon[101432]: pgmap v865: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 5.0 KiB/s wr, 67 op/s 2024-09-21T10:12:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:14 smithi119 ceph-mon[101432]: osdmap e797: 8 total, 8 up, 8 in 2024-09-21T10:12:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:14 smithi119 ceph-mon[101432]: osdmap e798: 8 total, 8 up, 8 in 2024-09-21T10:12:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:14 smithi119 ceph-mon[105288]: osdmap e797: 8 total, 8 up, 8 in 2024-09-21T10:12:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:14 smithi119 ceph-mon[105288]: osdmap e798: 8 total, 8 up, 8 in 2024-09-21T10:12:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:14 smithi145 ceph-mon[89359]: osdmap e797: 8 total, 8 up, 8 in 2024-09-21T10:12:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:14 smithi145 ceph-mon[89359]: osdmap e798: 8 total, 8 up, 8 in 2024-09-21T10:12:14.830 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:12:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:12:14] "GET /metrics HTTP/1.1" 200 89539 "" "Prometheus/2.51.0" 2024-09-21T10:12:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:16 smithi119 ceph-mon[105288]: pgmap v868: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 4.2 KiB/s wr, 70 op/s 2024-09-21T10:12:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:16 smithi119 ceph-mon[105288]: osdmap e799: 8 total, 8 up, 8 in 2024-09-21T10:12:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:16 smithi119 ceph-mon[101432]: pgmap v868: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 4.2 KiB/s wr, 70 op/s 2024-09-21T10:12:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:16 smithi119 ceph-mon[101432]: osdmap e799: 8 total, 8 up, 8 in 2024-09-21T10:12:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:16 smithi145 ceph-mon[89359]: pgmap v868: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 4.2 KiB/s wr, 70 op/s 2024-09-21T10:12:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:16 smithi145 ceph-mon[89359]: osdmap e799: 8 total, 8 up, 8 in 2024-09-21T10:12:17.210 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.QuiesceWatchPP (5946 ms) 2024-09-21T10:12:17.210 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.QuiesceWatchError 2024-09-21T10:12:17.211 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:17 smithi145 ceph-mon[89359]: osdmap e800: 8 total, 8 up, 8 in 2024-09-21T10:12:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:17 smithi119 ceph-mon[105288]: osdmap e800: 8 total, 8 up, 8 in 2024-09-21T10:12:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:17 smithi119 ceph-mon[101432]: osdmap e800: 8 total, 8 up, 8 in 2024-09-21T10:12:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:18 smithi145 ceph-mon[89359]: pgmap v871: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 4.7 KiB/s wr, 91 op/s 2024-09-21T10:12:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:18 smithi145 ceph-mon[89359]: osdmap e801: 8 total, 8 up, 8 in 2024-09-21T10:12:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[105288]: pgmap v871: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 4.7 KiB/s wr, 91 op/s 2024-09-21T10:12:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[105288]: osdmap e801: 8 total, 8 up, 8 in 2024-09-21T10:12:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[101432]: pgmap v871: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 69 KiB/s rd, 4.7 KiB/s wr, 91 op/s 2024-09-21T10:12:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[101432]: osdmap e801: 8 total, 8 up, 8 in 2024-09-21T10:12:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:18.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:19 smithi145 ceph-mon[89359]: osdmap e802: 8 total, 8 up, 8 in 2024-09-21T10:12:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:19 smithi145 ceph-mon[89359]: osdmap e803: 8 total, 8 up, 8 in 2024-09-21T10:12:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:19 smithi119 ceph-mon[105288]: osdmap e802: 8 total, 8 up, 8 in 2024-09-21T10:12:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:19 smithi119 ceph-mon[105288]: osdmap e803: 8 total, 8 up, 8 in 2024-09-21T10:12:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:19 smithi119 ceph-mon[101432]: osdmap e802: 8 total, 8 up, 8 in 2024-09-21T10:12:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:19 smithi119 ceph-mon[101432]: osdmap e803: 8 total, 8 up, 8 in 2024-09-21T10:12:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:20 smithi145 ceph-mon[89359]: pgmap v874: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 46 op/s 2024-09-21T10:12:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:12:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:12:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:20 smithi119 ceph-mon[105288]: pgmap v874: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 46 op/s 2024-09-21T10:12:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:20 smithi119 ceph-mon[101432]: pgmap v874: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 2.2 KiB/s wr, 46 op/s 2024-09-21T10:12:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:21 smithi145 ceph-mon[89359]: osdmap e804: 8 total, 8 up, 8 in 2024-09-21T10:12:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:21 smithi145 ceph-mon[89359]: pgmap v877: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 5.5 KiB/s wr, 109 op/s 2024-09-21T10:12:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:21 smithi119 ceph-mon[101432]: osdmap e804: 8 total, 8 up, 8 in 2024-09-21T10:12:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:21 smithi119 ceph-mon[101432]: pgmap v877: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 5.5 KiB/s wr, 109 op/s 2024-09-21T10:12:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:21 smithi119 ceph-mon[105288]: osdmap e804: 8 total, 8 up, 8 in 2024-09-21T10:12:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:21 smithi119 ceph-mon[105288]: pgmap v877: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 80 KiB/s rd, 5.5 KiB/s wr, 109 op/s 2024-09-21T10:12:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:22 smithi145 ceph-mon[89359]: osdmap e805: 8 total, 8 up, 8 in 2024-09-21T10:12:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:22 smithi119 ceph-mon[101432]: osdmap e805: 8 total, 8 up, 8 in 2024-09-21T10:12:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:22 smithi119 ceph-mon[105288]: osdmap e805: 8 total, 8 up, 8 in 2024-09-21T10:12:23.258 INFO:tasks.workunit.client.0.smithi119.stdout:10.00% done 2024-09-21T10:12:23.258 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.QuiesceWatchError (6048 ms) 2024-09-21T10:12:23.258 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.QuiesceWatchTimeout 2024-09-21T10:12:23.258 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:23.273 INFO:tasks.workunit.client.0.smithi119.stderr:test quiesce is not long enough to time out 2024-09-21T10:12:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:23 smithi145 ceph-mon[89359]: osdmap e806: 8 total, 8 up, 8 in 2024-09-21T10:12:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:23 smithi145 ceph-mon[89359]: pgmap v880: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 4.0 KiB/s wr, 92 op/s 2024-09-21T10:12:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:23 smithi119 ceph-mon[105288]: osdmap e806: 8 total, 8 up, 8 in 2024-09-21T10:12:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:23 smithi119 ceph-mon[105288]: pgmap v880: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 4.0 KiB/s wr, 92 op/s 2024-09-21T10:12:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:23 smithi119 ceph-mon[101432]: osdmap e806: 8 total, 8 up, 8 in 2024-09-21T10:12:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:23 smithi119 ceph-mon[101432]: pgmap v880: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 4.0 KiB/s wr, 92 op/s 2024-09-21T10:12:24.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:24 smithi119 ceph-mon[105288]: osdmap e807: 8 total, 8 up, 8 in 2024-09-21T10:12:24.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:24 smithi119 ceph-mon[105288]: osdmap e808: 8 total, 8 up, 8 in 2024-09-21T10:12:24.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:24 smithi119 ceph-mon[101432]: osdmap e807: 8 total, 8 up, 8 in 2024-09-21T10:12:24.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:24 smithi119 ceph-mon[101432]: osdmap e808: 8 total, 8 up, 8 in 2024-09-21T10:12:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:24 smithi145 ceph-mon[89359]: osdmap e807: 8 total, 8 up, 8 in 2024-09-21T10:12:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:24 smithi145 ceph-mon[89359]: osdmap e808: 8 total, 8 up, 8 in 2024-09-21T10:12:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:12:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:12:24] "GET /metrics HTTP/1.1" 200 89572 "" "Prometheus/2.51.0" 2024-09-21T10:12:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:25 smithi145 ceph-mon[89359]: pgmap v883: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 5.2 KiB/s wr, 79 op/s 2024-09-21T10:12:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:25 smithi119 ceph-mon[101432]: pgmap v883: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 5.2 KiB/s wr, 79 op/s 2024-09-21T10:12:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:25 smithi119 ceph-mon[105288]: pgmap v883: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 5.2 KiB/s wr, 79 op/s 2024-09-21T10:12:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:27 smithi145 ceph-mon[89359]: pgmap v884: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 4.2 KiB/s wr, 65 op/s 2024-09-21T10:12:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:27 smithi119 ceph-mon[101432]: pgmap v884: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 4.2 KiB/s wr, 65 op/s 2024-09-21T10:12:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:27 smithi119 ceph-mon[105288]: pgmap v884: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 4.2 KiB/s wr, 65 op/s 2024-09-21T10:12:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:30 smithi119 ceph-mon[101432]: pgmap v885: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 KiB/s wr, 55 op/s 2024-09-21T10:12:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:30 smithi119 ceph-mon[101432]: osdmap e809: 8 total, 8 up, 8 in 2024-09-21T10:12:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:30 smithi119 ceph-mon[105288]: pgmap v885: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 KiB/s wr, 55 op/s 2024-09-21T10:12:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:30 smithi119 ceph-mon[105288]: osdmap e809: 8 total, 8 up, 8 in 2024-09-21T10:12:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:30 smithi145 ceph-mon[89359]: pgmap v885: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 3.6 KiB/s wr, 55 op/s 2024-09-21T10:12:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:30 smithi145 ceph-mon[89359]: osdmap e809: 8 total, 8 up, 8 in 2024-09-21T10:12:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:12:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:12:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:32.143 INFO:tasks.workunit.client.0.smithi119.stderr:test quiesce is timed out 2024-09-21T10:12:32.145 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 0 2024-09-21T10:12:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:32 smithi145 ceph-mon[89359]: pgmap v887: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 3.0 KiB/s wr, 47 op/s 2024-09-21T10:12:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:32 smithi119 ceph-mon[101432]: pgmap v887: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 3.0 KiB/s wr, 47 op/s 2024-09-21T10:12:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:32 smithi119 ceph-mon[105288]: pgmap v887: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 3.0 KiB/s wr, 47 op/s 2024-09-21T10:12:33.145 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 1 2024-09-21T10:12:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:33 smithi145 ceph-mon[89359]: osdmap e810: 8 total, 8 up, 8 in 2024-09-21T10:12:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:33 smithi119 ceph-mon[105288]: osdmap e810: 8 total, 8 up, 8 in 2024-09-21T10:12:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:33 smithi119 ceph-mon[101432]: osdmap e810: 8 total, 8 up, 8 in 2024-09-21T10:12:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:34.145 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 2 2024-09-21T10:12:34.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:34 smithi119 ceph-mon[105288]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-21T10:12:34.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:34 smithi119 ceph-mon[101432]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-21T10:12:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:34 smithi145 ceph-mon[89359]: pgmap v889: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-21T10:12:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:12:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:12:34] "GET /metrics HTTP/1.1" 200 89579 "" "Prometheus/2.51.0" 2024-09-21T10:12:35.145 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 3 2024-09-21T10:12:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]} : dispatch 2024-09-21T10:12:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]} : dispatch 2024-09-21T10:12:36.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 4 2024-09-21T10:12:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:36 smithi145 ceph-mon[89359]: pgmap v890: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:12:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-09-21T10:12:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-09-21T10:12:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-09-21T10:12:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:36 smithi145 ceph-mon[89359]: osdmap e811: 8 total, 8 up, 8 in 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[105288]: pgmap v890: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[105288]: osdmap e811: 8 total, 8 up, 8 in 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[101432]: pgmap v890: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 127 B/s wr, 0 op/s 2024-09-21T10:12:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.2", "id": [3, 1]}]': finished 2024-09-21T10:12:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.d", "id": [3, 1]}]': finished 2024-09-21T10:12:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-09-21T10:12:36.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:36 smithi119 ceph-mon[101432]: osdmap e811: 8 total, 8 up, 8 in 2024-09-21T10:12:37.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 5 2024-09-21T10:12:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:37 smithi145 ceph-mon[89359]: osdmap e812: 8 total, 8 up, 8 in 2024-09-21T10:12:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:37 smithi145 ceph-mon[89359]: pgmap v893: 809 pgs: 2 peering, 807 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T10:12:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:37 smithi119 ceph-mon[101432]: osdmap e812: 8 total, 8 up, 8 in 2024-09-21T10:12:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:37 smithi119 ceph-mon[101432]: pgmap v893: 809 pgs: 2 peering, 807 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T10:12:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:37 smithi119 ceph-mon[105288]: osdmap e812: 8 total, 8 up, 8 in 2024-09-21T10:12:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:37 smithi119 ceph-mon[105288]: pgmap v893: 809 pgs: 2 peering, 807 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 511 B/s wr, 6 op/s 2024-09-21T10:12:38.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 6 2024-09-21T10:12:39.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 7 2024-09-21T10:12:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:39 smithi145 ceph-mon[89359]: pgmap v894: 809 pgs: 2 peering, 807 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 472 B/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:12:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:39 smithi119 ceph-mon[101432]: pgmap v894: 809 pgs: 2 peering, 807 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 472 B/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:12:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:39 smithi119 ceph-mon[105288]: pgmap v894: 809 pgs: 2 peering, 807 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.5 KiB/s rd, 472 B/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:12:40.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 8 2024-09-21T10:12:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:12:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:12:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:41.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 9 2024-09-21T10:12:42.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:41 smithi145 ceph-mon[89359]: pgmap v895: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 383 B/s wr, 4 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:41 smithi119 ceph-mon[105288]: pgmap v895: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 383 B/s wr, 4 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:41 smithi119 ceph-mon[101432]: pgmap v895: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.6 KiB/s rd, 383 B/s wr, 4 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:42.146 INFO:tasks.workunit.client.0.smithi119.stderr:waiting for timed out ... 10 2024-09-21T10:12:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:12:42.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:12:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:12:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:43 smithi119 ceph-mon[101432]: pgmap v896: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 255 B/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:43 smithi119 ceph-mon[105288]: pgmap v896: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 255 B/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:43 smithi145 ceph-mon[89359]: pgmap v896: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 KiB/s rd, 255 B/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:44.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:12:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:12:44] "GET /metrics HTTP/1.1" 200 89579 "" "Prometheus/2.51.0" 2024-09-21T10:12:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:45 smithi145 ceph-mon[89359]: pgmap v897: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.6 KiB/s rd, 216 B/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:45 smithi119 ceph-mon[105288]: pgmap v897: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.6 KiB/s rd, 216 B/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:45 smithi119 ceph-mon[101432]: pgmap v897: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.6 KiB/s rd, 216 B/s wr, 3 op/s; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:46.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:47 smithi145 ceph-mon[89359]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[101432]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:12:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:12:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:47 smithi119 ceph-mon[105288]: pgmap v898: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:12:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:12:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:49 smithi119 ceph-mon[105288]: pgmap v899: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:49 smithi119 ceph-mon[101432]: pgmap v899: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:49 smithi145 ceph-mon[89359]: pgmap v899: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1 B/s, 0 objects/s recovering 2024-09-21T10:12:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:12:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:12:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:12:52.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:51 smithi145 ceph-mon[89359]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:51 smithi119 ceph-mon[105288]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:51 smithi119 ceph-mon[101432]: pgmap v900: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:53.147 INFO:tasks.workunit.client.0.smithi119.stderr:test retry succeeds 2024-09-21T10:12:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:53 smithi145 ceph-mon[89359]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:53 smithi119 ceph-mon[105288]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:53 smithi119 ceph-mon[101432]: pgmap v901: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:12:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:54 smithi119 ceph-mon[101432]: osdmap e813: 8 total, 8 up, 8 in 2024-09-21T10:12:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:12:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:12:54] "GET /metrics HTTP/1.1" 200 89588 "" "Prometheus/2.51.0" 2024-09-21T10:12:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:54 smithi119 ceph-mon[105288]: osdmap e813: 8 total, 8 up, 8 in 2024-09-21T10:12:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:54 smithi145 ceph-mon[89359]: osdmap e813: 8 total, 8 up, 8 in 2024-09-21T10:12:55.805 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.QuiesceWatchTimeout (32546 ms) 2024-09-21T10:12:55.805 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.WriteZeroes 2024-09-21T10:12:55.805 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:55.866 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.WriteZeroes (61 ms) 2024-09-21T10:12:55.866 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.WriteZeroesThickProvision 2024-09-21T10:12:55.866 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:55.934 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.WriteZeroesThickProvision (69 ms) 2024-09-21T10:12:55.934 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestLibRBD.ConcurentOperations 2024-09-21T10:12:55.934 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:12:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:55 smithi145 ceph-mon[89359]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-09-21T10:12:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:55 smithi145 ceph-mon[89359]: osdmap e814: 8 total, 8 up, 8 in 2024-09-21T10:12:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:55 smithi119 ceph-mon[105288]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-09-21T10:12:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:55 smithi119 ceph-mon[105288]: osdmap e814: 8 total, 8 up, 8 in 2024-09-21T10:12:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:55 smithi119 ceph-mon[101432]: pgmap v903: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 409 B/s rd, 0 op/s 2024-09-21T10:12:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:55 smithi119 ceph-mon[101432]: osdmap e814: 8 total, 8 up, 8 in 2024-09-21T10:12:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:56 smithi145 ceph-mon[89359]: osdmap e815: 8 total, 8 up, 8 in 2024-09-21T10:12:57.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:56 smithi119 ceph-mon[105288]: osdmap e815: 8 total, 8 up, 8 in 2024-09-21T10:12:57.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:56 smithi119 ceph-mon[101432]: osdmap e815: 8 total, 8 up, 8 in 2024-09-21T10:12:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:57 smithi145 ceph-mon[89359]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-09-21T10:12:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:57 smithi145 ceph-mon[89359]: osdmap e816: 8 total, 8 up, 8 in 2024-09-21T10:12:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:57 smithi119 ceph-mon[105288]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-09-21T10:12:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:57 smithi119 ceph-mon[105288]: osdmap e816: 8 total, 8 up, 8 in 2024-09-21T10:12:58.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:57 smithi119 ceph-mon[101432]: pgmap v906: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-09-21T10:12:58.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:57 smithi119 ceph-mon[101432]: osdmap e816: 8 total, 8 up, 8 in 2024-09-21T10:12:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:58 smithi145 ceph-mon[89359]: osdmap e817: 8 total, 8 up, 8 in 2024-09-21T10:12:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:58 smithi119 ceph-mon[101432]: osdmap e817: 8 total, 8 up, 8 in 2024-09-21T10:12:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:58 smithi119 ceph-mon[105288]: osdmap e817: 8 total, 8 up, 8 in 2024-09-21T10:13:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:59 smithi145 ceph-mon[89359]: pgmap v909: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 91 KiB/s rd, 6.7 KiB/s wr, 123 op/s 2024-09-21T10:13:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:12:59 smithi145 ceph-mon[89359]: osdmap e818: 8 total, 8 up, 8 in 2024-09-21T10:13:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:59 smithi119 ceph-mon[101432]: pgmap v909: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 91 KiB/s rd, 6.7 KiB/s wr, 123 op/s 2024-09-21T10:13:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:12:59 smithi119 ceph-mon[101432]: osdmap e818: 8 total, 8 up, 8 in 2024-09-21T10:13:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:59 smithi119 ceph-mon[105288]: pgmap v909: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 91 KiB/s rd, 6.7 KiB/s wr, 123 op/s 2024-09-21T10:13:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:12:59 smithi119 ceph-mon[105288]: osdmap e818: 8 total, 8 up, 8 in 2024-09-21T10:13:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:13:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:13:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:00 smithi145 ceph-mon[89359]: osdmap e819: 8 total, 8 up, 8 in 2024-09-21T10:13:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:00 smithi119 ceph-mon[105288]: osdmap e819: 8 total, 8 up, 8 in 2024-09-21T10:13:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:00 smithi119 ceph-mon[101432]: osdmap e819: 8 total, 8 up, 8 in 2024-09-21T10:13:02.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:01 smithi119 ceph-mon[101432]: pgmap v912: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 113 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:13:02.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:01 smithi119 ceph-mon[101432]: osdmap e820: 8 total, 8 up, 8 in 2024-09-21T10:13:02.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:01 smithi119 ceph-mon[105288]: pgmap v912: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 113 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:13:02.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:01 smithi119 ceph-mon[105288]: osdmap e820: 8 total, 8 up, 8 in 2024-09-21T10:13:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:01 smithi145 ceph-mon[89359]: pgmap v912: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 113 KiB/s rd, 8.7 KiB/s wr, 160 op/s 2024-09-21T10:13:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:01 smithi145 ceph-mon[89359]: osdmap e820: 8 total, 8 up, 8 in 2024-09-21T10:13:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:02 smithi145 ceph-mon[89359]: osdmap e821: 8 total, 8 up, 8 in 2024-09-21T10:13:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:03.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:02 smithi119 ceph-mon[101432]: osdmap e821: 8 total, 8 up, 8 in 2024-09-21T10:13:03.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:02 smithi119 ceph-mon[105288]: osdmap e821: 8 total, 8 up, 8 in 2024-09-21T10:13:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:03 smithi145 ceph-mon[89359]: pgmap v915: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 4.2 KiB/s wr, 57 op/s 2024-09-21T10:13:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:03 smithi145 ceph-mon[89359]: osdmap e822: 8 total, 8 up, 8 in 2024-09-21T10:13:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:03 smithi119 ceph-mon[105288]: pgmap v915: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 4.2 KiB/s wr, 57 op/s 2024-09-21T10:13:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:03 smithi119 ceph-mon[105288]: osdmap e822: 8 total, 8 up, 8 in 2024-09-21T10:13:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:03 smithi119 ceph-mon[101432]: pgmap v915: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 4.2 KiB/s wr, 57 op/s 2024-09-21T10:13:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:03 smithi119 ceph-mon[101432]: osdmap e822: 8 total, 8 up, 8 in 2024-09-21T10:13:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:04 smithi119 ceph-mon[105288]: osdmap e823: 8 total, 8 up, 8 in 2024-09-21T10:13:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:04 smithi119 ceph-mon[105288]: osdmap e824: 8 total, 8 up, 8 in 2024-09-21T10:13:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:13:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:13:04] "GET /metrics HTTP/1.1" 200 89590 "" "Prometheus/2.51.0" 2024-09-21T10:13:04.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:04 smithi119 ceph-mon[101432]: osdmap e823: 8 total, 8 up, 8 in 2024-09-21T10:13:04.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:04 smithi119 ceph-mon[101432]: osdmap e824: 8 total, 8 up, 8 in 2024-09-21T10:13:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:04 smithi145 ceph-mon[89359]: osdmap e823: 8 total, 8 up, 8 in 2024-09-21T10:13:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:04 smithi145 ceph-mon[89359]: osdmap e824: 8 total, 8 up, 8 in 2024-09-21T10:13:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:06 smithi145 ceph-mon[89359]: pgmap v919: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 3.6 KiB/s wr, 16 op/s 2024-09-21T10:13:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:06 smithi145 ceph-mon[89359]: osdmap e825: 8 total, 8 up, 8 in 2024-09-21T10:13:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:06 smithi119 ceph-mon[105288]: pgmap v919: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 3.6 KiB/s wr, 16 op/s 2024-09-21T10:13:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:06 smithi119 ceph-mon[105288]: osdmap e825: 8 total, 8 up, 8 in 2024-09-21T10:13:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:06 smithi119 ceph-mon[101432]: pgmap v919: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 3.6 KiB/s wr, 16 op/s 2024-09-21T10:13:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:06 smithi119 ceph-mon[101432]: osdmap e825: 8 total, 8 up, 8 in 2024-09-21T10:13:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:07 smithi145 ceph-mon[89359]: osdmap e826: 8 total, 8 up, 8 in 2024-09-21T10:13:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:07 smithi119 ceph-mon[105288]: osdmap e826: 8 total, 8 up, 8 in 2024-09-21T10:13:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:07 smithi119 ceph-mon[101432]: osdmap e826: 8 total, 8 up, 8 in 2024-09-21T10:13:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:08 smithi145 ceph-mon[89359]: pgmap v922: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 3.9 KiB/s wr, 16 op/s 2024-09-21T10:13:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:08 smithi145 ceph-mon[89359]: osdmap e827: 8 total, 8 up, 8 in 2024-09-21T10:13:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:08 smithi119 ceph-mon[101432]: pgmap v922: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 3.9 KiB/s wr, 16 op/s 2024-09-21T10:13:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:08 smithi119 ceph-mon[101432]: osdmap e827: 8 total, 8 up, 8 in 2024-09-21T10:13:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:08 smithi119 ceph-mon[105288]: pgmap v922: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 3.9 KiB/s wr, 16 op/s 2024-09-21T10:13:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:08 smithi119 ceph-mon[105288]: osdmap e827: 8 total, 8 up, 8 in 2024-09-21T10:13:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:09 smithi145 ceph-mon[89359]: osdmap e828: 8 total, 8 up, 8 in 2024-09-21T10:13:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:09 smithi145 ceph-mon[89359]: pgmap v925: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 178 KiB/s rd, 6.0 KiB/s wr, 244 op/s 2024-09-21T10:13:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:09 smithi119 ceph-mon[105288]: osdmap e828: 8 total, 8 up, 8 in 2024-09-21T10:13:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:09 smithi119 ceph-mon[105288]: pgmap v925: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 178 KiB/s rd, 6.0 KiB/s wr, 244 op/s 2024-09-21T10:13:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:09 smithi119 ceph-mon[101432]: osdmap e828: 8 total, 8 up, 8 in 2024-09-21T10:13:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:09 smithi119 ceph-mon[101432]: pgmap v925: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 178 KiB/s rd, 6.0 KiB/s wr, 244 op/s 2024-09-21T10:13:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:10 smithi145 ceph-mon[89359]: osdmap e829: 8 total, 8 up, 8 in 2024-09-21T10:13:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:13:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:13:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:10 smithi119 ceph-mon[101432]: osdmap e829: 8 total, 8 up, 8 in 2024-09-21T10:13:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:10 smithi119 ceph-mon[105288]: osdmap e829: 8 total, 8 up, 8 in 2024-09-21T10:13:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:11 smithi145 ceph-mon[89359]: osdmap e830: 8 total, 8 up, 8 in 2024-09-21T10:13:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:11 smithi145 ceph-mon[89359]: pgmap v928: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 5.7 KiB/s wr, 251 op/s 2024-09-21T10:13:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:11 smithi119 ceph-mon[101432]: osdmap e830: 8 total, 8 up, 8 in 2024-09-21T10:13:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:11 smithi119 ceph-mon[101432]: pgmap v928: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 5.7 KiB/s wr, 251 op/s 2024-09-21T10:13:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:11 smithi119 ceph-mon[105288]: osdmap e830: 8 total, 8 up, 8 in 2024-09-21T10:13:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:11 smithi119 ceph-mon[105288]: pgmap v928: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 182 KiB/s rd, 5.7 KiB/s wr, 251 op/s 2024-09-21T10:13:12.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:12 smithi145 ceph-mon[89359]: osdmap e831: 8 total, 8 up, 8 in 2024-09-21T10:13:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:12 smithi119 ceph-mon[105288]: osdmap e831: 8 total, 8 up, 8 in 2024-09-21T10:13:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:12 smithi119 ceph-mon[101432]: osdmap e831: 8 total, 8 up, 8 in 2024-09-21T10:13:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:13 smithi145 ceph-mon[89359]: osdmap e832: 8 total, 8 up, 8 in 2024-09-21T10:13:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:13 smithi145 ceph-mon[89359]: pgmap v931: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T10:13:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:13 smithi119 ceph-mon[105288]: osdmap e832: 8 total, 8 up, 8 in 2024-09-21T10:13:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:13 smithi119 ceph-mon[105288]: pgmap v931: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T10:13:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:13 smithi119 ceph-mon[101432]: osdmap e832: 8 total, 8 up, 8 in 2024-09-21T10:13:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:13 smithi119 ceph-mon[101432]: pgmap v931: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T10:13:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:14 smithi119 ceph-mon[101432]: osdmap e833: 8 total, 8 up, 8 in 2024-09-21T10:13:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:14 smithi119 ceph-mon[101432]: osdmap e834: 8 total, 8 up, 8 in 2024-09-21T10:13:14.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:14 smithi119 ceph-mon[105288]: osdmap e833: 8 total, 8 up, 8 in 2024-09-21T10:13:14.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:14 smithi119 ceph-mon[105288]: osdmap e834: 8 total, 8 up, 8 in 2024-09-21T10:13:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:14 smithi145 ceph-mon[89359]: osdmap e833: 8 total, 8 up, 8 in 2024-09-21T10:13:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:14 smithi145 ceph-mon[89359]: osdmap e834: 8 total, 8 up, 8 in 2024-09-21T10:13:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:13:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:13:14] "GET /metrics HTTP/1.1" 200 89590 "" "Prometheus/2.51.0" 2024-09-21T10:13:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:16 smithi145 ceph-mon[89359]: pgmap v934: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 KiB/s wr, 23 op/s 2024-09-21T10:13:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:16 smithi145 ceph-mon[89359]: osdmap e835: 8 total, 8 up, 8 in 2024-09-21T10:13:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:16 smithi119 ceph-mon[101432]: pgmap v934: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 KiB/s wr, 23 op/s 2024-09-21T10:13:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:16 smithi119 ceph-mon[101432]: osdmap e835: 8 total, 8 up, 8 in 2024-09-21T10:13:16.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:16 smithi119 ceph-mon[105288]: pgmap v934: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 5.0 KiB/s wr, 23 op/s 2024-09-21T10:13:16.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:16 smithi119 ceph-mon[105288]: osdmap e835: 8 total, 8 up, 8 in 2024-09-21T10:13:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:17 smithi145 ceph-mon[89359]: osdmap e836: 8 total, 8 up, 8 in 2024-09-21T10:13:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:17 smithi119 ceph-mon[105288]: osdmap e836: 8 total, 8 up, 8 in 2024-09-21T10:13:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:17 smithi119 ceph-mon[101432]: osdmap e836: 8 total, 8 up, 8 in 2024-09-21T10:13:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:18 smithi145 ceph-mon[89359]: pgmap v937: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 5.0 KiB/s wr, 25 op/s 2024-09-21T10:13:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:18 smithi145 ceph-mon[89359]: osdmap e837: 8 total, 8 up, 8 in 2024-09-21T10:13:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:18 smithi119 ceph-mon[105288]: pgmap v937: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 5.0 KiB/s wr, 25 op/s 2024-09-21T10:13:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:18 smithi119 ceph-mon[105288]: osdmap e837: 8 total, 8 up, 8 in 2024-09-21T10:13:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:18 smithi119 ceph-mon[101432]: pgmap v937: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 5.0 KiB/s wr, 25 op/s 2024-09-21T10:13:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:18 smithi119 ceph-mon[101432]: osdmap e837: 8 total, 8 up, 8 in 2024-09-21T10:13:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:19 smithi145 ceph-mon[89359]: osdmap e838: 8 total, 8 up, 8 in 2024-09-21T10:13:19.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:19 smithi145 ceph-mon[89359]: osdmap e839: 8 total, 8 up, 8 in 2024-09-21T10:13:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:19 smithi119 ceph-mon[105288]: osdmap e838: 8 total, 8 up, 8 in 2024-09-21T10:13:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:19 smithi119 ceph-mon[105288]: osdmap e839: 8 total, 8 up, 8 in 2024-09-21T10:13:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:19 smithi119 ceph-mon[101432]: osdmap e838: 8 total, 8 up, 8 in 2024-09-21T10:13:19.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:19 smithi119 ceph-mon[101432]: osdmap e839: 8 total, 8 up, 8 in 2024-09-21T10:13:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:20 smithi145 ceph-mon[89359]: pgmap v940: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 72 op/s 2024-09-21T10:13:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:13:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:13:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:20 smithi119 ceph-mon[101432]: pgmap v940: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 72 op/s 2024-09-21T10:13:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:20 smithi119 ceph-mon[105288]: pgmap v940: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 72 op/s 2024-09-21T10:13:21.261 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestLibRBD.ConcurentOperations (25326 ms) 2024-09-21T10:13:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:21 smithi145 ceph-mon[89359]: osdmap e840: 8 total, 8 up, 8 in 2024-09-21T10:13:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:21 smithi145 ceph-mon[89359]: pgmap v943: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.2 KiB/s wr, 72 op/s 2024-09-21T10:13:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:21 smithi119 ceph-mon[101432]: osdmap e840: 8 total, 8 up, 8 in 2024-09-21T10:13:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:21 smithi119 ceph-mon[101432]: pgmap v943: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.2 KiB/s wr, 72 op/s 2024-09-21T10:13:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:21 smithi119 ceph-mon[105288]: osdmap e840: 8 total, 8 up, 8 in 2024-09-21T10:13:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:21 smithi119 ceph-mon[105288]: pgmap v943: 809 pgs: 809 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 4.2 KiB/s wr, 72 op/s 2024-09-21T10:13:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:22 smithi145 ceph-mon[89359]: osdmap e841: 8 total, 8 up, 8 in 2024-09-21T10:13:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:22 smithi119 ceph-mon[101432]: osdmap e841: 8 total, 8 up, 8 in 2024-09-21T10:13:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:22 smithi119 ceph-mon[105288]: osdmap e841: 8 total, 8 up, 8 in 2024-09-21T10:13:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:23 smithi145 ceph-mon[89359]: osdmap e842: 8 total, 8 up, 8 in 2024-09-21T10:13:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:23 smithi145 ceph-mon[89359]: pgmap v946: 777 pgs: 777 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:23 smithi145 ceph-mon[89359]: osdmap e843: 8 total, 8 up, 8 in 2024-09-21T10:13:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:23 smithi119 ceph-mon[101432]: osdmap e842: 8 total, 8 up, 8 in 2024-09-21T10:13:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:23 smithi119 ceph-mon[101432]: pgmap v946: 777 pgs: 777 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:23 smithi119 ceph-mon[101432]: osdmap e843: 8 total, 8 up, 8 in 2024-09-21T10:13:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:23 smithi119 ceph-mon[105288]: osdmap e842: 8 total, 8 up, 8 in 2024-09-21T10:13:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:23 smithi119 ceph-mon[105288]: pgmap v946: 777 pgs: 777 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:23 smithi119 ceph-mon[105288]: osdmap e843: 8 total, 8 up, 8 in 2024-09-21T10:13:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:13:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:13:24] "GET /metrics HTTP/1.1" 200 89590 "" "Prometheus/2.51.0" 2024-09-21T10:13:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:25 smithi145 ceph-mon[89359]: osdmap e844: 8 total, 8 up, 8 in 2024-09-21T10:13:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:25 smithi119 ceph-mon[105288]: osdmap e844: 8 total, 8 up, 8 in 2024-09-21T10:13:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:25 smithi119 ceph-mon[101432]: osdmap e844: 8 total, 8 up, 8 in 2024-09-21T10:13:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:26 smithi145 ceph-mon[89359]: pgmap v949: 713 pgs: 713 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:26 smithi145 ceph-mon[89359]: osdmap e845: 8 total, 8 up, 8 in 2024-09-21T10:13:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:26 smithi119 ceph-mon[101432]: pgmap v949: 713 pgs: 713 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:26 smithi119 ceph-mon[101432]: osdmap e845: 8 total, 8 up, 8 in 2024-09-21T10:13:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:26 smithi119 ceph-mon[105288]: pgmap v949: 713 pgs: 713 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:26 smithi119 ceph-mon[105288]: osdmap e845: 8 total, 8 up, 8 in 2024-09-21T10:13:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:27 smithi145 ceph-mon[89359]: osdmap e846: 8 total, 8 up, 8 in 2024-09-21T10:13:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:27 smithi145 ceph-mon[89359]: osdmap e847: 8 total, 8 up, 8 in 2024-09-21T10:13:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:27 smithi119 ceph-mon[101432]: osdmap e846: 8 total, 8 up, 8 in 2024-09-21T10:13:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:27 smithi119 ceph-mon[101432]: osdmap e847: 8 total, 8 up, 8 in 2024-09-21T10:13:27.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:27 smithi119 ceph-mon[105288]: osdmap e846: 8 total, 8 up, 8 in 2024-09-21T10:13:27.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:27 smithi119 ceph-mon[105288]: osdmap e847: 8 total, 8 up, 8 in 2024-09-21T10:13:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:28 smithi145 ceph-mon[89359]: pgmap v952: 649 pgs: 649 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:28 smithi145 ceph-mon[89359]: osdmap e848: 8 total, 8 up, 8 in 2024-09-21T10:13:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:28 smithi119 ceph-mon[105288]: pgmap v952: 649 pgs: 649 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:28 smithi119 ceph-mon[105288]: osdmap e848: 8 total, 8 up, 8 in 2024-09-21T10:13:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:28 smithi119 ceph-mon[101432]: pgmap v952: 649 pgs: 649 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:28 smithi119 ceph-mon[101432]: osdmap e848: 8 total, 8 up, 8 in 2024-09-21T10:13:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:30 smithi145 ceph-mon[89359]: pgmap v955: 585 pgs: 585 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:30 smithi145 ceph-mon[89359]: osdmap e849: 8 total, 8 up, 8 in 2024-09-21T10:13:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:13:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:13:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:30 smithi119 ceph-mon[105288]: pgmap v955: 585 pgs: 585 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:30 smithi119 ceph-mon[105288]: osdmap e849: 8 total, 8 up, 8 in 2024-09-21T10:13:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:30 smithi119 ceph-mon[101432]: pgmap v955: 585 pgs: 585 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:30 smithi119 ceph-mon[101432]: osdmap e849: 8 total, 8 up, 8 in 2024-09-21T10:13:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:31 smithi145 ceph-mon[89359]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:13:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:31 smithi145 ceph-mon[89359]: osdmap e850: 8 total, 8 up, 8 in 2024-09-21T10:13:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:31 smithi145 ceph-mon[89359]: osdmap e851: 8 total, 8 up, 8 in 2024-09-21T10:13:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:31 smithi119 ceph-mon[105288]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:13:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:31 smithi119 ceph-mon[105288]: osdmap e850: 8 total, 8 up, 8 in 2024-09-21T10:13:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:31 smithi119 ceph-mon[105288]: osdmap e851: 8 total, 8 up, 8 in 2024-09-21T10:13:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:31 smithi119 ceph-mon[101432]: Health check update: 1 pool(s) full (POOL_FULL) 2024-09-21T10:13:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:31 smithi119 ceph-mon[101432]: osdmap e850: 8 total, 8 up, 8 in 2024-09-21T10:13:31.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:31 smithi119 ceph-mon[101432]: osdmap e851: 8 total, 8 up, 8 in 2024-09-21T10:13:32.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:32 smithi119 ceph-mon[105288]: pgmap v958: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:32.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:32 smithi119 ceph-mon[105288]: osdmap e852: 8 total, 8 up, 8 in 2024-09-21T10:13:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:32 smithi119 ceph-mon[101432]: pgmap v958: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:32.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:32 smithi119 ceph-mon[101432]: osdmap e852: 8 total, 8 up, 8 in 2024-09-21T10:13:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:32 smithi145 ceph-mon[89359]: pgmap v958: 521 pgs: 521 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:32 smithi145 ceph-mon[89359]: osdmap e852: 8 total, 8 up, 8 in 2024-09-21T10:13:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:33 smithi145 ceph-mon[89359]: pgmap v961: 457 pgs: 457 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:33 smithi145 ceph-mon[89359]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:13:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:33 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:13:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:33 smithi145 ceph-mon[89359]: osdmap e853: 8 total, 8 up, 8 in 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[105288]: pgmap v961: 457 pgs: 457 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[105288]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[105288]: osdmap e853: 8 total, 8 up, 8 in 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[101432]: pgmap v961: 457 pgs: 457 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:33.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[101432]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-21T10:13:33.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:13:33.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:33 smithi119 ceph-mon[101432]: osdmap e853: 8 total, 8 up, 8 in 2024-09-21T10:13:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:13:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:13:34] "GET /metrics HTTP/1.1" 200 63109 "" "Prometheus/2.51.0" 2024-09-21T10:13:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:35 smithi145 ceph-mon[89359]: osdmap e854: 8 total, 8 up, 8 in 2024-09-21T10:13:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:35 smithi119 ceph-mon[105288]: osdmap e854: 8 total, 8 up, 8 in 2024-09-21T10:13:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:35 smithi119 ceph-mon[101432]: osdmap e854: 8 total, 8 up, 8 in 2024-09-21T10:13:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:36 smithi145 ceph-mon[89359]: pgmap v964: 393 pgs: 393 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:36 smithi145 ceph-mon[89359]: osdmap e855: 8 total, 8 up, 8 in 2024-09-21T10:13:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:36 smithi119 ceph-mon[105288]: pgmap v964: 393 pgs: 393 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:36 smithi119 ceph-mon[105288]: osdmap e855: 8 total, 8 up, 8 in 2024-09-21T10:13:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:36 smithi119 ceph-mon[101432]: pgmap v964: 393 pgs: 393 active+clean; 2.6 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:36 smithi119 ceph-mon[101432]: osdmap e855: 8 total, 8 up, 8 in 2024-09-21T10:13:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:37 smithi145 ceph-mon[89359]: osdmap e856: 8 total, 8 up, 8 in 2024-09-21T10:13:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:37 smithi119 ceph-mon[105288]: osdmap e856: 8 total, 8 up, 8 in 2024-09-21T10:13:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:37 smithi119 ceph-mon[101432]: osdmap e856: 8 total, 8 up, 8 in 2024-09-21T10:13:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:38 smithi145 ceph-mon[89359]: pgmap v967: 329 pgs: 329 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:38 smithi145 ceph-mon[89359]: osdmap e857: 8 total, 8 up, 8 in 2024-09-21T10:13:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:38 smithi145 ceph-mon[89359]: osdmap e858: 8 total, 8 up, 8 in 2024-09-21T10:13:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:38 smithi119 ceph-mon[105288]: pgmap v967: 329 pgs: 329 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:38 smithi119 ceph-mon[105288]: osdmap e857: 8 total, 8 up, 8 in 2024-09-21T10:13:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:38 smithi119 ceph-mon[105288]: osdmap e858: 8 total, 8 up, 8 in 2024-09-21T10:13:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:38 smithi119 ceph-mon[101432]: pgmap v967: 329 pgs: 329 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:38.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:38 smithi119 ceph-mon[101432]: osdmap e857: 8 total, 8 up, 8 in 2024-09-21T10:13:38.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:38 smithi119 ceph-mon[101432]: osdmap e858: 8 total, 8 up, 8 in 2024-09-21T10:13:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:13:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:13:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:40 smithi145 ceph-mon[89359]: pgmap v970: 265 pgs: 265 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:40 smithi145 ceph-mon[89359]: osdmap e859: 8 total, 8 up, 8 in 2024-09-21T10:13:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:40 smithi119 ceph-mon[105288]: pgmap v970: 265 pgs: 265 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:40 smithi119 ceph-mon[105288]: osdmap e859: 8 total, 8 up, 8 in 2024-09-21T10:13:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:40 smithi119 ceph-mon[101432]: pgmap v970: 265 pgs: 265 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:40 smithi119 ceph-mon[101432]: osdmap e859: 8 total, 8 up, 8 in 2024-09-21T10:13:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:41 smithi145 ceph-mon[89359]: osdmap e860: 8 total, 8 up, 8 in 2024-09-21T10:13:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:41 smithi145 ceph-mon[89359]: pgmap v973: 201 pgs: 201 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:41 smithi145 ceph-mon[89359]: osdmap e861: 8 total, 8 up, 8 in 2024-09-21T10:13:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:41 smithi119 ceph-mon[105288]: osdmap e860: 8 total, 8 up, 8 in 2024-09-21T10:13:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:41 smithi119 ceph-mon[105288]: pgmap v973: 201 pgs: 201 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:41 smithi119 ceph-mon[105288]: osdmap e861: 8 total, 8 up, 8 in 2024-09-21T10:13:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:41 smithi119 ceph-mon[101432]: osdmap e860: 8 total, 8 up, 8 in 2024-09-21T10:13:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:41 smithi119 ceph-mon[101432]: pgmap v973: 201 pgs: 201 active+clean; 587 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:41 smithi119 ceph-mon[101432]: osdmap e861: 8 total, 8 up, 8 in 2024-09-21T10:13:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:43 smithi145 ceph-mon[89359]: osdmap e862: 8 total, 8 up, 8 in 2024-09-21T10:13:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:43 smithi119 ceph-mon[105288]: osdmap e862: 8 total, 8 up, 8 in 2024-09-21T10:13:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:43 smithi119 ceph-mon[101432]: osdmap e862: 8 total, 8 up, 8 in 2024-09-21T10:13:44.178 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 161 tests from TestLibRBD (379938 ms total) 2024-09-21T10:13:44.178 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:13:44.178 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 19 tests from DiffIterateTest/0, where TypeParam = DiffIterateParams 2024-09-21T10:13:44.208 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterate 2024-09-21T10:13:44.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:44 smithi119 ceph-mon[101432]: pgmap v976: 137 pgs: 137 active+clean; 586 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:44.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:44 smithi119 ceph-mon[101432]: osdmap e863: 8 total, 8 up, 8 in 2024-09-21T10:13:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:44 smithi119 ceph-mon[105288]: pgmap v976: 137 pgs: 137 active+clean; 586 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:44 smithi119 ceph-mon[105288]: osdmap e863: 8 total, 8 up, 8 in 2024-09-21T10:13:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:44 smithi145 ceph-mon[89359]: pgmap v976: 137 pgs: 137 active+clean; 586 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:44 smithi145 ceph-mon[89359]: osdmap e863: 8 total, 8 up, 8 in 2024-09-21T10:13:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:13:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:13:44] "GET /metrics HTTP/1.1" 200 63109 "" "Prometheus/2.51.0" 2024-09-21T10:13:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:45 smithi145 ceph-mon[89359]: osdmap e864: 8 total, 8 up, 8 in 2024-09-21T10:13:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:45 smithi145 ceph-mon[89359]: pgmap v979: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:45 smithi119 ceph-mon[101432]: osdmap e864: 8 total, 8 up, 8 in 2024-09-21T10:13:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:45 smithi119 ceph-mon[101432]: pgmap v979: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:45 smithi119 ceph-mon[105288]: osdmap e864: 8 total, 8 up, 8 in 2024-09-21T10:13:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:45 smithi119 ceph-mon[105288]: pgmap v979: 73 pgs: 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:46.192 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:13:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:46 smithi145 ceph-mon[89359]: osdmap e865: 8 total, 8 up, 8 in 2024-09-21T10:13:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/474266996' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:46 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:46 smithi119 ceph-mon[101432]: osdmap e865: 8 total, 8 up, 8 in 2024-09-21T10:13:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/474266996' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:46 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:46 smithi119 ceph-mon[105288]: osdmap e865: 8 total, 8 up, 8 in 2024-09-21T10:13:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/474266996' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:46 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:13:47.485 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[101432]: osdmap e866: 8 total, 8 up, 8 in 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[101432]: pgmap v982: 105 pgs: 4 creating+peering, 28 unknown, 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[105288]: osdmap e866: 8 total, 8 up, 8 in 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:47.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:47 smithi119 ceph-mon[105288]: pgmap v982: 105 pgs: 4 creating+peering, 28 unknown, 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:47.512 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:47 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:13:47.512 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:47 smithi145 ceph-mon[89359]: osdmap e866: 8 total, 8 up, 8 in 2024-09-21T10:13:47.512 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:13:47.512 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:47 smithi145 ceph-mon[89359]: pgmap v982: 105 pgs: 4 creating+peering, 28 unknown, 73 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:13:48.218 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [1199503~23398,1307850~16020,4951613~90932,7804170~50654,11674025~2593,13589604~3616,13757128~86483,14276057~42118,17852495~49482,19293127~98601] 2024-09-21T10:13:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:48 smithi145 ceph-mon[89359]: osdmap e867: 8 total, 8 up, 8 in 2024-09-21T10:13:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:48 smithi119 ceph-mon[101432]: osdmap e867: 8 total, 8 up, 8 in 2024-09-21T10:13:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:48 smithi119 ceph-mon[105288]: osdmap e867: 8 total, 8 up, 8 in 2024-09-21T10:13:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:13:49.215 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [2664672~36046,4051055~40337,8026225~3624,8214043~73918,9911108~15076,11088343~42482,13647878~36335,14676580~75529,17993990~53087,18375929~42307] 2024-09-21T10:13:49.223 INFO:tasks.workunit.client.0.smithi119.stdout: diff was [1323870~2767522,7854824~433137,9911108~15076,11088343~42482,13647878~36335,14318175~433934,17993990~53087,18375929~42307] 2024-09-21T10:13:49.226 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterate (5018 ms) 2024-09-21T10:13:49.226 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministic 2024-09-21T10:13:49.227 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:13:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:49 smithi145 ceph-mon[89359]: osdmap e868: 8 total, 8 up, 8 in 2024-09-21T10:13:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:49 smithi145 ceph-mon[89359]: pgmap v985: 105 pgs: 4 creating+peering, 4 unknown, 97 active+clean; 9.7 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 76 KiB/s wr, 22 op/s 2024-09-21T10:13:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:49 smithi119 ceph-mon[105288]: osdmap e868: 8 total, 8 up, 8 in 2024-09-21T10:13:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:49 smithi119 ceph-mon[105288]: pgmap v985: 105 pgs: 4 creating+peering, 4 unknown, 97 active+clean; 9.7 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 76 KiB/s wr, 22 op/s 2024-09-21T10:13:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:49 smithi119 ceph-mon[101432]: osdmap e868: 8 total, 8 up, 8 in 2024-09-21T10:13:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:49 smithi119 ceph-mon[101432]: pgmap v985: 105 pgs: 4 creating+peering, 4 unknown, 97 active+clean; 9.7 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 76 KiB/s wr, 22 op/s 2024-09-21T10:13:50.518 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:50 smithi145 ceph-mon[89359]: osdmap e869: 8 total, 8 up, 8 in 2024-09-21T10:13:50.518 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:50.518 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:50.518 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:13:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:13:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:13:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:50 smithi119 ceph-mon[101432]: osdmap e869: 8 total, 8 up, 8 in 2024-09-21T10:13:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:50 smithi119 ceph-mon[105288]: osdmap e869: 8 total, 8 up, 8 in 2024-09-21T10:13:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:51 smithi145 ceph-mon[89359]: osdmap e870: 8 total, 8 up, 8 in 2024-09-21T10:13:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:51 smithi145 ceph-mon[89359]: pgmap v988: 105 pgs: 105 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 165 KiB/s wr, 25 op/s 2024-09-21T10:13:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[105288]: osdmap e870: 8 total, 8 up, 8 in 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[105288]: pgmap v988: 105 pgs: 105 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 165 KiB/s wr, 25 op/s 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[101432]: osdmap e870: 8 total, 8 up, 8 in 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[101432]: pgmap v988: 105 pgs: 105 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 165 KiB/s wr, 25 op/s 2024-09-21T10:13:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:51.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:52.247 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:13:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:52 smithi145 ceph-mon[89359]: osdmap e871: 8 total, 8 up, 8 in 2024-09-21T10:13:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:52 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[105288]: osdmap e871: 8 total, 8 up, 8 in 2024-09-21T10:13:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[101432]: osdmap e871: 8 total, 8 up, 8 in 2024-09-21T10:13:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:13:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:13:52.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:52 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:53 smithi145 ceph-mon[89359]: osdmap e872: 8 total, 8 up, 8 in 2024-09-21T10:13:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:53 smithi145 ceph-mon[89359]: pgmap v991: 105 pgs: 105 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 89 KiB/s wr, 2 op/s 2024-09-21T10:13:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:53 smithi119 ceph-mon[105288]: osdmap e872: 8 total, 8 up, 8 in 2024-09-21T10:13:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:53 smithi119 ceph-mon[105288]: pgmap v991: 105 pgs: 105 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 89 KiB/s wr, 2 op/s 2024-09-21T10:13:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:53 smithi119 ceph-mon[101432]: osdmap e872: 8 total, 8 up, 8 in 2024-09-21T10:13:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:53 smithi119 ceph-mon[101432]: pgmap v991: 105 pgs: 105 active+clean; 16 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 89 KiB/s wr, 2 op/s 2024-09-21T10:13:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:54 smithi119 ceph-mon[105288]: osdmap e873: 8 total, 8 up, 8 in 2024-09-21T10:13:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:54 smithi119 ceph-mon[105288]: osdmap e874: 8 total, 8 up, 8 in 2024-09-21T10:13:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:54 smithi119 ceph-mon[101432]: osdmap e873: 8 total, 8 up, 8 in 2024-09-21T10:13:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:54 smithi119 ceph-mon[101432]: osdmap e874: 8 total, 8 up, 8 in 2024-09-21T10:13:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:54 smithi145 ceph-mon[89359]: osdmap e873: 8 total, 8 up, 8 in 2024-09-21T10:13:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:54 smithi145 ceph-mon[89359]: osdmap e874: 8 total, 8 up, 8 in 2024-09-21T10:13:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:13:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:13:54] "GET /metrics HTTP/1.1" 200 36644 "" "Prometheus/2.51.0" 2024-09-21T10:13:55.161 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:13:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:56 smithi145 ceph-mon[89359]: pgmap v994: 105 pgs: 105 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 69 KiB/s wr, 116 op/s 2024-09-21T10:13:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:56 smithi145 ceph-mon[89359]: osdmap e875: 8 total, 8 up, 8 in 2024-09-21T10:13:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:56 smithi119 ceph-mon[105288]: pgmap v994: 105 pgs: 105 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 69 KiB/s wr, 116 op/s 2024-09-21T10:13:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:56 smithi119 ceph-mon[105288]: osdmap e875: 8 total, 8 up, 8 in 2024-09-21T10:13:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:56 smithi119 ceph-mon[101432]: pgmap v994: 105 pgs: 105 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 69 KiB/s wr, 116 op/s 2024-09-21T10:13:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:56 smithi119 ceph-mon[101432]: osdmap e875: 8 total, 8 up, 8 in 2024-09-21T10:13:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:57 smithi145 ceph-mon[89359]: osdmap e876: 8 total, 8 up, 8 in 2024-09-21T10:13:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:57 smithi119 ceph-mon[105288]: osdmap e876: 8 total, 8 up, 8 in 2024-09-21T10:13:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:57 smithi119 ceph-mon[101432]: osdmap e876: 8 total, 8 up, 8 in 2024-09-21T10:13:58.181 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:13:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:58 smithi145 ceph-mon[89359]: pgmap v997: 105 pgs: 105 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 70 KiB/s wr, 119 op/s 2024-09-21T10:13:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:58 smithi145 ceph-mon[89359]: osdmap e877: 8 total, 8 up, 8 in 2024-09-21T10:13:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:58 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:58 smithi119 ceph-mon[105288]: pgmap v997: 105 pgs: 105 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 70 KiB/s wr, 119 op/s 2024-09-21T10:13:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:58 smithi119 ceph-mon[105288]: osdmap e877: 8 total, 8 up, 8 in 2024-09-21T10:13:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:58 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:58 smithi119 ceph-mon[101432]: pgmap v997: 105 pgs: 105 active+clean; 17 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 76 KiB/s rd, 70 KiB/s wr, 119 op/s 2024-09-21T10:13:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:58 smithi119 ceph-mon[101432]: osdmap e877: 8 total, 8 up, 8 in 2024-09-21T10:13:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:58 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:13:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:13:59 smithi145 ceph-mon[89359]: osdmap e878: 8 total, 8 up, 8 in 2024-09-21T10:13:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:13:59 smithi119 ceph-mon[101432]: osdmap e878: 8 total, 8 up, 8 in 2024-09-21T10:13:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:13:59 smithi119 ceph-mon[105288]: osdmap e878: 8 total, 8 up, 8 in 2024-09-21T10:14:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:00 smithi145 ceph-mon[89359]: pgmap v1000: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 5.0 KiB/s wr, 66 op/s 2024-09-21T10:14:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:00 smithi145 ceph-mon[89359]: osdmap e879: 8 total, 8 up, 8 in 2024-09-21T10:14:00.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:14:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:14:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:00 smithi119 ceph-mon[105288]: pgmap v1000: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 5.0 KiB/s wr, 66 op/s 2024-09-21T10:14:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:00 smithi119 ceph-mon[105288]: osdmap e879: 8 total, 8 up, 8 in 2024-09-21T10:14:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:00 smithi119 ceph-mon[101432]: pgmap v1000: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 39 KiB/s rd, 5.0 KiB/s wr, 66 op/s 2024-09-21T10:14:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:00 smithi119 ceph-mon[101432]: osdmap e879: 8 total, 8 up, 8 in 2024-09-21T10:14:01.202 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:01 smithi145 ceph-mon[89359]: osdmap e880: 8 total, 8 up, 8 in 2024-09-21T10:14:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:01 smithi145 ceph-mon[89359]: pgmap v1003: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 8.5 KiB/s wr, 95 op/s 2024-09-21T10:14:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:01 smithi119 ceph-mon[105288]: osdmap e880: 8 total, 8 up, 8 in 2024-09-21T10:14:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:01 smithi119 ceph-mon[105288]: pgmap v1003: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 8.5 KiB/s wr, 95 op/s 2024-09-21T10:14:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:01 smithi119 ceph-mon[101432]: osdmap e880: 8 total, 8 up, 8 in 2024-09-21T10:14:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:01 smithi119 ceph-mon[101432]: pgmap v1003: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 8.5 KiB/s wr, 95 op/s 2024-09-21T10:14:02.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:02 smithi119 ceph-mon[101432]: osdmap e881: 8 total, 8 up, 8 in 2024-09-21T10:14:02.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:02 smithi119 ceph-mon[105288]: osdmap e881: 8 total, 8 up, 8 in 2024-09-21T10:14:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:02 smithi145 ceph-mon[89359]: osdmap e881: 8 total, 8 up, 8 in 2024-09-21T10:14:03.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:03 smithi119 ceph-mon[105288]: osdmap e882: 8 total, 8 up, 8 in 2024-09-21T10:14:03.536 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:03 smithi119 ceph-mon[105288]: pgmap v1006: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 KiB/s wr, 30 op/s 2024-09-21T10:14:03.536 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:03.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:03 smithi119 ceph-mon[101432]: osdmap e882: 8 total, 8 up, 8 in 2024-09-21T10:14:03.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:03 smithi119 ceph-mon[101432]: pgmap v1006: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 KiB/s wr, 30 op/s 2024-09-21T10:14:03.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:03 smithi145 ceph-mon[89359]: osdmap e882: 8 total, 8 up, 8 in 2024-09-21T10:14:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:03 smithi145 ceph-mon[89359]: pgmap v1006: 105 pgs: 105 active+clean; 20 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 4.0 KiB/s wr, 30 op/s 2024-09-21T10:14:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:04.216 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministic (14990 ms) 2024-09-21T10:14:04.216 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicPP 2024-09-21T10:14:04.217 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:04.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:04 smithi119 ceph-mon[105288]: osdmap e883: 8 total, 8 up, 8 in 2024-09-21T10:14:04.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:04 smithi119 ceph-mon[101432]: osdmap e883: 8 total, 8 up, 8 in 2024-09-21T10:14:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:04 smithi145 ceph-mon[89359]: osdmap e883: 8 total, 8 up, 8 in 2024-09-21T10:14:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:14:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:14:04] "GET /metrics HTTP/1.1" 200 36702 "" "Prometheus/2.51.0" 2024-09-21T10:14:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:05 smithi145 ceph-mon[89359]: osdmap e884: 8 total, 8 up, 8 in 2024-09-21T10:14:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:05 smithi145 ceph-mon[89359]: pgmap v1009: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 7.2 KiB/s wr, 76 op/s 2024-09-21T10:14:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:05 smithi119 ceph-mon[105288]: osdmap e884: 8 total, 8 up, 8 in 2024-09-21T10:14:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:05 smithi119 ceph-mon[105288]: pgmap v1009: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 7.2 KiB/s wr, 76 op/s 2024-09-21T10:14:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:05 smithi119 ceph-mon[101432]: osdmap e884: 8 total, 8 up, 8 in 2024-09-21T10:14:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:05 smithi119 ceph-mon[101432]: pgmap v1009: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 7.2 KiB/s wr, 76 op/s 2024-09-21T10:14:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:06 smithi145 ceph-mon[89359]: osdmap e885: 8 total, 8 up, 8 in 2024-09-21T10:14:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:06 smithi119 ceph-mon[105288]: osdmap e885: 8 total, 8 up, 8 in 2024-09-21T10:14:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:06 smithi119 ceph-mon[101432]: osdmap e885: 8 total, 8 up, 8 in 2024-09-21T10:14:07.245 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:07 smithi145 ceph-mon[89359]: osdmap e886: 8 total, 8 up, 8 in 2024-09-21T10:14:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:07 smithi145 ceph-mon[89359]: pgmap v1012: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 8.2 KiB/s wr, 87 op/s 2024-09-21T10:14:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:07 smithi119 ceph-mon[105288]: osdmap e886: 8 total, 8 up, 8 in 2024-09-21T10:14:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:07 smithi119 ceph-mon[105288]: pgmap v1012: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 8.2 KiB/s wr, 87 op/s 2024-09-21T10:14:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:07 smithi119 ceph-mon[101432]: osdmap e886: 8 total, 8 up, 8 in 2024-09-21T10:14:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:07 smithi119 ceph-mon[101432]: pgmap v1012: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 58 KiB/s rd, 8.2 KiB/s wr, 87 op/s 2024-09-21T10:14:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:08 smithi145 ceph-mon[89359]: osdmap e887: 8 total, 8 up, 8 in 2024-09-21T10:14:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:08 smithi119 ceph-mon[101432]: osdmap e887: 8 total, 8 up, 8 in 2024-09-21T10:14:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:08 smithi119 ceph-mon[105288]: osdmap e887: 8 total, 8 up, 8 in 2024-09-21T10:14:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:09 smithi145 ceph-mon[89359]: osdmap e888: 8 total, 8 up, 8 in 2024-09-21T10:14:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:09 smithi145 ceph-mon[89359]: pgmap v1015: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 6.2 KiB/s wr, 56 op/s 2024-09-21T10:14:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:09 smithi119 ceph-mon[105288]: osdmap e888: 8 total, 8 up, 8 in 2024-09-21T10:14:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:09 smithi119 ceph-mon[105288]: pgmap v1015: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 6.2 KiB/s wr, 56 op/s 2024-09-21T10:14:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:09 smithi119 ceph-mon[101432]: osdmap e888: 8 total, 8 up, 8 in 2024-09-21T10:14:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:09 smithi119 ceph-mon[101432]: pgmap v1015: 105 pgs: 105 active+clean; 30 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 31 KiB/s rd, 6.2 KiB/s wr, 56 op/s 2024-09-21T10:14:10.257 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:10 smithi145 ceph-mon[89359]: osdmap e889: 8 total, 8 up, 8 in 2024-09-21T10:14:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:14:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:14:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:10 smithi119 ceph-mon[101432]: osdmap e889: 8 total, 8 up, 8 in 2024-09-21T10:14:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:10 smithi119 ceph-mon[105288]: osdmap e889: 8 total, 8 up, 8 in 2024-09-21T10:14:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:11 smithi145 ceph-mon[89359]: osdmap e890: 8 total, 8 up, 8 in 2024-09-21T10:14:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:11 smithi145 ceph-mon[89359]: pgmap v1018: 105 pgs: 105 active+clean; 36 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 8.7 KiB/s wr, 102 op/s 2024-09-21T10:14:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:11 smithi119 ceph-mon[101432]: osdmap e890: 8 total, 8 up, 8 in 2024-09-21T10:14:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:11 smithi119 ceph-mon[101432]: pgmap v1018: 105 pgs: 105 active+clean; 36 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 8.7 KiB/s wr, 102 op/s 2024-09-21T10:14:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:11 smithi119 ceph-mon[105288]: osdmap e890: 8 total, 8 up, 8 in 2024-09-21T10:14:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:11 smithi119 ceph-mon[105288]: pgmap v1018: 105 pgs: 105 active+clean; 36 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 63 KiB/s rd, 8.7 KiB/s wr, 102 op/s 2024-09-21T10:14:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:12 smithi145 ceph-mon[89359]: osdmap e891: 8 total, 8 up, 8 in 2024-09-21T10:14:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:12 smithi119 ceph-mon[105288]: osdmap e891: 8 total, 8 up, 8 in 2024-09-21T10:14:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:12 smithi119 ceph-mon[101432]: osdmap e891: 8 total, 8 up, 8 in 2024-09-21T10:14:13.294 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:13 smithi145 ceph-mon[89359]: osdmap e892: 8 total, 8 up, 8 in 2024-09-21T10:14:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:13 smithi145 ceph-mon[89359]: pgmap v1021: 105 pgs: 105 active+clean; 36 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 3.5 KiB/s wr, 56 op/s 2024-09-21T10:14:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:13 smithi119 ceph-mon[105288]: osdmap e892: 8 total, 8 up, 8 in 2024-09-21T10:14:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:13 smithi119 ceph-mon[105288]: pgmap v1021: 105 pgs: 105 active+clean; 36 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 3.5 KiB/s wr, 56 op/s 2024-09-21T10:14:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:13 smithi119 ceph-mon[101432]: osdmap e892: 8 total, 8 up, 8 in 2024-09-21T10:14:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:13 smithi119 ceph-mon[101432]: pgmap v1021: 105 pgs: 105 active+clean; 36 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 40 KiB/s rd, 3.5 KiB/s wr, 56 op/s 2024-09-21T10:14:14.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:14 smithi119 ceph-mon[105288]: osdmap e893: 8 total, 8 up, 8 in 2024-09-21T10:14:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:14 smithi119 ceph-mon[101432]: osdmap e893: 8 total, 8 up, 8 in 2024-09-21T10:14:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:14 smithi145 ceph-mon[89359]: osdmap e893: 8 total, 8 up, 8 in 2024-09-21T10:14:14.815 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:14:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:14:14] "GET /metrics HTTP/1.1" 200 36702 "" "Prometheus/2.51.0" 2024-09-21T10:14:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:15 smithi145 ceph-mon[89359]: osdmap e894: 8 total, 8 up, 8 in 2024-09-21T10:14:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:15 smithi145 ceph-mon[89359]: pgmap v1024: 105 pgs: 105 active+clean; 42 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 7.2 KiB/s wr, 74 op/s 2024-09-21T10:14:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:15 smithi119 ceph-mon[101432]: osdmap e894: 8 total, 8 up, 8 in 2024-09-21T10:14:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:15 smithi119 ceph-mon[101432]: pgmap v1024: 105 pgs: 105 active+clean; 42 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 7.2 KiB/s wr, 74 op/s 2024-09-21T10:14:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:15 smithi119 ceph-mon[105288]: osdmap e894: 8 total, 8 up, 8 in 2024-09-21T10:14:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:15 smithi119 ceph-mon[105288]: pgmap v1024: 105 pgs: 105 active+clean; 42 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 7.2 KiB/s wr, 74 op/s 2024-09-21T10:14:16.307 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:16 smithi145 ceph-mon[89359]: osdmap e895: 8 total, 8 up, 8 in 2024-09-21T10:14:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:16 smithi119 ceph-mon[101432]: osdmap e895: 8 total, 8 up, 8 in 2024-09-21T10:14:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:16 smithi119 ceph-mon[105288]: osdmap e895: 8 total, 8 up, 8 in 2024-09-21T10:14:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:17 smithi145 ceph-mon[89359]: osdmap e896: 8 total, 8 up, 8 in 2024-09-21T10:14:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:17 smithi145 ceph-mon[89359]: pgmap v1027: 105 pgs: 105 active+clean; 42 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 8.0 KiB/s wr, 93 op/s 2024-09-21T10:14:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:17 smithi119 ceph-mon[105288]: osdmap e896: 8 total, 8 up, 8 in 2024-09-21T10:14:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:17 smithi119 ceph-mon[105288]: pgmap v1027: 105 pgs: 105 active+clean; 42 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 8.0 KiB/s wr, 93 op/s 2024-09-21T10:14:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:17 smithi119 ceph-mon[101432]: osdmap e896: 8 total, 8 up, 8 in 2024-09-21T10:14:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:17 smithi119 ceph-mon[101432]: pgmap v1027: 105 pgs: 105 active+clean; 42 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 8.0 KiB/s wr, 93 op/s 2024-09-21T10:14:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:18 smithi145 ceph-mon[89359]: osdmap e897: 8 total, 8 up, 8 in 2024-09-21T10:14:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:18 smithi119 ceph-mon[105288]: osdmap e897: 8 total, 8 up, 8 in 2024-09-21T10:14:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:18 smithi119 ceph-mon[101432]: osdmap e897: 8 total, 8 up, 8 in 2024-09-21T10:14:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:19.331 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicPP (15115 ms) 2024-09-21T10:14:19.331 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS1 2024-09-21T10:14:19.332 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:19 smithi119 ceph-mon[105288]: osdmap e898: 8 total, 8 up, 8 in 2024-09-21T10:14:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:19 smithi119 ceph-mon[105288]: pgmap v1030: 105 pgs: 105 active+clean; 52 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 8.7 KiB/s wr, 101 op/s 2024-09-21T10:14:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:19 smithi119 ceph-mon[101432]: osdmap e898: 8 total, 8 up, 8 in 2024-09-21T10:14:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:19 smithi119 ceph-mon[101432]: pgmap v1030: 105 pgs: 105 active+clean; 52 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 8.7 KiB/s wr, 101 op/s 2024-09-21T10:14:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:19 smithi145 ceph-mon[89359]: osdmap e898: 8 total, 8 up, 8 in 2024-09-21T10:14:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:19 smithi145 ceph-mon[89359]: pgmap v1030: 105 pgs: 105 active+clean; 52 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 8.7 KiB/s wr, 101 op/s 2024-09-21T10:14:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:20 smithi145 ceph-mon[89359]: osdmap e899: 8 total, 8 up, 8 in 2024-09-21T10:14:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:14:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:14:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:20 smithi119 ceph-mon[101432]: osdmap e899: 8 total, 8 up, 8 in 2024-09-21T10:14:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:20 smithi119 ceph-mon[105288]: osdmap e899: 8 total, 8 up, 8 in 2024-09-21T10:14:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:21 smithi119 ceph-mon[105288]: pgmap v1032: 105 pgs: 105 active+clean; 56 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 7.8 KiB/s wr, 76 op/s 2024-09-21T10:14:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:21 smithi119 ceph-mon[101432]: pgmap v1032: 105 pgs: 105 active+clean; 56 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 7.8 KiB/s wr, 76 op/s 2024-09-21T10:14:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:21 smithi145 ceph-mon[89359]: pgmap v1032: 105 pgs: 105 active+clean; 56 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 7.8 KiB/s wr, 76 op/s 2024-09-21T10:14:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:23 smithi145 ceph-mon[89359]: pgmap v1033: 105 pgs: 105 active+clean; 56 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 5.7 KiB/s wr, 55 op/s 2024-09-21T10:14:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:23 smithi119 ceph-mon[101432]: pgmap v1033: 105 pgs: 105 active+clean; 56 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 5.7 KiB/s wr, 55 op/s 2024-09-21T10:14:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:23 smithi119 ceph-mon[105288]: pgmap v1033: 105 pgs: 105 active+clean; 56 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 36 KiB/s rd, 5.7 KiB/s wr, 55 op/s 2024-09-21T10:14:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:24 smithi119 ceph-mon[101432]: osdmap e900: 8 total, 8 up, 8 in 2024-09-21T10:14:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:14:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:14:24] "GET /metrics HTTP/1.1" 200 36707 "" "Prometheus/2.51.0" 2024-09-21T10:14:24.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:24 smithi119 ceph-mon[105288]: osdmap e900: 8 total, 8 up, 8 in 2024-09-21T10:14:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:24 smithi145 ceph-mon[89359]: osdmap e900: 8 total, 8 up, 8 in 2024-09-21T10:14:25.765 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:25 smithi145 ceph-mon[89359]: pgmap v1035: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 650 KiB/s wr, 74 op/s 2024-09-21T10:14:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:25 smithi145 ceph-mon[89359]: osdmap e901: 8 total, 8 up, 8 in 2024-09-21T10:14:26.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:25 smithi119 ceph-mon[101432]: pgmap v1035: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 650 KiB/s wr, 74 op/s 2024-09-21T10:14:26.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:25 smithi119 ceph-mon[101432]: osdmap e901: 8 total, 8 up, 8 in 2024-09-21T10:14:26.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:25 smithi119 ceph-mon[105288]: pgmap v1035: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 47 KiB/s rd, 650 KiB/s wr, 74 op/s 2024-09-21T10:14:26.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:25 smithi119 ceph-mon[105288]: osdmap e901: 8 total, 8 up, 8 in 2024-09-21T10:14:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:26 smithi145 ceph-mon[89359]: osdmap e902: 8 total, 8 up, 8 in 2024-09-21T10:14:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:26 smithi119 ceph-mon[101432]: osdmap e902: 8 total, 8 up, 8 in 2024-09-21T10:14:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:26 smithi119 ceph-mon[105288]: osdmap e902: 8 total, 8 up, 8 in 2024-09-21T10:14:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:27 smithi145 ceph-mon[89359]: pgmap v1038: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 684 KiB/s wr, 22 op/s 2024-09-21T10:14:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:27 smithi119 ceph-mon[101432]: pgmap v1038: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 684 KiB/s wr, 22 op/s 2024-09-21T10:14:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:27 smithi119 ceph-mon[105288]: pgmap v1038: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 684 KiB/s wr, 22 op/s 2024-09-21T10:14:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:29 smithi145 ceph-mon[89359]: pgmap v1039: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 688 KiB/s wr, 60 op/s 2024-09-21T10:14:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:29 smithi119 ceph-mon[105288]: pgmap v1039: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 688 KiB/s wr, 60 op/s 2024-09-21T10:14:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:29 smithi119 ceph-mon[101432]: pgmap v1039: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 688 KiB/s wr, 60 op/s 2024-09-21T10:14:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:14:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:14:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:30 smithi145 ceph-mon[89359]: osdmap e903: 8 total, 8 up, 8 in 2024-09-21T10:14:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:30 smithi119 ceph-mon[101432]: osdmap e903: 8 total, 8 up, 8 in 2024-09-21T10:14:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:30 smithi119 ceph-mon[105288]: osdmap e903: 8 total, 8 up, 8 in 2024-09-21T10:14:31.790 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:32.017 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:31 smithi119 ceph-mon[105288]: pgmap v1041: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:14:32.017 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:31 smithi119 ceph-mon[105288]: osdmap e904: 8 total, 8 up, 8 in 2024-09-21T10:14:32.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:31 smithi119 ceph-mon[101432]: pgmap v1041: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:14:32.018 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:31 smithi119 ceph-mon[101432]: osdmap e904: 8 total, 8 up, 8 in 2024-09-21T10:14:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:31 smithi145 ceph-mon[89359]: pgmap v1041: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:14:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:31 smithi145 ceph-mon[89359]: osdmap e904: 8 total, 8 up, 8 in 2024-09-21T10:14:33.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:32 smithi145 ceph-mon[89359]: osdmap e905: 8 total, 8 up, 8 in 2024-09-21T10:14:33.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:33.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:32 smithi119 ceph-mon[105288]: osdmap e905: 8 total, 8 up, 8 in 2024-09-21T10:14:33.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:33.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:32 smithi119 ceph-mon[101432]: osdmap e905: 8 total, 8 up, 8 in 2024-09-21T10:14:33.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:33 smithi145 ceph-mon[89359]: pgmap v1044: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:14:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:33 smithi119 ceph-mon[101432]: pgmap v1044: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:14:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:33 smithi119 ceph-mon[105288]: pgmap v1044: 105 pgs: 105 active+clean; 60 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:14:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:14:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:14:34] "GET /metrics HTTP/1.1" 200 36704 "" "Prometheus/2.51.0" 2024-09-21T10:14:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:35 smithi145 ceph-mon[89359]: pgmap v1045: 105 pgs: 105 active+clean; 66 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:35 smithi119 ceph-mon[105288]: pgmap v1045: 105 pgs: 105 active+clean; 66 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:36.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:35 smithi119 ceph-mon[101432]: pgmap v1045: 105 pgs: 105 active+clean; 66 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:36 smithi145 ceph-mon[89359]: osdmap e906: 8 total, 8 up, 8 in 2024-09-21T10:14:37.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:36 smithi119 ceph-mon[101432]: osdmap e906: 8 total, 8 up, 8 in 2024-09-21T10:14:37.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:36 smithi119 ceph-mon[105288]: osdmap e906: 8 total, 8 up, 8 in 2024-09-21T10:14:37.823 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:37 smithi119 ceph-mon[105288]: pgmap v1047: 105 pgs: 105 active+clean; 66 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:37 smithi119 ceph-mon[105288]: osdmap e907: 8 total, 8 up, 8 in 2024-09-21T10:14:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:37 smithi119 ceph-mon[101432]: pgmap v1047: 105 pgs: 105 active+clean; 66 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:37 smithi119 ceph-mon[101432]: osdmap e907: 8 total, 8 up, 8 in 2024-09-21T10:14:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:37 smithi145 ceph-mon[89359]: pgmap v1047: 105 pgs: 105 active+clean; 66 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:37 smithi145 ceph-mon[89359]: osdmap e907: 8 total, 8 up, 8 in 2024-09-21T10:14:39.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:38 smithi119 ceph-mon[105288]: osdmap e908: 8 total, 8 up, 8 in 2024-09-21T10:14:39.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:38 smithi119 ceph-mon[101432]: osdmap e908: 8 total, 8 up, 8 in 2024-09-21T10:14:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:38 smithi145 ceph-mon[89359]: osdmap e908: 8 total, 8 up, 8 in 2024-09-21T10:14:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:39 smithi119 ceph-mon[101432]: pgmap v1050: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 1.3 MiB/s wr, 80 op/s 2024-09-21T10:14:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:39 smithi119 ceph-mon[105288]: pgmap v1050: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 1.3 MiB/s wr, 80 op/s 2024-09-21T10:14:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:39 smithi145 ceph-mon[89359]: pgmap v1050: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 1.3 MiB/s wr, 80 op/s 2024-09-21T10:14:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:14:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:14:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:41 smithi119 ceph-mon[101432]: pgmap v1051: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:41 smithi119 ceph-mon[105288]: pgmap v1051: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:41 smithi145 ceph-mon[89359]: pgmap v1051: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:42 smithi119 ceph-mon[101432]: osdmap e909: 8 total, 8 up, 8 in 2024-09-21T10:14:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:42 smithi119 ceph-mon[105288]: osdmap e909: 8 total, 8 up, 8 in 2024-09-21T10:14:43.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:42 smithi145 ceph-mon[89359]: osdmap e909: 8 total, 8 up, 8 in 2024-09-21T10:14:43.863 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:43 smithi119 ceph-mon[101432]: pgmap v1053: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:43 smithi119 ceph-mon[101432]: osdmap e910: 8 total, 8 up, 8 in 2024-09-21T10:14:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:43 smithi119 ceph-mon[105288]: pgmap v1053: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:43 smithi119 ceph-mon[105288]: osdmap e910: 8 total, 8 up, 8 in 2024-09-21T10:14:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:43 smithi145 ceph-mon[89359]: pgmap v1053: 105 pgs: 105 active+clean; 72 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:14:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:43 smithi145 ceph-mon[89359]: osdmap e910: 8 total, 8 up, 8 in 2024-09-21T10:14:44.841 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:14:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:14:44] "GET /metrics HTTP/1.1" 200 36704 "" "Prometheus/2.51.0" 2024-09-21T10:14:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:44 smithi119 ceph-mon[101432]: osdmap e911: 8 total, 8 up, 8 in 2024-09-21T10:14:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:44 smithi119 ceph-mon[105288]: osdmap e911: 8 total, 8 up, 8 in 2024-09-21T10:14:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:44 smithi145 ceph-mon[89359]: osdmap e911: 8 total, 8 up, 8 in 2024-09-21T10:14:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:45 smithi119 ceph-mon[105288]: pgmap v1056: 105 pgs: 105 active+clean; 78 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-09-21T10:14:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:45 smithi119 ceph-mon[101432]: pgmap v1056: 105 pgs: 105 active+clean; 78 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-09-21T10:14:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:45 smithi145 ceph-mon[89359]: pgmap v1056: 105 pgs: 105 active+clean; 78 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 21 op/s 2024-09-21T10:14:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:47 smithi119 ceph-mon[101432]: pgmap v1057: 105 pgs: 105 active+clean; 78 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 686 KiB/s wr, 23 op/s 2024-09-21T10:14:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:47 smithi119 ceph-mon[105288]: pgmap v1057: 105 pgs: 105 active+clean; 78 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 686 KiB/s wr, 23 op/s 2024-09-21T10:14:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:47 smithi145 ceph-mon[89359]: pgmap v1057: 105 pgs: 105 active+clean; 78 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 686 KiB/s wr, 23 op/s 2024-09-21T10:14:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:14:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:48 smithi119 ceph-mon[105288]: osdmap e912: 8 total, 8 up, 8 in 2024-09-21T10:14:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:48 smithi119 ceph-mon[101432]: osdmap e912: 8 total, 8 up, 8 in 2024-09-21T10:14:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:48 smithi145 ceph-mon[89359]: osdmap e912: 8 total, 8 up, 8 in 2024-09-21T10:14:49.889 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS1 (30557 ms) 2024-09-21T10:14:49.889 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS1PP 2024-09-21T10:14:49.889 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:49 smithi119 ceph-mon[105288]: pgmap v1059: 105 pgs: 105 active+clean; 82 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-09-21T10:14:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:49 smithi119 ceph-mon[105288]: osdmap e913: 8 total, 8 up, 8 in 2024-09-21T10:14:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:49 smithi119 ceph-mon[101432]: pgmap v1059: 105 pgs: 105 active+clean; 82 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-09-21T10:14:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:49 smithi119 ceph-mon[101432]: osdmap e913: 8 total, 8 up, 8 in 2024-09-21T10:14:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:49 smithi145 ceph-mon[89359]: pgmap v1059: 105 pgs: 105 active+clean; 82 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 29 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-09-21T10:14:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:49 smithi145 ceph-mon[89359]: osdmap e913: 8 total, 8 up, 8 in 2024-09-21T10:14:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:14:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:14:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:14:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:50 smithi119 ceph-mon[105288]: osdmap e914: 8 total, 8 up, 8 in 2024-09-21T10:14:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:50 smithi119 ceph-mon[101432]: osdmap e914: 8 total, 8 up, 8 in 2024-09-21T10:14:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:50 smithi145 ceph-mon[89359]: osdmap e914: 8 total, 8 up, 8 in 2024-09-21T10:14:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:51 smithi119 ceph-mon[101432]: pgmap v1062: 105 pgs: 105 active+clean; 90 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 687 KiB/s wr, 36 op/s 2024-09-21T10:14:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:51 smithi119 ceph-mon[105288]: pgmap v1062: 105 pgs: 105 active+clean; 90 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 687 KiB/s wr, 36 op/s 2024-09-21T10:14:52.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:52.194 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:51 smithi145 ceph-mon[89359]: pgmap v1062: 105 pgs: 105 active+clean; 90 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 687 KiB/s wr, 36 op/s 2024-09-21T10:14:52.194 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:14:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:53 smithi145 ceph-mon[89359]: pgmap v1063: 105 pgs: 105 active+clean; 90 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-09-21T10:14:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:53 smithi119 ceph-mon[105288]: pgmap v1063: 105 pgs: 105 active+clean; 90 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-09-21T10:14:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:53 smithi119 ceph-mon[101432]: pgmap v1063: 105 pgs: 105 active+clean; 90 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 20 KiB/s rd, 686 KiB/s wr, 34 op/s 2024-09-21T10:14:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:14:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:14:54] "GET /metrics HTTP/1.1" 200 36706 "" "Prometheus/2.51.0" 2024-09-21T10:14:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:54 smithi145 ceph-mon[89359]: osdmap e915: 8 total, 8 up, 8 in 2024-09-21T10:14:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:54 smithi119 ceph-mon[101432]: osdmap e915: 8 total, 8 up, 8 in 2024-09-21T10:14:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:54 smithi119 ceph-mon[105288]: osdmap e915: 8 total, 8 up, 8 in 2024-09-21T10:14:55.910 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:14:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:55 smithi145 ceph-mon[89359]: pgmap v1065: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 687 KiB/s wr, 36 op/s 2024-09-21T10:14:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:55 smithi145 ceph-mon[89359]: osdmap e916: 8 total, 8 up, 8 in 2024-09-21T10:14:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[105288]: pgmap v1065: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 687 KiB/s wr, 36 op/s 2024-09-21T10:14:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[105288]: osdmap e916: 8 total, 8 up, 8 in 2024-09-21T10:14:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[101432]: pgmap v1065: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 22 KiB/s rd, 687 KiB/s wr, 36 op/s 2024-09-21T10:14:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[101432]: osdmap e916: 8 total, 8 up, 8 in 2024-09-21T10:14:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:56 smithi145 ceph-mon[89359]: osdmap e917: 8 total, 8 up, 8 in 2024-09-21T10:14:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:14:57.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:14:57.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[101432]: osdmap e917: 8 total, 8 up, 8 in 2024-09-21T10:14:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:14:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:14:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[105288]: osdmap e917: 8 total, 8 up, 8 in 2024-09-21T10:14:57.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:57.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:14:57.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:14:57.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:14:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:57 smithi145 ceph-mon[89359]: pgmap v1068: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 685 KiB/s wr, 25 op/s 2024-09-21T10:14:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:57 smithi119 ceph-mon[101432]: pgmap v1068: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 685 KiB/s wr, 25 op/s 2024-09-21T10:14:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:57 smithi119 ceph-mon[105288]: pgmap v1068: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 685 KiB/s wr, 25 op/s 2024-09-21T10:15:00.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:14:59 smithi145 ceph-mon[89359]: pgmap v1069: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 41 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-09-21T10:15:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:14:59 smithi119 ceph-mon[101432]: pgmap v1069: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 41 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-09-21T10:15:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:14:59 smithi119 ceph-mon[105288]: pgmap v1069: 105 pgs: 105 active+clean; 94 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 41 KiB/s rd, 688 KiB/s wr, 61 op/s 2024-09-21T10:15:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:15:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:15:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:00 smithi145 ceph-mon[89359]: osdmap e918: 8 total, 8 up, 8 in 2024-09-21T10:15:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:00 smithi119 ceph-mon[101432]: osdmap e918: 8 total, 8 up, 8 in 2024-09-21T10:15:01.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:00 smithi119 ceph-mon[105288]: osdmap e918: 8 total, 8 up, 8 in 2024-09-21T10:15:01.947 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:02.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:01 smithi119 ceph-mon[101432]: pgmap v1071: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.8 KiB/s wr, 37 op/s 2024-09-21T10:15:02.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:01 smithi119 ceph-mon[101432]: osdmap e919: 8 total, 8 up, 8 in 2024-09-21T10:15:02.269 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:01 smithi119 ceph-mon[105288]: pgmap v1071: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.8 KiB/s wr, 37 op/s 2024-09-21T10:15:02.269 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:01 smithi119 ceph-mon[105288]: osdmap e919: 8 total, 8 up, 8 in 2024-09-21T10:15:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:01 smithi145 ceph-mon[89359]: pgmap v1071: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.8 KiB/s wr, 37 op/s 2024-09-21T10:15:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:01 smithi145 ceph-mon[89359]: osdmap e919: 8 total, 8 up, 8 in 2024-09-21T10:15:03.287 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:02 smithi119 ceph-mon[101432]: osdmap e920: 8 total, 8 up, 8 in 2024-09-21T10:15:03.287 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:03.287 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:02 smithi119 ceph-mon[105288]: osdmap e920: 8 total, 8 up, 8 in 2024-09-21T10:15:03.288 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:02 smithi145 ceph-mon[89359]: osdmap e920: 8 total, 8 up, 8 in 2024-09-21T10:15:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:03 smithi119 ceph-mon[105288]: pgmap v1074: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:15:04.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:03 smithi119 ceph-mon[101432]: pgmap v1074: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:15:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:03 smithi145 ceph-mon[89359]: pgmap v1074: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 3.7 KiB/s wr, 37 op/s 2024-09-21T10:15:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:15:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:15:04] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.51.0" 2024-09-21T10:15:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:05 smithi145 ceph-mon[89359]: pgmap v1075: 105 pgs: 105 active+clean; 104 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-09-21T10:15:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:05 smithi119 ceph-mon[105288]: pgmap v1075: 105 pgs: 105 active+clean; 104 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-09-21T10:15:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:05 smithi119 ceph-mon[101432]: pgmap v1075: 105 pgs: 105 active+clean; 104 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 38 op/s 2024-09-21T10:15:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:06 smithi145 ceph-mon[89359]: osdmap e921: 8 total, 8 up, 8 in 2024-09-21T10:15:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:06 smithi119 ceph-mon[105288]: osdmap e921: 8 total, 8 up, 8 in 2024-09-21T10:15:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:06 smithi119 ceph-mon[101432]: osdmap e921: 8 total, 8 up, 8 in 2024-09-21T10:15:07.973 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:07 smithi145 ceph-mon[89359]: pgmap v1077: 105 pgs: 105 active+clean; 104 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:15:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:07 smithi145 ceph-mon[89359]: osdmap e922: 8 total, 8 up, 8 in 2024-09-21T10:15:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:07 smithi119 ceph-mon[105288]: pgmap v1077: 105 pgs: 105 active+clean; 104 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:15:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:07 smithi119 ceph-mon[105288]: osdmap e922: 8 total, 8 up, 8 in 2024-09-21T10:15:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:07 smithi119 ceph-mon[101432]: pgmap v1077: 105 pgs: 105 active+clean; 104 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:15:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:07 smithi119 ceph-mon[101432]: osdmap e922: 8 total, 8 up, 8 in 2024-09-21T10:15:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:08 smithi145 ceph-mon[89359]: osdmap e923: 8 total, 8 up, 8 in 2024-09-21T10:15:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:08 smithi119 ceph-mon[105288]: osdmap e923: 8 total, 8 up, 8 in 2024-09-21T10:15:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:08 smithi119 ceph-mon[101432]: osdmap e923: 8 total, 8 up, 8 in 2024-09-21T10:15:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:09 smithi145 ceph-mon[89359]: pgmap v1080: 105 pgs: 105 active+clean; 104 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 690 KiB/s wr, 75 op/s 2024-09-21T10:15:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:09 smithi119 ceph-mon[101432]: pgmap v1080: 105 pgs: 105 active+clean; 104 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 690 KiB/s wr, 75 op/s 2024-09-21T10:15:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:09 smithi119 ceph-mon[105288]: pgmap v1080: 105 pgs: 105 active+clean; 104 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 50 KiB/s rd, 690 KiB/s wr, 75 op/s 2024-09-21T10:15:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:15:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:15:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:11 smithi145 ceph-mon[89359]: pgmap v1081: 105 pgs: 105 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:15:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:11 smithi119 ceph-mon[105288]: pgmap v1081: 105 pgs: 105 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:15:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:11 smithi119 ceph-mon[101432]: pgmap v1081: 105 pgs: 105 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:15:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:12 smithi145 ceph-mon[89359]: osdmap e924: 8 total, 8 up, 8 in 2024-09-21T10:15:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:12 smithi119 ceph-mon[101432]: osdmap e924: 8 total, 8 up, 8 in 2024-09-21T10:15:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:12 smithi119 ceph-mon[105288]: osdmap e924: 8 total, 8 up, 8 in 2024-09-21T10:15:13.999 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:13 smithi145 ceph-mon[89359]: pgmap v1083: 105 pgs: 105 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:15:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:13 smithi145 ceph-mon[89359]: osdmap e925: 8 total, 8 up, 8 in 2024-09-21T10:15:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:13 smithi119 ceph-mon[105288]: pgmap v1083: 105 pgs: 105 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:15:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:13 smithi119 ceph-mon[105288]: osdmap e925: 8 total, 8 up, 8 in 2024-09-21T10:15:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:13 smithi119 ceph-mon[101432]: pgmap v1083: 105 pgs: 105 active+clean; 114 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 687 KiB/s wr, 40 op/s 2024-09-21T10:15:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:13 smithi119 ceph-mon[101432]: osdmap e925: 8 total, 8 up, 8 in 2024-09-21T10:15:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:15:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:15:14] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.51.0" 2024-09-21T10:15:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:14 smithi145 ceph-mon[89359]: osdmap e926: 8 total, 8 up, 8 in 2024-09-21T10:15:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:14 smithi119 ceph-mon[105288]: osdmap e926: 8 total, 8 up, 8 in 2024-09-21T10:15:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:14 smithi119 ceph-mon[101432]: osdmap e926: 8 total, 8 up, 8 in 2024-09-21T10:15:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:15 smithi145 ceph-mon[89359]: pgmap v1086: 105 pgs: 105 active+clean; 124 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-09-21T10:15:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:15 smithi119 ceph-mon[105288]: pgmap v1086: 105 pgs: 105 active+clean; 124 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-09-21T10:15:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:15 smithi119 ceph-mon[101432]: pgmap v1086: 105 pgs: 105 active+clean; 124 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 1.3 MiB/s wr, 24 op/s 2024-09-21T10:15:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:17 smithi145 ceph-mon[89359]: pgmap v1087: 105 pgs: 105 active+clean; 124 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 19 op/s 2024-09-21T10:15:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:17 smithi119 ceph-mon[101432]: pgmap v1087: 105 pgs: 105 active+clean; 124 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 19 op/s 2024-09-21T10:15:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:17 smithi119 ceph-mon[105288]: pgmap v1087: 105 pgs: 105 active+clean; 124 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 19 op/s 2024-09-21T10:15:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:18 smithi145 ceph-mon[89359]: osdmap e927: 8 total, 8 up, 8 in 2024-09-21T10:15:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:18 smithi119 ceph-mon[105288]: osdmap e927: 8 total, 8 up, 8 in 2024-09-21T10:15:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:18 smithi119 ceph-mon[101432]: osdmap e927: 8 total, 8 up, 8 in 2024-09-21T10:15:20.030 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS1PP (30142 ms) 2024-09-21T10:15:20.030 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS2 2024-09-21T10:15:20.030 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:20 smithi145 ceph-mon[89359]: pgmap v1089: 105 pgs: 105 active+clean; 132 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 50 op/s 2024-09-21T10:15:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:20 smithi145 ceph-mon[89359]: osdmap e928: 8 total, 8 up, 8 in 2024-09-21T10:15:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:15:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:15:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:20 smithi119 ceph-mon[105288]: pgmap v1089: 105 pgs: 105 active+clean; 132 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 50 op/s 2024-09-21T10:15:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:20 smithi119 ceph-mon[105288]: osdmap e928: 8 total, 8 up, 8 in 2024-09-21T10:15:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:20 smithi119 ceph-mon[101432]: pgmap v1089: 105 pgs: 105 active+clean; 132 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 32 KiB/s rd, 1.3 MiB/s wr, 50 op/s 2024-09-21T10:15:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:20 smithi119 ceph-mon[101432]: osdmap e928: 8 total, 8 up, 8 in 2024-09-21T10:15:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:21 smithi145 ceph-mon[89359]: osdmap e929: 8 total, 8 up, 8 in 2024-09-21T10:15:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:21 smithi119 ceph-mon[101432]: osdmap e929: 8 total, 8 up, 8 in 2024-09-21T10:15:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:21 smithi119 ceph-mon[105288]: osdmap e929: 8 total, 8 up, 8 in 2024-09-21T10:15:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:22 smithi145 ceph-mon[89359]: pgmap v1092: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 33 op/s 2024-09-21T10:15:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:22 smithi119 ceph-mon[101432]: pgmap v1092: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 33 op/s 2024-09-21T10:15:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:22 smithi119 ceph-mon[105288]: pgmap v1092: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 33 op/s 2024-09-21T10:15:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:24 smithi145 ceph-mon[89359]: pgmap v1093: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 33 op/s 2024-09-21T10:15:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:24 smithi119 ceph-mon[105288]: pgmap v1093: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 33 op/s 2024-09-21T10:15:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:24 smithi119 ceph-mon[101432]: pgmap v1093: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 33 op/s 2024-09-21T10:15:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:15:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:15:24] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.51.0" 2024-09-21T10:15:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:26 smithi145 ceph-mon[89359]: pgmap v1094: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 617 KiB/s wr, 54 op/s 2024-09-21T10:15:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:26 smithi119 ceph-mon[101432]: pgmap v1094: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 617 KiB/s wr, 54 op/s 2024-09-21T10:15:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:26 smithi119 ceph-mon[105288]: pgmap v1094: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 617 KiB/s wr, 54 op/s 2024-09-21T10:15:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:28 smithi145 ceph-mon[89359]: pgmap v1095: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.9 KiB/s wr, 22 op/s 2024-09-21T10:15:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:28 smithi119 ceph-mon[105288]: pgmap v1095: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.9 KiB/s wr, 22 op/s 2024-09-21T10:15:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:28 smithi119 ceph-mon[101432]: pgmap v1095: 105 pgs: 105 active+clean; 136 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.9 KiB/s wr, 22 op/s 2024-09-21T10:15:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:29 smithi145 ceph-mon[89359]: osdmap e930: 8 total, 8 up, 8 in 2024-09-21T10:15:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:29 smithi119 ceph-mon[101432]: osdmap e930: 8 total, 8 up, 8 in 2024-09-21T10:15:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:29 smithi119 ceph-mon[105288]: osdmap e930: 8 total, 8 up, 8 in 2024-09-21T10:15:30.076 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:30.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:30 smithi145 ceph-mon[89359]: pgmap v1097: 105 pgs: 105 active+clean; 144 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 947 KiB/s wr, 30 op/s 2024-09-21T10:15:30.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:30 smithi145 ceph-mon[89359]: osdmap e931: 8 total, 8 up, 8 in 2024-09-21T10:15:30.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:15:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:15:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:30 smithi119 ceph-mon[101432]: pgmap v1097: 105 pgs: 105 active+clean; 144 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 947 KiB/s wr, 30 op/s 2024-09-21T10:15:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:30 smithi119 ceph-mon[101432]: osdmap e931: 8 total, 8 up, 8 in 2024-09-21T10:15:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:30 smithi119 ceph-mon[105288]: pgmap v1097: 105 pgs: 105 active+clean; 144 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 947 KiB/s wr, 30 op/s 2024-09-21T10:15:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:30 smithi119 ceph-mon[105288]: osdmap e931: 8 total, 8 up, 8 in 2024-09-21T10:15:31.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:15:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=sqlstore.transactions t=2024-09-21T10:15:30.746962531Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T10:15:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:31 smithi119 ceph-mon[105288]: osdmap e932: 8 total, 8 up, 8 in 2024-09-21T10:15:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:31 smithi119 ceph-mon[101432]: osdmap e932: 8 total, 8 up, 8 in 2024-09-21T10:15:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:31 smithi145 ceph-mon[89359]: osdmap e932: 8 total, 8 up, 8 in 2024-09-21T10:15:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:32 smithi119 ceph-mon[101432]: pgmap v1100: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:32 smithi119 ceph-mon[105288]: pgmap v1100: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:32 smithi145 ceph-mon[89359]: pgmap v1100: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:34 smithi119 ceph-mon[105288]: pgmap v1101: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:34 smithi119 ceph-mon[101432]: pgmap v1101: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:34 smithi145 ceph-mon[89359]: pgmap v1101: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:15:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:15:34] "GET /metrics HTTP/1.1" 200 36715 "" "Prometheus/2.51.0" 2024-09-21T10:15:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:36 smithi145 ceph-mon[89359]: pgmap v1102: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:15:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:36 smithi119 ceph-mon[105288]: pgmap v1102: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:15:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:36 smithi119 ceph-mon[101432]: pgmap v1102: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:15:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:38 smithi145 ceph-mon[89359]: pgmap v1103: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:15:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:38 smithi119 ceph-mon[101432]: pgmap v1103: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:15:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:38 smithi119 ceph-mon[105288]: pgmap v1103: 105 pgs: 105 active+clean; 152 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:15:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:39 smithi145 ceph-mon[89359]: osdmap e933: 8 total, 8 up, 8 in 2024-09-21T10:15:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:39 smithi119 ceph-mon[101432]: osdmap e933: 8 total, 8 up, 8 in 2024-09-21T10:15:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:39 smithi119 ceph-mon[105288]: osdmap e933: 8 total, 8 up, 8 in 2024-09-21T10:15:40.202 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:40 smithi145 ceph-mon[89359]: pgmap v1105: 105 pgs: 105 active+clean; 161 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 951 KiB/s wr, 25 op/s 2024-09-21T10:15:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:40 smithi145 ceph-mon[89359]: osdmap e934: 8 total, 8 up, 8 in 2024-09-21T10:15:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:15:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:15:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:40 smithi119 ceph-mon[101432]: pgmap v1105: 105 pgs: 105 active+clean; 161 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 951 KiB/s wr, 25 op/s 2024-09-21T10:15:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:40 smithi119 ceph-mon[101432]: osdmap e934: 8 total, 8 up, 8 in 2024-09-21T10:15:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:40 smithi119 ceph-mon[105288]: pgmap v1105: 105 pgs: 105 active+clean; 161 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 951 KiB/s wr, 25 op/s 2024-09-21T10:15:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:40 smithi119 ceph-mon[105288]: osdmap e934: 8 total, 8 up, 8 in 2024-09-21T10:15:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:41 smithi145 ceph-mon[89359]: osdmap e935: 8 total, 8 up, 8 in 2024-09-21T10:15:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:41 smithi145 ceph-mon[89359]: pgmap v1108: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:15:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:41 smithi119 ceph-mon[101432]: osdmap e935: 8 total, 8 up, 8 in 2024-09-21T10:15:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:41 smithi119 ceph-mon[101432]: pgmap v1108: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:15:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:41 smithi119 ceph-mon[105288]: osdmap e935: 8 total, 8 up, 8 in 2024-09-21T10:15:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:41 smithi119 ceph-mon[105288]: pgmap v1108: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:15:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:43 smithi145 ceph-mon[89359]: pgmap v1109: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:15:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:43 smithi119 ceph-mon[105288]: pgmap v1109: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:15:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:43 smithi119 ceph-mon[101432]: pgmap v1109: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 2.7 MiB/s wr, 18 op/s 2024-09-21T10:15:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:15:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:15:44] "GET /metrics HTTP/1.1" 200 36715 "" "Prometheus/2.51.0" 2024-09-21T10:15:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:45 smithi145 ceph-mon[89359]: pgmap v1110: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 41 op/s 2024-09-21T10:15:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:45 smithi119 ceph-mon[105288]: pgmap v1110: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 41 op/s 2024-09-21T10:15:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:45 smithi119 ceph-mon[101432]: pgmap v1110: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 2.5 MiB/s wr, 41 op/s 2024-09-21T10:15:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:47 smithi145 ceph-mon[89359]: pgmap v1111: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2024-09-21T10:15:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:47 smithi119 ceph-mon[105288]: pgmap v1111: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2024-09-21T10:15:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:47 smithi119 ceph-mon[101432]: pgmap v1111: 105 pgs: 105 active+clean; 170 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2024-09-21T10:15:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:15:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:48 smithi145 ceph-mon[89359]: osdmap e936: 8 total, 8 up, 8 in 2024-09-21T10:15:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:48 smithi119 ceph-mon[105288]: osdmap e936: 8 total, 8 up, 8 in 2024-09-21T10:15:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:48 smithi119 ceph-mon[101432]: osdmap e936: 8 total, 8 up, 8 in 2024-09-21T10:15:49.787 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:15:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:49 smithi145 ceph-mon[89359]: pgmap v1113: 105 pgs: 105 active+clean; 183 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.4 MiB/s wr, 37 op/s 2024-09-21T10:15:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:49 smithi145 ceph-mon[89359]: osdmap e937: 8 total, 8 up, 8 in 2024-09-21T10:15:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:49 smithi119 ceph-mon[105288]: pgmap v1113: 105 pgs: 105 active+clean; 183 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.4 MiB/s wr, 37 op/s 2024-09-21T10:15:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:49 smithi119 ceph-mon[105288]: osdmap e937: 8 total, 8 up, 8 in 2024-09-21T10:15:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:49 smithi119 ceph-mon[101432]: pgmap v1113: 105 pgs: 105 active+clean; 183 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.4 MiB/s wr, 37 op/s 2024-09-21T10:15:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:49 smithi119 ceph-mon[101432]: osdmap e937: 8 total, 8 up, 8 in 2024-09-21T10:15:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:15:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:15:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:15:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:50 smithi145 ceph-mon[89359]: osdmap e938: 8 total, 8 up, 8 in 2024-09-21T10:15:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:50 smithi119 ceph-mon[105288]: osdmap e938: 8 total, 8 up, 8 in 2024-09-21T10:15:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:50 smithi119 ceph-mon[101432]: osdmap e938: 8 total, 8 up, 8 in 2024-09-21T10:15:52.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:51 smithi145 ceph-mon[89359]: pgmap v1116: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:51 smithi119 ceph-mon[105288]: pgmap v1116: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:51 smithi119 ceph-mon[101432]: pgmap v1116: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:53 smithi145 ceph-mon[89359]: pgmap v1117: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:53 smithi119 ceph-mon[101432]: pgmap v1117: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:53 smithi119 ceph-mon[105288]: pgmap v1117: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:15:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:15:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:15:54] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.51.0" 2024-09-21T10:15:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:55 smithi145 ceph-mon[89359]: pgmap v1118: 105 pgs: 105 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-09-21T10:15:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:55 smithi119 ceph-mon[101432]: pgmap v1118: 105 pgs: 105 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-09-21T10:15:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:55 smithi119 ceph-mon[105288]: pgmap v1118: 105 pgs: 105 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-09-21T10:15:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:15:57.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:15:57.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:15:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:57 smithi145 ceph-mon[89359]: pgmap v1119: 105 pgs: 105 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 515 KiB/s wr, 22 op/s 2024-09-21T10:15:58.047 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:57 smithi119 ceph-mon[105288]: pgmap v1119: 105 pgs: 105 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 515 KiB/s wr, 22 op/s 2024-09-21T10:15:58.047 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:57 smithi119 ceph-mon[101432]: pgmap v1119: 105 pgs: 105 active+clean; 188 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 515 KiB/s wr, 22 op/s 2024-09-21T10:15:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:58 smithi145 ceph-mon[89359]: osdmap e939: 8 total, 8 up, 8 in 2024-09-21T10:15:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:58 smithi119 ceph-mon[105288]: osdmap e939: 8 total, 8 up, 8 in 2024-09-21T10:15:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:58 smithi119 ceph-mon[101432]: osdmap e939: 8 total, 8 up, 8 in 2024-09-21T10:15:59.822 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:16:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:59 smithi119 ceph-mon[101432]: pgmap v1121: 105 pgs: 105 active+clean; 197 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 922 KiB/s wr, 29 op/s 2024-09-21T10:16:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:15:59 smithi119 ceph-mon[101432]: osdmap e940: 8 total, 8 up, 8 in 2024-09-21T10:16:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:59 smithi119 ceph-mon[105288]: pgmap v1121: 105 pgs: 105 active+clean; 197 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 922 KiB/s wr, 29 op/s 2024-09-21T10:16:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:15:59 smithi119 ceph-mon[105288]: osdmap e940: 8 total, 8 up, 8 in 2024-09-21T10:16:00.212 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:59 smithi145 ceph-mon[89359]: pgmap v1121: 105 pgs: 105 active+clean; 197 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 922 KiB/s wr, 29 op/s 2024-09-21T10:16:00.212 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:15:59 smithi145 ceph-mon[89359]: osdmap e940: 8 total, 8 up, 8 in 2024-09-21T10:16:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:16:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:16:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[105288]: osdmap e941: 8 total, 8 up, 8 in 2024-09-21T10:16:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[101432]: osdmap e941: 8 total, 8 up, 8 in 2024-09-21T10:16:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:00 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:00 smithi145 ceph-mon[89359]: osdmap e941: 8 total, 8 up, 8 in 2024-09-21T10:16:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:00 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:00 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:00 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:00 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:01 smithi145 ceph-mon[89359]: pgmap v1124: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:01 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:01 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:01 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:02.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[105288]: pgmap v1124: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:02.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:02.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:02.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:02.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[101432]: pgmap v1124: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:02.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:16:02.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:16:02.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:16:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:03 smithi145 ceph-mon[89359]: pgmap v1125: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:03 smithi119 ceph-mon[105288]: pgmap v1125: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:03 smithi119 ceph-mon[101432]: pgmap v1125: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:16:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:16:04] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:16:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:05 smithi145 ceph-mon[89359]: pgmap v1126: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-09-21T10:16:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:05 smithi119 ceph-mon[105288]: pgmap v1126: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-09-21T10:16:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:05 smithi119 ceph-mon[101432]: pgmap v1126: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.3 MiB/s wr, 38 op/s 2024-09-21T10:16:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:07 smithi145 ceph-mon[89359]: pgmap v1127: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:16:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:07 smithi145 ceph-mon[89359]: osdmap e942: 8 total, 8 up, 8 in 2024-09-21T10:16:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:07 smithi119 ceph-mon[101432]: pgmap v1127: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:16:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:07 smithi119 ceph-mon[101432]: osdmap e942: 8 total, 8 up, 8 in 2024-09-21T10:16:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:07 smithi119 ceph-mon[105288]: pgmap v1127: 105 pgs: 105 active+clean; 206 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:16:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:07 smithi119 ceph-mon[105288]: osdmap e942: 8 total, 8 up, 8 in 2024-09-21T10:16:09.018 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS2 (48987 ms) 2024-09-21T10:16:09.018 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDeterministicLUKS2PP 2024-09-21T10:16:09.018 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:16:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:08 smithi145 ceph-mon[89359]: osdmap e943: 8 total, 8 up, 8 in 2024-09-21T10:16:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:08 smithi119 ceph-mon[101432]: osdmap e943: 8 total, 8 up, 8 in 2024-09-21T10:16:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:08 smithi119 ceph-mon[105288]: osdmap e943: 8 total, 8 up, 8 in 2024-09-21T10:16:10.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:09 smithi145 ceph-mon[89359]: pgmap v1130: 105 pgs: 105 active+clean; 226 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-09-21T10:16:10.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:09 smithi145 ceph-mon[89359]: osdmap e944: 8 total, 8 up, 8 in 2024-09-21T10:16:10.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:16:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:16:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:09 smithi119 ceph-mon[105288]: pgmap v1130: 105 pgs: 105 active+clean; 226 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-09-21T10:16:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:09 smithi119 ceph-mon[105288]: osdmap e944: 8 total, 8 up, 8 in 2024-09-21T10:16:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:09 smithi119 ceph-mon[101432]: pgmap v1130: 105 pgs: 105 active+clean; 226 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-09-21T10:16:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:09 smithi119 ceph-mon[101432]: osdmap e944: 8 total, 8 up, 8 in 2024-09-21T10:16:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:11 smithi145 ceph-mon[89359]: pgmap v1132: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:11 smithi119 ceph-mon[105288]: pgmap v1132: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:11 smithi119 ceph-mon[101432]: pgmap v1132: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:13 smithi145 ceph-mon[89359]: pgmap v1133: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:13 smithi119 ceph-mon[101432]: pgmap v1133: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:13 smithi119 ceph-mon[105288]: pgmap v1133: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:14.870 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:16:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:16:14] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:16:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:15 smithi145 ceph-mon[89359]: pgmap v1134: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-09-21T10:16:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:16 smithi119 ceph-mon[105288]: pgmap v1134: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-09-21T10:16:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:15 smithi119 ceph-mon[101432]: pgmap v1134: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-09-21T10:16:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:18 smithi145 ceph-mon[89359]: pgmap v1135: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.8 MiB/s wr, 30 op/s 2024-09-21T10:16:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:18 smithi145 ceph-mon[89359]: osdmap e945: 8 total, 8 up, 8 in 2024-09-21T10:16:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:18 smithi119 ceph-mon[105288]: pgmap v1135: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.8 MiB/s wr, 30 op/s 2024-09-21T10:16:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:18 smithi119 ceph-mon[105288]: osdmap e945: 8 total, 8 up, 8 in 2024-09-21T10:16:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:18 smithi119 ceph-mon[101432]: pgmap v1135: 105 pgs: 105 active+clean; 230 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.8 MiB/s wr, 30 op/s 2024-09-21T10:16:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:18 smithi119 ceph-mon[101432]: osdmap e945: 8 total, 8 up, 8 in 2024-09-21T10:16:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:19.050 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:16:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:19 smithi145 ceph-mon[89359]: osdmap e946: 8 total, 8 up, 8 in 2024-09-21T10:16:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:19 smithi119 ceph-mon[101432]: osdmap e946: 8 total, 8 up, 8 in 2024-09-21T10:16:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:19 smithi119 ceph-mon[105288]: osdmap e946: 8 total, 8 up, 8 in 2024-09-21T10:16:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:20 smithi145 ceph-mon[89359]: pgmap v1138: 105 pgs: 105 active+clean; 242 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-09-21T10:16:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:20 smithi145 ceph-mon[89359]: osdmap e947: 8 total, 8 up, 8 in 2024-09-21T10:16:20.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:16:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:16:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:20 smithi119 ceph-mon[105288]: pgmap v1138: 105 pgs: 105 active+clean; 242 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-09-21T10:16:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:20 smithi119 ceph-mon[105288]: osdmap e947: 8 total, 8 up, 8 in 2024-09-21T10:16:20.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:20 smithi119 ceph-mon[101432]: pgmap v1138: 105 pgs: 105 active+clean; 242 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-09-21T10:16:20.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:20 smithi119 ceph-mon[101432]: osdmap e947: 8 total, 8 up, 8 in 2024-09-21T10:16:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:22 smithi145 ceph-mon[89359]: pgmap v1140: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:22 smithi119 ceph-mon[101432]: pgmap v1140: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:22 smithi119 ceph-mon[105288]: pgmap v1140: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:24 smithi145 ceph-mon[89359]: pgmap v1141: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:24 smithi119 ceph-mon[105288]: pgmap v1141: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:24 smithi119 ceph-mon[101432]: pgmap v1141: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:16:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:16:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:16:24] "GET /metrics HTTP/1.1" 200 36711 "" "Prometheus/2.51.0" 2024-09-21T10:16:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:26 smithi145 ceph-mon[89359]: pgmap v1142: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-09-21T10:16:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:26 smithi119 ceph-mon[105288]: pgmap v1142: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-09-21T10:16:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:26 smithi119 ceph-mon[101432]: pgmap v1142: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 34 op/s 2024-09-21T10:16:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:28 smithi145 ceph-mon[89359]: pgmap v1143: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.8 MiB/s wr, 30 op/s 2024-09-21T10:16:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:28 smithi145 ceph-mon[89359]: osdmap e948: 8 total, 8 up, 8 in 2024-09-21T10:16:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:28 smithi119 ceph-mon[101432]: pgmap v1143: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.8 MiB/s wr, 30 op/s 2024-09-21T10:16:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:28 smithi119 ceph-mon[101432]: osdmap e948: 8 total, 8 up, 8 in 2024-09-21T10:16:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:28 smithi119 ceph-mon[105288]: pgmap v1143: 105 pgs: 105 active+clean; 246 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.8 MiB/s wr, 30 op/s 2024-09-21T10:16:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:28 smithi119 ceph-mon[105288]: osdmap e948: 8 total, 8 up, 8 in 2024-09-21T10:16:29.082 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:16:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:29 smithi119 ceph-mon[105288]: osdmap e949: 8 total, 8 up, 8 in 2024-09-21T10:16:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:29 smithi119 ceph-mon[101432]: osdmap e949: 8 total, 8 up, 8 in 2024-09-21T10:16:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:29 smithi145 ceph-mon[89359]: osdmap e949: 8 total, 8 up, 8 in 2024-09-21T10:16:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:30 smithi119 ceph-mon[105288]: pgmap v1146: 105 pgs: 105 active+clean; 257 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2024-09-21T10:16:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:30 smithi119 ceph-mon[105288]: osdmap e950: 8 total, 8 up, 8 in 2024-09-21T10:16:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:30 smithi119 ceph-mon[101432]: pgmap v1146: 105 pgs: 105 active+clean; 257 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2024-09-21T10:16:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:30 smithi119 ceph-mon[101432]: osdmap e950: 8 total, 8 up, 8 in 2024-09-21T10:16:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:30 smithi145 ceph-mon[89359]: pgmap v1146: 105 pgs: 105 active+clean; 257 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2024-09-21T10:16:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:30 smithi145 ceph-mon[89359]: osdmap e950: 8 total, 8 up, 8 in 2024-09-21T10:16:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:16:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:16:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:32 smithi119 ceph-mon[105288]: pgmap v1148: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:16:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:32 smithi119 ceph-mon[101432]: pgmap v1148: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:16:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:32 smithi145 ceph-mon[89359]: pgmap v1148: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:16:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:34 smithi119 ceph-mon[105288]: pgmap v1149: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:16:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:34 smithi119 ceph-mon[101432]: pgmap v1149: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:16:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:34 smithi145 ceph-mon[89359]: pgmap v1149: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 27 op/s 2024-09-21T10:16:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:16:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:16:34] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:16:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:36 smithi119 ceph-mon[105288]: pgmap v1150: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 35 op/s 2024-09-21T10:16:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:36 smithi119 ceph-mon[101432]: pgmap v1150: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 35 op/s 2024-09-21T10:16:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:36 smithi145 ceph-mon[89359]: pgmap v1150: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.1 MiB/s wr, 35 op/s 2024-09-21T10:16:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:38 smithi119 ceph-mon[101432]: pgmap v1151: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.9 MiB/s wr, 31 op/s 2024-09-21T10:16:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:38 smithi119 ceph-mon[105288]: pgmap v1151: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.9 MiB/s wr, 31 op/s 2024-09-21T10:16:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:38 smithi145 ceph-mon[89359]: pgmap v1151: 105 pgs: 105 active+clean; 268 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.9 MiB/s wr, 31 op/s 2024-09-21T10:16:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:39 smithi119 ceph-mon[105288]: osdmap e951: 8 total, 8 up, 8 in 2024-09-21T10:16:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:39 smithi119 ceph-mon[101432]: osdmap e951: 8 total, 8 up, 8 in 2024-09-21T10:16:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:39 smithi145 ceph-mon[89359]: osdmap e951: 8 total, 8 up, 8 in 2024-09-21T10:16:40.122 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:16:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:40 smithi119 ceph-mon[105288]: pgmap v1153: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-21T10:16:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:40 smithi119 ceph-mon[105288]: osdmap e952: 8 total, 8 up, 8 in 2024-09-21T10:16:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:40 smithi119 ceph-mon[101432]: pgmap v1153: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-21T10:16:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:40 smithi119 ceph-mon[101432]: osdmap e952: 8 total, 8 up, 8 in 2024-09-21T10:16:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:40 smithi145 ceph-mon[89359]: pgmap v1153: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 2.5 MiB/s wr, 22 op/s 2024-09-21T10:16:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:40 smithi145 ceph-mon[89359]: osdmap e952: 8 total, 8 up, 8 in 2024-09-21T10:16:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:16:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:16:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:41 smithi119 ceph-mon[105288]: osdmap e953: 8 total, 8 up, 8 in 2024-09-21T10:16:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:41 smithi119 ceph-mon[101432]: osdmap e953: 8 total, 8 up, 8 in 2024-09-21T10:16:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:41 smithi145 ceph-mon[89359]: osdmap e953: 8 total, 8 up, 8 in 2024-09-21T10:16:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:42 smithi119 ceph-mon[105288]: pgmap v1156: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:16:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:42 smithi119 ceph-mon[101432]: pgmap v1156: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:16:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:42 smithi145 ceph-mon[89359]: pgmap v1156: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:16:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:44 smithi119 ceph-mon[101432]: pgmap v1157: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:16:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:44 smithi119 ceph-mon[105288]: pgmap v1157: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:16:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:44 smithi145 ceph-mon[89359]: pgmap v1157: 105 pgs: 105 active+clean; 287 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:16:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:16:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:16:44] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:16:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:46 smithi119 ceph-mon[105288]: pgmap v1158: 105 pgs: 105 active+clean; 290 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:16:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:46 smithi119 ceph-mon[101432]: pgmap v1158: 105 pgs: 105 active+clean; 290 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:16:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:46 smithi145 ceph-mon[89359]: pgmap v1158: 105 pgs: 105 active+clean; 290 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:16:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:48 smithi119 ceph-mon[101432]: pgmap v1159: 105 pgs: 105 active+clean; 290 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 22 op/s 2024-09-21T10:16:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:48 smithi119 ceph-mon[105288]: pgmap v1159: 105 pgs: 105 active+clean; 290 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 22 op/s 2024-09-21T10:16:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:48 smithi145 ceph-mon[89359]: pgmap v1159: 105 pgs: 105 active+clean; 290 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.7 KiB/s wr, 22 op/s 2024-09-21T10:16:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:16:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:49 smithi145 ceph-mon[89359]: osdmap e954: 8 total, 8 up, 8 in 2024-09-21T10:16:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:49 smithi119 ceph-mon[101432]: osdmap e954: 8 total, 8 up, 8 in 2024-09-21T10:16:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:49 smithi119 ceph-mon[105288]: osdmap e954: 8 total, 8 up, 8 in 2024-09-21T10:16:50.151 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:16:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:50 smithi145 ceph-mon[89359]: pgmap v1161: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 956 KiB/s wr, 30 op/s 2024-09-21T10:16:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:50 smithi145 ceph-mon[89359]: osdmap e955: 8 total, 8 up, 8 in 2024-09-21T10:16:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:16:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:16:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:16:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:50 smithi119 ceph-mon[105288]: pgmap v1161: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 956 KiB/s wr, 30 op/s 2024-09-21T10:16:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:50 smithi119 ceph-mon[105288]: osdmap e955: 8 total, 8 up, 8 in 2024-09-21T10:16:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:50 smithi119 ceph-mon[101432]: pgmap v1161: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 956 KiB/s wr, 30 op/s 2024-09-21T10:16:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:50 smithi119 ceph-mon[101432]: osdmap e955: 8 total, 8 up, 8 in 2024-09-21T10:16:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:51 smithi145 ceph-mon[89359]: osdmap e956: 8 total, 8 up, 8 in 2024-09-21T10:16:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:51 smithi119 ceph-mon[105288]: osdmap e956: 8 total, 8 up, 8 in 2024-09-21T10:16:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:51 smithi119 ceph-mon[101432]: osdmap e956: 8 total, 8 up, 8 in 2024-09-21T10:16:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:52 smithi145 ceph-mon[89359]: pgmap v1164: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:52 smithi119 ceph-mon[101432]: pgmap v1164: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:52 smithi119 ceph-mon[105288]: pgmap v1164: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:54.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:54 smithi119 ceph-mon[105288]: pgmap v1165: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:54 smithi119 ceph-mon[101432]: pgmap v1165: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:54 smithi145 ceph-mon[89359]: pgmap v1165: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 17 op/s 2024-09-21T10:16:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:16:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:16:54] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:16:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:56 smithi145 ceph-mon[89359]: pgmap v1166: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:16:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:56 smithi119 ceph-mon[105288]: pgmap v1166: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:16:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:56 smithi119 ceph-mon[101432]: pgmap v1166: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 40 op/s 2024-09-21T10:16:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:58 smithi145 ceph-mon[89359]: pgmap v1167: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:16:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:58 smithi119 ceph-mon[105288]: pgmap v1167: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:16:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:58 smithi119 ceph-mon[101432]: pgmap v1167: 105 pgs: 105 active+clean; 312 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 MiB/s wr, 22 op/s 2024-09-21T10:16:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:16:59 smithi145 ceph-mon[89359]: osdmap e957: 8 total, 8 up, 8 in 2024-09-21T10:16:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:16:59 smithi119 ceph-mon[101432]: osdmap e957: 8 total, 8 up, 8 in 2024-09-21T10:16:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:16:59 smithi119 ceph-mon[105288]: osdmap e957: 8 total, 8 up, 8 in 2024-09-21T10:17:00.212 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDeterministicLUKS2PP (51195 ms) 2024-09-21T10:17:00.213 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateDiscard 2024-09-21T10:17:00.213 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:00 smithi145 ceph-mon[89359]: pgmap v1169: 105 pgs: 105 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 2.3 MiB/s wr, 31 op/s 2024-09-21T10:17:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:00 smithi145 ceph-mon[89359]: osdmap e958: 8 total, 8 up, 8 in 2024-09-21T10:17:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:17:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:17:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:00 smithi119 ceph-mon[101432]: pgmap v1169: 105 pgs: 105 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 2.3 MiB/s wr, 31 op/s 2024-09-21T10:17:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:00 smithi119 ceph-mon[101432]: osdmap e958: 8 total, 8 up, 8 in 2024-09-21T10:17:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:00 smithi119 ceph-mon[105288]: pgmap v1169: 105 pgs: 105 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 2.3 MiB/s wr, 31 op/s 2024-09-21T10:17:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:00 smithi119 ceph-mon[105288]: osdmap e958: 8 total, 8 up, 8 in 2024-09-21T10:17:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:01 smithi145 ceph-mon[89359]: osdmap e959: 8 total, 8 up, 8 in 2024-09-21T10:17:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:01 smithi145 ceph-mon[89359]: pgmap v1172: 105 pgs: 105 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:17:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:01 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:01 smithi119 ceph-mon[105288]: osdmap e959: 8 total, 8 up, 8 in 2024-09-21T10:17:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:01 smithi119 ceph-mon[105288]: pgmap v1172: 105 pgs: 105 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:17:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:01 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:01 smithi119 ceph-mon[101432]: osdmap e959: 8 total, 8 up, 8 in 2024-09-21T10:17:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:01 smithi119 ceph-mon[101432]: pgmap v1172: 105 pgs: 105 active+clean; 332 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.2 KiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-21T10:17:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:01 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:17:02.270 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:02 smithi119 ceph-mon[105288]: osdmap e960: 8 total, 8 up, 8 in 2024-09-21T10:17:02.271 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:02 smithi119 ceph-mon[101432]: osdmap e960: 8 total, 8 up, 8 in 2024-09-21T10:17:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:02 smithi145 ceph-mon[89359]: osdmap e960: 8 total, 8 up, 8 in 2024-09-21T10:17:03.234 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateDiscard (3021 ms) 2024-09-21T10:17:03.234 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateTruncate 2024-09-21T10:17:03.234 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:03 smithi145 ceph-mon[89359]: osdmap e961: 8 total, 8 up, 8 in 2024-09-21T10:17:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:03 smithi145 ceph-mon[89359]: pgmap v1175: 105 pgs: 105 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:17:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:03 smithi119 ceph-mon[101432]: osdmap e961: 8 total, 8 up, 8 in 2024-09-21T10:17:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:03 smithi119 ceph-mon[101432]: pgmap v1175: 105 pgs: 105 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:17:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:03 smithi119 ceph-mon[105288]: osdmap e961: 8 total, 8 up, 8 in 2024-09-21T10:17:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:03 smithi119 ceph-mon[105288]: pgmap v1175: 105 pgs: 105 active+clean; 332 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-09-21T10:17:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:04 smithi119 ceph-mon[101432]: osdmap e962: 8 total, 8 up, 8 in 2024-09-21T10:17:04.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:04 smithi119 ceph-mon[105288]: osdmap e962: 8 total, 8 up, 8 in 2024-09-21T10:17:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:04 smithi145 ceph-mon[89359]: osdmap e962: 8 total, 8 up, 8 in 2024-09-21T10:17:04.813 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:17:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:17:04] "GET /metrics HTTP/1.1" 200 36717 "" "Prometheus/2.51.0" 2024-09-21T10:17:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:05 smithi145 ceph-mon[89359]: osdmap e963: 8 total, 8 up, 8 in 2024-09-21T10:17:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:05 smithi145 ceph-mon[89359]: pgmap v1178: 105 pgs: 105 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 14 KiB/s wr, 116 op/s 2024-09-21T10:17:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[105288]: osdmap e963: 8 total, 8 up, 8 in 2024-09-21T10:17:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[105288]: pgmap v1178: 105 pgs: 105 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 14 KiB/s wr, 116 op/s 2024-09-21T10:17:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[101432]: osdmap e963: 8 total, 8 up, 8 in 2024-09-21T10:17:05.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[101432]: pgmap v1178: 105 pgs: 105 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 71 KiB/s rd, 14 KiB/s wr, 116 op/s 2024-09-21T10:17:05.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:05.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.261 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateTruncate (3028 ms) 2024-09-21T10:17:06.262 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateWriteAndTruncate 2024-09-21T10:17:06.262 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:06 smithi145 ceph-mon[89359]: osdmap e964: 8 total, 8 up, 8 in 2024-09-21T10:17:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:17:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:17:06.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[105288]: osdmap e964: 8 total, 8 up, 8 in 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[101432]: osdmap e964: 8 total, 8 up, 8 in 2024-09-21T10:17:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:17:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:17:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:17:07.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:07 smithi145 ceph-mon[89359]: osdmap e965: 8 total, 8 up, 8 in 2024-09-21T10:17:07.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:07 smithi145 ceph-mon[89359]: pgmap v1181: 105 pgs: 105 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 15 KiB/s wr, 134 op/s 2024-09-21T10:17:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:07 smithi119 ceph-mon[101432]: osdmap e965: 8 total, 8 up, 8 in 2024-09-21T10:17:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:07 smithi119 ceph-mon[101432]: pgmap v1181: 105 pgs: 105 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 15 KiB/s wr, 134 op/s 2024-09-21T10:17:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:07 smithi119 ceph-mon[105288]: osdmap e965: 8 total, 8 up, 8 in 2024-09-21T10:17:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:07 smithi119 ceph-mon[105288]: pgmap v1181: 105 pgs: 105 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 84 KiB/s rd, 15 KiB/s wr, 134 op/s 2024-09-21T10:17:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:08 smithi145 ceph-mon[89359]: osdmap e966: 8 total, 8 up, 8 in 2024-09-21T10:17:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:08 smithi119 ceph-mon[101432]: osdmap e966: 8 total, 8 up, 8 in 2024-09-21T10:17:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:08 smithi119 ceph-mon[105288]: osdmap e966: 8 total, 8 up, 8 in 2024-09-21T10:17:09.297 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateWriteAndTruncate (3036 ms) 2024-09-21T10:17:09.298 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStress 2024-09-21T10:17:09.298 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:09.415 INFO:tasks.workunit.client.0.smithi119.stdout: i=0 exists [987023~7718531,48470354~2859832,63487045~3789375,168602994~8810553,295227730~2977882,304036132~6026099,338657068~8947460] wrote [987023~7718531,48470354~2859832,63487045~3789375,168602994~8810553,295227730~2977882,304036132~6026099,338657068~8947460] 2024-09-21T10:17:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:09 smithi145 ceph-mon[89359]: osdmap e967: 8 total, 8 up, 8 in 2024-09-21T10:17:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:09 smithi145 ceph-mon[89359]: pgmap v1184: 105 pgs: 105 active+clean; 338 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 518 KiB/s wr, 63 op/s 2024-09-21T10:17:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:09 smithi119 ceph-mon[101432]: osdmap e967: 8 total, 8 up, 8 in 2024-09-21T10:17:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:09 smithi119 ceph-mon[101432]: pgmap v1184: 105 pgs: 105 active+clean; 338 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 518 KiB/s wr, 63 op/s 2024-09-21T10:17:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:09 smithi119 ceph-mon[105288]: osdmap e967: 8 total, 8 up, 8 in 2024-09-21T10:17:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:09 smithi119 ceph-mon[105288]: pgmap v1184: 105 pgs: 105 active+clean; 338 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 40 KiB/s rd, 518 KiB/s wr, 63 op/s 2024-09-21T10:17:10.321 INFO:tasks.workunit.client.0.smithi119.stdout: i=1 exists [987023~7718531,48470354~2859832,63487045~3789375,73357425~130357,123250767~308257,126258979~8030873,168602994~8810553,187819226~1568752,291722063~1204351,295227730~2977882,304036132~6026099,323584036~3186360,337140860~10463668,371282199~4549840,381676868~1924010] wrote [73357425~130357,123250767~308257,126258979~8030873,187819226~1568752,291722063~1204351,323584036~3186360,337140860~4197101,371282199~4549840,381676868~1924010] 2024-09-21T10:17:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:10 smithi145 ceph-mon[89359]: osdmap e968: 8 total, 8 up, 8 in 2024-09-21T10:17:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:17:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:17:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:10 smithi119 ceph-mon[105288]: osdmap e968: 8 total, 8 up, 8 in 2024-09-21T10:17:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:10 smithi119 ceph-mon[101432]: osdmap e968: 8 total, 8 up, 8 in 2024-09-21T10:17:11.348 INFO:tasks.workunit.client.0.smithi119.stdout: i=2 exists [987023~10882921,24109271~8164295,46114405~5215781,63487045~3789375,73357425~130357,90690282~540466,123250767~308257,124256485~10033367,168602994~8810553,187819226~1568752,194851207~6720279,218818238~1747253,240781071~4755025,291722063~1204351,295227730~2977882,304036132~6026099,314194184~3108995,323584036~3186360,337140860~10463668,371282199~4549840,381676868~1924010] wrote [4920468~6949476,24109271~8164295,46114405~4416020,90690282~540466,124256485~8003279,194851207~6720279,218818238~1747253,240781071~4755025,314194184~3108995,381827012~1351784] 2024-09-21T10:17:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:11 smithi145 ceph-mon[89359]: osdmap e969: 8 total, 8 up, 8 in 2024-09-21T10:17:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:11 smithi145 ceph-mon[89359]: pgmap v1187: 105 pgs: 105 active+clean; 354 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 3.6 MiB/s wr, 107 op/s 2024-09-21T10:17:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:11 smithi119 ceph-mon[105288]: osdmap e969: 8 total, 8 up, 8 in 2024-09-21T10:17:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:11 smithi119 ceph-mon[105288]: pgmap v1187: 105 pgs: 105 active+clean; 354 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 3.6 MiB/s wr, 107 op/s 2024-09-21T10:17:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:11 smithi119 ceph-mon[101432]: osdmap e969: 8 total, 8 up, 8 in 2024-09-21T10:17:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:11 smithi119 ceph-mon[101432]: pgmap v1187: 105 pgs: 105 active+clean; 354 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 67 KiB/s rd, 3.6 MiB/s wr, 107 op/s 2024-09-21T10:17:12.389 INFO:tasks.workunit.client.0.smithi119.stdout: i=3 exists [987023~10882921,24109271~8164295,32577525~5846129,46114405~5215781,52808135~14468285,73357425~130357,90681367~1732321,94348578~2489555,123250767~308257,124256485~10033367,161611977~6757093,168602994~8810553,187819226~1568752,194851207~6720279,218818238~1747253,240781071~4755025,291722063~1204351,295227730~7268735,304036132~6026099,314194184~3108995,323584036~3186360,336005050~17729215,371282199~4549840,380821839~3925984] wrote [32577525~5846129,52808135~11361339,90681367~1732321,94348578~2489555,161611977~6757093,296708045~5788420,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:12 smithi145 ceph-mon[89359]: osdmap e970: 8 total, 8 up, 8 in 2024-09-21T10:17:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:12 smithi119 ceph-mon[105288]: osdmap e970: 8 total, 8 up, 8 in 2024-09-21T10:17:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:12 smithi119 ceph-mon[101432]: osdmap e970: 8 total, 8 up, 8 in 2024-09-21T10:17:13.388 INFO:tasks.workunit.client.0.smithi119.stdout: i=4 exists [987023~10882921,24109271~8164295,32577525~5846129,46114405~5215781,52808135~14468285,73357425~130357,90681367~1732321,94348578~2489555,106726996~7825270,123250767~308257,123829650~10460202,146381793~5644987,161611977~6757093,168602994~8810553,187819226~1568752,193444193~8127293,218818238~1747253,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,295227730~7268735,304036132~6026099,314194184~3108995,323584036~8233466,336005050~17729215,363488934~7299761,371282199~4549840,380821839~3925984] wrote [106726996~7825270,123829650~6594126,146381793~5644987,193444193~5178520,232289009~4460589,238432676~9225993,268409630~3522630,324210499~7607003,363488934~7299761] 2024-09-21T10:17:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:13 smithi145 ceph-mon[89359]: osdmap e971: 8 total, 8 up, 8 in 2024-09-21T10:17:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:13 smithi145 ceph-mon[89359]: pgmap v1190: 105 pgs: 105 active+clean; 354 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 3.1 MiB/s wr, 62 op/s 2024-09-21T10:17:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:13 smithi119 ceph-mon[101432]: osdmap e971: 8 total, 8 up, 8 in 2024-09-21T10:17:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:13 smithi119 ceph-mon[101432]: pgmap v1190: 105 pgs: 105 active+clean; 354 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 3.1 MiB/s wr, 62 op/s 2024-09-21T10:17:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:13 smithi119 ceph-mon[105288]: osdmap e971: 8 total, 8 up, 8 in 2024-09-21T10:17:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:13 smithi119 ceph-mon[105288]: pgmap v1190: 105 pgs: 105 active+clean; 354 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 41 KiB/s rd, 3.1 MiB/s wr, 62 op/s 2024-09-21T10:17:14.367 INFO:tasks.workunit.client.0.smithi119.stdout: i=5 exists [987023~10882921,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~5215781,52808135~14468285,73357425~130357,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123250767~308257,123829650~10460202,146381793~5644987,161611977~6757093,168602994~8810553,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,295227730~7268735,303111601~6950630,314194184~3108995,320886570~983977,323584036~8233466,336005050~22232311,363488934~7299761,371282199~4549840,380821839~3925984,402396115~6360928,410461141~1101111] wrote [13177842~2866366,39443894~4770339,55845980~5219038,88009097~1929038,223540318~707294,303111601~3118728,320886570~983977,351518511~6718850,402396115~6360928,410461141~1101111] 2024-09-21T10:17:14.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:14 smithi119 ceph-mon[105288]: osdmap e972: 8 total, 8 up, 8 in 2024-09-21T10:17:14.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:14 smithi119 ceph-mon[101432]: osdmap e972: 8 total, 8 up, 8 in 2024-09-21T10:17:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:14 smithi145 ceph-mon[89359]: osdmap e972: 8 total, 8 up, 8 in 2024-09-21T10:17:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:17:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:17:14] "GET /metrics HTTP/1.1" 200 36717 "" "Prometheus/2.51.0" 2024-09-21T10:17:15.409 INFO:tasks.workunit.client.0.smithi119.stdout: i=6 exists [987023~10882921,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~5215781,52808135~14468285,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10460202,144031262~7995518,161611977~6757093,168602994~8810553,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,295227730~7268735,303111601~14191578,320886570~983977,323584036~8233466,332868862~25804438,363488934~7299761,371282199~13465624,385039936~6349854,402396115~6360928,410461141~1101111] wrote [76370336~7503356,99139515~7728324,144031262~6739415,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,374202738~8382468,385039936~6349854] 2024-09-21T10:17:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:15 smithi119 ceph-mon[101432]: osdmap e973: 8 total, 8 up, 8 in 2024-09-21T10:17:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:15 smithi119 ceph-mon[101432]: pgmap v1193: 105 pgs: 105 active+clean; 593 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 49 MiB/s wr, 99 op/s 2024-09-21T10:17:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:15 smithi119 ceph-mon[105288]: osdmap e973: 8 total, 8 up, 8 in 2024-09-21T10:17:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:15 smithi119 ceph-mon[105288]: pgmap v1193: 105 pgs: 105 active+clean; 593 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 49 MiB/s wr, 99 op/s 2024-09-21T10:17:15.687 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:15 smithi145 ceph-mon[89359]: osdmap e973: 8 total, 8 up, 8 in 2024-09-21T10:17:15.687 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:15 smithi145 ceph-mon[89359]: pgmap v1193: 105 pgs: 105 active+clean; 593 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 49 MiB/s wr, 99 op/s 2024-09-21T10:17:16.363 INFO:tasks.workunit.client.0.smithi119.stdout: i=7 exists [987023~10882921,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~5215781,52808135~14468285,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,168602994~8810553,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,295227730~22075449,320886570~983977,323584036~8233466,332868862~25804438,359901878~1171315,363488934~7299761,371282199~13465624,385039936~6349854,402396115~7541723,410461141~1101111] wrote [13033324~2616660,17903366~4986190,25823944~2571315,69648386~1173386,133610121~798745,140433786~462019,152640544~4522823,301562997~3599492,359901878~1171315,403544798~6393040] 2024-09-21T10:17:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:16 smithi119 ceph-mon[101432]: osdmap e974: 8 total, 8 up, 8 in 2024-09-21T10:17:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:16 smithi119 ceph-mon[105288]: osdmap e974: 8 total, 8 up, 8 in 2024-09-21T10:17:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:16 smithi145 ceph-mon[89359]: osdmap e974: 8 total, 8 up, 8 in 2024-09-21T10:17:17.412 INFO:tasks.workunit.client.0.smithi119.stdout: i=8 exists [987023~10882921,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~5215781,52808135~14468285,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,168602994~8810553,185246972~826100,187819226~1568752,189676270~11895216,218818238~1747253,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,295227730~22075449,320886570~983977,323584036~8233466,332868862~25804438,359901878~1171315,363488934~7299761,371282199~27881433,402396115~7541723,410461141~1101111] wrote [53157964~7891177,152132680~1977901,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,379369400~7767668,388300573~10863059] 2024-09-21T10:17:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:17 smithi119 ceph-mon[101432]: osdmap e975: 8 total, 8 up, 8 in 2024-09-21T10:17:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:17 smithi119 ceph-mon[101432]: pgmap v1196: 105 pgs: 105 active+clean; 627 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 182 KiB/s rd, 57 MiB/s wr, 311 op/s 2024-09-21T10:17:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:17 smithi119 ceph-mon[105288]: osdmap e975: 8 total, 8 up, 8 in 2024-09-21T10:17:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:17 smithi119 ceph-mon[105288]: pgmap v1196: 105 pgs: 105 active+clean; 627 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 182 KiB/s rd, 57 MiB/s wr, 311 op/s 2024-09-21T10:17:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:17 smithi145 ceph-mon[89359]: osdmap e975: 8 total, 8 up, 8 in 2024-09-21T10:17:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:17 smithi145 ceph-mon[89359]: pgmap v1196: 105 pgs: 105 active+clean; 627 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 182 KiB/s rd, 57 MiB/s wr, 311 op/s 2024-09-21T10:17:18.409 INFO:tasks.workunit.client.0.smithi119.stdout: i=9 exists [987023~10882921,13033324~4328551,17903366~5163012,24109271~14314383,39443894~4770339,46114405~5215781,52808135~14468285,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,168602994~8810553,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,295227730~22075449,320886570~983977,323584036~8233466,332868862~25804438,359901878~1171315,363488934~7299761,371282199~27881433,402396115~7541723,410461141~1101111] wrote [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,113327605~1146979,212176573~2340741,224922963~5398542,308825230~5425531,373691010~1041998] 2024-09-21T10:17:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:18 smithi119 ceph-mon[105288]: osdmap e976: 8 total, 8 up, 8 in 2024-09-21T10:17:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:18 smithi119 ceph-mon[101432]: osdmap e976: 8 total, 8 up, 8 in 2024-09-21T10:17:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:18 smithi145 ceph-mon[89359]: osdmap e976: 8 total, 8 up, 8 in 2024-09-21T10:17:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:19.360 INFO:tasks.workunit.client.0.smithi119.stdout: i=10 exists [987023~16374852,17903366~5163012,24109271~14314383,39443894~4770339,46114405~5215781,52808135~14468285,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,168602994~8810553,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332868862~25804438,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] wrote [10137230~3491714,55496708~3205502,109886548~39489,130891915~1515516,169726147~3786678,197256444~2704019,318091802~2618148,337769986~374855,359615966~5871094,410712466~88886] 2024-09-21T10:17:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:19 smithi119 ceph-mon[105288]: osdmap e977: 8 total, 8 up, 8 in 2024-09-21T10:17:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:19 smithi119 ceph-mon[105288]: pgmap v1199: 105 pgs: 105 active+clean; 787 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 42 MiB/s wr, 323 op/s 2024-09-21T10:17:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:19 smithi119 ceph-mon[101432]: osdmap e977: 8 total, 8 up, 8 in 2024-09-21T10:17:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:19 smithi119 ceph-mon[101432]: pgmap v1199: 105 pgs: 105 active+clean; 787 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 42 MiB/s wr, 323 op/s 2024-09-21T10:17:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:19 smithi145 ceph-mon[89359]: osdmap e977: 8 total, 8 up, 8 in 2024-09-21T10:17:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:19 smithi145 ceph-mon[89359]: pgmap v1199: 105 pgs: 105 active+clean; 787 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 200 KiB/s rd, 42 MiB/s wr, 323 op/s 2024-09-21T10:17:20.477 INFO:tasks.workunit.client.0.smithi119.stdout: i=11 exists [987023~16374852,17903366~5163012,23077889~15345765,39443894~4770339,46114405~5215781,52808135~14468285,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123250767~308257,123829650~10579216,140433786~462019,144031262~13132105,161611977~6757093,168602994~8810553,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332868862~25804438,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] wrote [23077889~1786236,71187875~2918851,91636093~10962678,146820042~6886028,193062700~7513532,272525416~8033351,310542780~6335230,403122934~1766297] 2024-09-21T10:17:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:20 smithi145 ceph-mon[89359]: osdmap e978: 8 total, 8 up, 8 in 2024-09-21T10:17:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:17:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:17:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:20 smithi119 ceph-mon[101432]: osdmap e978: 8 total, 8 up, 8 in 2024-09-21T10:17:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:20 smithi119 ceph-mon[105288]: osdmap e978: 8 total, 8 up, 8 in 2024-09-21T10:17:21.442 INFO:tasks.workunit.client.0.smithi119.stdout: i=12 exists [987023~16374852,17903366~5163012,23077889~15345765,39443894~4770339,46114405~21162015,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123250767~308257,123829650~10579216,140433786~462019,144031262~13132105,161611977~15801570,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~26084992,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] wrote [28477050~4613848,51254163~2799606,81640613~1197037,167981878~2908531,191882768~4277993,231013148~5168624,277827050~797905,305787002~9390867,332588308~827816] 2024-09-21T10:17:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:21 smithi119 ceph-mon[101432]: osdmap e979: 8 total, 8 up, 8 in 2024-09-21T10:17:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:21 smithi119 ceph-mon[101432]: pgmap v1202: 105 pgs: 105 active+clean; 840 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 179 KiB/s rd, 47 MiB/s wr, 291 op/s 2024-09-21T10:17:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:21 smithi119 ceph-mon[105288]: osdmap e979: 8 total, 8 up, 8 in 2024-09-21T10:17:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:21 smithi119 ceph-mon[105288]: pgmap v1202: 105 pgs: 105 active+clean; 840 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 179 KiB/s rd, 47 MiB/s wr, 291 op/s 2024-09-21T10:17:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:21 smithi145 ceph-mon[89359]: osdmap e979: 8 total, 8 up, 8 in 2024-09-21T10:17:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:21 smithi145 ceph-mon[89359]: pgmap v1202: 105 pgs: 105 active+clean; 840 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 179 KiB/s rd, 47 MiB/s wr, 291 op/s 2024-09-21T10:17:22.468 INFO:tasks.workunit.client.0.smithi119.stdout: i=13 exists [987023~16374852,17903366~5163012,23077889~15345765,39443894~4770339,46114405~21162015,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~13132105,161611977~15801570,185246972~826100,187819226~1568752,189676270~15460646,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~38200387,371282199~27881433,402396115~7541723,410461141~1101111] wrote [31740612~3461096,87497139~7774339,107841497~8162972,135450794~2524008,145811637~8122332,199738595~5398321,223107934~1074781,284670827~5801079,352853425~7795887,402573262~5559752] 2024-09-21T10:17:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:22 smithi119 ceph-mon[101432]: osdmap e980: 8 total, 8 up, 8 in 2024-09-21T10:17:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:22 smithi119 ceph-mon[105288]: osdmap e980: 8 total, 8 up, 8 in 2024-09-21T10:17:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:22 smithi145 ceph-mon[89359]: osdmap e980: 8 total, 8 up, 8 in 2024-09-21T10:17:23.438 INFO:tasks.workunit.client.0.smithi119.stdout: i=14 exists [987023~16374852,17903366~5163012,23077889~15345765,39443894~4770339,46114405~21162015,69648386~4458340,76370336~7503356,87497139~28507330,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~15801570,185246972~826100,186816738~18320178,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~38200387,371282199~27881433,402396115~7541723,410461141~1101111] wrote [70391236~2357106,153095864~7375224,169289246~106547,186816738~7215627,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,361646431~2900270,375410865~6845743] 2024-09-21T10:17:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:23 smithi119 ceph-mon[101432]: osdmap e981: 8 total, 8 up, 8 in 2024-09-21T10:17:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:23 smithi119 ceph-mon[101432]: pgmap v1205: 105 pgs: 105 active+clean; 840 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 13 MiB/s wr, 180 op/s 2024-09-21T10:17:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:23 smithi119 ceph-mon[105288]: osdmap e981: 8 total, 8 up, 8 in 2024-09-21T10:17:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:23 smithi119 ceph-mon[105288]: pgmap v1205: 105 pgs: 105 active+clean; 840 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 13 MiB/s wr, 180 op/s 2024-09-21T10:17:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:23 smithi145 ceph-mon[89359]: osdmap e981: 8 total, 8 up, 8 in 2024-09-21T10:17:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:23 smithi145 ceph-mon[89359]: pgmap v1205: 105 pgs: 105 active+clean; 840 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 109 KiB/s rd, 13 MiB/s wr, 180 op/s 2024-09-21T10:17:24.426 INFO:tasks.workunit.client.0.smithi119.stdout: i=15 exists [987023~16374852,17903366~5163012,23077889~15345765,39443894~27832526,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~18320178,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~38200387,371282199~27881433,402396115~7541723,410461141~1101111] wrote [2543338~2707254,34940153~3153596,43960151~4773045,118600721~3501566,147119839~4572824,163757885~4407634,175596367~2528958,213967659~2691695,386800431~4472496,402901262~853192] 2024-09-21T10:17:24.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:24 smithi119 ceph-mon[101432]: osdmap e982: 8 total, 8 up, 8 in 2024-09-21T10:17:24.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:17:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:17:24] "GET /metrics HTTP/1.1" 200 36717 "" "Prometheus/2.51.0" 2024-09-21T10:17:24.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:24 smithi119 ceph-mon[105288]: osdmap e982: 8 total, 8 up, 8 in 2024-09-21T10:17:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:24 smithi145 ceph-mon[89359]: osdmap e982: 8 total, 8 up, 8 in 2024-09-21T10:17:25.470 INFO:tasks.workunit.client.0.smithi119.stdout: i=16 exists [987023~22079355,23077889~15345765,39443894~27832526,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~38200387,371282199~27881433,402396115~7541723,410461141~1101111] wrote [15885326~5190114,26686115~8062042,81873196~2901335,150410923~4222509,171480060~4147314,202498566~3702769,261441010~4291601,336572672~4979196,388641439~3935572] 2024-09-21T10:17:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:25 smithi145 ceph-mon[89359]: osdmap e983: 8 total, 8 up, 8 in 2024-09-21T10:17:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:25 smithi145 ceph-mon[89359]: pgmap v1208: 105 pgs: 105 active+clean; 1020 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 46 MiB/s wr, 158 op/s 2024-09-21T10:17:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:25 smithi119 ceph-mon[105288]: osdmap e983: 8 total, 8 up, 8 in 2024-09-21T10:17:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:25 smithi119 ceph-mon[105288]: pgmap v1208: 105 pgs: 105 active+clean; 1020 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 46 MiB/s wr, 158 op/s 2024-09-21T10:17:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:25 smithi119 ceph-mon[101432]: osdmap e983: 8 total, 8 up, 8 in 2024-09-21T10:17:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:25 smithi119 ceph-mon[101432]: pgmap v1208: 105 pgs: 105 active+clean; 1020 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 103 KiB/s rd, 46 MiB/s wr, 158 op/s 2024-09-21T10:17:26.490 INFO:tasks.workunit.client.0.smithi119.stdout: i=17 exists [987023~22079355,23077889~15345765,39443894~27832526,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~38200387,371282199~27881433,402396115~7541723,410461141~1101111] wrote [44880910~4677553,56306953~6791971,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,194992372~7533021,349881700~3109379,393772170~4230048] 2024-09-21T10:17:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:26 smithi145 ceph-mon[89359]: osdmap e984: 8 total, 8 up, 8 in 2024-09-21T10:17:26.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:26 smithi119 ceph-mon[105288]: osdmap e984: 8 total, 8 up, 8 in 2024-09-21T10:17:26.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:26 smithi119 ceph-mon[101432]: osdmap e984: 8 total, 8 up, 8 in 2024-09-21T10:17:27.469 INFO:tasks.workunit.client.0.smithi119.stdout: i=18 exists [987023~22079355,23077889~15345765,39443894~27832526,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,295227730~22075449,318091802~2618148,320886570~983977,323584036~8233466,332588308~38200387,371282199~27881433,402396115~7541723,410461141~1101111] wrote [13101064~1721180,56247609~9616106,69192860~770245,126747906~5612252,161966455~2167834,220548498~966115,283728297~7471077,299831556~7055694,389422554~2983188] 2024-09-21T10:17:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:27 smithi145 ceph-mon[89359]: osdmap e985: 8 total, 8 up, 8 in 2024-09-21T10:17:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:27 smithi145 ceph-mon[89359]: pgmap v1211: 105 pgs: 105 active+clean; 1.0 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 225 KiB/s rd, 51 MiB/s wr, 358 op/s 2024-09-21T10:17:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:27 smithi119 ceph-mon[101432]: osdmap e985: 8 total, 8 up, 8 in 2024-09-21T10:17:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:27 smithi119 ceph-mon[101432]: pgmap v1211: 105 pgs: 105 active+clean; 1.0 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 225 KiB/s rd, 51 MiB/s wr, 358 op/s 2024-09-21T10:17:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:27 smithi119 ceph-mon[105288]: osdmap e985: 8 total, 8 up, 8 in 2024-09-21T10:17:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:27 smithi119 ceph-mon[105288]: pgmap v1211: 105 pgs: 105 active+clean; 1.0 GiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 225 KiB/s rd, 51 MiB/s wr, 358 op/s 2024-09-21T10:17:28.485 INFO:tasks.workunit.client.0.smithi119.stdout: i=19 exists [987023~22079355,23077889~15345765,39443894~27832526,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,295227730~25482220,320886570~983977,323584036~8233466,332588308~38200387,371282199~30060093,402396115~7541723,410461141~1101111] wrote [1334775~10302970,31331118~6084841,77530326~3165641,203271544~1782031,254681153~667359,313949869~4832006,350071290~3241518,387129528~4653735,397313563~4028729] 2024-09-21T10:17:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:28 smithi145 ceph-mon[89359]: osdmap e986: 8 total, 8 up, 8 in 2024-09-21T10:17:28.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:28 smithi119 ceph-mon[101432]: osdmap e986: 8 total, 8 up, 8 in 2024-09-21T10:17:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:28 smithi119 ceph-mon[105288]: osdmap e986: 8 total, 8 up, 8 in 2024-09-21T10:17:29.422 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 1 diff [73357425~130357,123250767~308257,126258979~8030873,187819226~1568752,291722063~1204351,323584036~3186360,337140860~4197101,371282199~4549840,381676868~1924010] 2024-09-21T10:17:29.422 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [73357425~130357,123250767~308257,126258979~8030873,187819226~1568752,291722063~1204351,323584036~3186360,337140860~4197101,371282199~4549840,381676868~1924010] 2024-09-21T10:17:29.443 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [71303168~2184614,121634816~1924208,125829120~8460732,184549376~4838602,289406976~3519438,322961408~3808988,337140860~4197101,369098752~6733287,377487360~6113518] 2024-09-21T10:17:29.443 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 2 diff [4920468~6949476,24109271~8164295,46114405~4416020,73357425~130357,90690282~540466,123250767~308257,124256485~10033367,187819226~1568752,194851207~6720279,218818238~1747253,240781071~4755025,291722063~1204351,314194184~3108995,323584036~3186360,337140860~4197101,371282199~4549840,381676868~1924010] 2024-09-21T10:17:29.443 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,24109271~8164295,46114405~4416020,73357425~130357,90690282~540466,123250767~308257,124256485~10033367,187819226~1568752,194851207~6720279,218818238~1747253,240781071~4755025,291722063~1204351,314194184~3108995,323584036~3186360,337140860~4197101,371282199~4549840,381676868~1924010] 2024-09-21T10:17:29.449 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,20971520~11302046,41943040~8587385,71303168~2184614,88080384~3150364,121634816~12655036,184549376~4838602,192937984~8633502,218103808~2461683,239075328~6460768,289406976~3519438,310378496~6924683,322961408~3808988,337140860~4197101,369098752~6733287,377487360~6113518] 2024-09-21T10:17:29.449 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 3 diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,73357425~130357,90681367~1732321,94348578~2489555,123250767~308257,124256485~10033367,161611977~6757093,187819226~1568752,194851207~6720279,218818238~1747253,240781071~4755025,291722063~1204351,296708045~5788420,314194184~3108995,323584036~3186360,336005050~7495021,346583587~7150678,371282199~4549840,380821839~3925984] 2024-09-21T10:17:29.449 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,73357425~130357,90681367~1732321,94348578~2489555,123250767~308257,124256485~10033367,161611977~6757093,187819226~1568752,194851207~6720279,218818238~1747253,240781071~4755025,291722063~1204351,296708045~5788420,314194184~3108995,323584036~3186360,336005050~7495021,346583587~7150678,371282199~4549840,380821839~3925984] 2024-09-21T10:17:29.455 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,20971520~17452134,41943040~8587385,51330186~12839288,71303168~2184614,88080384~8757749,121634816~12655036,159383552~8985518,184549376~4838602,192937984~8633502,218103808~2461683,239075328~6460768,289406976~3519438,296708045~5788420,310378496~6924683,322961408~3808988,336005050~7495021,346583587~7150678,369098752~6733287,377487360~7260463] 2024-09-21T10:17:29.456 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 4 diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,73357425~130357,90681367~1732321,94348578~2489555,106726996~7825270,123250767~308257,123829650~10460202,146381793~5644987,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~5788420,314194184~3108995,323584036~8233466,336005050~7495021,346583587~7150678,363488934~7299761,371282199~4549840,380821839~3925984] 2024-09-21T10:17:29.456 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,73357425~130357,90681367~1732321,94348578~2489555,106726996~7825270,123250767~308257,123829650~10460202,146381793~5644987,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~5788420,314194184~3108995,323584036~8233466,336005050~7495021,346583587~7150678,363488934~7299761,371282199~4549840,380821839~3925984] 2024-09-21T10:17:29.461 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,20971520~17452134,41943040~8587385,51330186~12839288,71303168~2184614,88080384~8757749,104857600~9694666,121634816~12655036,142606336~9420444,159383552~8985518,184549376~4838602,192937984~8633502,218103808~2461683,230686720~16971949,264241152~7691108,289406976~3519438,296708045~5788420,310378496~6924683,322961408~8856094,336005050~7495021,346583587~7150678,360710144~15121895,377487360~7260463] 2024-09-21T10:17:29.462 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 5 diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,73357425~130357,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123250767~308257,123829650~10460202,146381793~5644987,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~5788420,303111601~3118728,314194184~3108995,320886570~983977,323584036~8233466,336005050~7495021,346583587~11653774,363488934~7299761,371282199~4549840,380821839~3925984,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.462 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,73357425~130357,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123250767~308257,123829650~10460202,146381793~5644987,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~5788420,303111601~3118728,314194184~3108995,320886570~983977,323584036~8233466,336005050~7495021,346583587~11653774,363488934~7299761,371282199~4549840,380821839~3925984,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.468 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,20971520~29558905,51330186~12839288,71303168~2184614,83886080~12952053,104857600~9694666,121634816~12655036,142606336~9420444,159383552~8985518,184549376~4838602,192937984~8633502,218103808~2461683,222298112~1949500,230686720~16971949,264241152~7691108,289406976~3519438,296708045~5788420,303111601~3118728,310378496~6924683,318767104~3103443,322961408~8856094,336005050~7495021,346583587~11653774,360710144~15121895,377487360~7260463,398458880~13103372] 2024-09-21T10:17:29.468 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 6 diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10460202,144031262~7995518,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~5788420,303111601~3118728,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,363488934~7299761,371282199~13465624,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.468 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10460202,144031262~7995518,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~5788420,303111601~3118728,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,363488934~7299761,371282199~13465624,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.474 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,20971520~29558905,51330186~12839288,71303168~2184614,75497472~8376220,83886080~30666186,121634816~12655036,142606336~9420444,159383552~8985518,184549376~4838602,192937984~8633502,218103808~2461683,222298112~1949500,230686720~16971949,264241152~7691108,289406976~3519438,296708045~5788420,303111601~3118728,307946002~9357177,318767104~3103443,322961408~20538663,346583587~12089713,360710144~30679646,398458880~13103372] 2024-09-21T10:17:29.474 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 7 diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,371282199~13465624,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.474 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,187819226~1568752,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,371282199~13465624,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.480 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,16777216~33753209,51330186~12839288,67276420~3545352,71303168~2184614,75497472~8376220,83886080~30666186,121634816~12774050,138412032~2483773,142606336~14557031,159383552~8985518,184549376~4838602,192937984~8633502,218103808~2461683,222298112~1949500,230686720~16971949,264241152~7691108,289406976~3519438,296708045~9522284,307946002~9357177,318767104~3103443,322961408~20538663,346583587~44806203,398458880~13103372] 2024-09-21T10:17:29.480 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 8 diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,187819226~1568752,189676270~11895216,218818238~1747253,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.480 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,187819226~1568752,189676270~11895216,218818238~1747253,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.486 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,16777216~33753209,51330186~12839288,67276420~3545352,71303168~2184614,75497472~8376220,83886080~30666186,121634816~12774050,138412032~2483773,142606336~14557031,159383552~8985518,184549376~17022110,218103808~2461683,222298112~1949500,230686720~18971178,264241152~7691108,289406976~3519438,296708045~9522284,307946002~9357177,318767104~3103443,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.486 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 9 diff [4920468~6949476,13033324~4328551,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.486 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13033324~4328551,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.492 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~37947513,51330186~12839288,67276420~3545352,71303168~2184614,75497472~8376220,83886080~30666186,121634816~12774050,138412032~2483773,142606336~14557031,159383552~8985518,184549376~17022110,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~7691108,289406976~3519438,296708045~9522284,307946002~9357177,318767104~3103443,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.492 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 10 diff [4920468~12441407,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,169726147~3786678,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.492 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,73357425~130357,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123250767~308257,123829650~10579216,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,169726147~3786678,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,291722063~1204351,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.498 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51330186~12839288,67276420~3545352,71303168~2184614,75497472~8376220,83886080~30666186,121634816~12774050,138412032~2483773,142606336~14557031,159383552~8985518,169726147~3786678,184549376~17022110,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~7691108,289406976~3519438,296708045~9522284,307946002~13924545,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.498 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 11 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123250767~308257,123829650~10579216,140433786~462019,144031262~13132105,161611977~6757093,169726147~3786678,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,291722063~1204351,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.498 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123250767~308257,123829650~10579216,140433786~462019,144031262~13132105,161611977~6757093,169726147~3786678,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,291722063~1204351,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,323584036~8233466,332868862~10631209,346583587~12089713,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.504 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51330186~12839288,67276420~6830306,75497472~8376220,83886080~30666186,121634816~12774050,138412032~2483773,142606336~14557031,159383552~8985518,169726147~3786678,184549376~17022110,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~16317615,289406976~3519438,296708045~9522284,307946002~13924545,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.504 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 12 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123250767~308257,123829650~10579216,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~12089713,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.504 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123250767~308257,123829650~10579216,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,187819226~1568752,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~12089713,359615966~11172729,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.509 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51254163~12915311,67276420~6830306,75497472~8376220,83886080~30666186,121634816~12774050,138412032~2483773,142606336~14557031,159383552~14129273,184549376~17022110,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~16317615,289406976~3519438,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.509 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 13 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,187819226~1568752,189676270~15460646,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.510 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,187819226~1568752,189676270~15460646,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.515 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,121634816~16339986,138412032~2483773,142606336~14557031,159383552~14129273,184549376~20587540,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~16317615,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.515 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 14 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,185246972~826100,186816738~18320178,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.515 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,185246972~826100,186816738~18320178,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.521 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,121634816~16339986,138412032~2483773,142606336~30906489,184549376~20587540,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.521 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 15 diff [2543338~14818537,17903366~5163012,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.521 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~14818537,17903366~5163012,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.527 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,117440512~20534290,138412032~2483773,142606336~30906489,175596367~2528958,184549376~20587540,209715200~6944154,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.527 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 16 diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.527 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.533 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~12915311,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.533 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 17 diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.533 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.538 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~12915311,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.538 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 18 diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.538 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~20595134,318091802~2618148,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~27881433,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.544 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.544 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 19 diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.544 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.550 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:29.550 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 2 diff [4920468~6949476,24109271~8164295,46114405~4416020,90690282~540466,124256485~8003279,194851207~6720279,218818238~1747253,240781071~4755025,314194184~3108995,381827012~1351784] 2024-09-21T10:17:29.550 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,24109271~8164295,46114405~4416020,90690282~540466,124256485~8003279,194851207~6720279,218818238~1747253,240781071~4755025,314194184~3108995,381827012~1351784] 2024-09-21T10:17:29.555 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,20971520~11302046,41943040~8587385,88080384~3150364,123559024~8700740,192937984~8633502,218103808~2461683,239075328~6460768,310378496~6924683,381827012~1351784] 2024-09-21T10:17:29.555 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 3 diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,90681367~1732321,94348578~2489555,124256485~8003279,161611977~6757093,194851207~6720279,218818238~1747253,240781071~4755025,296708045~5788420,314194184~3108995,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:29.555 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,90681367~1732321,94348578~2489555,124256485~8003279,161611977~6757093,194851207~6720279,218818238~1747253,240781071~4755025,296708045~5788420,314194184~3108995,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:29.560 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,20971520~17452134,41943040~8587385,51330186~12839288,88080384~8757749,123559024~8700740,159383552~8985518,192937984~8633502,218103808~2461683,239075328~6460768,296708045~5788420,310378496~6924683,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:29.560 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 4 diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,90681367~1732321,94348578~2489555,106726996~7825270,123829650~8430114,146381793~5644987,161611977~6757093,193444193~8127293,218818238~1747253,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,314194184~3108995,324210499~7607003,336005050~7495021,346583587~7150678,363488934~7299761,380821839~3925984] 2024-09-21T10:17:29.560 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,24109271~8164295,32577525~5846129,46114405~4416020,52808135~11361339,90681367~1732321,94348578~2489555,106726996~7825270,123829650~8430114,146381793~5644987,161611977~6757093,193444193~8127293,218818238~1747253,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,314194184~3108995,324210499~7607003,336005050~7495021,346583587~7150678,363488934~7299761,380821839~3925984] 2024-09-21T10:17:29.565 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,20971520~17452134,41943040~8587385,51330186~12839288,88080384~8757749,104857600~9694666,123559024~8700740,142606336~9420444,159383552~8985518,192937984~8633502,218103808~2461683,230686720~16971949,264241152~7691108,296708045~5788420,310378496~6924683,324210499~7607003,336005050~7495021,346583587~7150678,360710144~10078551,380821839~3925984] 2024-09-21T10:17:29.566 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 5 diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123829650~8430114,146381793~5644987,161611977~6757093,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,314194184~3108995,320886570~983977,324210499~7607003,336005050~7495021,346583587~11653774,363488934~7299761,380821839~3925984,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.566 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123829650~8430114,146381793~5644987,161611977~6757093,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,314194184~3108995,320886570~983977,324210499~7607003,336005050~7495021,346583587~11653774,363488934~7299761,380821839~3925984,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.571 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,20971520~29558905,51330186~12839288,83886080~12952053,104857600~9694666,123559024~8700740,142606336~9420444,159383552~8985518,192937984~8633502,218103808~2461683,222298112~1949500,230686720~16971949,264241152~7691108,296708045~5788420,303111601~3118728,310378496~6924683,318767104~3103443,324210499~7607003,336005050~7495021,346583587~11653774,360710144~10078551,380821839~3925984,398458880~13103372] 2024-09-21T10:17:29.571 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 6 diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,144031262~7995518,161611977~6757093,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,363488934~7299761,374202738~10545085,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.571 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13177842~2866366,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,144031262~7995518,161611977~6757093,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,363488934~7299761,374202738~10545085,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.576 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,20971520~29558905,51330186~12839288,75497472~8376220,83886080~30666186,123559024~8700740,142606336~9420444,159383552~8985518,192937984~8633502,218103808~2461683,222298112~1949500,230686720~16971949,264241152~7691108,296708045~5788420,303111601~3118728,307946002~9357177,318767104~3103443,324210499~19289572,346583587~12089713,360710144~10078551,374202738~17187052,398458880~13103372] 2024-09-21T10:17:29.576 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 7 diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,133610121~798745,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~9522284,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~10545085,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.577 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,133610121~798745,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,193444193~8127293,218818238~1747253,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~9522284,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~10545085,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.583 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,16777216~33753209,51330186~12839288,67276420~3545352,75497472~8376220,83886080~30666186,123559024~8700740,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,192937984~8633502,218103808~2461683,222298112~1949500,230686720~16971949,264241152~7691108,296708045~9522284,307946002~9357177,318767104~3103443,324210499~19289572,346583587~24205108,374202738~17187052,398458880~13103372] 2024-09-21T10:17:29.583 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 8 diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~11895216,218818238~1747253,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.583 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13033324~3010884,17903366~4986190,24109271~8164295,32577525~5846129,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~11895216,218818238~1747253,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.587 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~3461296,16777216~33753209,51330186~12839288,67276420~3545352,75497472~8376220,83886080~30666186,123559024~8700740,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,185246972~826100,189387978~12183508,218103808~2461683,222298112~1949500,230686720~18971178,264241152~7691108,296708045~9522284,307946002~9357177,318767104~3103443,324210499~19289572,346583587~24205108,374202738~37359514] 2024-09-21T10:17:29.587 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 9 diff [4920468~6949476,13033324~4328551,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.587 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~6949476,13033324~4328551,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8430114,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~9357177,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.592 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~6949476,12582912~37947513,51330186~12839288,67276420~3545352,75497472~8376220,83886080~30666186,123559024~8700740,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,185246972~826100,189387978~12183508,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~7691108,296708045~9522284,307946002~9357177,318767104~3103443,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.592 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 10 diff [4920468~12441407,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8577781,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,169726147~3786678,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.592 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,24109271~14314383,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~8577781,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,169726147~3786678,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.597 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51330186~12839288,67276420~3545352,75497472~8376220,83886080~30666186,123559024~8848407,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,169726147~3786678,185246972~826100,189387978~12183508,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~7691108,296708045~9522284,307946002~13924545,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.598 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 11 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~8577781,133610121~798745,140433786~462019,144031262~13132105,161611977~6757093,169726147~3786678,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.598 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,52808135~11361339,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~8577781,133610121~798745,140433786~462019,144031262~13132105,161611977~6757093,169726147~3786678,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~9522284,307946002~9357177,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.603 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51330186~12839288,67276420~6830306,75497472~8376220,83886080~30666186,123559024~8848407,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,169726147~3786678,185246972~826100,189387978~12183508,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~16317615,296708045~9522284,307946002~13924545,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.603 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 12 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~8577781,133610121~798745,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.603 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~8577781,133610121~798745,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~11895216,212176573~2340741,218818238~1747253,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.608 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51254163~12915311,67276420~6830306,75497472~8376220,83886080~30666186,123559024~8848407,133610121~798745,138412032~2483773,142606336~14557031,159383552~14129273,185246972~826100,189387978~12183508,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~16317615,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.608 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 13 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~15460646,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.608 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~15460646,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.614 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,123559024~8848407,133610121~4364681,138412032~2483773,142606336~14557031,159383552~14129273,185246972~826100,189387978~15748938,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,264241152~16317615,281018368~9453538,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.614 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 14 diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,185246972~826100,186816738~18320178,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.614 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4920468~12441407,17903366~5163012,23077889~15345765,39443894~4770339,46114405~4416020,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,185246972~826100,186816738~18320178,212176573~2340741,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.619 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4920468~45609957,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,123559024~8848407,133610121~4364681,138412032~2483773,142606336~30906489,185246972~826100,186816738~18320178,209715200~4802114,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~9453538,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.619 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 15 diff [2543338~14818537,17903366~5163012,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.619 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~14818537,17903366~5163012,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.625 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~30906489,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~9453538,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.625 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 16 diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.625 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.631 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~12915311,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~9453538,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.631 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 17 diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.631 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~12915311,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~1747253,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.636 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~12915311,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~2461683,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~9453538,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.636 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 18 diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.636 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~20523040,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~20595134,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.641 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~47987087,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.642 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 19 diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.642 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.648 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.648 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 3 diff [32577525~5846129,52808135~11361339,90681367~1732321,94348578~2489555,161611977~6757093,296708045~5788420,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:29.648 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [32577525~5846129,52808135~11361339,90681367~1732321,94348578~2489555,161611977~6757093,296708045~5788420,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:29.653 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [32273566~6150088,51330186~12839288,90681367~6156766,159383552~8985518,296708045~5788420,336005050~7495021,346583587~7150678,380821839~3925984] 2024-09-21T10:17:29.653 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 4 diff [32577525~5846129,52808135~11361339,90681367~1732321,94348578~2489555,106726996~7825270,123829650~6594126,146381793~5644987,161611977~6757093,193444193~5178520,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,324210499~7607003,336005050~7495021,346583587~7150678,363488934~7299761,380821839~3925984] 2024-09-21T10:17:29.653 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [32577525~5846129,52808135~11361339,90681367~1732321,94348578~2489555,106726996~7825270,123829650~6594126,146381793~5644987,161611977~6757093,193444193~5178520,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,324210499~7607003,336005050~7495021,346583587~7150678,363488934~7299761,380821839~3925984] 2024-09-21T10:17:29.658 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [32273566~6150088,51330186~12839288,90681367~6156766,104857600~9694666,123829650~6594126,142606336~9420444,159383552~8985518,193444193~5178520,230686720~16971949,264241152~7691108,296708045~5788420,324210499~7607003,336005050~7495021,346583587~7150678,360710144~10078551,380821839~3925984] 2024-09-21T10:17:29.658 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 5 diff [13177842~2866366,32577525~5846129,39443894~4770339,52808135~11361339,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123829650~6594126,146381793~5644987,161611977~6757093,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,320886570~983977,324210499~7607003,336005050~7495021,346583587~11653774,363488934~7299761,380821839~3925984,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.658 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13177842~2866366,32577525~5846129,39443894~4770339,52808135~11361339,88009097~1929038,90681367~1732321,94348578~2489555,106726996~7825270,123829650~6594126,146381793~5644987,161611977~6757093,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,320886570~983977,324210499~7607003,336005050~7495021,346583587~11653774,363488934~7299761,380821839~3925984,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.663 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,32273566~11940667,51330186~12839288,83886080~6052055,90681367~6156766,104857600~9694666,123829650~6594126,142606336~9420444,159383552~8985518,193444193~5178520,222298112~1949500,230686720~16971949,264241152~7691108,296708045~5788420,303111601~3118728,318767104~3103443,324210499~7607003,336005050~7495021,346583587~11653774,360710144~10078551,380821839~3925984,398458880~13103372] 2024-09-21T10:17:29.663 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 6 diff [13177842~2866366,32577525~5846129,39443894~4770339,52808135~11361339,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,144031262~7995518,161611977~6757093,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,363488934~7299761,374202738~10545085,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.663 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13177842~2866366,32577525~5846129,39443894~4770339,52808135~11361339,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,144031262~7995518,161611977~6757093,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~5788420,303111601~3118728,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,363488934~7299761,374202738~10545085,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.668 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,32273566~11940667,51330186~12839288,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,142606336~9420444,159383552~8985518,193444193~5178520,222298112~1949500,230686720~16971949,264241152~7691108,296708045~5788420,303111601~3118728,307946002~6659332,318767104~3103443,324210499~19289572,346583587~12089713,360710144~10078551,374202738~17187052,398458880~13103372] 2024-09-21T10:17:29.668 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 7 diff [13033324~3010884,17903366~4986190,25823944~2571315,32577525~5846129,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~9522284,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~10545085,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.668 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~3010884,17903366~4986190,25823944~2571315,32577525~5846129,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152640544~4522823,161611977~6757093,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,296708045~9522284,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~10545085,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.674 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,16777216~6112340,25823944~2571315,32273566~11940667,51330186~12839288,67276420~3545352,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,193444193~5178520,222298112~1949500,230686720~16971949,264241152~7691108,296708045~9522284,307946002~6659332,318767104~3103443,324210499~19289572,346583587~24205108,374202738~17187052,398458880~13103372] 2024-09-21T10:17:29.674 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 8 diff [13033324~3010884,17903366~4986190,25823944~2571315,32577525~5846129,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~8946443,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.674 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~3010884,17903366~4986190,25823944~2571315,32577525~5846129,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~8946443,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.679 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,16777216~6112340,25823944~2571315,32273566~11940667,51330186~12839288,67276420~3545352,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,185246972~826100,189387978~9234735,222298112~1949500,230686720~18971178,264241152~7691108,296708045~9522284,307946002~6659332,318767104~3103443,324210499~19289572,346583587~24205108,374202738~37359514] 2024-09-21T10:17:29.679 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 9 diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~12599710,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~8946443,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.679 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~12599710,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,185246972~826100,189676270~8946443,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~6659332,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359901878~1171315,363488934~7299761,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.684 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,12582912~10483466,25823944~18390289,51330186~12839288,67276420~3545352,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,185246972~826100,189387978~9234735,209715200~4802114,222298112~8023393,230686720~18971178,264241152~7691108,296708045~9522284,307946002~6659332,318767104~3103443,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.685 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 10 diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~12599710,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,169726147~3786678,185246972~826100,189676270~10284193,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~6659332,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.685 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~12599710,39443894~4770339,52808135~11361339,69648386~1173386,76370336~7503356,88009097~1929038,90681367~1732321,94348578~2489555,99139515~15412751,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,161611977~6757093,169726147~3786678,185246972~826100,189676270~10284193,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,296708045~9522284,307946002~6659332,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.690 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,25823944~18390289,51330186~12839288,67276420~3545352,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,169726147~3786678,185246972~826100,189387978~10572485,209715200~4802114,222298112~8023393,230686720~18971178,264241152~7691108,296708045~9522284,307946002~6659332,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.690 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 11 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,52808135~11361339,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,161611977~6757093,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~9522284,307946002~8932008,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.690 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,52808135~11361339,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,161611977~6757093,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~9522284,307946002~8932008,318091802~2618148,320886570~983977,324210499~7607003,332868862~10631209,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.695 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~18390289,51330186~12839288,67276420~6830306,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,138412032~2483773,142606336~14557031,159383552~8985518,169726147~3786678,185246972~826100,189387978~11188254,209715200~4802114,222298112~8023393,230686720~18971178,264241152~16317615,296708045~9522284,307946002~8932008,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.695 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 12 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.695 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~12089713,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.700 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~18390289,51254163~12915311,67276420~6830306,75497472~8376220,83886080~6052055,90681367~23870899,123829650~6594126,130891915~1515516,133610121~798745,138412032~2483773,142606336~14557031,159383552~14129273,185246972~826100,189387978~11188254,209715200~4802114,222298112~8023393,230686720~18971178,264241152~16317615,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.701 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 13 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~15460646,212176573~2340741,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.701 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,51254163~12915311,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,161611977~11900848,185246972~826100,189676270~15460646,212176573~2340741,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.706 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~18390289,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~14557031,159383552~14129273,185246972~826100,189387978~15748938,209715200~4802114,222298112~8023393,230686720~18971178,264241152~16317615,281018368~9453538,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.706 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 14 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,185246972~826100,186816738~18320178,212176573~2340741,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.706 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~4770339,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,185246972~826100,186816738~18320178,212176573~2340741,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.711 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~18390289,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~30906489,185246972~826100,186816738~18320178,209715200~4802114,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~9453538,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.711 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~9289302,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.711 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12599710,39443894~9289302,51254163~12915311,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~11900848,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.716 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~22909252,51254163~12915311,67276420~6830306,75497472~8376220,83886080~32118389,117440512~4661775,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~30906489,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~9453538,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.717 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12599710,39443894~9289302,51254163~12915311,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.717 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12599710,39443894~9289302,51254163~12915311,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.722 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~22909252,51254163~12915311,67276420~6830306,75497472~40506997,117440512~4661775,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~9453538,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.722 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12599710,39443894~10114569,51254163~12915311,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.723 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12599710,39443894~10114569,51254163~12915311,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.728 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~23734519,51254163~12915311,67276420~6830306,75497472~40506997,117440512~4661775,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~9453538,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.728 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.728 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~20169965,318091802~2618148,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.733 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~20169965,317303179~4567368,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.733 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 19 diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.734 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.739 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:29.739 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 4 diff [106726996~7825270,123829650~6594126,146381793~5644987,193444193~5178520,232289009~4460589,238432676~9225993,268409630~3522630,324210499~7607003,363488934~7299761] 2024-09-21T10:17:29.739 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [106726996~7825270,123829650~6594126,146381793~5644987,193444193~5178520,232289009~4460589,238432676~9225993,268409630~3522630,324210499~7607003,363488934~7299761] 2024-09-21T10:17:29.744 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [104857600~9694666,123829650~6594126,142606336~9420444,193444193~5178520,230686720~16971949,264241152~7691108,324210499~7607003,360710144~10078551] 2024-09-21T10:17:29.744 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 5 diff [13177842~2866366,39443894~4770339,55845980~5219038,88009097~1929038,106726996~7825270,123829650~6594126,146381793~5644987,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,303111601~3118728,320886570~983977,324210499~7607003,351518511~6718850,363488934~7299761,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.744 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13177842~2866366,39443894~4770339,55845980~5219038,88009097~1929038,106726996~7825270,123829650~6594126,146381793~5644987,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,303111601~3118728,320886570~983977,324210499~7607003,351518511~6718850,363488934~7299761,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.749 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,38423654~5790579,55845980~5219038,83886080~6052055,104857600~9694666,123829650~6594126,142606336~9420444,193444193~5178520,222298112~1949500,230686720~16971949,264241152~7691108,303111601~3118728,318767104~3103443,324210499~7607003,351518511~6718850,360710144~10078551,398458880~13103372] 2024-09-21T10:17:29.749 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 6 diff [13177842~2866366,39443894~4770339,55845980~5219038,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,144031262~7995518,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,303111601~3118728,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,363488934~7299761,374202738~8382468,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.749 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13177842~2866366,39443894~4770339,55845980~5219038,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,144031262~7995518,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,303111601~3118728,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,363488934~7299761,374202738~8382468,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.755 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,38423654~5790579,55845980~5219038,75497472~8376220,83886080~6052055,96838133~17714133,123829650~6594126,142606336~9420444,193444193~5178520,222298112~1949500,230686720~16971949,264241152~7691108,303111601~3118728,307946002~6659332,318767104~3103443,324210499~12228701,350953324~7719976,360710144~10078551,374202738~8382468,384747823~6641967,398458880~13103372] 2024-09-21T10:17:29.755 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 7 diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,55845980~5219038,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152640544~4522823,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,301562997~4667332,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,359901878~1171315,363488934~7299761,374202738~8382468,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.755 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,55845980~5219038,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152640544~4522823,193444193~5178520,223540318~707294,232289009~4460589,238432676~9225993,268409630~3522630,301562997~4667332,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,359901878~1171315,363488934~7299761,374202738~8382468,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.760 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,16777216~6112340,25823944~2571315,38423654~5790579,55845980~5219038,67276420~3545352,75497472~8376220,83886080~6052055,96838133~17714133,123829650~6594126,133610121~798745,138412032~2483773,142606336~14557031,193444193~5178520,222298112~1949500,230686720~16971949,264241152~7691108,301562997~4667332,307946002~6659332,318767104~3103443,324210499~12228701,350953324~19835371,374202738~8382468,384747823~6641967,398458880~13103372] 2024-09-21T10:17:29.760 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 8 diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,185246972~826100,189676270~8946443,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,301562997~4667332,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,359901878~1171315,363488934~7299761,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.760 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,185246972~826100,189676270~8946443,223540318~707294,230873414~6179084,237202911~12454987,268409630~3522630,301562997~4667332,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,359901878~1171315,363488934~7299761,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.765 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,16777216~6112340,25823944~2571315,38423654~5790579,53157964~7907054,67276420~3545352,75497472~8376220,83886080~6052055,96838133~17714133,123829650~6594126,133610121~798745,138412032~2483773,142606336~14557031,185246972~826100,189387978~9234735,222298112~1949500,230686720~18971178,264241152~7691108,301562997~4667332,307946002~6659332,318767104~3103443,324210499~12228701,350953324~19835371,374202738~37359514] 2024-09-21T10:17:29.765 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 9 diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,185246972~826100,189676270~8946443,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,301562997~4667332,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,359901878~1171315,363488934~7299761,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.765 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,185246972~826100,189676270~8946443,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,301562997~4667332,307946002~6659332,320886570~983977,324210499~7607003,332868862~3570338,350953324~7719976,359901878~1171315,363488934~7299761,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.771 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,12582912~10483466,25823944~9584048,38423654~5790579,53157964~7907054,67276420~3545352,75497472~8376220,83886080~6052055,96838133~17714133,123829650~6594126,133610121~798745,138412032~2483773,142606336~14557031,185246972~826100,189387978~9234735,209715200~4802114,222298112~8023393,230686720~18971178,264241152~7691108,301562997~4667332,307946002~6659332,318767104~3103443,324210499~12228701,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.771 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 10 diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,169726147~3786678,185246972~826100,189676270~10284193,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,301562997~4667332,307946002~6659332,318091802~2618148,320886570~983977,324210499~7607003,332868862~3570338,337769986~374855,350953324~7719976,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.771 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~15412751,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~7995518,152132680~5030687,169726147~3786678,185246972~826100,189676270~10284193,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,301562997~4667332,307946002~6659332,318091802~2618148,320886570~983977,324210499~7607003,332868862~3570338,337769986~374855,350953324~7719976,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.776 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,25823944~9584048,38423654~5790579,53157964~7907054,67276420~3545352,75497472~8376220,83886080~6052055,96838133~17714133,123829650~6594126,130891915~1515516,133610121~798745,138412032~2483773,142606336~14557031,169726147~3786678,185246972~826100,189387978~10572485,209715200~4802114,222298112~8023393,230686720~18971178,264241152~7691108,301562997~4667332,307946002~6659332,317303179~4567368,324210499~12228701,337769986~374855,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.776 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 11 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~22916173,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,301562997~4667332,307946002~8932008,318091802~2618148,320886570~983977,324210499~7607003,332868862~3570338,337769986~374855,350953324~7719976,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.776 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~22916173,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,301562997~4667332,307946002~8932008,318091802~2618148,320886570~983977,324210499~7607003,332868862~3570338,337769986~374855,350953324~7719976,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.782 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,53157964~7907054,67276420~6830306,75497472~8376220,83886080~6052055,91636093~22916173,123829650~6594126,130891915~1515516,133610121~798745,138412032~2483773,142606336~14557031,169726147~3786678,185246972~826100,189387978~11188254,209715200~4802114,222298112~8023393,230686720~18971178,264241152~16317615,301562997~4667332,307946002~8932008,317303179~4567368,324210499~12228701,337769986~374855,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.782 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 12 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~22916173,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~7719976,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.782 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~22916173,123829650~6594126,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~7719976,359615966~11172729,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.787 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,51254163~9810855,67276420~6830306,75497472~8376220,83886080~6052055,91636093~22916173,123829650~6594126,130891915~1515516,133610121~798745,138412032~2483773,142606336~14557031,167981878~5530947,185246972~826100,189387978~11188254,209715200~4802114,222298112~8023393,230686720~18971178,264241152~16317615,301562997~15315013,317303179~4567368,324210499~12228701,337769986~374855,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.787 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 13 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.787 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1139678,224922963~5398542,230873414~6179084,237202911~12454987,268409630~3522630,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.792 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,51254163~9810855,67276420~6830306,75497472~8376220,83886080~32118389,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~14557031,167981878~5530947,185246972~826100,189387978~15748938,209715200~4802114,222298112~8023393,230686720~18971178,264241152~16317615,281018368~9453538,301562997~15315013,317303179~4567368,324210499~12228701,337769986~374855,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.792 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 14 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~4458340,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.793 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~4458340,76370336~7503356,87497139~28507330,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:29 smithi145 ceph-mon[89359]: osdmap e987: 8 total, 8 up, 8 in 2024-09-21T10:17:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:29 smithi145 ceph-mon[89359]: pgmap v1214: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 257 KiB/s rd, 32 MiB/s wr, 394 op/s 2024-09-21T10:17:29.798 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,51254163~9810855,67276420~6830306,75497472~8376220,83886080~32118389,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~17864752,167981878~5530947,185246972~826100,186816738~18320178,209715200~4802114,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~9453538,301562997~15315013,317303179~4567368,324210499~12228701,337769986~374855,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.798 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.798 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~7503356,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,337769986~374855,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.803 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~10309542,51254163~9810855,67276420~6830306,75497472~8376220,83886080~32118389,117440512~4661775,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~17864752,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,222298112~8023393,230686720~18971178,251658240~8092188,264241152~16317615,281018368~9453538,301562997~15315013,317303179~4567368,324210499~12228701,337769986~374855,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.803 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.803 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~8404195,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,350953324~19835371,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.808 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~10309542,51254163~9810855,67276420~6830306,75497472~40506997,117440512~4661775,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~17864752,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~9453538,301562997~15315013,317303179~4567368,324210499~12228701,336572672~4979196,350953324~19835371,373691010~37871242] 2024-09-21T10:17:29.808 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~11844761,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.808 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~11844761,69648386~4458340,76370336~10705678,87497139~28507330,118600721~3501566,123829650~6594126,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.814 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~11134809,51254163~11844761,67276420~6830306,75497472~40506997,117440512~4661775,123829650~6594126,130891915~1515516,133610121~4364681,138412032~2483773,142606336~17864752,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~9453538,301562997~15315013,317303179~4567368,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:29.814 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.814 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.822 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~17046454,317303179~4567368,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:29.822 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 19 diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.822 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.824 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:29.824 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 5 diff [13177842~2866366,39443894~4770339,55845980~5219038,88009097~1929038,223540318~707294,303111601~3118728,320886570~983977,351518511~6718850,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.824 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13177842~2866366,39443894~4770339,55845980~5219038,88009097~1929038,223540318~707294,303111601~3118728,320886570~983977,351518511~6718850,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.830 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,38423654~5790579,55845980~5219038,83886080~6052055,222298112~1949500,303111601~3118728,318767104~3103443,351518511~6718850,398458880~13103372] 2024-09-21T10:17:29.830 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 6 diff [13177842~2866366,39443894~4770339,55845980~5219038,76370336~7503356,88009097~1929038,99139515~7728324,144031262~6739415,223540318~707294,303111601~3118728,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,374202738~8382468,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.830 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13177842~2866366,39443894~4770339,55845980~5219038,76370336~7503356,88009097~1929038,99139515~7728324,144031262~6739415,223540318~707294,303111601~3118728,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,374202738~8382468,385039936~6349854,402396115~6360928,410461141~1101111] 2024-09-21T10:17:29.835 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,38423654~5790579,55845980~5219038,75497472~8376220,83886080~6052055,96838133~10029706,144031262~6739415,222298112~1949500,303111601~3118728,307946002~6659332,318767104~3103443,331817502~4621698,350953324~7719976,374202738~8382468,384747823~6641967,398458880~13103372] 2024-09-21T10:17:29.835 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 7 diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,55845980~5219038,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152640544~4522823,223540318~707294,301562997~4667332,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,359901878~1171315,374202738~8382468,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.835 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,55845980~5219038,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152640544~4522823,223540318~707294,301562997~4667332,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,359901878~1171315,374202738~8382468,385039936~6349854,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.840 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,16777216~6112340,25823944~2571315,38423654~5790579,55845980~5219038,67276420~3545352,75497472~8376220,83886080~6052055,96838133~10029706,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,222298112~1949500,301562997~4667332,307946002~6659332,318767104~3103443,331817502~4621698,350953324~10119869,374202738~8382468,384747823~6641967,398458880~13103372] 2024-09-21T10:17:29.840 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 8 diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,223540318~707294,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,359901878~1171315,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.841 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~3010884,17903366~4986190,25823944~2571315,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,223540318~707294,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,359901878~1171315,374202738~24960894,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.846 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~3461296,16777216~6112340,25823944~2571315,38423654~5790579,53157964~7907054,67276420~3545352,75497472~8376220,83886080~6052055,96838133~10029706,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,185246972~826100,189387978~5230782,222298112~1949500,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,318767104~3103443,331817502~4621698,350953324~10119869,374202738~37359514] 2024-09-21T10:17:29.846 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 9 diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,113327605~1146979,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,359901878~1171315,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.846 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,113327605~1146979,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,320886570~983977,332868862~3570338,350953324~7719976,359901878~1171315,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.851 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,12582912~10483466,25823944~9584048,38423654~5790579,53157964~7907054,67276420~3545352,75497472~8376220,83886080~6052055,96838133~10029706,113327605~1146979,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,185246972~826100,189387978~5230782,209715200~4802114,222298112~8023393,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,318767104~3103443,331817502~4621698,350953324~10119869,373691010~37871242] 2024-09-21T10:17:29.851 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 10 diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,318091802~2618148,320886570~983977,332868862~3570338,337769986~374855,350953324~7719976,359615966~5871094,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.851 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,76370336~7503356,88009097~1929038,99139515~7728324,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,318091802~2618148,320886570~983977,332868862~3570338,337769986~374855,350953324~7719976,359615966~5871094,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.856 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,25823944~9584048,38423654~5790579,53157964~7907054,67276420~3545352,75497472~8376220,83886080~6052055,96838133~10029706,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,169726147~3786678,185246972~826100,189387978~5230782,197256444~2704019,209715200~4802114,222298112~8023393,230873414~6179084,237202911~1629605,241574681~8083217,301562997~4667332,307946002~6659332,317303179~4567368,331817502~4621698,337769986~374855,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.856 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 11 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~4667332,307946002~8932008,318091802~2618148,320886570~983977,332868862~3570338,337769986~374855,350953324~7719976,359615966~5871094,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.856 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,53157964~7907054,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~4667332,307946002~8932008,318091802~2618148,320886570~983977,332868862~3570338,337769986~374855,350953324~7719976,359615966~5871094,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.862 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,53157964~7907054,67276420~6830306,75497472~8376220,83886080~6052055,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,144031262~13132105,169726147~3786678,185246972~826100,189387978~11188254,209715200~4802114,222298112~8023393,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,301562997~4667332,307946002~8932008,317303179~4567368,331817502~4621698,337769986~374855,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.862 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 12 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~7719976,359615966~5871094,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.862 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,88009097~1929038,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,223540318~707294,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~7719976,359615966~5871094,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:29 smithi119 ceph-mon[105288]: osdmap e987: 8 total, 8 up, 8 in 2024-09-21T10:17:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:29 smithi119 ceph-mon[105288]: pgmap v1214: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 257 KiB/s rd, 32 MiB/s wr, 394 op/s 2024-09-21T10:17:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:29 smithi119 ceph-mon[101432]: osdmap e987: 8 total, 8 up, 8 in 2024-09-21T10:17:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:29 smithi119 ceph-mon[101432]: pgmap v1214: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 257 KiB/s rd, 32 MiB/s wr, 394 op/s 2024-09-21T10:17:29.868 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,51254163~9810855,67276420~6830306,75497472~8376220,83886080~6052055,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,144031262~13132105,167981878~5530947,185246972~826100,189387978~11188254,209715200~4802114,222298112~8023393,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,301562997~15315013,317303179~4567368,331817502~4621698,337769986~374855,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.868 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 13 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1139678,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.868 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~1173386,71187875~2918851,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1139678,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.873 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,51254163~9810855,67276420~6830306,75497472~8376220,83886080~22981759,107841497~8162972,130891915~1515516,133610121~4364681,138412032~2483773,144031262~13132105,167981878~5530947,185246972~826100,189387978~15748938,209715200~4802114,222298112~8023393,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,281018368~9453538,301562997~15315013,317303179~4567368,331817502~4621698,337769986~374855,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.873 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 14 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.873 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,39443894~4770339,51254163~9810855,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.878 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,38423654~5790579,51254163~9810855,67276420~6830306,75497472~8376220,83886080~22981759,107841497~8162972,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,209715200~4802114,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,301562997~15315013,317303179~4567368,331817502~4621698,337769986~374855,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.878 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.879 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.883 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~10309542,51254163~9810855,67276420~6830306,75497472~8376220,83886080~22981759,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,301562997~15315013,317303179~4567368,331817502~4621698,337769986~374855,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.884 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~8404195,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,336572672~4979196,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.884 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~9289302,51254163~9810855,69648386~4458340,76370336~8404195,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,336572672~4979196,350953324~14533736,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.889 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~10309542,51254163~9810855,67276420~6830306,75497472~31370367,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,301562997~15315013,317303179~4567368,331817502~4621698,336572672~4979196,350953324~14533736,373691010~37871242] 2024-09-21T10:17:29.889 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~11844761,69648386~4458340,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.889 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~11844761,69648386~4458340,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~15315013,318091802~2618148,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.894 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~11134809,51254163~11844761,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,301562997~15315013,317303179~4567368,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:29.894 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.894 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.899 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,317303179~4567368,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:29.900 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 19 diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.900 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:29.905 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:29.905 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 6 diff [76370336~7503356,99139515~7728324,144031262~6739415,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,374202738~8382468,385039936~6349854] 2024-09-21T10:17:29.905 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [76370336~7503356,99139515~7728324,144031262~6739415,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,374202738~8382468,385039936~6349854] 2024-09-21T10:17:29.910 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [75497472~8376220,96838133~10029706,144031262~6739415,307946002~6659332,331817502~4621698,350953324~3781463,355372960~3300340,374202738~8382468,384747823~6641967] 2024-09-21T10:17:29.910 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 7 diff [13033324~2616660,17903366~4986190,25823944~2571315,69648386~1173386,76370336~7503356,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152640544~4522823,301562997~3599492,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,359901878~1171315,374202738~8382468,385039936~6349854,403544798~6393040] 2024-09-21T10:17:29.910 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~2616660,17903366~4986190,25823944~2571315,69648386~1173386,76370336~7503356,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152640544~4522823,301562997~3599492,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,359901878~1171315,374202738~8382468,385039936~6349854,403544798~6393040] 2024-09-21T10:17:29.915 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13033324~2616660,16777216~6112340,25823944~2571315,67276420~3545352,75497472~8376220,96838133~10029706,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,301562997~3599492,307946002~6659332,331817502~4621698,350953324~3781463,355372960~5700233,374202738~8382468,384747823~6641967,403544798~6393040] 2024-09-21T10:17:29.915 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 8 diff [13033324~2616660,17903366~4986190,25823944~2571315,53157964~7891177,69648386~1173386,76370336~7503356,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,359901878~1171315,374202738~24960894,403544798~6393040] 2024-09-21T10:17:29.915 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~2616660,17903366~4986190,25823944~2571315,53157964~7891177,69648386~1173386,76370336~7503356,99139515~7728324,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,359901878~1171315,374202738~24960894,403544798~6393040] 2024-09-21T10:17:29.920 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13033324~2616660,16777216~6112340,25823944~2571315,53157964~7891177,67276420~3545352,75497472~8376220,96838133~10029706,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,185246972~826100,189387978~5230782,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,331817502~4621698,350953324~3781463,355372960~5700233,374202738~24960894,403544798~6393040] 2024-09-21T10:17:29.920 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 9 diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,76370336~7503356,99139515~7728324,113327605~1146979,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,359901878~1171315,373691010~25472622,403544798~6393040] 2024-09-21T10:17:29.921 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,76370336~7503356,99139515~7728324,113327605~1146979,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,185246972~826100,189676270~4942490,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,332868862~3570338,350953324~3781463,355372960~3300340,359901878~1171315,373691010~25472622,403544798~6393040] 2024-09-21T10:17:29.926 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,13033324~10033054,25823944~9584048,53157964~7891177,67276420~3545352,75497472~8376220,96838133~10029706,113327605~1146979,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,185246972~826100,189387978~5230782,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,331817502~4621698,350953324~3781463,355372960~5700233,373691010~25472622,403544798~6393040] 2024-09-21T10:17:29.926 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 10 diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,76370336~7503356,99139515~7728324,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,318091802~2618148,332868862~3570338,337769986~374855,350953324~3781463,355372960~3300340,359615966~5871094,373691010~25472622,403544798~6393040,410712466~88886] 2024-09-21T10:17:29.926 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,76370336~7503356,99139515~7728324,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~6739415,152132680~5030687,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,318091802~2618148,332868862~3570338,337769986~374855,350953324~3781463,355372960~3300340,359615966~5871094,373691010~25472622,403544798~6393040,410712466~88886] 2024-09-21T10:17:29.931 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,25823944~9584048,53157964~7891177,67276420~3545352,75497472~8376220,96838133~10029706,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,144031262~6739415,152026780~5136587,169726147~3786678,185246972~826100,189387978~5230782,197256444~2704019,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,307946002~6659332,317303179~3406771,331817502~4621698,337769986~374855,350953324~3781463,355372960~10114100,373691010~25472622,403544798~6393040,410712466~88886] 2024-09-21T10:17:29.931 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 11 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,53157964~7891177,69648386~1173386,71187875~2918851,76370336~7503356,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,307946002~8932008,318091802~2618148,332868862~3570338,337769986~374855,350953324~3781463,355372960~3300340,359615966~5871094,373691010~25472622,403122934~6814904,410712466~88886] 2024-09-21T10:17:29.931 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,53157964~7891177,69648386~1173386,71187875~2918851,76370336~7503356,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,307946002~8932008,318091802~2618148,332868862~3570338,337769986~374855,350953324~3781463,355372960~3300340,359615966~5871094,373691010~25472622,403122934~6814904,410712466~88886] 2024-09-21T10:17:29.937 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,53157964~7891177,67276420~6830306,75497472~8376220,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,144031262~13132105,169726147~3786678,185246972~826100,189387978~11188254,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,301562997~3599492,307946002~8932008,317303179~3406771,331817502~4621698,337769986~374855,350953324~3781463,355372960~10114100,373691010~25472622,403122934~6814904,410712466~88886] 2024-09-21T10:17:29.937 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 12 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,76370336~7503356,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~3781463,355372960~3300340,359615966~5871094,373691010~25472622,403122934~6814904,410712466~88886] 2024-09-21T10:17:29.937 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,76370336~7503356,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~3781463,355372960~3300340,359615966~5871094,373691010~25472622,403122934~6814904,410712466~88886] 2024-09-21T10:17:29.942 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,51254163~9794978,67276420~6830306,75497472~8376220,91636093~15231746,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,144031262~13132105,167981878~5530947,185246972~826100,189387978~11188254,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,301562997~3599492,305787002~11091008,317303179~3406771,331817502~4621698,337769986~374855,350953324~3781463,355372960~10114100,373691010~25472622,403122934~6814904,410712466~88886] 2024-09-21T10:17:29.942 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 13 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1074781,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.942 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~13132105,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1074781,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.947 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,51254163~9794978,67276420~6830306,75497472~8376220,87497139~19370700,107841497~8162972,130891915~1515516,133610121~4364681,138412032~2483773,144031262~13132105,167981878~5530947,185246972~826100,189387978~15748938,209715200~4802114,223107934~1074781,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,331817502~4621698,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.947 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 14 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.947 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.952 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,51254163~9794978,67276420~6830306,75497472~8376220,87497139~19370700,107841497~8162972,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,167981878~5530947,185246972~826100,186816738~18320178,209715200~4802114,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,331817502~4621698,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.952 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.952 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,76370336~7503356,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.958 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,67276420~6830306,75497472~8376220,87497139~19370700,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,331817502~4621698,337769986~374855,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.958 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,76370336~8404195,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,336572672~4979196,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.958 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,76370336~8404195,87497139~19370700,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,336572672~4979196,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.963 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,67276420~6830306,75497472~9277059,87497139~19370700,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,331817502~4621698,336572672~4979196,350953324~14533736,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.963 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~11844761,69648386~4458340,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.963 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~11844761,69648386~4458340,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.969 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~11844761,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,144031262~16439826,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,331817502~4621698,336572672~4979196,349881700~15605360,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.969 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.969 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~3850892,336572672~4979196,349881700~15605360,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.974 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,317303179~3406771,331817502~4621698,336572672~4979196,349881700~15605360,373691010~25472622,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.974 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 19 diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.974 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.979 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:29.979 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 7 diff [13033324~2616660,17903366~4986190,25823944~2571315,69648386~1173386,133610121~798745,140433786~462019,152640544~4522823,301562997~3599492,359901878~1171315,403544798~6393040] 2024-09-21T10:17:29.979 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~2616660,17903366~4986190,25823944~2571315,69648386~1173386,133610121~798745,140433786~462019,152640544~4522823,301562997~3599492,359901878~1171315,403544798~6393040] 2024-09-21T10:17:29.985 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13033324~2616660,16777216~6112340,25823944~2571315,67276420~3545352,133610121~798745,138412032~2483773,152026780~5136587,301562997~3599492,358673300~2399893,403544798~6393040] 2024-09-21T10:17:29.985 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 8 diff [13033324~2616660,17903366~4986190,25823944~2571315,53157964~7891177,69648386~1173386,133610121~798745,140433786~462019,152132680~5030687,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,359901878~1171315,379369400~7767668,388300573~10863059,403544798~6393040] 2024-09-21T10:17:29.985 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13033324~2616660,17903366~4986190,25823944~2571315,53157964~7891177,69648386~1173386,133610121~798745,140433786~462019,152132680~5030687,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,359901878~1171315,379369400~7767668,388300573~10863059,403544798~6393040] 2024-09-21T10:17:29.990 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13033324~2616660,16777216~6112340,25823944~2571315,53157964~7891177,67276420~3545352,133610121~798745,138412032~2483773,152026780~5136587,185246972~826100,189387978~5230782,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,358673300~2399893,379369400~7767668,388300573~10863059,403544798~6393040] 2024-09-21T10:17:29.990 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 9 diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,113327605~1146979,133610121~798745,140433786~462019,152132680~5030687,185246972~826100,189676270~4942490,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,308825230~5425531,359901878~1171315,373691010~1041998,379369400~7767668,388300573~10863059,403544798~6393040] 2024-09-21T10:17:29.990 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13033324~4328551,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,113327605~1146979,133610121~798745,140433786~462019,152132680~5030687,185246972~826100,189676270~4942490,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,308825230~5425531,359901878~1171315,373691010~1041998,379369400~7767668,388300573~10863059,403544798~6393040] 2024-09-21T10:17:29.996 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,13033324~10033054,25823944~9584048,53157964~7891177,67276420~3545352,113327605~1146979,133610121~798745,138412032~2483773,152026780~5136587,185246972~826100,189387978~5230782,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,308825230~5425531,358673300~2399893,373691010~1041998,379369400~7767668,388300573~10863059,403544798~6393040] 2024-09-21T10:17:29.996 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 10 diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,152132680~5030687,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,308825230~5425531,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403544798~6393040,410712466~88886] 2024-09-21T10:17:29.996 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,25823944~9584048,53157964~7891177,69648386~1173386,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,152132680~5030687,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,308825230~5425531,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403544798~6393040,410712466~88886] 2024-09-21T10:17:30.001 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,25823944~9584048,53157964~7891177,67276420~3545352,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,152026780~5136587,169726147~3786678,185246972~826100,189387978~5230782,197256444~2704019,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,301562997~3599492,308825230~5425531,317303179~3406771,337769986~374855,358673300~6813760,373691010~1041998,379369400~7767668,388300573~10863059,403544798~6393040,410712466~88886] 2024-09-21T10:17:30.001 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 11 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,53157964~7891177,69648386~1173386,71187875~2918851,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,146820042~10343325,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,308825230~8052780,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~6814904,410712466~88886] 2024-09-21T10:17:30.001 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,53157964~7891177,69648386~1173386,71187875~2918851,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,146820042~10343325,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,308825230~8052780,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~6814904,410712466~88886] 2024-09-21T10:17:30.006 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,53157964~7891177,67276420~6830306,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,146820042~10343325,169726147~3786678,185246972~826100,189387978~11188254,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,301562997~3599492,308825230~8052780,317303179~3406771,337769986~374855,358673300~6813760,373691010~1041998,379369400~7767668,388300573~10863059,403122934~6814904,410712466~88886] 2024-09-21T10:17:30.006 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 12 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,146820042~10343325,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~6814904,410712466~88886] 2024-09-21T10:17:30.007 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,140433786~462019,146820042~10343325,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~6814904,410712466~88886] 2024-09-21T10:17:30.012 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,51254163~9794978,67276420~6830306,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,133610121~798745,138412032~2483773,146820042~10343325,167981878~5530947,185246972~826100,189387978~11188254,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,301562997~3599492,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,358673300~6813760,373691010~1041998,379369400~7767668,388300573~10863059,403122934~6814904,410712466~88886] 2024-09-21T10:17:30.012 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 13 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~11351730,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1074781,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,379369400~7767668,388300573~10863059,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.012 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~1173386,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~11351730,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1074781,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,379369400~7767668,388300573~10863059,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.017 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,51254163~9794978,67276420~6830306,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,133610121~4364681,138412032~2483773,145811637~11351730,167981878~5530947,185246972~826100,189387978~15748938,209715200~4802114,223107934~1074781,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,379369400~7767668,388300573~10863059,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.017 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 14 diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~4458340,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~11726203,388300573~10863059,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.017 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~9584048,51254163~9794978,69648386~4458340,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~11726203,388300573~10863059,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.022 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~12929148,23077889~1786236,25823944~9584048,51254163~9794978,67276420~6830306,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,133610121~4364681,138412032~2483773,145811637~14659451,167981878~5530947,185246972~826100,186816738~18320178,209715200~4802114,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~11726203,388300573~10863059,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.023 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.023 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,17903366~5163012,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.028 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,67276420~6830306,81640613~1197037,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,145811637~14659451,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.028 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.028 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,69648386~4458340,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.033 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~4773045,51254163~9794978,67276420~6830306,81640613~3133918,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.033 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~11844761,69648386~4458340,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.033 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~11844761,69648386~4458340,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,301562997~3599492,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.039 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~11844761,67276420~6830306,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,130891915~1515516,133610121~4364681,138412032~2483773,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,301562997~3599492,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.039 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.039 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.044 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.044 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 19 diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.045 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.050 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:30.050 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 8 diff [53157964~7891177,152132680~1977901,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,379369400~7767668,388300573~10863059] 2024-09-21T10:17:30.050 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [53157964~7891177,152132680~1977901,185246972~826100,189676270~4942490,230873414~6179084,237202911~1629605,241574681~8083217,379369400~7767668,388300573~10863059] 2024-09-21T10:17:30.055 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [53157964~7891177,152132680~1977901,185246972~826100,189387978~5230782,230873414~6179084,237202911~1629605,241574681~8083217,379369400~7767668,388300573~10863059] 2024-09-21T10:17:30.055 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 9 diff [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,53157964~7891177,113327605~1146979,152132680~1977901,185246972~826100,189676270~4942490,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,308825230~5425531,373691010~1041998,379369400~7767668,388300573~10863059] 2024-09-21T10:17:30.055 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,53157964~7891177,113327605~1146979,152132680~1977901,185246972~826100,189676270~4942490,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,308825230~5425531,373691010~1041998,379369400~7767668,388300573~10863059] 2024-09-21T10:17:30.060 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,53157964~7891177,113327605~1146979,152132680~1977901,185246972~826100,189387978~5230782,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,308825230~5425531,373691010~1041998,379369400~7767668,388300573~10863059] 2024-09-21T10:17:30.060 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 10 diff [5605158~470501,10137230~7224645,19186984~3879394,28294828~7113164,53157964~7891177,109886548~39489,113327605~1146979,130891915~1515516,152132680~1977901,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,308825230~5425531,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,410712466~88886] 2024-09-21T10:17:30.061 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,28294828~7113164,53157964~7891177,109886548~39489,113327605~1146979,130891915~1515516,152132680~1977901,169726147~3786678,185246972~826100,189676270~4942490,197256444~2704019,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,308825230~5425531,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,410712466~88886] 2024-09-21T10:17:30.065 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,28294828~7113164,53157964~7891177,109886548~39489,113327605~1146979,130891915~1515516,152132680~1977901,169726147~3786678,185246972~826100,189387978~5230782,197256444~2704019,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,308825230~5425531,317303179~3406771,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,410712466~88886] 2024-09-21T10:17:30.065 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 11 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,53157964~7891177,71187875~2918851,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~7290539,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,308825230~8052780,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.066 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,53157964~7891177,71187875~2918851,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~7290539,169726147~3786678,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,308825230~8052780,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.071 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,53157964~7891177,70821772~3284954,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~7290539,169726147~3786678,185246972~826100,189387978~11188254,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,308825230~8052780,317303179~3406771,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.071 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 12 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~7290539,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.071 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~7290539,167981878~5530947,185246972~826100,189676270~10899962,212176573~2340741,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.076 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,70821772~3284954,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~7290539,167981878~5530947,185246972~826100,189387978~11188254,209715200~4802114,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,379369400~7767668,388300573~10863059,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.076 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 13 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~8298944,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1074781,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,379369400~7767668,388300573~10863059,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.076 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~8298944,167981878~5530947,185246972~826100,189676270~15460646,212176573~2340741,223107934~1074781,224922963~5398542,230873414~6179084,237202911~1629605,241574681~8083217,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,379369400~7767668,388300573~10863059,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.081 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,70821772~3284954,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,134408866~3565936,145811637~8298944,167981878~5530947,185246972~826100,189387978~15748938,209715200~4802114,223107934~1074781,224247612~6073893,230873414~6179084,237202911~1629605,241574681~8083217,271932260~8626507,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,379369400~7767668,388300573~10863059,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.081 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 14 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~14659451,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~11726203,388300573~10863059,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.082 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~14659451,167981878~5530947,185246972~826100,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~11726203,388300573~10863059,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.087 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~9794978,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,134408866~3565936,145811637~14659451,167981878~5530947,185246972~826100,186816738~18320178,209715200~4802114,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~11726203,388300573~10863059,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.087 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~9798921,43960151~4773045,51254163~9794978,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.087 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~9798921,43960151~4773045,51254163~9794978,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.092 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~9798921,43960151~4773045,51254163~9794978,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~9754940,175596367~2528958,185246972~826100,186816738~18320178,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.092 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~4773045,51254163~9794978,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.092 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~4773045,51254163~9794978,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.098 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~4773045,51254163~9794978,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.098 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~11844761,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.098 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~11844761,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.103 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~11844761,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~14367440,185246972~826100,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.103 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.103 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.109 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~23752767,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.109 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 19 diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.109 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.114 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.114 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 9 diff [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,113327605~1146979,212176573~2340741,224922963~5398542,308825230~5425531,373691010~1041998] 2024-09-21T10:17:30.114 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,113327605~1146979,212176573~2340741,224922963~5398542,308825230~5425531,373691010~1041998] 2024-09-21T10:17:30.120 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,13420027~3941848,19186984~3879394,28294828~7113164,113327605~1146979,209715200~4802114,224247612~6073893,308825230~5425531,373691010~1041998] 2024-09-21T10:17:30.120 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 10 diff [5605158~470501,10137230~7224645,19186984~3879394,28294828~7113164,55496708~3205502,109886548~39489,113327605~1146979,130891915~1515516,169726147~3786678,197256444~2704019,212176573~2340741,224922963~5398542,308825230~5425531,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,410712466~88886] 2024-09-21T10:17:30.120 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,28294828~7113164,55496708~3205502,109886548~39489,113327605~1146979,130891915~1515516,169726147~3786678,197256444~2704019,212176573~2340741,224922963~5398542,308825230~5425531,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,410712466~88886] 2024-09-21T10:17:30.125 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,28294828~7113164,55496708~3205502,109886548~39489,113327605~1146979,130891915~1515516,169726147~3786678,197256444~2704019,209715200~4802114,224247612~6073893,308825230~5425531,317303179~3406771,337769986~374855,359615966~5871094,373691010~1041998,410712466~88886] 2024-09-21T10:17:30.125 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 11 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,55496708~3205502,71187875~2918851,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~6886028,169726147~3786678,193062700~7513532,212176573~2340741,224922963~5398542,272525416~8033351,308825230~8052780,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.125 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,55496708~3205502,71187875~2918851,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~6886028,169726147~3786678,193062700~7513532,212176573~2340741,224922963~5398542,272525416~8033351,308825230~8052780,318091802~2618148,337769986~374855,359615966~5871094,373691010~1041998,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.130 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,55496708~3205502,70821772~3284954,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~6886028,169726147~3786678,193062700~7513532,209715200~4802114,224247612~6073893,271932260~8626507,308825230~8052780,317303179~3406771,337769986~374855,359615966~5871094,373691010~1041998,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.130 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 12 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~6886028,167981878~5530947,191882768~8693464,212176573~2340741,224922963~5398542,231013148~5168624,272525416~8033351,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.131 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~6886028,167981878~5530947,191882768~8693464,212176573~2340741,224922963~5398542,231013148~5168624,272525416~8033351,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.136 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,70821772~3284954,81640613~1197037,91636093~10962678,109886548~39489,113327605~1146979,130891915~1515516,146820042~6886028,167981878~5530947,191882768~8693464,209715200~4802114,224247612~6073893,231013148~5168624,271932260~8626507,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,359615966~5871094,373691010~1041998,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.136 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 13 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~8122332,167981878~5530947,191882768~13254148,212176573~2340741,223107934~1074781,224922963~5398542,231013148~5168624,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.136 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~8122332,167981878~5530947,191882768~13254148,212176573~2340741,223107934~1074781,224922963~5398542,231013148~5168624,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.141 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,70821772~3284954,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,134408866~3565936,145811637~8122332,167981878~5530947,191882768~13254148,209715200~4802114,223107934~1074781,224247612~6073893,231013148~5168624,271932260~8626507,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.141 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 14 diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~14659451,167981878~5530947,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~6845743,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.141 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~14659451,167981878~5530947,186816738~18320178,212176573~2340741,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~6845743,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.147 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~7113164,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,134408866~3565936,145811637~14659451,167981878~5530947,186816738~18320178,209715200~4802114,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~6845743,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.147 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 15 diff [2543338~2707254,5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~9798921,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~9754940,175596367~2528958,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~6845743,386800431~4472496,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.147 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~9798921,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~9754940,175596367~2528958,186816738~18320178,212176573~4482781,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~6845743,386800431~4472496,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.152 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~7224645,19186984~3879394,23077889~1786236,28294828~9798921,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~9754940,175596367~2528958,186816738~18320178,209715200~6944154,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,373691010~1041998,375410865~6845743,386800431~4472496,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.152 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 16 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~6845743,386800431~5776580,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.152 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~6845743,386800431~5776580,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.157 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~14367440,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,352853425~12633635,373691010~1041998,375410865~6845743,386800431~5776580,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.158 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 17 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~7602216,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.158 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~7602216,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,212176573~4482781,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.163 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~7602216,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~14367440,186816738~19384597,209715200~6944154,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.163 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 18 diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.163 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.168 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,5605158~470501,10137230~12929148,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.168 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 19 diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.168 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.173 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.173 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 10 diff [10137230~3491714,55496708~3205502,109886548~39489,130891915~1515516,169726147~3786678,197256444~2704019,318091802~2618148,337769986~374855,359615966~5871094,410712466~88886] 2024-09-21T10:17:30.174 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [10137230~3491714,55496708~3205502,109886548~39489,130891915~1515516,169726147~3786678,197256444~2704019,318091802~2618148,337769986~374855,359615966~5871094,410712466~88886] 2024-09-21T10:17:30.179 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [10137230~3491714,55496708~3205502,109886548~39489,130891915~1515516,169726147~3786678,197256444~2704019,317303179~3406771,337769986~374855,359615966~5871094,410712466~88886] 2024-09-21T10:17:30.179 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 11 diff [10137230~3491714,23077889~1786236,55496708~3205502,71187875~2918851,91636093~10962678,109886548~39489,130891915~1515516,146820042~6886028,169726147~3786678,193062700~7513532,272525416~8033351,310542780~6335230,318091802~2618148,337769986~374855,359615966~5871094,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.179 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [10137230~3491714,23077889~1786236,55496708~3205502,71187875~2918851,91636093~10962678,109886548~39489,130891915~1515516,146820042~6886028,169726147~3786678,193062700~7513532,272525416~8033351,310542780~6335230,318091802~2618148,337769986~374855,359615966~5871094,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.184 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [10137230~3491714,23077889~1786236,55496708~3205502,70821772~3284954,91636093~10962678,109886548~39489,130891915~1515516,146820042~6886028,169726147~3786678,193062700~7513532,271932260~8626507,310542780~6335230,317303179~3406771,337769986~374855,359615966~5871094,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.184 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 12 diff [10137230~3491714,23077889~1786236,28477050~4613848,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,130891915~1515516,146820042~6886028,167981878~5530947,191882768~8693464,231013148~5168624,272525416~8033351,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.184 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [10137230~3491714,23077889~1786236,28477050~4613848,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,91636093~10962678,109886548~39489,130891915~1515516,146820042~6886028,167981878~5530947,191882768~8693464,231013148~5168624,272525416~8033351,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,359615966~5871094,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.189 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [10137230~3491714,23077889~1786236,28477050~4613848,51254163~2799606,55496708~3205502,70821772~3284954,81640613~1197037,91636093~10962678,109886548~39489,130891915~1515516,146820042~6886028,167981878~5530947,191882768~8693464,231013148~5168624,271932260~8626507,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,359615966~5871094,403122934~1766297,410712466~88886] 2024-09-21T10:17:30.189 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 13 diff [10137230~3491714,23077889~1786236,28477050~6724658,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~8122332,167981878~5530947,191882768~13254148,223107934~1074781,231013148~5168624,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.189 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [10137230~3491714,23077889~1786236,28477050~6724658,51254163~2799606,55496708~3205502,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~8122332,167981878~5530947,191882768~13254148,223107934~1074781,231013148~5168624,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.195 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [10137230~3491714,23077889~1786236,28477050~6724658,51254163~2799606,55496708~3205502,70821772~3284954,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,134408866~3565936,145811637~8122332,167981878~5530947,191882768~13254148,223107934~1074781,231013148~5168624,271932260~8626507,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.195 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 14 diff [10137230~3491714,23077889~1786236,28477050~6724658,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~14659451,167981878~5530947,186816738~18320178,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,375410865~6845743,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.195 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [10137230~3491714,23077889~1786236,28477050~6724658,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,135450794~2524008,145811637~14659451,167981878~5530947,186816738~18320178,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,375410865~6845743,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.200 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [10137230~3491714,23077889~1786236,28477050~6724658,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,130891915~1515516,134408866~3565936,145811637~14659451,167981878~5530947,186816738~18320178,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,375410865~6845743,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.200 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 15 diff [2543338~2707254,10137230~3491714,23077889~1786236,28477050~9616699,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~9754940,175596367~2528958,186816738~18320178,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,375410865~6845743,386800431~4472496,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.200 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,10137230~3491714,23077889~1786236,28477050~9616699,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~9754940,175596367~2528958,186816738~18320178,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,337769986~374855,352853425~12633635,375410865~6845743,386800431~4472496,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.206 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,10137230~3491714,23077889~1786236,28477050~9616699,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~9754940,175596367~2528958,186816738~18320178,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,337769986~374855,352853425~12633635,375410865~6845743,386800431~4472496,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.207 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 16 diff [2543338~2707254,10137230~3491714,15885326~5190114,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,375410865~6845743,386800431~5776580,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.207 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,10137230~3491714,15885326~5190114,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,352853425~12633635,375410865~6845743,386800431~5776580,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.212 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,10137230~3491714,15885326~5190114,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,55496708~3205502,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~14367440,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,352853425~12633635,375410865~6845743,386800431~5776580,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.212 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 17 diff [2543338~2707254,10137230~3491714,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~7602216,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.212 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,10137230~3491714,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~7602216,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,130891915~1515516,135450794~2524008,145811637~14659451,163757885~14367440,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.217 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,10137230~3491714,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~7602216,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,130891915~1515516,134408866~3565936,145811637~14659451,163757885~14367440,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.217 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 18 diff [2543338~2707254,10137230~4685014,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.218 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,10137230~4685014,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,318091802~2618148,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.223 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,10137230~4685014,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,317303179~3406771,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.223 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 19 diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.223 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.230 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:30.230 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 11 diff [23077889~1786236,71187875~2918851,91636093~10962678,146820042~6886028,193062700~7513532,272525416~8033351,310542780~6335230,403122934~1766297] 2024-09-21T10:17:30.230 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [23077889~1786236,71187875~2918851,91636093~10962678,146820042~6886028,193062700~7513532,272525416~8033351,310542780~6335230,403122934~1766297] 2024-09-21T10:17:30.233 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [23077889~1786236,70821772~3284954,91636093~10962678,146820042~6886028,193062700~7513532,271932260~8626507,310542780~6335230,403122934~1766297] 2024-09-21T10:17:30.233 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 12 diff [23077889~1786236,28477050~4613848,51254163~2799606,71187875~2918851,81640613~1197037,91636093~10962678,146820042~6886028,167981878~2908531,191882768~8693464,231013148~5168624,272525416~8033351,305787002~11091008,332588308~827816,403122934~1766297] 2024-09-21T10:17:30.233 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [23077889~1786236,28477050~4613848,51254163~2799606,71187875~2918851,81640613~1197037,91636093~10962678,146820042~6886028,167981878~2908531,191882768~8693464,231013148~5168624,272525416~8033351,305787002~11091008,332588308~827816,403122934~1766297] 2024-09-21T10:17:30.239 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [23077889~1786236,28477050~4613848,51254163~2799606,70821772~3284954,81640613~1197037,91636093~10962678,146820042~6886028,167981878~2908531,191882768~8693464,231013148~5168624,271932260~8626507,305787002~11091008,332588308~827816,403122934~1766297] 2024-09-21T10:17:30.239 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 13 diff [23077889~1786236,28477050~6724658,51254163~2799606,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,135450794~2524008,145811637~8122332,167981878~2908531,191882768~13254148,223107934~1074781,231013148~5168624,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.239 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [23077889~1786236,28477050~6724658,51254163~2799606,71187875~2918851,81640613~1197037,87497139~15101632,107841497~8162972,135450794~2524008,145811637~8122332,167981878~2908531,191882768~13254148,223107934~1074781,231013148~5168624,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.244 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [23077889~1786236,28477050~6724658,51254163~2799606,70821772~3284954,81640613~1197037,87497139~15101632,107841497~8162972,134408866~3565936,145811637~8122332,167981878~2908531,191882768~13254148,223107934~1074781,231013148~5168624,271932260~8626507,281018368~9453538,305787002~11091008,332588308~827816,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.244 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 14 diff [23077889~1786236,28477050~6724658,51254163~2799606,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,135450794~2524008,145811637~14659451,167981878~2908531,186816738~18320178,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.244 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [23077889~1786236,28477050~6724658,51254163~2799606,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,135450794~2524008,145811637~14659451,167981878~2908531,186816738~18320178,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.249 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [23077889~1786236,28477050~6724658,51254163~2799606,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,134408866~3565936,145811637~14659451,167981878~2908531,186816738~18320178,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.249 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 15 diff [2543338~2707254,23077889~1786236,28477050~9616699,43960151~4773045,51254163~2799606,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,175596367~2528958,186816738~18320178,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.249 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,23077889~1786236,28477050~9616699,43960151~4773045,51254163~2799606,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,175596367~2528958,186816738~18320178,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.255 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,23077889~1786236,28477050~9616699,43960151~4773045,51254163~2799606,70391236~3715490,81640613~1197037,87497139~15101632,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~7132524,175596367~2528958,186816738~18320178,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,266388628~14170139,281018368~9453538,305787002~11091008,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.255 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 16 diff [2543338~2707254,15885326~5190114,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.255 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.260 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,23077889~1786236,26686115~11407634,43960151~4773045,51254163~2799606,70391236~3715490,81640613~3133918,87497139~15101632,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,332588308~827816,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.260 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 17 diff [2543338~2707254,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56306953~6791971,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.260 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56306953~6791971,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,284670827~5801079,305787002~11091008,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.265 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56306953~6791971,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~9453538,305787002~11091008,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.265 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 18 diff [2543338~2707254,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.265 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~17046454,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.271 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~17046454,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.271 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 19 diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.271 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.276 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.276 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 12 diff [28477050~4613848,51254163~2799606,81640613~1197037,167981878~2908531,191882768~4277993,231013148~5168624,277827050~797905,305787002~9390867,332588308~827816] 2024-09-21T10:17:30.276 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [28477050~4613848,51254163~2799606,81640613~1197037,167981878~2908531,191882768~4277993,231013148~5168624,277827050~797905,305787002~9390867,332588308~827816] 2024-09-21T10:17:30.281 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [28477050~4613848,51254163~2799606,81640613~1197037,167981878~2908531,191882768~4277993,231013148~5168624,277827050~797905,305787002~9390867,332588308~827816] 2024-09-21T10:17:30.281 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 13 diff [28477050~6724658,51254163~2799606,81640613~1197037,87497139~7774339,107841497~8162972,135450794~2524008,145811637~8122332,167981878~2908531,191882768~4277993,199738595~5398321,223107934~1074781,231013148~5168624,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.281 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [28477050~6724658,51254163~2799606,81640613~1197037,87497139~7774339,107841497~8162972,135450794~2524008,145811637~8122332,167981878~2908531,191882768~4277993,199738595~5398321,223107934~1074781,231013148~5168624,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.287 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [28477050~6724658,51254163~2799606,81640613~1197037,87497139~7774339,107841497~8162972,134408866~3565936,145811637~8122332,167981878~2908531,191882768~4277993,199738595~5398321,223107934~1074781,231013148~5168624,277827050~797905,281018368~9453538,305787002~9390867,332588308~827816,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.287 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 14 diff [28477050~6724658,51254163~2799606,70391236~2357106,81640613~1197037,87497139~7774339,107841497~8162972,135450794~2524008,145811637~14659451,167981878~2908531,186816738~9344023,199738595~5398321,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.287 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [28477050~6724658,51254163~2799606,70391236~2357106,81640613~1197037,87497139~7774339,107841497~8162972,135450794~2524008,145811637~14659451,167981878~2908531,186816738~9344023,199738595~5398321,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.292 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [28477050~6724658,51254163~2799606,70391236~2357106,81640613~1197037,87497139~7774339,107841497~8162972,134408866~3565936,145811637~14659451,167981878~2908531,186816738~9344023,199738595~5398321,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,266388628~6010521,277827050~797905,281018368~9453538,305787002~9390867,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.292 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 15 diff [2543338~2707254,28477050~9616699,43960151~4773045,51254163~2799606,70391236~2357106,81640613~1197037,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,175596367~2528958,186816738~9344023,199738595~5398321,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.292 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,28477050~9616699,43960151~4773045,51254163~2799606,70391236~2357106,81640613~1197037,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,175596367~2528958,186816738~9344023,199738595~5398321,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.297 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,28477050~9616699,43960151~4773045,51254163~2799606,70391236~2357106,81640613~1197037,87497139~7774339,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~7132524,175596367~2528958,186816738~9344023,199738595~5398321,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,266388628~6010521,277827050~797905,281018368~9453538,305787002~9390867,332588308~827816,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.297 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 16 diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~4773045,51254163~2799606,70391236~2357106,81640613~3133918,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~9344023,199738595~6462740,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.297 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~4773045,51254163~2799606,70391236~2357106,81640613~3133918,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~9344023,199738595~6462740,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.303 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~11407634,43960151~4773045,51254163~2799606,70391236~2357106,81640613~3133918,87497139~7774339,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~7132524,171480060~6645265,186816738~9344023,199738595~6462740,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~9453538,305787002~9390867,332588308~827816,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.303 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 17 diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56306953~6791971,70391236~2357106,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.303 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56306953~6791971,70391236~2357106,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,277827050~797905,284670827~5801079,305787002~9390867,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.308 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56306953~6791971,70391236~2357106,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~7132524,171480060~6645265,186816738~19384597,213967659~2691695,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~9453538,305787002~9390867,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.308 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 18 diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~15346313,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.308 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~15346313,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.314 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~15346313,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.314 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 19 diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 13 diff [31740612~3461096,87497139~7774339,107841497~8162972,135450794~2524008,145811637~8122332,199738595~5398321,223107934~1074781,284670827~5801079,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.319 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [31740612~3461096,87497139~7774339,107841497~8162972,135450794~2524008,145811637~8122332,199738595~5398321,223107934~1074781,284670827~5801079,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.324 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [31740612~3461096,87497139~7774339,107841497~8162972,134408866~3565936,145811637~8122332,199738595~5398321,223107934~1074781,281018368~9453538,352853425~7795887,402573262~5559752] 2024-09-21T10:17:30.325 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 14 diff [31740612~3461096,70391236~2357106,87497139~7774339,107841497~8162972,135450794~2524008,145811637~14659451,169289246~106547,186816738~7215627,199738595~5398321,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,284670827~5801079,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.325 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [31740612~3461096,70391236~2357106,87497139~7774339,107841497~8162972,135450794~2524008,145811637~14659451,169289246~106547,186816738~7215627,199738595~5398321,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,284670827~5801079,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.330 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [31740612~3461096,70391236~2357106,87497139~7774339,107841497~8162972,134408866~3565936,145811637~14659451,169289246~106547,186816738~7215627,199738595~5398321,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,266388628~6010521,281018368~9453538,352853425~7795887,361646431~2900270,375410865~6845743,402573262~5559752] 2024-09-21T10:17:30.330 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 15 diff [2543338~2707254,31740612~6353137,43960151~4773045,70391236~2357106,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~4407634,169289246~106547,175596367~2528958,186816738~7215627,199738595~5398321,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,284670827~5801079,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.330 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,31740612~6353137,43960151~4773045,70391236~2357106,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~4407634,169289246~106547,175596367~2528958,186816738~7215627,199738595~5398321,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,284670827~5801079,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.335 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,31740612~6353137,43960151~4773045,70391236~2357106,87497139~7774339,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~4407634,169289246~106547,175596367~2528958,186816738~7215627,199738595~5398321,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,266388628~6010521,281018368~9453538,352853425~7795887,361646431~2900270,375410865~6845743,386800431~4472496,402573262~5559752] 2024-09-21T10:17:30.336 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 16 diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~4773045,70391236~2357106,81873196~2901335,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,199738595~6462740,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,284670827~5801079,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.336 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~4773045,70391236~2357106,81873196~2901335,87497139~7774339,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,199738595~6462740,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,284670827~5801079,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.341 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~11407634,43960151~4773045,70391236~2357106,81873196~2901335,87497139~7774339,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,199738595~6462740,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~9453538,336572672~4979196,352853425~7795887,361646431~2900270,375410865~6845743,386800431~5776580,402573262~5559752] 2024-09-21T10:17:30.341 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 17 diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~5598312,56306953~6791971,70391236~2357106,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,284670827~5801079,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.341 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~11407634,43960151~5598312,56306953~6791971,70391236~2357106,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,135450794~2524008,145811637~14659451,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,284670827~5801079,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.346 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~11407634,43960151~5598312,56306953~6791971,70391236~2357106,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,134408866~3565936,145811637~14659451,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~9453538,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.346 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 18 diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.347 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.352 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402573262~5559752] 2024-09-21T10:17:30.352 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 19 diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.352 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.357 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:30.358 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 14 diff [70391236~2357106,153095864~7375224,169289246~106547,186816738~7215627,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,361646431~2900270,375410865~6845743] 2024-09-21T10:17:30.358 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [70391236~2357106,153095864~7375224,169289246~106547,186816738~7215627,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,361646431~2900270,375410865~6845743] 2024-09-21T10:17:30.363 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [70391236~2357106,153095864~7375224,169289246~106547,186816738~7215627,234393369~6292088,246511951~1009810,251658240~8092188,266388628~6010521,361646431~2900270,375410865~6845743] 2024-09-21T10:17:30.363 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 15 diff [2543338~2707254,34940153~3153596,43960151~4773045,70391236~2357106,118600721~3501566,147119839~4572824,153095864~7375224,163757885~4407634,169289246~106547,175596367~2528958,186816738~7215627,213967659~2691695,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,361646431~2900270,375410865~6845743,386800431~4472496,402901262~853192] 2024-09-21T10:17:30.363 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,34940153~3153596,43960151~4773045,70391236~2357106,118600721~3501566,147119839~4572824,153095864~7375224,163757885~4407634,169289246~106547,175596367~2528958,186816738~7215627,213967659~2691695,234393369~6292088,246511951~1009810,255330136~4420292,266388628~6010521,361646431~2900270,375410865~6845743,386800431~4472496,402901262~853192] 2024-09-21T10:17:30.368 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,34940153~3153596,43960151~4773045,70391236~2357106,117440512~4661775,147119839~4572824,153095864~7375224,163757885~4407634,169289246~106547,175596367~2528958,186816738~7215627,213967659~2691695,234393369~6292088,246511951~1009810,251658240~8092188,266388628~6010521,361646431~2900270,375410865~6845743,386800431~4472496,402901262~853192] 2024-09-21T10:17:30.368 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 16 diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~4773045,70391236~2357106,81873196~2901335,118600721~3501566,147119839~13351249,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,202498566~3702769,213967659~2691695,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,336572672~4979196,361646431~2900270,375410865~6845743,386800431~5776580,402901262~853192] 2024-09-21T10:17:30.368 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~4773045,70391236~2357106,81873196~2901335,118600721~3501566,147119839~13351249,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,202498566~3702769,213967659~2691695,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,336572672~4979196,361646431~2900270,375410865~6845743,386800431~5776580,402901262~853192] 2024-09-21T10:17:30.373 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~4773045,70391236~2357106,81873196~2901335,117440512~4661775,147119839~13351249,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,202498566~3702769,213967659~2691695,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,336572672~4979196,361646431~2900270,375410865~6845743,386800431~5776580,402901262~853192] 2024-09-21T10:17:30.373 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 17 diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56306953~6791971,70391236~2357106,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,147119839~13351249,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,336572672~4979196,349881700~3109379,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.374 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56306953~6791971,70391236~2357106,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,147119839~13351249,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,336572672~4979196,349881700~3109379,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.379 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56306953~6791971,70391236~2357106,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,147119839~13351249,163757885~4407634,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,336572672~4979196,349881700~3109379,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.379 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 18 diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.379 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,255330136~4420292,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.384 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,13101064~1721180,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,361646431~2900270,375410865~6845743,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.385 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 19 diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:30.385 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:30.390 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:30.390 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 15 diff [2543338~2707254,34940153~3153596,43960151~4773045,118600721~3501566,147119839~4572824,163757885~4407634,175596367~2528958,213967659~2691695,386800431~4472496,402901262~853192] 2024-09-21T10:17:30.390 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,34940153~3153596,43960151~4773045,118600721~3501566,147119839~4572824,163757885~4407634,175596367~2528958,213967659~2691695,386800431~4472496,402901262~853192] 2024-09-21T10:17:30.395 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,34940153~3153596,43960151~4773045,117440512~4661775,147119839~4572824,163757885~4407634,175596367~2528958,213967659~2691695,386800431~4472496,402901262~853192] 2024-09-21T10:17:30.395 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 16 diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~4773045,81873196~2901335,118600721~3501566,147119839~7513593,163757885~4407634,171480060~6645265,202498566~3702769,213967659~2691695,261441010~4291601,336572672~4979196,386800431~5776580,402901262~853192] 2024-09-21T10:17:30.396 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~4773045,81873196~2901335,118600721~3501566,147119839~7513593,163757885~4407634,171480060~6645265,202498566~3702769,213967659~2691695,261441010~4291601,336572672~4979196,386800431~5776580,402901262~853192] 2024-09-21T10:17:30.401 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~4773045,81873196~2901335,117440512~4661775,147119839~7513593,163757885~4407634,171480060~6645265,202498566~3702769,213967659~2691695,260046848~5685763,336572672~4979196,386800431~5776580,402901262~853192] 2024-09-21T10:17:30.401 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 17 diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56306953~6791971,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,147119839~7513593,163757885~4407634,171480060~6645265,194992372~11208963,213967659~2691695,261441010~4291601,336572672~4979196,349881700~3109379,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.401 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56306953~6791971,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,147119839~7513593,163757885~4407634,171480060~6645265,194992372~11208963,213967659~2691695,261441010~4291601,336572672~4979196,349881700~3109379,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.406 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56306953~6791971,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,147119839~7513593,163757885~4407634,171480060~6645265,194992372~11208963,213967659~2691695,260046848~5685763,336572672~4979196,349881700~3109379,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.406 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 18 diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56247609~9616106,69192860~770245,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,261441010~4291601,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.406 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [2543338~2707254,13101064~1721180,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56247609~9616106,69192860~770245,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,261441010~4291601,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.412 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [2543338~2707254,13101064~1721180,15885326~5190114,26686115~8062042,34940153~3153596,43960151~5598312,56247609~9616106,69192860~770245,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,260046848~5685763,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,386800431~5776580,393772170~4230048,402901262~853192] 2024-09-21T10:17:30.412 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 19 diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:30.413 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:30.417 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:30.417 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 16 diff [15885326~5190114,26686115~8062042,81873196~2901335,150410923~4222509,171480060~4147314,202498566~3702769,261441010~4291601,336572672~4979196,388641439~3935572] 2024-09-21T10:17:30.418 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [15885326~5190114,26686115~8062042,81873196~2901335,150410923~4222509,171480060~4147314,202498566~3702769,261441010~4291601,336572672~4979196,388641439~3935572] 2024-09-21T10:17:30.423 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [15885326~5190114,26686115~8062042,81873196~2901335,150410923~4222509,171480060~4147314,202498566~3702769,260046848~5685763,336572672~4979196,388641439~3935572] 2024-09-21T10:17:30.423 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 17 diff [15885326~5190114,26686115~8062042,44880910~4677553,56306953~6791971,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,150410923~4222509,171480060~4147314,194992372~11208963,261441010~4291601,336572672~4979196,349881700~3109379,388641439~3935572,393772170~4230048] 2024-09-21T10:17:30.423 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [15885326~5190114,26686115~8062042,44880910~4677553,56306953~6791971,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,150410923~4222509,171480060~4147314,194992372~11208963,261441010~4291601,336572672~4979196,349881700~3109379,388641439~3935572,393772170~4230048] 2024-09-21T10:17:30.428 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [15885326~5190114,26686115~8062042,44880910~4677553,56306953~6791971,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,150410923~4222509,171480060~4147314,194992372~11208963,260046848~5685763,336572672~4979196,349881700~3109379,388641439~3935572,393772170~4230048] 2024-09-21T10:17:30.428 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 18 diff [13101064~1721180,15885326~5190114,26686115~8062042,44880910~4677553,56247609~9616106,69192860~770245,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,261441010~4291601,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,388641439~3935572,393772170~4230048] 2024-09-21T10:17:30.428 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13101064~1721180,15885326~5190114,26686115~8062042,44880910~4677553,56247609~9616106,69192860~770245,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,261441010~4291601,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,388641439~3935572,393772170~4230048] 2024-09-21T10:17:30.434 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13101064~1721180,15885326~5190114,26686115~8062042,44880910~4677553,56247609~9616106,69192860~770245,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,260046848~5685763,283728297~7471077,299831556~7055694,336572672~4979196,349881700~3109379,388641439~3935572,393772170~4230048] 2024-09-21T10:17:30.434 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 19 diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:30.434 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:30.439 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:30.439 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to 17 diff [44880910~4677553,56306953~6791971,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,194992372~7533021,349881700~3109379,393772170~4230048] 2024-09-21T10:17:30.439 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [44880910~4677553,56306953~6791971,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,194992372~7533021,349881700~3109379,393772170~4230048] 2024-09-21T10:17:30.444 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [44880910~4677553,56306953~6791971,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,194992372~7533021,349881700~3109379,393772170~4230048] 2024-09-21T10:17:30.444 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to 18 diff [13101064~1721180,44880910~4677553,56247609~9616106,69192860~770245,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,220548498~966115,283728297~7471077,299831556~7055694,349881700~3109379,389422554~2983188,393772170~4230048] 2024-09-21T10:17:30.444 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13101064~1721180,44880910~4677553,56247609~9616106,69192860~770245,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,220548498~966115,283728297~7471077,299831556~7055694,349881700~3109379,389422554~2983188,393772170~4230048] 2024-09-21T10:17:30.449 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13101064~1721180,44880910~4677553,56247609~9616106,69192860~770245,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,220548498~966115,283728297~7471077,299831556~7055694,349881700~3109379,389422554~2983188,393772170~4230048] 2024-09-21T10:17:30.449 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to 19 diff [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:30.449 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:30.455 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:30.455 INFO:tasks.workunit.client.0.smithi119.stdout:from 17 to 18 diff [13101064~1721180,56247609~9616106,69192860~770245,126747906~5612252,161966455~2167834,220548498~966115,283728297~7471077,299831556~7055694,389422554~2983188] 2024-09-21T10:17:30.455 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [13101064~1721180,56247609~9616106,69192860~770245,126747906~5612252,161966455~2167834,220548498~966115,283728297~7471077,299831556~7055694,389422554~2983188] 2024-09-21T10:17:30.460 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [13101064~1721180,56247609~9616106,69192860~770245,126747906~5612252,161966455~2167834,220548498~966115,283728297~7471077,299831556~7055694,389422554~2983188] 2024-09-21T10:17:30.460 INFO:tasks.workunit.client.0.smithi119.stdout:from 17 to 19 diff [1334775~10302970,13101064~1721180,31331118~6084841,56247609~9616106,69192860~770245,77530326~3165641,126747906~5612252,161966455~2167834,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,350071290~3241518,387129528~5276214,397313563~4028729] 2024-09-21T10:17:30.460 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,31331118~6084841,56247609~9616106,69192860~770245,77530326~3165641,126747906~5612252,161966455~2167834,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,350071290~3241518,387129528~5276214,397313563~4028729] 2024-09-21T10:17:30.465 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,31331118~6084841,56247609~9616106,69192860~770245,77530326~3165641,126747906~5612252,161966455~2167834,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,350071290~3241518,387129528~5276214,397313563~4028729] 2024-09-21T10:17:30.465 INFO:tasks.workunit.client.0.smithi119.stdout:from 18 to 19 diff [1334775~10302970,31331118~6084841,77530326~3165641,203271544~1782031,254681153~667359,313949869~4832006,350071290~3241518,387129528~4653735,397313563~4028729] 2024-09-21T10:17:30.465 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,31331118~6084841,77530326~3165641,203271544~1782031,254681153~667359,313949869~4832006,350071290~3241518,387129528~4653735,397313563~4028729] 2024-09-21T10:17:30.470 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,31331118~6084841,77530326~3165641,203271544~1782031,254681153~667359,313949869~4832006,350071290~3241518,387129528~4653735,397313563~4028729] 2024-09-21T10:17:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:30 smithi145 ceph-mon[89359]: osdmap e988: 8 total, 8 up, 8 in 2024-09-21T10:17:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:17:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:17:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:30.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:30 smithi119 ceph-mon[105288]: osdmap e988: 8 total, 8 up, 8 in 2024-09-21T10:17:30.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:30 smithi119 ceph-mon[101432]: osdmap e988: 8 total, 8 up, 8 in 2024-09-21T10:17:31.422 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.423 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.439 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:31.440 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.440 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.446 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:31.446 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.446 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.453 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:31.453 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.453 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.458 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:31.458 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.458 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:31.463 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:31.463 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:31.464 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:31.469 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:31.469 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:31.469 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:31.475 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:31.475 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.475 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.481 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.481 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.481 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.487 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.487 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.487 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.492 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:31.492 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.492 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.498 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.498 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.498 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.503 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.503 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.503 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.509 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:31.509 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:31.509 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:31.514 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:31.514 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:31.514 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:31.519 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:31.520 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:31.520 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:31.525 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:31.525 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to HEAD diff [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:31.525 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:31.531 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:31.531 INFO:tasks.workunit.client.0.smithi119.stdout:from 17 to HEAD diff [1334775~10302970,13101064~1721180,31331118~6084841,56247609~9616106,69192860~770245,77530326~3165641,126747906~5612252,161966455~2167834,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,350071290~3241518,387129528~5276214,397313563~4028729] 2024-09-21T10:17:31.531 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,31331118~6084841,56247609~9616106,69192860~770245,77530326~3165641,126747906~5612252,161966455~2167834,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,350071290~3241518,387129528~5276214,397313563~4028729] 2024-09-21T10:17:31.536 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,31331118~6084841,56247609~9616106,69192860~770245,77530326~3165641,126747906~5612252,161966455~2167834,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,350071290~3241518,387129528~5276214,397313563~4028729] 2024-09-21T10:17:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:31 smithi145 ceph-mon[89359]: pgmap v1216: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 217 KiB/s rd, 42 MiB/s wr, 741 op/s 2024-09-21T10:17:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:31 smithi119 ceph-mon[105288]: pgmap v1216: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 217 KiB/s rd, 42 MiB/s wr, 741 op/s 2024-09-21T10:17:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:31 smithi119 ceph-mon[101432]: pgmap v1216: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 217 KiB/s rd, 42 MiB/s wr, 741 op/s 2024-09-21T10:17:32.438 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.438 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.450 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:32.450 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.450 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.456 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:32.456 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.456 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.462 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:32.462 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.462 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.467 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:32.468 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.468 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:32.473 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:32.473 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:32.473 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:32.479 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:32.479 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:32.479 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:32.484 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:32.485 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.485 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.490 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.490 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.490 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.496 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.496 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.496 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.501 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:32.501 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.501 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.506 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.506 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.506 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.511 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.512 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.512 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.517 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:32.517 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:32.517 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:32.522 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:32.522 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:32.522 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:32.528 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:32.528 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:32.528 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:32.533 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:32.533 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to HEAD diff [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:32.533 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:32.538 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,31331118~6084841,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,82770359~4305655,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,161966455~2167834,194992372~7533021,203271544~1782031,220548498~966115,254681153~667359,283728297~7471077,299831556~7055694,313949869~4832006,349881700~3431108,387129528~5276214,393772170~7570122] 2024-09-21T10:17:32.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:32 smithi119 ceph-mon[101432]: osdmap e989: 8 total, 8 up, 8 in 2024-09-21T10:17:32.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:32 smithi119 ceph-mon[105288]: osdmap e989: 8 total, 8 up, 8 in 2024-09-21T10:17:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:32 smithi145 ceph-mon[89359]: osdmap e989: 8 total, 8 up, 8 in 2024-09-21T10:17:33.445 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.445 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.459 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:33.459 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.459 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.465 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:33.465 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.465 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.470 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:33.471 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.471 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.476 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:33.476 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.476 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:33.482 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:33.482 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:33.482 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:33.488 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:33.488 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:33.488 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:33.493 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:33.493 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.493 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.499 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.499 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.499 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.505 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.505 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.505 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.510 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:33.510 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.510 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.515 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.515 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.515 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.520 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.521 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.521 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.527 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:33.527 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:33.527 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:33.531 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:33.531 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:33.532 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:33.538 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:33.538 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:33.538 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:33.542 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~10729844,44880910~4677553,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,126747906~5612252,150410923~4222509,161966455~2167834,171480060~4147314,194992372~11208963,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,387129528~5447483,393772170~7570122] 2024-09-21T10:17:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:33 smithi145 ceph-mon[89359]: osdmap e990: 8 total, 8 up, 8 in 2024-09-21T10:17:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:33 smithi145 ceph-mon[89359]: pgmap v1219: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 217 KiB/s rd, 42 MiB/s wr, 742 op/s 2024-09-21T10:17:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:33 smithi119 ceph-mon[101432]: osdmap e990: 8 total, 8 up, 8 in 2024-09-21T10:17:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:33 smithi119 ceph-mon[101432]: pgmap v1219: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 217 KiB/s rd, 42 MiB/s wr, 742 op/s 2024-09-21T10:17:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:33 smithi119 ceph-mon[105288]: osdmap e990: 8 total, 8 up, 8 in 2024-09-21T10:17:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:33 smithi119 ceph-mon[105288]: pgmap v1219: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 217 KiB/s rd, 42 MiB/s wr, 742 op/s 2024-09-21T10:17:34.450 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.450 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.464 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:34.464 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.464 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.470 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:34.470 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.470 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.476 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:34.476 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.476 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.482 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:34.482 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.482 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:34.488 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:34.488 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:34.488 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:34.493 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:34.493 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:34.493 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:34.499 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:34.499 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.499 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.504 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.505 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.505 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.510 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.510 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.510 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.515 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:34.516 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.516 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.521 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.521 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.521 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.527 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.527 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.527 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.532 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:34.532 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:34.532 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:34.537 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:34.538 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:34.538 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,261441010~4291601,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:34.543 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~7513593,161966455~6199064,171480060~6645265,194992372~11208963,213967659~2691695,220548498~966115,254681153~667359,260046848~5685763,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:34 smithi145 ceph-mon[89359]: osdmap e991: 8 total, 8 up, 8 in 2024-09-21T10:17:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:17:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:17:34] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.51.0" 2024-09-21T10:17:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:34 smithi119 ceph-mon[105288]: osdmap e991: 8 total, 8 up, 8 in 2024-09-21T10:17:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:34 smithi119 ceph-mon[101432]: osdmap e991: 8 total, 8 up, 8 in 2024-09-21T10:17:35.459 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.459 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.479 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:35.479 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.479 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.485 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:35.485 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.485 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.490 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:35.491 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.491 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.496 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:35.496 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.496 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:35.502 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:35.502 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:35.502 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:35.507 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:35.508 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:35.508 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:35.513 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:35.513 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.513 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.519 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.519 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.519 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.524 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.524 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.525 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.530 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:35.530 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.530 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.535 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.535 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.535 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.540 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.540 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.540 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.546 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:35.546 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:35.546 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,118600721~3501566,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:35.550 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,88240944~3902398,96201188~862024,107031566~187623,117440512~4661775,126747906~5612252,147119839~13351249,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~3431108,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402901262~853192] 2024-09-21T10:17:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:35 smithi145 ceph-mon[89359]: osdmap e992: 8 total, 8 up, 8 in 2024-09-21T10:17:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:35 smithi145 ceph-mon[89359]: pgmap v1222: 105 pgs: 105 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 163 KiB/s rd, 1.4 MiB/s wr, 5.32k op/s 2024-09-21T10:17:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:35 smithi119 ceph-mon[101432]: osdmap e992: 8 total, 8 up, 8 in 2024-09-21T10:17:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:35 smithi119 ceph-mon[101432]: pgmap v1222: 105 pgs: 105 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 163 KiB/s rd, 1.4 MiB/s wr, 5.32k op/s 2024-09-21T10:17:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:35 smithi119 ceph-mon[105288]: osdmap e992: 8 total, 8 up, 8 in 2024-09-21T10:17:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:35 smithi119 ceph-mon[105288]: pgmap v1222: 105 pgs: 105 active+clean; 1.2 GiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 163 KiB/s rd, 1.4 MiB/s wr, 5.32k op/s 2024-09-21T10:17:36.478 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.478 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.496 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:36.496 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.496 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.501 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:36.501 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.502 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.507 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:36.507 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.507 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.513 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:36.513 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.513 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:36.518 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:36.518 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:36.518 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:36.523 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:36.523 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:36.524 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:36.529 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:36.529 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.529 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.535 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.535 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.535 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.540 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.540 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.540 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.545 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:36.545 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.546 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.550 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.550 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.551 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.556 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.556 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.556 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,283728297~7471077,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.561 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81873196~5202818,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~6199064,169289246~106547,171480060~6645265,186816738~7215627,194992372~11208963,213967659~2691695,220548498~966115,223107934~1074781,234393369~6292088,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,281018368~10181006,299831556~7055694,313949869~4832006,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:36 smithi145 ceph-mon[89359]: osdmap e993: 8 total, 8 up, 8 in 2024-09-21T10:17:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:36 smithi119 ceph-mon[101432]: osdmap e993: 8 total, 8 up, 8 in 2024-09-21T10:17:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:36 smithi119 ceph-mon[105288]: osdmap e993: 8 total, 8 up, 8 in 2024-09-21T10:17:37.479 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.479 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.499 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:37.499 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.499 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.505 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:37.505 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.505 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.511 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:37.511 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.511 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.516 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:37.516 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.517 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:37.522 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:37.522 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:37.522 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:37.528 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:37.528 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:37.528 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:37.534 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:37.534 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.534 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.539 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.539 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.539 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.544 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.544 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.545 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.550 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:37.550 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:37.550 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:37.555 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:37.555 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:37.555 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,277827050~797905,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:37.560 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~2357106,77530326~3165641,81640613~5435401,87497139~7774339,96201188~862024,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~6010521,277827050~797905,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:37 smithi145 ceph-mon[89359]: osdmap e994: 8 total, 8 up, 8 in 2024-09-21T10:17:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:37 smithi145 ceph-mon[89359]: pgmap v1225: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 168 KiB/s rd, 1.4 MiB/s wr, 5.66k op/s 2024-09-21T10:17:37.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:37 smithi119 ceph-mon[101432]: osdmap e994: 8 total, 8 up, 8 in 2024-09-21T10:17:37.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:37 smithi119 ceph-mon[101432]: pgmap v1225: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 168 KiB/s rd, 1.4 MiB/s wr, 5.66k op/s 2024-09-21T10:17:37.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:37 smithi119 ceph-mon[105288]: osdmap e994: 8 total, 8 up, 8 in 2024-09-21T10:17:37.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:37 smithi119 ceph-mon[105288]: pgmap v1225: 105 pgs: 105 active+clean; 1.2 GiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 168 KiB/s rd, 1.4 MiB/s wr, 5.66k op/s 2024-09-21T10:17:38.488 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.488 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.505 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:38.506 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.506 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.511 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:38.512 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.512 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.517 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:38.517 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.517 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.522 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:38.523 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.523 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:38.528 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:38.528 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:38.528 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:38.533 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:38.533 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:38.533 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:38.539 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:38.539 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.539 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.544 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.544 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.544 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.549 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.549 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.550 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.555 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:38.555 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:38.555 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5612252,135450794~2524008,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:38.560 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~10302970,13101064~1721180,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,56247609~9616106,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5612252,134408866~3565936,145811637~14659451,161966455~8923954,171480060~6645265,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~18950319,332588308~827816,336572672~4979196,349881700~10767612,361646431~2900270,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752] 2024-09-21T10:17:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:38 smithi145 ceph-mon[89359]: osdmap e995: 8 total, 8 up, 8 in 2024-09-21T10:17:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:38 smithi119 ceph-mon[101432]: osdmap e995: 8 total, 8 up, 8 in 2024-09-21T10:17:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:38 smithi119 ceph-mon[105288]: osdmap e995: 8 total, 8 up, 8 in 2024-09-21T10:17:39.192 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.193 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.216 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:39.216 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.216 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.222 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:39.222 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.222 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.230 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:39.230 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.230 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.233 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:39.233 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.234 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:39.239 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:39.239 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:39.240 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:39.245 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:39.250 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:39.251 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.251 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.256 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.256 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.256 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.261 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.261 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.261 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.267 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~13487469,15885326~5190114,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,213967659~2691695,220548498~966115,223107934~1074781,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:39 smithi145 ceph-mon[89359]: osdmap e996: 8 total, 8 up, 8 in 2024-09-21T10:17:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:39 smithi145 ceph-mon[89359]: pgmap v1228: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 125 KiB/s rd, 6.2 KiB/s wr, 1.63k op/s 2024-09-21T10:17:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:39 smithi145 ceph-mon[89359]: osdmap e997: 8 total, 8 up, 8 in 2024-09-21T10:17:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:39 smithi119 ceph-mon[101432]: osdmap e996: 8 total, 8 up, 8 in 2024-09-21T10:17:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:39 smithi119 ceph-mon[101432]: pgmap v1228: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 125 KiB/s rd, 6.2 KiB/s wr, 1.63k op/s 2024-09-21T10:17:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:39 smithi119 ceph-mon[101432]: osdmap e997: 8 total, 8 up, 8 in 2024-09-21T10:17:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:39 smithi119 ceph-mon[105288]: osdmap e996: 8 total, 8 up, 8 in 2024-09-21T10:17:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:39 smithi119 ceph-mon[105288]: pgmap v1228: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 125 KiB/s rd, 6.2 KiB/s wr, 1.63k op/s 2024-09-21T10:17:39.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:39 smithi119 ceph-mon[105288]: osdmap e997: 8 total, 8 up, 8 in 2024-09-21T10:17:40.193 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.194 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.215 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:40.215 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.215 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.220 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:40.220 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.221 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.226 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:40.226 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.226 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.231 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:40.232 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.232 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:40.237 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:40.237 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:40.237 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:40.243 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:40.243 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:40.243 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:40.248 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:40.248 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:40.248 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:40.253 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:40.253 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:40.253 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,231013148~9672309,246511951~1009810,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:40.259 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~2799606,55496708~10367007,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,231013148~9672309,246511951~1009810,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~6845743,386800431~5776580,393772170~7570122,402573262~5559752,410712466~88886] 2024-09-21T10:17:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:17:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:17:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:41.206 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.206 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.222 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:41.222 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.222 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.228 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:41.228 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.228 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.234 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:41.234 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.234 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.240 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:41.240 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.240 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:41.245 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:41.245 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:41.245 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:41.251 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:41.251 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:41.251 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:41.256 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:41.256 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:41.256 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,135450794~2524008,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:41.261 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,26686115~11407634,43960151~5598312,51254163~14609552,69192860~770245,70391236~3715490,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,134408866~3565936,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~5559752,410712466~88886] 2024-09-21T10:17:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:41 smithi145 ceph-mon[89359]: osdmap e998: 8 total, 8 up, 8 in 2024-09-21T10:17:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:41 smithi119 ceph-mon[101432]: osdmap e998: 8 total, 8 up, 8 in 2024-09-21T10:17:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:41 smithi119 ceph-mon[105288]: osdmap e998: 8 total, 8 up, 8 in 2024-09-21T10:17:42.207 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.207 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.225 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:42.225 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.225 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.231 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:42.231 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.231 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.236 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:42.236 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.236 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.242 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:42.242 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.242 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:42.247 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:42.247 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:42.248 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:42.253 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:42.253 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:42.253 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:42.258 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,77530326~3165641,81640613~5435401,87497139~15101632,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,145811637~14659451,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,332588308~827816,336572672~4979196,349881700~15605360,373691010~1041998,375410865~25931427,402573262~7364576,410712466~88886] 2024-09-21T10:17:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:42 smithi145 ceph-mon[89359]: pgmap v1231: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 123 KiB/s rd, 6.2 KiB/s wr, 1.73k op/s 2024-09-21T10:17:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:42 smithi145 ceph-mon[89359]: osdmap e999: 8 total, 8 up, 8 in 2024-09-21T10:17:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:42 smithi119 ceph-mon[105288]: pgmap v1231: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 123 KiB/s rd, 6.2 KiB/s wr, 1.73k op/s 2024-09-21T10:17:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:42 smithi119 ceph-mon[105288]: osdmap e999: 8 total, 8 up, 8 in 2024-09-21T10:17:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:42 smithi119 ceph-mon[101432]: pgmap v1231: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 123 KiB/s rd, 6.2 KiB/s wr, 1.73k op/s 2024-09-21T10:17:42.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:42 smithi119 ceph-mon[101432]: osdmap e999: 8 total, 8 up, 8 in 2024-09-21T10:17:43.219 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.219 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.236 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:43.236 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.236 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.241 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:43.242 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.242 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.247 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:43.247 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.247 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.252 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:43.252 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.253 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:43.258 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:43.258 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:43.258 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1074781,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:43.262 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,43960151~5598312,51254163~14609552,67276420~6830306,75497472~11578542,87497139~19370700,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,223107934~1074781,224247612~6073893,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~20878394,331817502~4621698,336572672~4979196,349881700~15605360,373691010~27651282,402573262~7364576,410712466~88886] 2024-09-21T10:17:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:43 smithi145 ceph-mon[89359]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-21T10:17:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:43 smithi145 ceph-mon[89359]: pgmap v1234: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 440 op/s 2024-09-21T10:17:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:43 smithi119 ceph-mon[105288]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-21T10:17:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:43 smithi119 ceph-mon[105288]: pgmap v1234: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 440 op/s 2024-09-21T10:17:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:43 smithi119 ceph-mon[101432]: osdmap e1000: 8 total, 8 up, 8 in 2024-09-21T10:17:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:43 smithi119 ceph-mon[101432]: pgmap v1234: 105 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 102 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 440 op/s 2024-09-21T10:17:44.195 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.195 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.216 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:44.217 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.217 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.222 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:44.222 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.222 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.227 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:44.227 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.227 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.232 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:44.233 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.233 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~19370700,107031566~187623,107841497~8162972,118600721~3501566,126747906~5659525,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~9812043,241574681~8083217,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,332588308~3850892,336572672~4979196,349881700~15605360,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:44.238 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~31370367,107031566~187623,107841497~8162972,117440512~4661775,126747906~5659525,133610121~4364681,138412032~2483773,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230873414~9812043,241574681~8083217,251658240~8092188,260046848~5685763,266388628~14170139,281018368~10181006,299831556~22038991,331817502~4621698,336572672~4979196,349881700~15605360,373691010~37871242] 2024-09-21T10:17:44.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:44 smithi119 ceph-mon[105288]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-21T10:17:44.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:44 smithi119 ceph-mon[105288]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-21T10:17:44.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:44 smithi119 ceph-mon[101432]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-21T10:17:44.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:44 smithi119 ceph-mon[101432]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-21T10:17:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:44 smithi145 ceph-mon[89359]: osdmap e1001: 8 total, 8 up, 8 in 2024-09-21T10:17:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:44 smithi145 ceph-mon[89359]: osdmap e1002: 8 total, 8 up, 8 in 2024-09-21T10:17:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:17:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:17:44] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.51.0" 2024-09-21T10:17:45.227 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.228 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.250 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:45.250 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.250 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.255 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:45.255 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.255 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.261 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:45.261 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.261 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12269805,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161966455~16158870,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,299831556~20878394,320886570~983977,324210499~7607003,332588308~3850892,336572672~4979196,349881700~20906995,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:45.266 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~12269805,38423654~11134809,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~17864752,161966455~16158870,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,299831556~22038991,324210499~12228701,336572672~4979196,349881700~20906995,373691010~37871242] 2024-09-21T10:17:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:45 smithi145 ceph-mon[89359]: pgmap v1237: 105 pgs: 105 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 89 KiB/s rd, 5.7 KiB/s wr, 1.14k op/s 2024-09-21T10:17:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:45 smithi119 ceph-mon[105288]: pgmap v1237: 105 pgs: 105 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 89 KiB/s rd, 5.7 KiB/s wr, 1.14k op/s 2024-09-21T10:17:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:45 smithi119 ceph-mon[101432]: pgmap v1237: 105 pgs: 105 active+clean; 1.0 GiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 89 KiB/s rd, 5.7 KiB/s wr, 1.14k op/s 2024-09-21T10:17:46.229 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:46.230 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:46.248 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:46.248 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:46.248 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:46.254 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:46.254 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:46.254 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~1786236,25823944~12599710,39443894~10114569,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,220548498~966115,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:46.259 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~21731603,23077889~1786236,25823944~23734519,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123829650~8577781,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,220548498~966115,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:46 smithi145 ceph-mon[89359]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-21T10:17:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:46 smithi119 ceph-mon[101432]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-21T10:17:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:46 smithi119 ceph-mon[105288]: osdmap e1003: 8 total, 8 up, 8 in 2024-09-21T10:17:47.246 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:47.246 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:47.258 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:47.258 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:47.258 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123829650~8577781,133610121~798745,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,296708045~24001905,320886570~983977,324210499~7607003,332588308~10911763,346583587~24205108,373691010~27651282,402396115~7541723,410461141~1101111] 2024-09-21T10:17:47.263 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~4661775,123559024~8848407,133610121~4364681,138412032~2483773,142606336~35518989,185246972~826100,186816738~19384597,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~10181006,296708045~25162502,324210499~19289572,346583587~24205108,373691010~37871242] 2024-09-21T10:17:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:47 smithi145 ceph-mon[89359]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-21T10:17:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:47 smithi145 ceph-mon[89359]: pgmap v1240: 105 pgs: 2 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 102 active+clean; 1003 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 92 KiB/s rd, 6.5 KiB/s wr, 1.24k op/s 2024-09-21T10:17:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:47 smithi119 ceph-mon[101432]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-21T10:17:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:47 smithi119 ceph-mon[101432]: pgmap v1240: 105 pgs: 2 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 102 active+clean; 1003 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 92 KiB/s rd, 6.5 KiB/s wr, 1.24k op/s 2024-09-21T10:17:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:47 smithi119 ceph-mon[105288]: osdmap e1004: 8 total, 8 up, 8 in 2024-09-21T10:17:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:47 smithi119 ceph-mon[105288]: pgmap v1240: 105 pgs: 2 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 102 active+clean; 1003 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 92 KiB/s rd, 6.5 KiB/s wr, 1.24k op/s 2024-09-21T10:17:48.247 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:48.247 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [1334775~21731603,23077889~15345765,39443894~11086531,51254163~14609552,69192860~4913866,76370336~10705678,87497139~28507330,118600721~3501566,123250767~308257,123829650~10579216,135450794~2524008,140433786~462019,144031262~16439826,161611977~16513348,185246972~826100,186816738~19384597,212176573~4482781,218818238~2696375,223107934~1139678,224922963~5398542,230873414~18784484,254681153~5069275,261441010~4291601,266388628~6010521,272525416~8033351,283728297~7471077,291722063~1204351,296708045~24001905,320886570~983977,323584036~8233466,332588308~10911763,346583587~24205108,371282199~30060093,402396115~7541723,410461141~1101111] 2024-09-21T10:17:48.260 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [1334775~49195650,51254163~14609552,67276420~6830306,75497472~40506997,117440512~20534290,138412032~2483773,142606336~35518989,184549376~21651959,209715200~6944154,218103808~3410805,222298112~8023393,230686720~18971178,251658240~8092188,260046848~20511919,281018368~11908046,296708045~25162502,322961408~20538663,346583587~64978665] 2024-09-21T10:17:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:48 smithi145 ceph-mon[89359]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-21T10:17:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:48 smithi119 ceph-mon[101432]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-21T10:17:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:48 smithi119 ceph-mon[105288]: osdmap e1005: 8 total, 8 up, 8 in 2024-09-21T10:17:48.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:17:49.211 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateStress (39913 ms) 2024-09-21T10:17:49.211 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParent 2024-09-21T10:17:49.211 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:49 smithi145 ceph-mon[89359]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-21T10:17:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:49 smithi145 ceph-mon[89359]: pgmap v1243: 105 pgs: 2 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 102 active+clean; 822 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 5.7 KiB/s wr, 445 op/s 2024-09-21T10:17:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:49 smithi145 ceph-mon[89359]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-21T10:17:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:49 smithi119 ceph-mon[101432]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-21T10:17:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:49 smithi119 ceph-mon[101432]: pgmap v1243: 105 pgs: 2 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 102 active+clean; 822 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 5.7 KiB/s wr, 445 op/s 2024-09-21T10:17:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:49 smithi119 ceph-mon[101432]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-21T10:17:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:49 smithi119 ceph-mon[105288]: osdmap e1006: 8 total, 8 up, 8 in 2024-09-21T10:17:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:49 smithi119 ceph-mon[105288]: pgmap v1243: 105 pgs: 2 active+clean+snaptrim, 1 active+clean+snaptrim_wait, 102 active+clean; 822 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 59 KiB/s rd, 5.7 KiB/s wr, 445 op/s 2024-09-21T10:17:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:49 smithi119 ceph-mon[105288]: osdmap e1007: 8 total, 8 up, 8 in 2024-09-21T10:17:50.393 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateParent (1183 ms) 2024-09-21T10:17:50.394 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateIgnoreParent 2024-09-21T10:17:50.394 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:17:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:17:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:17:51.410 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [707810~12937,9922753~76824,11418325~32584,12458297~21790,14282496~15281,15636722~54821,16531938~50230,17040867~11287,17338949~66835,20127530~72771] to clone 2024-09-21T10:17:51.410 INFO:tasks.workunit.client.0.smithi119.stdout: diff was [0~720747,8388608~4091479,12582912~3999256,16777216~3423085] 2024-09-21T10:17:51.414 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateIgnoreParent (1021 ms) 2024-09-21T10:17:51.414 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateCallbackError 2024-09-21T10:17:51.414 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:51.433 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [1147580~15559,3796467~100127,7039700~42092,7159224~63508,7925377~31610,8919257~39937,10261991~43802,14478606~14066,17460508~11465,20490435~1553] 2024-09-21T10:17:51.442 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateCallbackError (28 ms) 2024-09-21T10:17:51.442 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateParentDiscard 2024-09-21T10:17:51.442 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:51 smithi145 ceph-mon[89359]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-21T10:17:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:51 smithi145 ceph-mon[89359]: pgmap v1246: 105 pgs: 105 active+clean; 773 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 8.2 KiB/s wr, 431 op/s 2024-09-21T10:17:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:51 smithi119 ceph-mon[101432]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-21T10:17:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:51 smithi119 ceph-mon[101432]: pgmap v1246: 105 pgs: 105 active+clean; 773 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 8.2 KiB/s wr, 431 op/s 2024-09-21T10:17:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:51 smithi119 ceph-mon[105288]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-21T10:17:51.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:51 smithi119 ceph-mon[105288]: pgmap v1246: 105 pgs: 105 active+clean; 773 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 8.2 KiB/s wr, 431 op/s 2024-09-21T10:17:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:52 smithi145 ceph-mon[89359]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-21T10:17:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:52 smithi119 ceph-mon[105288]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-21T10:17:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:52 smithi119 ceph-mon[101432]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-21T10:17:53.410 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateParentDiscard (1968 ms) 2024-09-21T10:17:53.410 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnalignedSmall 2024-09-21T10:17:53.410 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:53.459 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnalignedSmall (48 ms) 2024-09-21T10:17:53.459 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateUnaligned 2024-09-21T10:17:53.459 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:53.532 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateUnaligned (74 ms) 2024-09-21T10:17:53.533 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateTryAcquireLock 2024-09-21T10:17:53.533 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:17:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:53 smithi145 ceph-mon[89359]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-21T10:17:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:53 smithi145 ceph-mon[89359]: pgmap v1249: 105 pgs: 105 active+clean; 773 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.2 KiB/s wr, 83 op/s 2024-09-21T10:17:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:53 smithi119 ceph-mon[105288]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-21T10:17:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:53 smithi119 ceph-mon[105288]: pgmap v1249: 105 pgs: 105 active+clean; 773 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.2 KiB/s wr, 83 op/s 2024-09-21T10:17:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:53 smithi119 ceph-mon[101432]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-21T10:17:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:53 smithi119 ceph-mon[101432]: pgmap v1249: 105 pgs: 105 active+clean; 773 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.2 KiB/s wr, 83 op/s 2024-09-21T10:17:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:54 smithi119 ceph-mon[101432]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-21T10:17:54.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:54 smithi119 ceph-mon[101432]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-21T10:17:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:54 smithi119 ceph-mon[105288]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-21T10:17:54.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:54 smithi119 ceph-mon[105288]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-21T10:17:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:54 smithi145 ceph-mon[89359]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-21T10:17:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:54 smithi145 ceph-mon[89359]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-21T10:17:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:17:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:17:54] "GET /metrics HTTP/1.1" 200 36715 "" "Prometheus/2.51.0" 2024-09-21T10:17:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:55 smithi119 ceph-mon[105288]: pgmap v1252: 105 pgs: 105 active+clean; 856 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 15 MiB/s wr, 331 op/s 2024-09-21T10:17:55.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:55 smithi119 ceph-mon[101432]: pgmap v1252: 105 pgs: 105 active+clean; 856 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 15 MiB/s wr, 331 op/s 2024-09-21T10:17:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:55 smithi145 ceph-mon[89359]: pgmap v1252: 105 pgs: 105 active+clean; 856 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.3 MiB/s rd, 15 MiB/s wr, 331 op/s 2024-09-21T10:17:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:57 smithi145 ceph-mon[89359]: pgmap v1253: 105 pgs: 105 active+clean; 862 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 285 op/s 2024-09-21T10:17:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:57 smithi119 ceph-mon[105288]: pgmap v1253: 105 pgs: 105 active+clean; 862 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 285 op/s 2024-09-21T10:17:58.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:57 smithi119 ceph-mon[101432]: pgmap v1253: 105 pgs: 105 active+clean; 862 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.9 MiB/s rd, 12 MiB/s wr, 285 op/s 2024-09-21T10:17:58.567 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateTryAcquireLock (5035 ms) 2024-09-21T10:17:58.567 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/0.DiffIterateStriping 2024-09-21T10:17:58.567 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:17:58.567 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/0.DiffIterateStriping (0 ms) 2024-09-21T10:17:58.787 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 19 tests from DiffIterateTest/0 (254359 ms total) 2024-09-21T10:17:58.787 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:17:58.787 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 19 tests from DiffIterateTest/1, where TypeParam = DiffIterateParams 2024-09-21T10:17:58.821 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterate 2024-09-21T10:18:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:59 smithi145 ceph-mon[89359]: pgmap v1254: 105 pgs: 105 active+clean; 870 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 254 op/s 2024-09-21T10:18:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:17:59 smithi145 ceph-mon[89359]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-21T10:18:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:59 smithi119 ceph-mon[105288]: pgmap v1254: 105 pgs: 105 active+clean; 870 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 254 op/s 2024-09-21T10:18:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:17:59 smithi119 ceph-mon[105288]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-21T10:18:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:59 smithi119 ceph-mon[101432]: pgmap v1254: 105 pgs: 105 active+clean; 870 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.3 MiB/s rd, 11 MiB/s wr, 254 op/s 2024-09-21T10:18:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:17:59 smithi119 ceph-mon[101432]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-21T10:18:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:18:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:18:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:00.793 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[101432]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-21T10:18:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2787303351' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2787303351' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[101432]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-21T10:18:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[105288]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-21T10:18:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2787303351' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2787303351' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:00 smithi119 ceph-mon[105288]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-21T10:18:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:00 smithi145 ceph-mon[89359]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-21T10:18:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2787303351' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-40","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:18:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2787303351' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-librbd-smithi119-84733-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:18:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:00 smithi145 ceph-mon[89359]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-21T10:18:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:01 smithi119 ceph-mon[101432]: pgmap v1257: 105 pgs: 3 creating+peering, 29 unknown, 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:18:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:01 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:18:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:01 smithi119 ceph-mon[101432]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-21T10:18:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:01 smithi119 ceph-mon[105288]: pgmap v1257: 105 pgs: 3 creating+peering, 29 unknown, 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:18:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:01 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:18:02.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:01 smithi119 ceph-mon[105288]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-21T10:18:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:01 smithi145 ceph-mon[89359]: pgmap v1257: 105 pgs: 3 creating+peering, 29 unknown, 73 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:18:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:01 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:18:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:01 smithi145 ceph-mon[89359]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-21T10:18:02.826 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [779379~83176,2752319~1633,4619817~45918,7041304~98484,7250279~11311,10263257~82463,12844509~31561,13739542~62331,16126962~80651,18441303~1159] 2024-09-21T10:18:03.811 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [0~20971520] 2024-09-21T10:18:03.820 INFO:tasks.workunit.client.0.smithi119.stdout: diff was [0~20971520] 2024-09-21T10:18:03.823 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterate (5003 ms) 2024-09-21T10:18:03.823 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministic 2024-09-21T10:18:03.824 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:03 smithi145 ceph-mon[89359]: pgmap v1260: 105 pgs: 3 creating+peering, 25 unknown, 77 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:18:04.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:03 smithi145 ceph-mon[89359]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-21T10:18:04.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:04.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:04.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[101432]: pgmap v1260: 105 pgs: 3 creating+peering, 25 unknown, 77 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:18:04.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[101432]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-21T10:18:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:04.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[105288]: pgmap v1260: 105 pgs: 3 creating+peering, 25 unknown, 77 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:18:04.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:03 smithi119 ceph-mon[105288]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-21T10:18:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:18:04] "GET /metrics HTTP/1.1" 200 36645 "" "Prometheus/2.51.0" 2024-09-21T10:18:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:18:04.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:18:04.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-mon[101432]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-21T10:18:04.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:18:04.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:18:04.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:04 smithi119 ceph-mon[105288]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-21T10:18:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:04 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:18:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:04 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:18:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:04 smithi145 ceph-mon[89359]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-21T10:18:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:05 smithi119 ceph-mon[101432]: pgmap v1263: 105 pgs: 3 creating+peering, 102 active+clean; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 148 KiB/s wr, 31 op/s 2024-09-21T10:18:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:05 smithi119 ceph-mon[101432]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-21T10:18:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:05 smithi119 ceph-mon[105288]: pgmap v1263: 105 pgs: 3 creating+peering, 102 active+clean; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 148 KiB/s wr, 31 op/s 2024-09-21T10:18:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:05 smithi119 ceph-mon[105288]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-21T10:18:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:05 smithi145 ceph-mon[89359]: pgmap v1263: 105 pgs: 3 creating+peering, 102 active+clean; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 148 KiB/s wr, 31 op/s 2024-09-21T10:18:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:05 smithi145 ceph-mon[89359]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-21T10:18:06.839 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:06 smithi119 ceph-mon[101432]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-21T10:18:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:06 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:18:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:06 smithi119 ceph-mon[105288]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-21T10:18:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:06 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:18:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:06 smithi145 ceph-mon[89359]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-21T10:18:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:06 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:18:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:07 smithi145 ceph-mon[89359]: pgmap v1266: 105 pgs: 105 active+clean; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 148 KiB/s wr, 31 op/s 2024-09-21T10:18:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:07 smithi145 ceph-mon[89359]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-21T10:18:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:07 smithi119 ceph-mon[101432]: pgmap v1266: 105 pgs: 105 active+clean; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 148 KiB/s wr, 31 op/s 2024-09-21T10:18:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:07 smithi119 ceph-mon[101432]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-21T10:18:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:07 smithi119 ceph-mon[105288]: pgmap v1266: 105 pgs: 105 active+clean; 13 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 148 KiB/s wr, 31 op/s 2024-09-21T10:18:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:07 smithi119 ceph-mon[105288]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-21T10:18:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:08 smithi119 ceph-mon[105288]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-21T10:18:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:08 smithi119 ceph-mon[101432]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-21T10:18:09.220 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:09.240 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:08 smithi145 ceph-mon[89359]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-21T10:18:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[101432]: pgmap v1269: 105 pgs: 105 active+clean; 15 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 142 KiB/s wr, 70 op/s 2024-09-21T10:18:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[101432]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-21T10:18:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[101432]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-21T10:18:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:10.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[105288]: pgmap v1269: 105 pgs: 105 active+clean; 15 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 142 KiB/s wr, 70 op/s 2024-09-21T10:18:10.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[105288]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-21T10:18:10.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[105288]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-21T10:18:10.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:10.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:10.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:09 smithi145 ceph-mon[89359]: pgmap v1269: 105 pgs: 105 active+clean; 15 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 42 KiB/s rd, 142 KiB/s wr, 70 op/s 2024-09-21T10:18:10.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:09 smithi145 ceph-mon[89359]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-21T10:18:10.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:09 smithi145 ceph-mon[89359]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-21T10:18:10.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:10.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:18:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:18:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:11 smithi145 ceph-mon[89359]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-21T10:18:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[105288]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[101432]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-21T10:18:11.648 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.649 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:11.649 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:18:11.649 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:18:11.649 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:12.232 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:12 smithi145 ceph-mon[89359]: pgmap v1273: 105 pgs: 105 active+clean; 17 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 149 KiB/s wr, 115 op/s 2024-09-21T10:18:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:12 smithi145 ceph-mon[89359]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-21T10:18:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:12 smithi119 ceph-mon[101432]: pgmap v1273: 105 pgs: 105 active+clean; 17 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 149 KiB/s wr, 115 op/s 2024-09-21T10:18:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:12 smithi119 ceph-mon[101432]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-21T10:18:12.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:12 smithi119 ceph-mon[105288]: pgmap v1273: 105 pgs: 105 active+clean; 17 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 149 KiB/s wr, 115 op/s 2024-09-21T10:18:12.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:12 smithi119 ceph-mon[105288]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-21T10:18:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:13 smithi145 ceph-mon[89359]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-21T10:18:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:13 smithi145 ceph-mon[89359]: pgmap v1276: 105 pgs: 105 active+clean; 17 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 3.6 KiB/s wr, 50 op/s 2024-09-21T10:18:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:13 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:13 smithi119 ceph-mon[101432]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-21T10:18:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:13 smithi119 ceph-mon[101432]: pgmap v1276: 105 pgs: 105 active+clean; 17 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 3.6 KiB/s wr, 50 op/s 2024-09-21T10:18:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:13 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:13.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:13 smithi119 ceph-mon[105288]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-21T10:18:13.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:13 smithi119 ceph-mon[105288]: pgmap v1276: 105 pgs: 105 active+clean; 17 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 3.6 KiB/s wr, 50 op/s 2024-09-21T10:18:13.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:13 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:14 smithi119 ceph-mon[105288]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-21T10:18:14.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:14 smithi119 ceph-mon[101432]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-21T10:18:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:14 smithi145 ceph-mon[89359]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-21T10:18:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:18:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:18:14] "GET /metrics HTTP/1.1" 200 36645 "" "Prometheus/2.51.0" 2024-09-21T10:18:15.255 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:15 smithi145 ceph-mon[89359]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-21T10:18:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:15 smithi145 ceph-mon[89359]: pgmap v1279: 105 pgs: 105 active+clean; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 11 KiB/s wr, 111 op/s 2024-09-21T10:18:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:15 smithi119 ceph-mon[105288]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-21T10:18:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:15 smithi119 ceph-mon[105288]: pgmap v1279: 105 pgs: 105 active+clean; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 11 KiB/s wr, 111 op/s 2024-09-21T10:18:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:15 smithi119 ceph-mon[101432]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-21T10:18:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:15 smithi119 ceph-mon[101432]: pgmap v1279: 105 pgs: 105 active+clean; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 11 KiB/s wr, 111 op/s 2024-09-21T10:18:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:16 smithi145 ceph-mon[89359]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-21T10:18:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:16 smithi119 ceph-mon[105288]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-21T10:18:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:16 smithi119 ceph-mon[101432]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-21T10:18:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:17 smithi145 ceph-mon[89359]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-21T10:18:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:17 smithi145 ceph-mon[89359]: pgmap v1282: 105 pgs: 105 active+clean; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 10 KiB/s wr, 104 op/s 2024-09-21T10:18:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:17 smithi119 ceph-mon[105288]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-21T10:18:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:17 smithi119 ceph-mon[105288]: pgmap v1282: 105 pgs: 105 active+clean; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 10 KiB/s wr, 104 op/s 2024-09-21T10:18:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:17 smithi119 ceph-mon[101432]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-21T10:18:17.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:17 smithi119 ceph-mon[101432]: pgmap v1282: 105 pgs: 105 active+clean; 21 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 72 KiB/s rd, 10 KiB/s wr, 104 op/s 2024-09-21T10:18:18.271 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministic (14448 ms) 2024-09-21T10:18:18.271 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicPP 2024-09-21T10:18:18.271 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:18 smithi145 ceph-mon[89359]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-21T10:18:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:18 smithi119 ceph-mon[105288]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-21T10:18:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:18 smithi119 ceph-mon[101432]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-21T10:18:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:18:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:19 smithi145 ceph-mon[89359]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-21T10:18:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:19 smithi145 ceph-mon[89359]: pgmap v1285: 105 pgs: 105 active+clean; 29 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 6.7 KiB/s wr, 90 op/s 2024-09-21T10:18:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:19 smithi119 ceph-mon[105288]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-21T10:18:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:19 smithi119 ceph-mon[105288]: pgmap v1285: 105 pgs: 105 active+clean; 29 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 6.7 KiB/s wr, 90 op/s 2024-09-21T10:18:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:19 smithi119 ceph-mon[101432]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-21T10:18:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:19 smithi119 ceph-mon[101432]: pgmap v1285: 105 pgs: 105 active+clean; 29 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 64 KiB/s rd, 6.7 KiB/s wr, 90 op/s 2024-09-21T10:18:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:20 smithi145 ceph-mon[89359]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-21T10:18:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:18:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:18:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:20.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:20 smithi119 ceph-mon[101432]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-21T10:18:20.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:20 smithi119 ceph-mon[105288]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-21T10:18:21.291 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:21 smithi145 ceph-mon[89359]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-21T10:18:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:21 smithi145 ceph-mon[89359]: pgmap v1288: 105 pgs: 105 active+clean; 29 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 KiB/s wr, 105 op/s 2024-09-21T10:18:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:21 smithi119 ceph-mon[105288]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-21T10:18:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:21 smithi119 ceph-mon[105288]: pgmap v1288: 105 pgs: 105 active+clean; 29 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 KiB/s wr, 105 op/s 2024-09-21T10:18:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:21 smithi119 ceph-mon[101432]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-21T10:18:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:21 smithi119 ceph-mon[101432]: pgmap v1288: 105 pgs: 105 active+clean; 29 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 70 KiB/s rd, 8.0 KiB/s wr, 105 op/s 2024-09-21T10:18:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:22 smithi145 ceph-mon[89359]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-21T10:18:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:22 smithi119 ceph-mon[105288]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-21T10:18:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:22 smithi119 ceph-mon[101432]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-21T10:18:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:23 smithi145 ceph-mon[89359]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-21T10:18:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:23 smithi145 ceph-mon[89359]: pgmap v1291: 105 pgs: 105 active+clean; 29 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-09-21T10:18:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:23 smithi119 ceph-mon[101432]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-21T10:18:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:23 smithi119 ceph-mon[101432]: pgmap v1291: 105 pgs: 105 active+clean; 29 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-09-21T10:18:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:23 smithi119 ceph-mon[105288]: osdmap e1037: 8 total, 8 up, 8 in 2024-09-21T10:18:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:23 smithi119 ceph-mon[105288]: pgmap v1291: 105 pgs: 105 active+clean; 29 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-09-21T10:18:24.306 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:24.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:24 smithi119 ceph-mon[101432]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-21T10:18:24.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:24 smithi119 ceph-mon[105288]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-21T10:18:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:24 smithi145 ceph-mon[89359]: osdmap e1038: 8 total, 8 up, 8 in 2024-09-21T10:18:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:18:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:18:24] "GET /metrics HTTP/1.1" 200 36699 "" "Prometheus/2.51.0" 2024-09-21T10:18:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:25 smithi145 ceph-mon[89359]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-21T10:18:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:25 smithi145 ceph-mon[89359]: pgmap v1294: 105 pgs: 105 active+clean; 35 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 8.2 KiB/s wr, 81 op/s 2024-09-21T10:18:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:25 smithi119 ceph-mon[101432]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-21T10:18:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:25 smithi119 ceph-mon[101432]: pgmap v1294: 105 pgs: 105 active+clean; 35 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 8.2 KiB/s wr, 81 op/s 2024-09-21T10:18:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:25 smithi119 ceph-mon[105288]: osdmap e1039: 8 total, 8 up, 8 in 2024-09-21T10:18:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:25 smithi119 ceph-mon[105288]: pgmap v1294: 105 pgs: 105 active+clean; 35 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 57 KiB/s rd, 8.2 KiB/s wr, 81 op/s 2024-09-21T10:18:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:26 smithi145 ceph-mon[89359]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-21T10:18:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:26 smithi119 ceph-mon[101432]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-21T10:18:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:26 smithi119 ceph-mon[105288]: osdmap e1040: 8 total, 8 up, 8 in 2024-09-21T10:18:27.343 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:27 smithi119 ceph-mon[101432]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-21T10:18:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:27 smithi119 ceph-mon[101432]: pgmap v1297: 105 pgs: 105 active+clean; 35 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 8.5 KiB/s wr, 85 op/s 2024-09-21T10:18:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:27 smithi119 ceph-mon[105288]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-21T10:18:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:27 smithi119 ceph-mon[105288]: pgmap v1297: 105 pgs: 105 active+clean; 35 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 8.5 KiB/s wr, 85 op/s 2024-09-21T10:18:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:27 smithi145 ceph-mon[89359]: osdmap e1041: 8 total, 8 up, 8 in 2024-09-21T10:18:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:27 smithi145 ceph-mon[89359]: pgmap v1297: 105 pgs: 105 active+clean; 35 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 59 KiB/s rd, 8.5 KiB/s wr, 85 op/s 2024-09-21T10:18:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:28 smithi119 ceph-mon[105288]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-21T10:18:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:28 smithi119 ceph-mon[101432]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-21T10:18:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:28 smithi145 ceph-mon[89359]: osdmap e1042: 8 total, 8 up, 8 in 2024-09-21T10:18:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:29 smithi119 ceph-mon[105288]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-21T10:18:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:29 smithi119 ceph-mon[105288]: pgmap v1300: 105 pgs: 105 active+clean; 38 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 49 KiB/s rd, 5.5 KiB/s wr, 72 op/s 2024-09-21T10:18:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:29 smithi119 ceph-mon[101432]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-21T10:18:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:29 smithi119 ceph-mon[101432]: pgmap v1300: 105 pgs: 105 active+clean; 38 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 49 KiB/s rd, 5.5 KiB/s wr, 72 op/s 2024-09-21T10:18:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:29 smithi145 ceph-mon[89359]: osdmap e1043: 8 total, 8 up, 8 in 2024-09-21T10:18:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:29 smithi145 ceph-mon[89359]: pgmap v1300: 105 pgs: 105 active+clean; 38 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 49 KiB/s rd, 5.5 KiB/s wr, 72 op/s 2024-09-21T10:18:30.360 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:30 smithi145 ceph-mon[89359]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-21T10:18:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:18:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:18:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:30 smithi119 ceph-mon[101432]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-21T10:18:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:30 smithi119 ceph-mon[105288]: osdmap e1044: 8 total, 8 up, 8 in 2024-09-21T10:18:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:31 smithi119 ceph-mon[105288]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-21T10:18:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:31 smithi119 ceph-mon[105288]: pgmap v1303: 105 pgs: 105 active+clean; 44 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 77 KiB/s rd, 9.0 KiB/s wr, 111 op/s 2024-09-21T10:18:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:31 smithi119 ceph-mon[101432]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-21T10:18:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:31 smithi119 ceph-mon[101432]: pgmap v1303: 105 pgs: 105 active+clean; 44 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 77 KiB/s rd, 9.0 KiB/s wr, 111 op/s 2024-09-21T10:18:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:31 smithi145 ceph-mon[89359]: osdmap e1045: 8 total, 8 up, 8 in 2024-09-21T10:18:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:31 smithi145 ceph-mon[89359]: pgmap v1303: 105 pgs: 105 active+clean; 44 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 77 KiB/s rd, 9.0 KiB/s wr, 111 op/s 2024-09-21T10:18:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:32 smithi119 ceph-mon[105288]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-21T10:18:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:32 smithi119 ceph-mon[101432]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-21T10:18:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:32 smithi145 ceph-mon[89359]: osdmap e1046: 8 total, 8 up, 8 in 2024-09-21T10:18:33.379 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicPP (15108 ms) 2024-09-21T10:18:33.379 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS1 2024-09-21T10:18:33.379 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:33 smithi119 ceph-mon[101432]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-21T10:18:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:33 smithi119 ceph-mon[101432]: pgmap v1306: 105 pgs: 105 active+clean; 44 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 4.5 KiB/s wr, 53 op/s 2024-09-21T10:18:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:33 smithi119 ceph-mon[105288]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-21T10:18:33.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:33.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:33 smithi119 ceph-mon[105288]: pgmap v1306: 105 pgs: 105 active+clean; 44 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 4.5 KiB/s wr, 53 op/s 2024-09-21T10:18:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:33 smithi145 ceph-mon[89359]: osdmap e1047: 8 total, 8 up, 8 in 2024-09-21T10:18:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:33 smithi145 ceph-mon[89359]: pgmap v1306: 105 pgs: 105 active+clean; 44 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 4.5 KiB/s wr, 53 op/s 2024-09-21T10:18:34.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:34 smithi119 ceph-mon[101432]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-21T10:18:34.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:18:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:18:34] "GET /metrics HTTP/1.1" 200 36705 "" "Prometheus/2.51.0" 2024-09-21T10:18:34.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:34 smithi119 ceph-mon[105288]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-21T10:18:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:34 smithi145 ceph-mon[89359]: osdmap e1048: 8 total, 8 up, 8 in 2024-09-21T10:18:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:35 smithi119 ceph-mon[101432]: pgmap v1308: 105 pgs: 105 active+clean; 55 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 52 KiB/s rd, 7.0 KiB/s wr, 80 op/s 2024-09-21T10:18:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:35 smithi119 ceph-mon[105288]: pgmap v1308: 105 pgs: 105 active+clean; 55 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 52 KiB/s rd, 7.0 KiB/s wr, 80 op/s 2024-09-21T10:18:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:35 smithi145 ceph-mon[89359]: pgmap v1308: 105 pgs: 105 active+clean; 55 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 52 KiB/s rd, 7.0 KiB/s wr, 80 op/s 2024-09-21T10:18:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:37 smithi145 ceph-mon[89359]: pgmap v1309: 105 pgs: 105 active+clean; 55 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 5.5 KiB/s wr, 66 op/s 2024-09-21T10:18:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:37 smithi119 ceph-mon[105288]: pgmap v1309: 105 pgs: 105 active+clean; 55 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 5.5 KiB/s wr, 66 op/s 2024-09-21T10:18:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:37 smithi119 ceph-mon[101432]: pgmap v1309: 105 pgs: 105 active+clean; 55 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 5.5 KiB/s wr, 66 op/s 2024-09-21T10:18:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:38 smithi145 ceph-mon[89359]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-21T10:18:39.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:38 smithi119 ceph-mon[105288]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-21T10:18:39.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:38 smithi119 ceph-mon[101432]: osdmap e1049: 8 total, 8 up, 8 in 2024-09-21T10:18:39.830 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:39 smithi119 ceph-mon[101432]: pgmap v1311: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 647 KiB/s wr, 64 op/s 2024-09-21T10:18:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:39 smithi119 ceph-mon[101432]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-21T10:18:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:39 smithi119 ceph-mon[105288]: pgmap v1311: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 647 KiB/s wr, 64 op/s 2024-09-21T10:18:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:39 smithi119 ceph-mon[105288]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-21T10:18:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:39 smithi145 ceph-mon[89359]: pgmap v1311: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 647 KiB/s wr, 64 op/s 2024-09-21T10:18:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:39 smithi145 ceph-mon[89359]: osdmap e1050: 8 total, 8 up, 8 in 2024-09-21T10:18:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:18:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:18:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:41.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:40 smithi119 ceph-mon[101432]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-21T10:18:41.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:40 smithi119 ceph-mon[105288]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-21T10:18:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:40 smithi145 ceph-mon[89359]: osdmap e1051: 8 total, 8 up, 8 in 2024-09-21T10:18:42.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:41 smithi119 ceph-mon[101432]: pgmap v1314: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 684 KiB/s wr, 16 op/s 2024-09-21T10:18:42.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:41 smithi119 ceph-mon[105288]: pgmap v1314: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 684 KiB/s wr, 16 op/s 2024-09-21T10:18:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:41 smithi145 ceph-mon[89359]: pgmap v1314: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 684 KiB/s wr, 16 op/s 2024-09-21T10:18:44.096 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:43 smithi145 ceph-mon[89359]: pgmap v1315: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 685 KiB/s wr, 15 op/s 2024-09-21T10:18:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:43 smithi119 ceph-mon[105288]: pgmap v1315: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 685 KiB/s wr, 15 op/s 2024-09-21T10:18:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:43 smithi119 ceph-mon[101432]: pgmap v1315: 105 pgs: 105 active+clean; 59 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 685 KiB/s wr, 15 op/s 2024-09-21T10:18:44.810 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:18:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:18:44] "GET /metrics HTTP/1.1" 200 36705 "" "Prometheus/2.51.0" 2024-09-21T10:18:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:44 smithi119 ceph-mon[105288]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-21T10:18:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:44 smithi119 ceph-mon[101432]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-21T10:18:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:44 smithi145 ceph-mon[89359]: osdmap e1052: 8 total, 8 up, 8 in 2024-09-21T10:18:45.853 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:45 smithi119 ceph-mon[105288]: pgmap v1317: 105 pgs: 105 active+clean; 63 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:18:46.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:45 smithi119 ceph-mon[105288]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-21T10:18:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:45 smithi119 ceph-mon[101432]: pgmap v1317: 105 pgs: 105 active+clean; 63 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:18:46.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:45 smithi119 ceph-mon[101432]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-21T10:18:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:45 smithi145 ceph-mon[89359]: pgmap v1317: 105 pgs: 105 active+clean; 63 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:18:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:45 smithi145 ceph-mon[89359]: osdmap e1053: 8 total, 8 up, 8 in 2024-09-21T10:18:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:46 smithi119 ceph-mon[105288]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-21T10:18:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:46 smithi119 ceph-mon[101432]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-21T10:18:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:46 smithi145 ceph-mon[89359]: osdmap e1054: 8 total, 8 up, 8 in 2024-09-21T10:18:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:47 smithi119 ceph-mon[101432]: pgmap v1320: 105 pgs: 105 active+clean; 63 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:18:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:47 smithi119 ceph-mon[105288]: pgmap v1320: 105 pgs: 105 active+clean; 63 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:18:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:47 smithi145 ceph-mon[89359]: pgmap v1320: 105 pgs: 105 active+clean; 63 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 686 KiB/s wr, 36 op/s 2024-09-21T10:18:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:18:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:49 smithi119 ceph-mon[101432]: pgmap v1321: 105 pgs: 105 active+clean; 65 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 688 KiB/s wr, 67 op/s 2024-09-21T10:18:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:49 smithi119 ceph-mon[105288]: pgmap v1321: 105 pgs: 105 active+clean; 65 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 688 KiB/s wr, 67 op/s 2024-09-21T10:18:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:49 smithi145 ceph-mon[89359]: pgmap v1321: 105 pgs: 105 active+clean; 65 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 46 KiB/s rd, 688 KiB/s wr, 67 op/s 2024-09-21T10:18:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:18:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:18:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:18:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:50 smithi119 ceph-mon[101432]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-21T10:18:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:50 smithi119 ceph-mon[105288]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-21T10:18:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:50 smithi145 ceph-mon[89359]: osdmap e1055: 8 total, 8 up, 8 in 2024-09-21T10:18:51.890 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:51 smithi119 ceph-mon[105288]: pgmap v1323: 105 pgs: 105 active+clean; 65 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 KiB/s wr, 39 op/s 2024-09-21T10:18:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:51 smithi119 ceph-mon[105288]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-21T10:18:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:51 smithi119 ceph-mon[101432]: pgmap v1323: 105 pgs: 105 active+clean; 65 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 KiB/s wr, 39 op/s 2024-09-21T10:18:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:51 smithi119 ceph-mon[101432]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-21T10:18:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:51 smithi145 ceph-mon[89359]: pgmap v1323: 105 pgs: 105 active+clean; 65 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 KiB/s wr, 39 op/s 2024-09-21T10:18:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:51 smithi145 ceph-mon[89359]: osdmap e1056: 8 total, 8 up, 8 in 2024-09-21T10:18:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:52 smithi119 ceph-mon[105288]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-21T10:18:53.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:52 smithi119 ceph-mon[101432]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-21T10:18:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:52 smithi145 ceph-mon[89359]: osdmap e1057: 8 total, 8 up, 8 in 2024-09-21T10:18:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:53 smithi145 ceph-mon[89359]: pgmap v1326: 105 pgs: 105 active+clean; 70 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:18:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:53 smithi119 ceph-mon[105288]: pgmap v1326: 105 pgs: 105 active+clean; 70 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:18:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:53 smithi119 ceph-mon[101432]: pgmap v1326: 105 pgs: 105 active+clean; 70 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:18:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:18:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:18:54] "GET /metrics HTTP/1.1" 200 36702 "" "Prometheus/2.51.0" 2024-09-21T10:18:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:55 smithi145 ceph-mon[89359]: pgmap v1327: 105 pgs: 105 active+clean; 71 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:18:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:55 smithi119 ceph-mon[105288]: pgmap v1327: 105 pgs: 105 active+clean; 71 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:18:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:55 smithi119 ceph-mon[101432]: pgmap v1327: 105 pgs: 105 active+clean; 71 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 43 op/s 2024-09-21T10:18:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:56 smithi145 ceph-mon[89359]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-21T10:18:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:56 smithi119 ceph-mon[101432]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-21T10:18:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:56 smithi119 ceph-mon[105288]: osdmap e1058: 8 total, 8 up, 8 in 2024-09-21T10:18:57.925 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:18:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:57 smithi145 ceph-mon[89359]: pgmap v1329: 105 pgs: 105 active+clean; 71 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:18:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:57 smithi145 ceph-mon[89359]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-21T10:18:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:57 smithi119 ceph-mon[105288]: pgmap v1329: 105 pgs: 105 active+clean; 71 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:18:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:57 smithi119 ceph-mon[105288]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-21T10:18:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:57 smithi119 ceph-mon[101432]: pgmap v1329: 105 pgs: 105 active+clean; 71 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:18:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:57 smithi119 ceph-mon[101432]: osdmap e1059: 8 total, 8 up, 8 in 2024-09-21T10:18:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:58 smithi145 ceph-mon[89359]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-21T10:18:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:58 smithi119 ceph-mon[105288]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-21T10:18:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:58 smithi119 ceph-mon[101432]: osdmap e1060: 8 total, 8 up, 8 in 2024-09-21T10:19:00.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:18:59 smithi145 ceph-mon[89359]: pgmap v1332: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:19:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:18:59 smithi119 ceph-mon[101432]: pgmap v1332: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:19:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:18:59 smithi119 ceph-mon[105288]: pgmap v1332: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 34 KiB/s rd, 687 KiB/s wr, 50 op/s 2024-09-21T10:19:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:19:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:19:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:02.169 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:01 smithi119 ceph-mon[101432]: pgmap v1333: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:19:02.169 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:01 smithi119 ceph-mon[105288]: pgmap v1333: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:19:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:01 smithi145 ceph-mon[89359]: pgmap v1333: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 686 KiB/s wr, 41 op/s 2024-09-21T10:19:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:02 smithi145 ceph-mon[89359]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-21T10:19:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:02 smithi119 ceph-mon[101432]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-21T10:19:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:02 smithi119 ceph-mon[105288]: osdmap e1061: 8 total, 8 up, 8 in 2024-09-21T10:19:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:03.945 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS1 (30566 ms) 2024-09-21T10:19:03.945 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS1PP 2024-09-21T10:19:03.945 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:03 smithi145 ceph-mon[89359]: pgmap v1335: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:03 smithi145 ceph-mon[89359]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-21T10:19:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:03 smithi119 ceph-mon[101432]: pgmap v1335: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:03 smithi119 ceph-mon[101432]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-21T10:19:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:03 smithi119 ceph-mon[105288]: pgmap v1335: 105 pgs: 105 active+clean; 77 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:03 smithi119 ceph-mon[105288]: osdmap e1062: 8 total, 8 up, 8 in 2024-09-21T10:19:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:19:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:19:04] "GET /metrics HTTP/1.1" 200 36708 "" "Prometheus/2.51.0" 2024-09-21T10:19:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:04 smithi145 ceph-mon[89359]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-21T10:19:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:04 smithi119 ceph-mon[101432]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-21T10:19:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:04 smithi119 ceph-mon[105288]: osdmap e1063: 8 total, 8 up, 8 in 2024-09-21T10:19:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:05 smithi145 ceph-mon[89359]: pgmap v1338: 105 pgs: 105 active+clean; 89 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 688 KiB/s wr, 48 op/s 2024-09-21T10:19:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:05 smithi119 ceph-mon[105288]: pgmap v1338: 105 pgs: 105 active+clean; 89 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 688 KiB/s wr, 48 op/s 2024-09-21T10:19:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:05 smithi119 ceph-mon[101432]: pgmap v1338: 105 pgs: 105 active+clean; 89 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 688 KiB/s wr, 48 op/s 2024-09-21T10:19:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:07 smithi145 ceph-mon[89359]: pgmap v1339: 105 pgs: 105 active+clean; 89 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 19 op/s 2024-09-21T10:19:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:07 smithi119 ceph-mon[105288]: pgmap v1339: 105 pgs: 105 active+clean; 89 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 19 op/s 2024-09-21T10:19:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:07 smithi119 ceph-mon[101432]: pgmap v1339: 105 pgs: 105 active+clean; 89 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 685 KiB/s wr, 19 op/s 2024-09-21T10:19:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:08 smithi145 ceph-mon[89359]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-21T10:19:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:08 smithi119 ceph-mon[105288]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-21T10:19:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:08 smithi119 ceph-mon[101432]: osdmap e1064: 8 total, 8 up, 8 in 2024-09-21T10:19:09.975 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:10.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:09 smithi145 ceph-mon[89359]: pgmap v1341: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-09-21T10:19:10.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:09 smithi145 ceph-mon[89359]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-21T10:19:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:09 smithi119 ceph-mon[105288]: pgmap v1341: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-09-21T10:19:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:09 smithi119 ceph-mon[105288]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-21T10:19:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:09 smithi119 ceph-mon[101432]: pgmap v1341: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 1.3 MiB/s wr, 46 op/s 2024-09-21T10:19:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:09 smithi119 ceph-mon[101432]: osdmap e1065: 8 total, 8 up, 8 in 2024-09-21T10:19:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:19:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:19:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:10 smithi145 ceph-mon[89359]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-21T10:19:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:10 smithi119 ceph-mon[101432]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-21T10:19:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:10 smithi119 ceph-mon[105288]: osdmap e1066: 8 total, 8 up, 8 in 2024-09-21T10:19:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:19:12.247 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:11 smithi145 ceph-mon[89359]: pgmap v1344: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 685 KiB/s wr, 27 op/s 2024-09-21T10:19:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:11 smithi119 ceph-mon[105288]: pgmap v1344: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 685 KiB/s wr, 27 op/s 2024-09-21T10:19:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:11 smithi119 ceph-mon[101432]: pgmap v1344: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 685 KiB/s wr, 27 op/s 2024-09-21T10:19:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:13 smithi145 ceph-mon[89359]: pgmap v1345: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 686 KiB/s wr, 48 op/s 2024-09-21T10:19:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:13 smithi119 ceph-mon[105288]: pgmap v1345: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 686 KiB/s wr, 48 op/s 2024-09-21T10:19:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:13 smithi119 ceph-mon[101432]: pgmap v1345: 105 pgs: 105 active+clean; 93 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 686 KiB/s wr, 48 op/s 2024-09-21T10:19:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:19:14] "GET /metrics HTTP/1.1" 200 36708 "" "Prometheus/2.51.0" 2024-09-21T10:19:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:14 smithi145 ceph-mon[89359]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-21T10:19:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-mon[101432]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-21T10:19:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-mon[105288]: osdmap e1067: 8 total, 8 up, 8 in 2024-09-21T10:19:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.002 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:16.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: pgmap v1347: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:19:16.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-21T10:19:16.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:16.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:16.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: pgmap v1347: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:19:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-21T10:19:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: pgmap v1347: 105 pgs: 105 active+clean; 97 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: osdmap e1068: 8 total, 8 up, 8 in 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:19:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:19:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:16 smithi145 ceph-mon[89359]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-21T10:19:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:16 smithi119 ceph-mon[105288]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-21T10:19:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:16 smithi119 ceph-mon[101432]: osdmap e1069: 8 total, 8 up, 8 in 2024-09-21T10:19:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:17 smithi145 ceph-mon[89359]: pgmap v1350: 105 pgs: 105 active+clean; 100 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:19:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:17 smithi119 ceph-mon[105288]: pgmap v1350: 105 pgs: 105 active+clean; 100 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:19:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:17 smithi119 ceph-mon[101432]: pgmap v1350: 105 pgs: 105 active+clean; 100 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 26 KiB/s rd, 686 KiB/s wr, 39 op/s 2024-09-21T10:19:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:19 smithi145 ceph-mon[89359]: pgmap v1351: 105 pgs: 105 active+clean; 103 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 688 KiB/s wr, 57 op/s 2024-09-21T10:19:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:19:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:19:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:19 smithi119 ceph-mon[105288]: pgmap v1351: 105 pgs: 105 active+clean; 103 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 688 KiB/s wr, 57 op/s 2024-09-21T10:19:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:19 smithi119 ceph-mon[101432]: pgmap v1351: 105 pgs: 105 active+clean; 103 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 688 KiB/s wr, 57 op/s 2024-09-21T10:19:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:21 smithi145 ceph-mon[89359]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-21T10:19:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:21 smithi119 ceph-mon[105288]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-21T10:19:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:21 smithi119 ceph-mon[101432]: osdmap e1070: 8 total, 8 up, 8 in 2024-09-21T10:19:22.034 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:22 smithi145 ceph-mon[89359]: pgmap v1353: 105 pgs: 105 active+clean; 103 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 KiB/s wr, 39 op/s 2024-09-21T10:19:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:22 smithi145 ceph-mon[89359]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-21T10:19:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:22 smithi119 ceph-mon[101432]: pgmap v1353: 105 pgs: 105 active+clean; 103 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 KiB/s wr, 39 op/s 2024-09-21T10:19:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:22 smithi119 ceph-mon[101432]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-21T10:19:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:22 smithi119 ceph-mon[105288]: pgmap v1353: 105 pgs: 105 active+clean; 103 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 3.7 KiB/s wr, 39 op/s 2024-09-21T10:19:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:22 smithi119 ceph-mon[105288]: osdmap e1071: 8 total, 8 up, 8 in 2024-09-21T10:19:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:23 smithi145 ceph-mon[89359]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-21T10:19:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:23 smithi119 ceph-mon[105288]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-21T10:19:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:23 smithi119 ceph-mon[101432]: osdmap e1072: 8 total, 8 up, 8 in 2024-09-21T10:19:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:24 smithi145 ceph-mon[89359]: pgmap v1356: 105 pgs: 105 active+clean; 106 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 3.8 KiB/s wr, 41 op/s 2024-09-21T10:19:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:24 smithi119 ceph-mon[101432]: pgmap v1356: 105 pgs: 105 active+clean; 106 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 3.8 KiB/s wr, 41 op/s 2024-09-21T10:19:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:24 smithi119 ceph-mon[105288]: pgmap v1356: 105 pgs: 105 active+clean; 106 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 3.8 KiB/s wr, 41 op/s 2024-09-21T10:19:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:19:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:19:24] "GET /metrics HTTP/1.1" 200 36709 "" "Prometheus/2.51.0" 2024-09-21T10:19:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:26 smithi145 ceph-mon[89359]: pgmap v1357: 105 pgs: 105 active+clean; 113 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:26 smithi119 ceph-mon[105288]: pgmap v1357: 105 pgs: 105 active+clean; 113 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:26 smithi119 ceph-mon[101432]: pgmap v1357: 105 pgs: 105 active+clean; 113 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:27 smithi145 ceph-mon[89359]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-21T10:19:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:27 smithi119 ceph-mon[105288]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-21T10:19:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:27 smithi119 ceph-mon[101432]: osdmap e1073: 8 total, 8 up, 8 in 2024-09-21T10:19:28.071 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:28 smithi145 ceph-mon[89359]: pgmap v1359: 105 pgs: 105 active+clean; 113 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:28 smithi145 ceph-mon[89359]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-21T10:19:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:28 smithi119 ceph-mon[105288]: pgmap v1359: 105 pgs: 105 active+clean; 113 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:28 smithi119 ceph-mon[105288]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-21T10:19:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:28 smithi119 ceph-mon[101432]: pgmap v1359: 105 pgs: 105 active+clean; 113 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:28 smithi119 ceph-mon[101432]: osdmap e1074: 8 total, 8 up, 8 in 2024-09-21T10:19:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:29 smithi119 ceph-mon[105288]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-21T10:19:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:29 smithi119 ceph-mon[101432]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-21T10:19:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:29 smithi145 ceph-mon[89359]: osdmap e1075: 8 total, 8 up, 8 in 2024-09-21T10:19:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:30 smithi119 ceph-mon[101432]: pgmap v1362: 105 pgs: 105 active+clean; 117 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-09-21T10:19:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:30 smithi119 ceph-mon[105288]: pgmap v1362: 105 pgs: 105 active+clean; 117 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-09-21T10:19:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:30 smithi145 ceph-mon[89359]: pgmap v1362: 105 pgs: 105 active+clean; 117 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 44 KiB/s rd, 1.3 MiB/s wr, 63 op/s 2024-09-21T10:19:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:19:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:19:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:32 smithi119 ceph-mon[105288]: pgmap v1363: 105 pgs: 105 active+clean; 123 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:19:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:32 smithi119 ceph-mon[101432]: pgmap v1363: 105 pgs: 105 active+clean; 123 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:19:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:32 smithi145 ceph-mon[89359]: pgmap v1363: 105 pgs: 105 active+clean; 123 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 42 op/s 2024-09-21T10:19:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:33 smithi119 ceph-mon[101432]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-21T10:19:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:33 smithi119 ceph-mon[105288]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-21T10:19:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:33 smithi145 ceph-mon[89359]: osdmap e1076: 8 total, 8 up, 8 in 2024-09-21T10:19:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:34.096 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS1PP (30151 ms) 2024-09-21T10:19:34.097 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS2 2024-09-21T10:19:34.097 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:34 smithi119 ceph-mon[101432]: pgmap v1365: 105 pgs: 105 active+clean; 123 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:34 smithi119 ceph-mon[101432]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-21T10:19:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:34 smithi119 ceph-mon[105288]: pgmap v1365: 105 pgs: 105 active+clean; 123 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:34 smithi119 ceph-mon[105288]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-21T10:19:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:34 smithi145 ceph-mon[89359]: pgmap v1365: 105 pgs: 105 active+clean; 123 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 687 KiB/s wr, 41 op/s 2024-09-21T10:19:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:34 smithi145 ceph-mon[89359]: osdmap e1077: 8 total, 8 up, 8 in 2024-09-21T10:19:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:19:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:19:34] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:19:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:35 smithi119 ceph-mon[105288]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-21T10:19:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:35 smithi119 ceph-mon[101432]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-21T10:19:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:35 smithi145 ceph-mon[89359]: osdmap e1078: 8 total, 8 up, 8 in 2024-09-21T10:19:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:36 smithi119 ceph-mon[101432]: pgmap v1368: 105 pgs: 105 active+clean; 131 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 5.5 KiB/s wr, 45 op/s 2024-09-21T10:19:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:36 smithi119 ceph-mon[105288]: pgmap v1368: 105 pgs: 105 active+clean; 131 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 5.5 KiB/s wr, 45 op/s 2024-09-21T10:19:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:36 smithi145 ceph-mon[89359]: pgmap v1368: 105 pgs: 105 active+clean; 131 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 30 KiB/s rd, 5.5 KiB/s wr, 45 op/s 2024-09-21T10:19:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:38 smithi119 ceph-mon[105288]: pgmap v1369: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-09-21T10:19:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:38 smithi119 ceph-mon[101432]: pgmap v1369: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-09-21T10:19:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:38 smithi145 ceph-mon[89359]: pgmap v1369: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 19 KiB/s rd, 686 KiB/s wr, 28 op/s 2024-09-21T10:19:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:40 smithi119 ceph-mon[101432]: pgmap v1370: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 616 KiB/s wr, 37 op/s 2024-09-21T10:19:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:40 smithi119 ceph-mon[105288]: pgmap v1370: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 616 KiB/s wr, 37 op/s 2024-09-21T10:19:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:40 smithi145 ceph-mon[89359]: pgmap v1370: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 25 KiB/s rd, 616 KiB/s wr, 37 op/s 2024-09-21T10:19:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:19:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:19:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:42 smithi119 ceph-mon[101432]: pgmap v1371: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 515 KiB/s wr, 31 op/s 2024-09-21T10:19:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:42 smithi119 ceph-mon[105288]: pgmap v1371: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 515 KiB/s wr, 31 op/s 2024-09-21T10:19:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:42 smithi145 ceph-mon[89359]: pgmap v1371: 105 pgs: 105 active+clean; 135 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 21 KiB/s rd, 515 KiB/s wr, 31 op/s 2024-09-21T10:19:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:43 smithi119 ceph-mon[101432]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-21T10:19:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:43 smithi119 ceph-mon[105288]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-21T10:19:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:43 smithi145 ceph-mon[89359]: osdmap e1079: 8 total, 8 up, 8 in 2024-09-21T10:19:44.136 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:44 smithi119 ceph-mon[105288]: pgmap v1373: 105 pgs: 105 active+clean; 139 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 946 KiB/s wr, 24 op/s 2024-09-21T10:19:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:44 smithi119 ceph-mon[105288]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-21T10:19:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:44 smithi119 ceph-mon[101432]: pgmap v1373: 105 pgs: 105 active+clean; 139 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 946 KiB/s wr, 24 op/s 2024-09-21T10:19:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:44 smithi119 ceph-mon[101432]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-21T10:19:44.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:44 smithi145 ceph-mon[89359]: pgmap v1373: 105 pgs: 105 active+clean; 139 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 946 KiB/s wr, 24 op/s 2024-09-21T10:19:44.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:44 smithi145 ceph-mon[89359]: osdmap e1080: 8 total, 8 up, 8 in 2024-09-21T10:19:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:19:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:19:44] "GET /metrics HTTP/1.1" 200 36710 "" "Prometheus/2.51.0" 2024-09-21T10:19:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:45 smithi119 ceph-mon[105288]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-21T10:19:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:45 smithi119 ceph-mon[101432]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-21T10:19:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:45 smithi145 ceph-mon[89359]: osdmap e1081: 8 total, 8 up, 8 in 2024-09-21T10:19:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:46 smithi119 ceph-mon[101432]: pgmap v1376: 105 pgs: 105 active+clean; 143 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 23 op/s 2024-09-21T10:19:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:46 smithi119 ceph-mon[105288]: pgmap v1376: 105 pgs: 105 active+clean; 143 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 23 op/s 2024-09-21T10:19:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:46 smithi145 ceph-mon[89359]: pgmap v1376: 105 pgs: 105 active+clean; 143 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 1.3 MiB/s wr, 23 op/s 2024-09-21T10:19:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:48 smithi119 ceph-mon[101432]: pgmap v1377: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:19:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:48 smithi119 ceph-mon[105288]: pgmap v1377: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:19:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:48 smithi145 ceph-mon[89359]: pgmap v1377: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:19:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:19:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:50 smithi145 ceph-mon[89359]: pgmap v1378: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.4 MiB/s wr, 41 op/s 2024-09-21T10:19:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:19:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:19:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:19:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:50 smithi119 ceph-mon[101432]: pgmap v1378: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.4 MiB/s wr, 41 op/s 2024-09-21T10:19:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:50 smithi119 ceph-mon[105288]: pgmap v1378: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.4 MiB/s wr, 41 op/s 2024-09-21T10:19:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:52 smithi145 ceph-mon[89359]: pgmap v1379: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:19:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:52 smithi119 ceph-mon[105288]: pgmap v1379: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:19:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:52 smithi119 ceph-mon[101432]: pgmap v1379: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 1.5 MiB/s wr, 34 op/s 2024-09-21T10:19:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:53 smithi145 ceph-mon[89359]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-21T10:19:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:53 smithi119 ceph-mon[105288]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-21T10:19:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:53 smithi119 ceph-mon[101432]: osdmap e1082: 8 total, 8 up, 8 in 2024-09-21T10:19:54.181 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:19:54.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:54 smithi119 ceph-mon[101432]: pgmap v1381: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 948 KiB/s wr, 15 op/s 2024-09-21T10:19:54.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:54 smithi119 ceph-mon[101432]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-21T10:19:54.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:54 smithi119 ceph-mon[105288]: pgmap v1381: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 948 KiB/s wr, 15 op/s 2024-09-21T10:19:54.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:54 smithi119 ceph-mon[105288]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-21T10:19:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:54 smithi145 ceph-mon[89359]: pgmap v1381: 105 pgs: 105 active+clean; 151 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 948 KiB/s wr, 15 op/s 2024-09-21T10:19:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:54 smithi145 ceph-mon[89359]: osdmap e1083: 8 total, 8 up, 8 in 2024-09-21T10:19:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:19:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:19:54] "GET /metrics HTTP/1.1" 200 36706 "" "Prometheus/2.51.0" 2024-09-21T10:19:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:55 smithi145 ceph-mon[89359]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-21T10:19:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:55 smithi119 ceph-mon[105288]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-21T10:19:55.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:55 smithi119 ceph-mon[101432]: osdmap e1084: 8 total, 8 up, 8 in 2024-09-21T10:19:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:56 smithi145 ceph-mon[89359]: pgmap v1384: 105 pgs: 105 active+clean; 168 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:19:56.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:56 smithi119 ceph-mon[105288]: pgmap v1384: 105 pgs: 105 active+clean; 168 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:19:56.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:56 smithi119 ceph-mon[101432]: pgmap v1384: 105 pgs: 105 active+clean; 168 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 23 op/s 2024-09-21T10:19:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:19:58 smithi145 ceph-mon[89359]: pgmap v1385: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:19:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:19:58 smithi119 ceph-mon[101432]: pgmap v1385: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:19:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:19:58 smithi119 ceph-mon[105288]: pgmap v1385: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 24 op/s 2024-09-21T10:20:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:00 smithi145 ceph-mon[89359]: pgmap v1386: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.4 MiB/s wr, 41 op/s 2024-09-21T10:20:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:00 smithi145 ceph-mon[89359]: overall HEALTH_OK 2024-09-21T10:20:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:20:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:20:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:00 smithi119 ceph-mon[101432]: pgmap v1386: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.4 MiB/s wr, 41 op/s 2024-09-21T10:20:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:00 smithi119 ceph-mon[101432]: overall HEALTH_OK 2024-09-21T10:20:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:00 smithi119 ceph-mon[105288]: pgmap v1386: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 27 KiB/s rd, 2.4 MiB/s wr, 41 op/s 2024-09-21T10:20:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:00 smithi119 ceph-mon[105288]: overall HEALTH_OK 2024-09-21T10:20:02.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:02 smithi119 ceph-mon[101432]: pgmap v1387: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-09-21T10:20:02.420 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:02 smithi119 ceph-mon[105288]: pgmap v1387: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-09-21T10:20:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:02 smithi145 ceph-mon[89359]: pgmap v1387: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-09-21T10:20:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:03 smithi145 ceph-mon[89359]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-21T10:20:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:03 smithi119 ceph-mon[105288]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-21T10:20:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:03 smithi119 ceph-mon[101432]: osdmap e1085: 8 total, 8 up, 8 in 2024-09-21T10:20:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:04.220 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:20:04.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:04 smithi119 ceph-mon[101432]: pgmap v1389: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 478 KiB/s wr, 15 op/s 2024-09-21T10:20:04.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:04 smithi119 ceph-mon[101432]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-21T10:20:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:04 smithi119 ceph-mon[105288]: pgmap v1389: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 478 KiB/s wr, 15 op/s 2024-09-21T10:20:04.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:04 smithi119 ceph-mon[105288]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-21T10:20:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:04 smithi145 ceph-mon[89359]: pgmap v1389: 105 pgs: 105 active+clean; 169 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 478 KiB/s wr, 15 op/s 2024-09-21T10:20:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:04 smithi145 ceph-mon[89359]: osdmap e1086: 8 total, 8 up, 8 in 2024-09-21T10:20:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:20:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:20:04] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.51.0" 2024-09-21T10:20:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:05 smithi145 ceph-mon[89359]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-21T10:20:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:05 smithi119 ceph-mon[101432]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-21T10:20:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:05 smithi119 ceph-mon[105288]: osdmap e1087: 8 total, 8 up, 8 in 2024-09-21T10:20:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:06 smithi145 ceph-mon[89359]: pgmap v1392: 105 pgs: 105 active+clean; 177 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:20:06.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:06 smithi119 ceph-mon[105288]: pgmap v1392: 105 pgs: 105 active+clean; 177 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:20:06.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:06 smithi119 ceph-mon[101432]: pgmap v1392: 105 pgs: 105 active+clean; 177 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 1.3 MiB/s wr, 15 op/s 2024-09-21T10:20:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:08 smithi145 ceph-mon[89359]: pgmap v1393: 105 pgs: 105 active+clean; 186 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:20:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:08 smithi119 ceph-mon[101432]: pgmap v1393: 105 pgs: 105 active+clean; 186 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:20:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:08 smithi119 ceph-mon[105288]: pgmap v1393: 105 pgs: 105 active+clean; 186 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 28 op/s 2024-09-21T10:20:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:10 smithi145 ceph-mon[89359]: pgmap v1394: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.4 MiB/s wr, 42 op/s 2024-09-21T10:20:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:20:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:20:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:10 smithi119 ceph-mon[105288]: pgmap v1394: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.4 MiB/s wr, 42 op/s 2024-09-21T10:20:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:10 smithi119 ceph-mon[101432]: pgmap v1394: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 28 KiB/s rd, 2.4 MiB/s wr, 42 op/s 2024-09-21T10:20:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:11 smithi145 ceph-mon[89359]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-21T10:20:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:11 smithi145 ceph-mon[89359]: pgmap v1396: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.1 MiB/s wr, 36 op/s 2024-09-21T10:20:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:11 smithi119 ceph-mon[101432]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-21T10:20:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:11 smithi119 ceph-mon[101432]: pgmap v1396: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.1 MiB/s wr, 36 op/s 2024-09-21T10:20:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:11 smithi119 ceph-mon[105288]: osdmap e1088: 8 total, 8 up, 8 in 2024-09-21T10:20:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:11 smithi119 ceph-mon[105288]: pgmap v1396: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 24 KiB/s rd, 2.1 MiB/s wr, 36 op/s 2024-09-21T10:20:12.261 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:20:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:12 smithi145 ceph-mon[89359]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-21T10:20:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:12 smithi119 ceph-mon[105288]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-21T10:20:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:12 smithi119 ceph-mon[101432]: osdmap e1089: 8 total, 8 up, 8 in 2024-09-21T10:20:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:13 smithi145 ceph-mon[89359]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-21T10:20:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:13 smithi145 ceph-mon[89359]: pgmap v1399: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 2.3 KiB/s wr, 18 op/s 2024-09-21T10:20:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:13 smithi119 ceph-mon[101432]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-21T10:20:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:13 smithi119 ceph-mon[101432]: pgmap v1399: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 2.3 KiB/s wr, 18 op/s 2024-09-21T10:20:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:13 smithi119 ceph-mon[105288]: osdmap e1090: 8 total, 8 up, 8 in 2024-09-21T10:20:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:13 smithi119 ceph-mon[105288]: pgmap v1399: 105 pgs: 105 active+clean; 187 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 2.3 KiB/s wr, 18 op/s 2024-09-21T10:20:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:20:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:20:14] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.51.0" 2024-09-21T10:20:16.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:15 smithi119 ceph-mon[105288]: pgmap v1400: 105 pgs: 105 active+clean; 205 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 45 op/s 2024-09-21T10:20:16.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:16.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:15 smithi119 ceph-mon[101432]: pgmap v1400: 105 pgs: 105 active+clean; 205 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 45 op/s 2024-09-21T10:20:16.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:16.207 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:15 smithi145 ceph-mon[89359]: pgmap v1400: 105 pgs: 105 active+clean; 205 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 2.7 MiB/s wr, 45 op/s 2024-09-21T10:20:16.207 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:20:17.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:17 smithi119 ceph-mon[101432]: pgmap v1401: 105 pgs: 105 active+clean; 205 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.5 MiB/s wr, 42 op/s 2024-09-21T10:20:17.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:18.087 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:17 smithi145 ceph-mon[89359]: pgmap v1401: 105 pgs: 105 active+clean; 205 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.5 MiB/s wr, 42 op/s 2024-09-21T10:20:18.087 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:17 smithi119 ceph-mon[105288]: pgmap v1401: 105 pgs: 105 active+clean; 205 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.5 MiB/s wr, 42 op/s 2024-09-21T10:20:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:19 smithi119 ceph-mon[105288]: pgmap v1402: 105 pgs: 105 active+clean; 205 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-09-21T10:20:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:19 smithi119 ceph-mon[101432]: pgmap v1402: 105 pgs: 105 active+clean; 205 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-09-21T10:20:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:19 smithi145 ceph-mon[89359]: pgmap v1402: 105 pgs: 105 active+clean; 205 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.0 MiB/s wr, 34 op/s 2024-09-21T10:20:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:19 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:19 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:20:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[101432]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[105288]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-21T10:20:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-mon[89359]: osdmap e1091: 8 total, 8 up, 8 in 2024-09-21T10:20:21.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:20:21.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:20:21.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:20:21.862 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS2 (47765 ms) 2024-09-21T10:20:21.862 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDeterministicLUKS2PP 2024-09-21T10:20:21.862 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:20:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:21 smithi119 ceph-mon[105288]: pgmap v1404: 105 pgs: 105 active+clean; 209 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 38 op/s 2024-09-21T10:20:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:21 smithi119 ceph-mon[105288]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-21T10:20:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:21 smithi119 ceph-mon[101432]: pgmap v1404: 105 pgs: 105 active+clean; 209 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 38 op/s 2024-09-21T10:20:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:21 smithi119 ceph-mon[101432]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-21T10:20:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:21 smithi145 ceph-mon[89359]: pgmap v1404: 105 pgs: 105 active+clean; 209 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 2.4 MiB/s wr, 38 op/s 2024-09-21T10:20:22.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:21 smithi145 ceph-mon[89359]: osdmap e1092: 8 total, 8 up, 8 in 2024-09-21T10:20:23.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:22 smithi119 ceph-mon[105288]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-21T10:20:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:22 smithi119 ceph-mon[101432]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-21T10:20:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:22 smithi145 ceph-mon[89359]: osdmap e1093: 8 total, 8 up, 8 in 2024-09-21T10:20:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:23 smithi119 ceph-mon[105288]: pgmap v1407: 105 pgs: 105 active+clean; 209 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:20:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:23 smithi119 ceph-mon[101432]: pgmap v1407: 105 pgs: 105 active+clean; 209 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:20:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:23 smithi145 ceph-mon[89359]: pgmap v1407: 105 pgs: 105 active+clean; 209 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 684 KiB/s wr, 10 op/s 2024-09-21T10:20:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:20:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:20:24] "GET /metrics HTTP/1.1" 200 36713 "" "Prometheus/2.51.0" 2024-09-21T10:20:26.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:25 smithi119 ceph-mon[101432]: pgmap v1408: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:26.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:25 smithi119 ceph-mon[105288]: pgmap v1408: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:25 smithi145 ceph-mon[89359]: pgmap v1408: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:27 smithi119 ceph-mon[105288]: pgmap v1409: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-09-21T10:20:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:27 smithi119 ceph-mon[101432]: pgmap v1409: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-09-21T10:20:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:27 smithi145 ceph-mon[89359]: pgmap v1409: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.7 MiB/s wr, 39 op/s 2024-09-21T10:20:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:29 smithi119 ceph-mon[101432]: pgmap v1410: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-09-21T10:20:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:29 smithi119 ceph-mon[105288]: pgmap v1410: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-09-21T10:20:30.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:29 smithi145 ceph-mon[89359]: pgmap v1410: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 1.5 MiB/s wr, 28 op/s 2024-09-21T10:20:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:20:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:20:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:31 smithi145 ceph-mon[89359]: pgmap v1411: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-09-21T10:20:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:31 smithi145 ceph-mon[89359]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-21T10:20:32.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:20:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=cleanup t=2024-09-21T10:20:31.650993352Z level=info msg="Completed cleanup jobs" duration=12.881554ms 2024-09-21T10:20:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:31 smithi119 ceph-mon[101432]: pgmap v1411: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-09-21T10:20:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:31 smithi119 ceph-mon[101432]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-21T10:20:32.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:31 smithi119 ceph-mon[105288]: pgmap v1411: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-09-21T10:20:32.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:31 smithi119 ceph-mon[105288]: osdmap e1094: 8 total, 8 up, 8 in 2024-09-21T10:20:32.904 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:20:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:32 smithi145 ceph-mon[89359]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-21T10:20:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:32 smithi119 ceph-mon[101432]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-21T10:20:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:32 smithi119 ceph-mon[105288]: osdmap e1095: 8 total, 8 up, 8 in 2024-09-21T10:20:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:33 smithi145 ceph-mon[89359]: pgmap v1414: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-21T10:20:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:33 smithi145 ceph-mon[89359]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-21T10:20:34.315 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:33 smithi119 ceph-mon[101432]: pgmap v1414: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-21T10:20:34.315 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:33 smithi119 ceph-mon[101432]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-21T10:20:34.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:33 smithi119 ceph-mon[105288]: pgmap v1414: 105 pgs: 105 active+clean; 229 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 511 B/s wr, 3 op/s 2024-09-21T10:20:34.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:33 smithi119 ceph-mon[105288]: osdmap e1096: 8 total, 8 up, 8 in 2024-09-21T10:20:34.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:20:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:20:34] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.51.0" 2024-09-21T10:20:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:35 smithi145 ceph-mon[89359]: pgmap v1416: 105 pgs: 105 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:35 smithi119 ceph-mon[101432]: pgmap v1416: 105 pgs: 105 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:36.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:35 smithi119 ceph-mon[105288]: pgmap v1416: 105 pgs: 105 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:37 smithi145 ceph-mon[89359]: pgmap v1417: 105 pgs: 105 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:37 smithi119 ceph-mon[105288]: pgmap v1417: 105 pgs: 105 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:37 smithi119 ceph-mon[101432]: pgmap v1417: 105 pgs: 105 active+clean; 245 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 2.7 MiB/s wr, 41 op/s 2024-09-21T10:20:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:39 smithi145 ceph-mon[89359]: pgmap v1418: 105 pgs: 105 active+clean; 253 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 MiB/s wr, 40 op/s 2024-09-21T10:20:40.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:39 smithi145 ceph-mon[89359]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-21T10:20:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:39 smithi119 ceph-mon[105288]: pgmap v1418: 105 pgs: 105 active+clean; 253 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 MiB/s wr, 40 op/s 2024-09-21T10:20:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:39 smithi119 ceph-mon[105288]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-21T10:20:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:39 smithi119 ceph-mon[101432]: pgmap v1418: 105 pgs: 105 active+clean; 253 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 3.0 MiB/s wr, 40 op/s 2024-09-21T10:20:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:39 smithi119 ceph-mon[101432]: osdmap e1097: 8 total, 8 up, 8 in 2024-09-21T10:20:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:20:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:20:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:40.935 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:20:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:40 smithi145 ceph-mon[89359]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-21T10:20:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:40 smithi119 ceph-mon[101432]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-21T10:20:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:40 smithi119 ceph-mon[105288]: osdmap e1098: 8 total, 8 up, 8 in 2024-09-21T10:20:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:41 smithi145 ceph-mon[89359]: pgmap v1421: 105 pgs: 105 active+clean; 261 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-09-21T10:20:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:41 smithi145 ceph-mon[89359]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-21T10:20:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:41 smithi119 ceph-mon[101432]: pgmap v1421: 105 pgs: 105 active+clean; 261 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-09-21T10:20:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:41 smithi119 ceph-mon[101432]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-21T10:20:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:41 smithi119 ceph-mon[105288]: pgmap v1421: 105 pgs: 105 active+clean; 261 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 4.1 MiB/s wr, 40 op/s 2024-09-21T10:20:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:41 smithi119 ceph-mon[105288]: osdmap e1099: 8 total, 8 up, 8 in 2024-09-21T10:20:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:43 smithi145 ceph-mon[89359]: pgmap v1423: 105 pgs: 105 active+clean; 261 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-09-21T10:20:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:43 smithi119 ceph-mon[101432]: pgmap v1423: 105 pgs: 105 active+clean; 261 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-09-21T10:20:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:43 smithi119 ceph-mon[105288]: pgmap v1423: 105 pgs: 105 active+clean; 261 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 23 KiB/s rd, 2.7 MiB/s wr, 32 op/s 2024-09-21T10:20:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:20:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:20:44] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.51.0" 2024-09-21T10:20:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:45 smithi145 ceph-mon[89359]: pgmap v1424: 105 pgs: 105 active+clean; 267 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 39 op/s 2024-09-21T10:20:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:45 smithi119 ceph-mon[105288]: pgmap v1424: 105 pgs: 105 active+clean; 267 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 39 op/s 2024-09-21T10:20:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:45 smithi119 ceph-mon[101432]: pgmap v1424: 105 pgs: 105 active+clean; 267 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 39 op/s 2024-09-21T10:20:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:47 smithi145 ceph-mon[89359]: pgmap v1425: 105 pgs: 105 active+clean; 267 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2024-09-21T10:20:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:47 smithi119 ceph-mon[101432]: pgmap v1425: 105 pgs: 105 active+clean; 267 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2024-09-21T10:20:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:47 smithi119 ceph-mon[105288]: pgmap v1425: 105 pgs: 105 active+clean; 267 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2024-09-21T10:20:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:20:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:49 smithi145 ceph-mon[89359]: pgmap v1426: 105 pgs: 105 active+clean; 275 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 MiB/s wr, 31 op/s 2024-09-21T10:20:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:49 smithi145 ceph-mon[89359]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-21T10:20:50.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:49 smithi119 ceph-mon[101432]: pgmap v1426: 105 pgs: 105 active+clean; 275 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 MiB/s wr, 31 op/s 2024-09-21T10:20:50.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:49 smithi119 ceph-mon[101432]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-21T10:20:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:49 smithi119 ceph-mon[105288]: pgmap v1426: 105 pgs: 105 active+clean; 275 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 MiB/s wr, 31 op/s 2024-09-21T10:20:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:49 smithi119 ceph-mon[105288]: osdmap e1100: 8 total, 8 up, 8 in 2024-09-21T10:20:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:20:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:20:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:20:50.959 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:20:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:50 smithi145 ceph-mon[89359]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-21T10:20:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:50 smithi119 ceph-mon[101432]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-21T10:20:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:50 smithi119 ceph-mon[105288]: osdmap e1101: 8 total, 8 up, 8 in 2024-09-21T10:20:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:51 smithi145 ceph-mon[89359]: pgmap v1429: 105 pgs: 105 active+clean; 275 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 1.0 MiB/s wr, 18 op/s 2024-09-21T10:20:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:51 smithi145 ceph-mon[89359]: osdmap e1102: 8 total, 8 up, 8 in 2024-09-21T10:20:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:51 smithi119 ceph-mon[101432]: pgmap v1429: 105 pgs: 105 active+clean; 275 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 1.0 MiB/s wr, 18 op/s 2024-09-21T10:20:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:51 smithi119 ceph-mon[101432]: osdmap e1102: 8 total, 8 up, 8 in 2024-09-21T10:20:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:51 smithi119 ceph-mon[105288]: pgmap v1429: 105 pgs: 105 active+clean; 275 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 1.0 MiB/s wr, 18 op/s 2024-09-21T10:20:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:51 smithi119 ceph-mon[105288]: osdmap e1102: 8 total, 8 up, 8 in 2024-09-21T10:20:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:53 smithi145 ceph-mon[89359]: pgmap v1431: 105 pgs: 105 active+clean; 283 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 8 op/s 2024-09-21T10:20:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:53 smithi119 ceph-mon[105288]: pgmap v1431: 105 pgs: 105 active+clean; 283 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 8 op/s 2024-09-21T10:20:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:53 smithi119 ceph-mon[101432]: pgmap v1431: 105 pgs: 105 active+clean; 283 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 8 op/s 2024-09-21T10:20:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:20:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:20:54] "GET /metrics HTTP/1.1" 200 36712 "" "Prometheus/2.51.0" 2024-09-21T10:20:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:55 smithi145 ceph-mon[89359]: pgmap v1432: 105 pgs: 105 active+clean; 289 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 39 op/s 2024-09-21T10:20:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:55 smithi119 ceph-mon[105288]: pgmap v1432: 105 pgs: 105 active+clean; 289 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 39 op/s 2024-09-21T10:20:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:55 smithi119 ceph-mon[101432]: pgmap v1432: 105 pgs: 105 active+clean; 289 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 39 op/s 2024-09-21T10:20:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:57 smithi145 ceph-mon[89359]: pgmap v1433: 105 pgs: 105 active+clean; 289 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2024-09-21T10:20:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:57 smithi119 ceph-mon[101432]: pgmap v1433: 105 pgs: 105 active+clean; 289 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2024-09-21T10:20:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:57 smithi119 ceph-mon[105288]: pgmap v1433: 105 pgs: 105 active+clean; 289 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2024-09-21T10:21:00.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:59 smithi145 ceph-mon[89359]: pgmap v1434: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 2.3 MiB/s wr, 31 op/s 2024-09-21T10:21:00.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:20:59 smithi145 ceph-mon[89359]: osdmap e1103: 8 total, 8 up, 8 in 2024-09-21T10:21:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:59 smithi119 ceph-mon[105288]: pgmap v1434: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 2.3 MiB/s wr, 31 op/s 2024-09-21T10:21:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:20:59 smithi119 ceph-mon[105288]: osdmap e1103: 8 total, 8 up, 8 in 2024-09-21T10:21:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:59 smithi119 ceph-mon[101432]: pgmap v1434: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 2.3 MiB/s wr, 31 op/s 2024-09-21T10:21:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:20:59 smithi119 ceph-mon[101432]: osdmap e1103: 8 total, 8 up, 8 in 2024-09-21T10:21:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:21:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:21:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:00.991 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:21:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:00 smithi145 ceph-mon[89359]: osdmap e1104: 8 total, 8 up, 8 in 2024-09-21T10:21:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:00 smithi119 ceph-mon[105288]: osdmap e1104: 8 total, 8 up, 8 in 2024-09-21T10:21:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:00 smithi119 ceph-mon[101432]: osdmap e1104: 8 total, 8 up, 8 in 2024-09-21T10:21:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:01 smithi145 ceph-mon[89359]: pgmap v1437: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-09-21T10:21:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:01 smithi145 ceph-mon[89359]: osdmap e1105: 8 total, 8 up, 8 in 2024-09-21T10:21:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:01 smithi119 ceph-mon[101432]: pgmap v1437: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-09-21T10:21:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:01 smithi119 ceph-mon[101432]: osdmap e1105: 8 total, 8 up, 8 in 2024-09-21T10:21:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:01 smithi119 ceph-mon[105288]: pgmap v1437: 105 pgs: 105 active+clean; 301 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-09-21T10:21:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:01 smithi119 ceph-mon[105288]: osdmap e1105: 8 total, 8 up, 8 in 2024-09-21T10:21:03.268 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:03.269 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:03 smithi145 ceph-mon[89359]: pgmap v1439: 105 pgs: 105 active+clean; 305 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 7 op/s 2024-09-21T10:21:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:03 smithi119 ceph-mon[105288]: pgmap v1439: 105 pgs: 105 active+clean; 305 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 7 op/s 2024-09-21T10:21:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:03 smithi119 ceph-mon[101432]: pgmap v1439: 105 pgs: 105 active+clean; 305 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 7 op/s 2024-09-21T10:21:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:21:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:21:04] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.51.0" 2024-09-21T10:21:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:05 smithi145 ceph-mon[89359]: pgmap v1440: 105 pgs: 105 active+clean; 311 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 37 op/s 2024-09-21T10:21:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:05 smithi119 ceph-mon[101432]: pgmap v1440: 105 pgs: 105 active+clean; 311 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 37 op/s 2024-09-21T10:21:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:05 smithi119 ceph-mon[105288]: pgmap v1440: 105 pgs: 105 active+clean; 311 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 687 KiB/s wr, 37 op/s 2024-09-21T10:21:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:07 smithi145 ceph-mon[89359]: pgmap v1441: 105 pgs: 105 active+clean; 311 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 527 KiB/s wr, 30 op/s 2024-09-21T10:21:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:07 smithi119 ceph-mon[101432]: pgmap v1441: 105 pgs: 105 active+clean; 311 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 527 KiB/s wr, 30 op/s 2024-09-21T10:21:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:07 smithi119 ceph-mon[105288]: pgmap v1441: 105 pgs: 105 active+clean; 311 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 527 KiB/s wr, 30 op/s 2024-09-21T10:21:10.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:09 smithi145 ceph-mon[89359]: pgmap v1442: 105 pgs: 105 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 MiB/s wr, 30 op/s 2024-09-21T10:21:10.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:09 smithi145 ceph-mon[89359]: osdmap e1106: 8 total, 8 up, 8 in 2024-09-21T10:21:10.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:21:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:21:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:09 smithi119 ceph-mon[105288]: pgmap v1442: 105 pgs: 105 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 MiB/s wr, 30 op/s 2024-09-21T10:21:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:09 smithi119 ceph-mon[105288]: osdmap e1106: 8 total, 8 up, 8 in 2024-09-21T10:21:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:09 smithi119 ceph-mon[101432]: pgmap v1442: 105 pgs: 105 active+clean; 319 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 1.4 MiB/s wr, 30 op/s 2024-09-21T10:21:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:09 smithi119 ceph-mon[101432]: osdmap e1106: 8 total, 8 up, 8 in 2024-09-21T10:21:11.037 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDeterministicLUKS2PP (49176 ms) 2024-09-21T10:21:11.037 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateDiscard 2024-09-21T10:21:11.037 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:21:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:10 smithi145 ceph-mon[89359]: osdmap e1107: 8 total, 8 up, 8 in 2024-09-21T10:21:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:10 smithi119 ceph-mon[101432]: osdmap e1107: 8 total, 8 up, 8 in 2024-09-21T10:21:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:10 smithi119 ceph-mon[105288]: osdmap e1107: 8 total, 8 up, 8 in 2024-09-21T10:21:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:12 smithi145 ceph-mon[89359]: pgmap v1445: 105 pgs: 105 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-09-21T10:21:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:12 smithi145 ceph-mon[89359]: osdmap e1108: 8 total, 8 up, 8 in 2024-09-21T10:21:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:12 smithi119 ceph-mon[105288]: pgmap v1445: 105 pgs: 105 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-09-21T10:21:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:12 smithi119 ceph-mon[105288]: osdmap e1108: 8 total, 8 up, 8 in 2024-09-21T10:21:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:12 smithi119 ceph-mon[101432]: pgmap v1445: 105 pgs: 105 active+clean; 331 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 1.5 MiB/s wr, 38 op/s 2024-09-21T10:21:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:12 smithi119 ceph-mon[101432]: osdmap e1108: 8 total, 8 up, 8 in 2024-09-21T10:21:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:13 smithi145 ceph-mon[89359]: osdmap e1109: 8 total, 8 up, 8 in 2024-09-21T10:21:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:13 smithi119 ceph-mon[105288]: osdmap e1109: 8 total, 8 up, 8 in 2024-09-21T10:21:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:13 smithi119 ceph-mon[101432]: osdmap e1109: 8 total, 8 up, 8 in 2024-09-21T10:21:14.046 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateDiscard (3008 ms) 2024-09-21T10:21:14.046 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateTruncate 2024-09-21T10:21:14.046 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:21:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:14 smithi145 ceph-mon[89359]: pgmap v1448: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T10:21:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:14 smithi145 ceph-mon[89359]: osdmap e1110: 8 total, 8 up, 8 in 2024-09-21T10:21:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:14 smithi119 ceph-mon[101432]: pgmap v1448: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T10:21:14.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:14 smithi119 ceph-mon[101432]: osdmap e1110: 8 total, 8 up, 8 in 2024-09-21T10:21:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:14 smithi119 ceph-mon[105288]: pgmap v1448: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 6.2 KiB/s rd, 2.0 MiB/s wr, 10 op/s 2024-09-21T10:21:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:14 smithi119 ceph-mon[105288]: osdmap e1110: 8 total, 8 up, 8 in 2024-09-21T10:21:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:21:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:21:14] "GET /metrics HTTP/1.1" 200 36719 "" "Prometheus/2.51.0" 2024-09-21T10:21:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:15 smithi145 ceph-mon[89359]: osdmap e1111: 8 total, 8 up, 8 in 2024-09-21T10:21:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:15 smithi119 ceph-mon[105288]: osdmap e1111: 8 total, 8 up, 8 in 2024-09-21T10:21:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:15 smithi119 ceph-mon[101432]: osdmap e1111: 8 total, 8 up, 8 in 2024-09-21T10:21:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:16 smithi145 ceph-mon[89359]: pgmap v1451: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-09-21T10:21:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:16 smithi145 ceph-mon[89359]: osdmap e1112: 8 total, 8 up, 8 in 2024-09-21T10:21:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:16 smithi119 ceph-mon[105288]: pgmap v1451: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-09-21T10:21:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:16 smithi119 ceph-mon[105288]: osdmap e1112: 8 total, 8 up, 8 in 2024-09-21T10:21:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:16 smithi119 ceph-mon[101432]: pgmap v1451: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-09-21T10:21:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:16 smithi119 ceph-mon[101432]: osdmap e1112: 8 total, 8 up, 8 in 2024-09-21T10:21:17.085 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateTruncate (3039 ms) 2024-09-21T10:21:17.085 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateWriteAndTruncate 2024-09-21T10:21:17.085 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:21:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:17 smithi145 ceph-mon[89359]: osdmap e1113: 8 total, 8 up, 8 in 2024-09-21T10:21:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:17 smithi119 ceph-mon[105288]: osdmap e1113: 8 total, 8 up, 8 in 2024-09-21T10:21:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:17 smithi119 ceph-mon[101432]: osdmap e1113: 8 total, 8 up, 8 in 2024-09-21T10:21:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:18 smithi119 ceph-mon[105288]: pgmap v1454: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 12 KiB/s wr, 93 op/s 2024-09-21T10:21:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:18 smithi119 ceph-mon[105288]: osdmap e1114: 8 total, 8 up, 8 in 2024-09-21T10:21:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:18 smithi119 ceph-mon[101432]: pgmap v1454: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 12 KiB/s wr, 93 op/s 2024-09-21T10:21:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:18 smithi119 ceph-mon[101432]: osdmap e1114: 8 total, 8 up, 8 in 2024-09-21T10:21:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:18 smithi145 ceph-mon[89359]: pgmap v1454: 105 pgs: 105 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 12 KiB/s wr, 93 op/s 2024-09-21T10:21:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:18 smithi145 ceph-mon[89359]: osdmap e1114: 8 total, 8 up, 8 in 2024-09-21T10:21:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:19 smithi119 ceph-mon[105288]: osdmap e1115: 8 total, 8 up, 8 in 2024-09-21T10:21:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:19 smithi119 ceph-mon[101432]: osdmap e1115: 8 total, 8 up, 8 in 2024-09-21T10:21:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:19 smithi145 ceph-mon[89359]: osdmap e1115: 8 total, 8 up, 8 in 2024-09-21T10:21:20.116 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateWriteAndTruncate (3032 ms) 2024-09-21T10:21:20.116 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStress 2024-09-21T10:21:20.116 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:21:20.252 INFO:tasks.workunit.client.0.smithi119.stdout: i=0 exists [29981650~6718567,153570589~3933400,161557009~1622158,169003728~798567,235482698~5119978,393686301~7562933,408086187~7779835] wrote [29981650~6718567,153570589~3933400,161557009~1622158,169003728~798567,235482698~5119978,393686301~7562933,408086187~7779835] 2024-09-21T10:21:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:20 smithi119 ceph-mon[101432]: pgmap v1457: 105 pgs: 105 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 392 KiB/s wr, 114 op/s 2024-09-21T10:21:20.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:20 smithi119 ceph-mon[101432]: osdmap e1116: 8 total, 8 up, 8 in 2024-09-21T10:21:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:20 smithi119 ceph-mon[105288]: pgmap v1457: 105 pgs: 105 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 392 KiB/s wr, 114 op/s 2024-09-21T10:21:20.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:20 smithi119 ceph-mon[105288]: osdmap e1116: 8 total, 8 up, 8 in 2024-09-21T10:21:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:20 smithi145 ceph-mon[89359]: pgmap v1457: 105 pgs: 105 active+clean; 337 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 73 KiB/s rd, 392 KiB/s wr, 114 op/s 2024-09-21T10:21:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:20 smithi145 ceph-mon[89359]: osdmap e1116: 8 total, 8 up, 8 in 2024-09-21T10:21:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:21:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:21:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:21.154 INFO:tasks.workunit.client.0.smithi119.stdout: i=1 exists [19939885~2180127,23795927~12904290,52460119~2707469,117632188~3523956,139346825~3893399,153570589~3933400,161557009~1622158,169003728~798567,206554317~7565484,228146076~6383951,235482698~5119978,322115027~4377816,379176051~2904412,393686301~7562933,401854282~135665,408086187~7779835] wrote [19939885~2180127,23795927~8184429,52460119~2707469,117632188~3523956,139346825~3893399,206554317~7565484,228146076~6383951,322115027~4377816,379176051~2904412,401854282~135665] 2024-09-21T10:21:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:21 smithi145 ceph-mon[89359]: osdmap e1117: 8 total, 8 up, 8 in 2024-09-21T10:21:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:21 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:21.352 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[105288]: osdmap e1117: 8 total, 8 up, 8 in 2024-09-21T10:21:21.352 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:21.352 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:21.352 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:21.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[101432]: osdmap e1117: 8 total, 8 up, 8 in 2024-09-21T10:21:21.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:21:21.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:21.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:21 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:22.158 INFO:tasks.workunit.client.0.smithi119.stdout: i=2 exists [15418248~7029160,23795927~12904290,52460119~6023091,88164909~449586,117632188~3523956,139346825~3893399,153570589~3933400,161557009~1622158,168126409~3826441,197446742~1889394,206554317~7565484,228146076~6383951,235482698~5119978,242834170~826705,298603335~6158547,322115027~4377816,332714618~7607330,379176051~2904412,383803862~6494145,393686301~7562933,401854282~135665,408086187~7779835] wrote [15418248~7029160,53329564~5153646,88164909~449586,168126409~3826441,197446742~1889394,242834170~826705,298603335~6158547,332714618~7607330,383803862~6494145] 2024-09-21T10:21:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:22 smithi145 ceph-mon[89359]: pgmap v1460: 105 pgs: 105 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 78 KiB/s rd, 919 KiB/s wr, 122 op/s 2024-09-21T10:21:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:22 smithi145 ceph-mon[89359]: osdmap e1118: 8 total, 8 up, 8 in 2024-09-21T10:21:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:22 smithi119 ceph-mon[105288]: pgmap v1460: 105 pgs: 105 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 78 KiB/s rd, 919 KiB/s wr, 122 op/s 2024-09-21T10:21:22.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:22 smithi119 ceph-mon[105288]: osdmap e1118: 8 total, 8 up, 8 in 2024-09-21T10:21:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:22 smithi119 ceph-mon[101432]: pgmap v1460: 105 pgs: 105 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 78 KiB/s rd, 919 KiB/s wr, 122 op/s 2024-09-21T10:21:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:22 smithi119 ceph-mon[101432]: osdmap e1118: 8 total, 8 up, 8 in 2024-09-21T10:21:23.157 INFO:tasks.workunit.client.0.smithi119.stdout: i=3 exists [15418248~7029160,23795927~12904290,52460119~13529773,85382137~6249500,117632188~3523956,139346825~3893399,153570589~3933400,161557009~1622158,168126409~3826441,189123779~6264039,197446742~1889394,200787355~3578059,206554317~7565484,228146076~12456600,242834170~826705,252284483~4291897,298603335~6447412,322115027~4377816,332714618~7607330,379176051~2904412,383803862~6494145,393686301~7562933,401854282~135665,408086187~7779835] wrote [19273872~1465606,58239674~7750218,85382137~6249500,189123779~6264039,200787355~3578059,209416257~1117831,232270566~4353491,252284483~4291897,298850952~6199795,395466343~321857] 2024-09-21T10:21:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:23 smithi119 ceph-mon[101432]: osdmap e1119: 8 total, 8 up, 8 in 2024-09-21T10:21:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:23 smithi119 ceph-mon[105288]: osdmap e1119: 8 total, 8 up, 8 in 2024-09-21T10:21:23.449 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:23 smithi145 ceph-mon[89359]: osdmap e1119: 8 total, 8 up, 8 in 2024-09-21T10:21:24.145 INFO:tasks.workunit.client.0.smithi119.stdout: i=4 exists [15418248~7029160,23795927~12904290,52460119~18973367,85382137~6249500,117632188~3523956,138569964~4670260,151417002~6086987,161557009~1622158,168126409~3826441,183393811~1371859,189123779~6264039,197446742~1889394,200787355~3578059,206554317~7565484,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,298603335~6447412,322115027~4377816,332714618~7818504,355080918~529553,379176051~2904412,383803862~6494145,393686301~7562933,401854282~135665,408086187~7779835] wrote [31539191~3942287,64814076~6619410,138569964~2485600,151417002~4072370,183393811~1371859,229869466~3030386,240750794~6163539,270173586~2678004,333943490~6589632,355080918~529553] 2024-09-21T10:21:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[101432]: pgmap v1463: 105 pgs: 105 active+clean; 342 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 1.6 MiB/s wr, 41 op/s 2024-09-21T10:21:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[101432]: osdmap e1120: 8 total, 8 up, 8 in 2024-09-21T10:21:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:24.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[105288]: pgmap v1463: 105 pgs: 105 active+clean; 342 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 1.6 MiB/s wr, 41 op/s 2024-09-21T10:21:24.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[105288]: osdmap e1120: 8 total, 8 up, 8 in 2024-09-21T10:21:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:24.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:24.395 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:24 smithi145 ceph-mon[89359]: pgmap v1463: 105 pgs: 105 active+clean; 342 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 1.6 MiB/s wr, 41 op/s 2024-09-21T10:21:24.396 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:24 smithi145 ceph-mon[89359]: osdmap e1120: 8 total, 8 up, 8 in 2024-09-21T10:21:24.396 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:24 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:24.396 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:24 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:21:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:21:24] "GET /metrics HTTP/1.1" 200 36713 "" "Prometheus/2.51.0" 2024-09-21T10:21:25.172 INFO:tasks.workunit.client.0.smithi119.stdout: i=5 exists [15418248~7029160,23795927~12904290,52460119~18973367,73054359~6304344,85382137~6249500,117632188~3523956,138569964~4670260,151417002~6086987,159070184~7209693,168126409~3826441,176533389~382502,183393811~1371859,189123779~6264039,197446742~1889394,200787355~3578059,206554317~13174349,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,298603335~6447412,317927452~8565391,332714618~7818504,355080918~529553,365013721~4661196,375709646~14588361,393686301~7562933,401854282~135665,408086187~7779835] wrote [73054359~6304344,159070184~7209693,176533389~382502,213479444~6249222,252504001~1621375,317927452~5163545,365013721~4661196,375709646~8147125,400390208~15275] 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[105288]: osdmap e1121: 8 total, 8 up, 8 in 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[101432]: osdmap e1121: 8 total, 8 up, 8 in 2024-09-21T10:21:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:21:25.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:21:25.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:25 smithi145 ceph-mon[89359]: osdmap e1121: 8 total, 8 up, 8 in 2024-09-21T10:21:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:21:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:21:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:21:26.175 INFO:tasks.workunit.client.0.smithi119.stdout: i=6 exists [11779571~302924,15418248~7029160,23795927~12904290,42413085~357444,50871480~20562006,73054359~6304344,85382137~7443022,117632188~3523956,138569964~4670260,151417002~6086987,159070184~7209693,168126409~3826441,176533389~382502,183393811~1371859,189123779~6264039,197446742~1889394,200787355~3578059,206554317~13174349,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,298603335~6447412,317927452~8565391,332714618~8019148,350875578~374443,355080918~529553,356136685~1013531,365013721~7043275,375709646~14588361,393686301~7562933,401854282~135665,408086187~7779835] wrote [11779571~302924,24624770~3325285,42413085~357444,50871480~7554806,68390092~847381,89024864~3800295,335427015~5306751,350875578~374443,356136685~1013531,366708946~5348050] 2024-09-21T10:21:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:26 smithi119 ceph-mon[101432]: pgmap v1466: 105 pgs: 105 active+clean; 516 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 129 KiB/s rd, 38 MiB/s wr, 224 op/s 2024-09-21T10:21:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:26 smithi119 ceph-mon[101432]: osdmap e1122: 8 total, 8 up, 8 in 2024-09-21T10:21:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:26 smithi119 ceph-mon[105288]: pgmap v1466: 105 pgs: 105 active+clean; 516 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 129 KiB/s rd, 38 MiB/s wr, 224 op/s 2024-09-21T10:21:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:26 smithi119 ceph-mon[105288]: osdmap e1122: 8 total, 8 up, 8 in 2024-09-21T10:21:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:26 smithi145 ceph-mon[89359]: pgmap v1466: 105 pgs: 105 active+clean; 516 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 129 KiB/s rd, 38 MiB/s wr, 224 op/s 2024-09-21T10:21:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:26 smithi145 ceph-mon[89359]: osdmap e1122: 8 total, 8 up, 8 in 2024-09-21T10:21:27.184 INFO:tasks.workunit.client.0.smithi119.stdout: i=7 exists [11779571~302924,13369180~9078228,23795927~12904290,42037868~2750187,49235280~76667,50871480~20562006,73054359~6304344,83517131~9308028,117632188~3523956,138569964~4670260,151417002~7558251,159070184~7209693,167440486~4512364,176533389~382502,183393811~1371859,189123779~6264039,196374037~3899920,200787355~3578059,206554317~13174349,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,298603335~6447412,317927452~8565391,332714618~9746626,350875578~374443,355080918~529553,356136685~1013531,365013721~7043275,375709646~14588361,393686301~7562933,401854282~135665,408086187~7779835] wrote [13369180~3865045,42037868~2750187,49235280~76667,83517131~2154387,151788765~7186488,167440486~1104740,196374037~3899920,235470784~1192158,335839633~6621611,377374997~4432208] 2024-09-21T10:21:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:27 smithi145 ceph-mon[89359]: osdmap e1123: 8 total, 8 up, 8 in 2024-09-21T10:21:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:27 smithi119 ceph-mon[105288]: osdmap e1123: 8 total, 8 up, 8 in 2024-09-21T10:21:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:27 smithi119 ceph-mon[101432]: osdmap e1123: 8 total, 8 up, 8 in 2024-09-21T10:21:28.174 INFO:tasks.workunit.client.0.smithi119.stdout: i=8 exists [11779571~302924,13369180~23331037,42037868~2750187,49235280~76667,50871480~20562006,73054359~6304344,83517131~9308028,117632188~3523956,138569964~4670260,151417002~7558251,159070184~7209693,167440486~4512364,176533389~382502,183393811~1371859,189123779~6264039,196374037~3899920,200787355~3578059,206554317~13174349,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,274092665~1398104,286842367~2887797,296109836~1440273,298603335~6447412,317927452~8565391,332714618~9746626,343898237~5922339,350875578~374443,355080918~529553,356136685~1013531,365013721~7043275,375709646~14588361,393686301~7562933,401854282~135665,404747422~11118600] wrote [21757572~5987847,28992920~4663129,43280308~1460130,56754004~681851,86271944~1459819,274092665~1398104,286842367~2887797,296109836~1440273,343898237~5922339,404747422~3435218] 2024-09-21T10:21:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:28 smithi145 ceph-mon[89359]: pgmap v1469: 105 pgs: 105 active+clean; 533 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 148 KiB/s rd, 40 MiB/s wr, 248 op/s 2024-09-21T10:21:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:28 smithi145 ceph-mon[89359]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-21T10:21:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:28 smithi119 ceph-mon[101432]: pgmap v1469: 105 pgs: 105 active+clean; 533 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 148 KiB/s rd, 40 MiB/s wr, 248 op/s 2024-09-21T10:21:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:28 smithi119 ceph-mon[101432]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-21T10:21:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:28 smithi119 ceph-mon[105288]: pgmap v1469: 105 pgs: 105 active+clean; 533 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 148 KiB/s rd, 40 MiB/s wr, 248 op/s 2024-09-21T10:21:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:28 smithi119 ceph-mon[105288]: osdmap e1124: 8 total, 8 up, 8 in 2024-09-21T10:21:29.256 INFO:tasks.workunit.client.0.smithi119.stdout: i=9 exists [5982079~34120474,42037868~2750187,49235280~76667,50871480~20562006,73054359~8130033,82098553~10726606,117632188~3523956,138569964~4670260,151417002~7558251,159070184~7209693,167440486~4512364,176533389~382502,183393811~1371859,189123779~15289560,206554317~13174349,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,274092665~1398104,286842367~2887797,296109836~1440273,298603335~6447412,317927452~8565391,332714618~9746626,343898237~9383032,355080918~529553,356136685~1013531,365013721~7416070,372977457~17320550,393686301~7562933,401854282~135665,404747422~11118600] wrote [5982079~7753386,33845414~6257139,73501286~7683106,82098553~3508464,90960866~1617186,190942346~6244688,198649714~5763625,347887843~5393426,365261282~7168509,372977457~7988310] 2024-09-21T10:21:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:29 smithi145 ceph-mon[89359]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-21T10:21:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:29 smithi119 ceph-mon[105288]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-21T10:21:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:29 smithi119 ceph-mon[101432]: osdmap e1125: 8 total, 8 up, 8 in 2024-09-21T10:21:30.211 INFO:tasks.workunit.client.0.smithi119.stdout: i=10 exists [4658718~35443835,42037868~2750187,49235280~76667,50871480~20562006,73054359~8130033,82098553~16819402,117632188~3523956,138569964~4670260,151417002~7558251,159070184~7209693,167440486~4512364,176533389~382502,183393811~1371859,189123779~15289560,206554317~13174349,221546668~5261584,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,274092665~1398104,286842367~2887797,296109836~8940911,310680101~860466,311808142~5347710,317927452~8565391,332714618~9746626,343898237~9383032,355080918~529553,356136685~1013531,364317837~8111954,372977457~17320550,393686301~7562933,401854282~135665,404747422~11118600] wrote [4658718~3715385,10437405~5070702,92105820~6812135,221546668~5261584,297441532~1653696,310680101~860466,311808142~5347710,348293072~1243371,364317837~5015561] 2024-09-21T10:21:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:30 smithi145 ceph-mon[89359]: pgmap v1472: 105 pgs: 105 active+clean; 691 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 161 KiB/s rd, 35 MiB/s wr, 264 op/s 2024-09-21T10:21:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:30 smithi145 ceph-mon[89359]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-21T10:21:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:21:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:21:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:30 smithi119 ceph-mon[105288]: pgmap v1472: 105 pgs: 105 active+clean; 691 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 161 KiB/s rd, 35 MiB/s wr, 264 op/s 2024-09-21T10:21:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:30 smithi119 ceph-mon[105288]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-21T10:21:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:30 smithi119 ceph-mon[101432]: pgmap v1472: 105 pgs: 105 active+clean; 691 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 161 KiB/s rd, 35 MiB/s wr, 264 op/s 2024-09-21T10:21:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:30 smithi119 ceph-mon[101432]: osdmap e1126: 8 total, 8 up, 8 in 2024-09-21T10:21:31.275 INFO:tasks.workunit.client.0.smithi119.stdout: i=11 exists [453390~39649163,42037868~2750187,49235280~76667,50871480~20562006,73054359~8130033,82098553~16819402,117632188~3523956,123786540~5770526,138569964~4670260,145839658~4102649,151417002~7558251,159070184~7209693,167440486~4512364,176533389~382502,183393811~1371859,189123779~15289560,206554317~13174349,221546668~5261584,228146076~12456600,240750794~6163539,252284483~4291897,270173586~2678004,274092665~1398104,286842367~2887797,296109836~8940911,310680101~860466,311808142~5347710,317927452~24533792,343898237~9383032,355080918~529553,356136685~1013531,360459668~11970123,372977457~17320550,393686301~7562933,401854282~135665,404747422~13241238] wrote [453390~18968762,92093581~6228187,123786540~5770526,145839658~4102649,155597618~2993990,325873327~7779328,360459668~7182668,410165847~7822813] 2024-09-21T10:21:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:31 smithi145 ceph-mon[89359]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-21T10:21:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:31 smithi119 ceph-mon[105288]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-21T10:21:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:31 smithi119 ceph-mon[101432]: osdmap e1127: 8 total, 8 up, 8 in 2024-09-21T10:21:32.211 INFO:tasks.workunit.client.0.smithi119.stdout: i=12 exists [453390~39649163,42037868~2750187,49235280~76667,49925057~21508429,73054359~8130033,82098553~16819402,100811185~6742788,117632188~3523956,123786540~5770526,138569964~4670260,145839658~4102649,151417002~7558251,159070184~7209693,166383692~5569158,176533389~382502,183393811~1371859,189123779~15289560,206554317~13174349,221546668~5261584,228146076~12456600,240750794~6163539,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,286842367~2887797,296109836~8940911,310680101~860466,311808142~5347710,317927452~24533792,343898237~9383032,355080918~529553,356136685~1013531,360459668~11970123,372977457~17320550,393686301~7562933,401854282~135665,404747422~13241238] wrote [13840644~7837290,49925057~5531983,82278598~3283996,93738399~638303,100811185~6742788,166383692~5309940,249384618~2637003,266551876~5153282,324204899~5129146] 2024-09-21T10:21:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:32 smithi145 ceph-mon[89359]: pgmap v1475: 105 pgs: 105 active+clean; 731 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 192 KiB/s rd, 40 MiB/s wr, 311 op/s 2024-09-21T10:21:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:32 smithi145 ceph-mon[89359]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-21T10:21:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:32 smithi119 ceph-mon[101432]: pgmap v1475: 105 pgs: 105 active+clean; 731 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 192 KiB/s rd, 40 MiB/s wr, 311 op/s 2024-09-21T10:21:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:32 smithi119 ceph-mon[101432]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-21T10:21:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:32 smithi119 ceph-mon[105288]: pgmap v1475: 105 pgs: 105 active+clean; 731 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 192 KiB/s rd, 40 MiB/s wr, 311 op/s 2024-09-21T10:21:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:32 smithi119 ceph-mon[105288]: osdmap e1128: 8 total, 8 up, 8 in 2024-09-21T10:21:33.239 INFO:tasks.workunit.client.0.smithi119.stdout: i=13 exists [453390~39649163,42037868~2750187,49235280~76667,49925057~21508429,73054359~8130033,82098553~16819402,100811185~6742788,117632188~3523956,123786540~7507354,138569964~4670260,145839658~13135595,159070184~7209693,166383692~5569158,176533389~382502,178495302~8381360,189123779~15289560,206554317~13174349,221546668~5261584,228146076~12456600,240750794~6163539,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,286842367~2887797,296109836~8940911,310680101~860466,311808142~5347710,317927452~24533792,343898237~9383032,355080918~529553,356136685~1013531,360459668~11970123,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [14469210~4146069,27198089~7426413,60332289~1057684,126157279~5136615,147858386~3667319,178495302~8381360,206751286~312827,279614755~745233,389320381~2423791] 2024-09-21T10:21:33.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:33 smithi119 ceph-mon[105288]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-21T10:21:33.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:33.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:33 smithi119 ceph-mon[101432]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-21T10:21:33.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:33 smithi145 ceph-mon[89359]: osdmap e1129: 8 total, 8 up, 8 in 2024-09-21T10:21:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:34.277 INFO:tasks.workunit.client.0.smithi119.stdout: i=14 exists [453390~46018812,49235280~76667,49925057~21508429,73054359~8130033,82098553~16819402,100811185~6742788,117632188~3523956,123786540~7507354,138569964~5721505,145839658~13135595,159070184~7209693,166383692~20492970,189123779~15289560,206554317~13174349,221546668~19056008,240750794~6163539,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,286842367~2887797,296109836~8940911,310680101~860466,311808142~5347710,317927452~24533792,343898237~9383032,355080918~529553,356136685~1013531,360459668~12060430,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [38473645~7998557,104380924~2328035,126349261~2338552,142555402~1736067,170141667~9219905,225830439~5569603,366551290~5968808] 2024-09-21T10:21:34.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:34 smithi119 ceph-mon[105288]: pgmap v1478: 105 pgs: 105 active+clean; 753 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 13 MiB/s wr, 107 op/s 2024-09-21T10:21:34.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:34 smithi119 ceph-mon[105288]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-21T10:21:34.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:34 smithi119 ceph-mon[101432]: pgmap v1478: 105 pgs: 105 active+clean; 753 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 13 MiB/s wr, 107 op/s 2024-09-21T10:21:34.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:34 smithi119 ceph-mon[101432]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-21T10:21:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:34 smithi145 ceph-mon[89359]: pgmap v1478: 105 pgs: 105 active+clean; 753 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 75 KiB/s rd, 13 MiB/s wr, 107 op/s 2024-09-21T10:21:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:34 smithi145 ceph-mon[89359]: osdmap e1130: 8 total, 8 up, 8 in 2024-09-21T10:21:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:21:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:21:34] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.51.0" 2024-09-21T10:21:35.277 INFO:tasks.workunit.client.0.smithi119.stdout: i=15 exists [453390~46018812,49235280~76667,49925057~21508429,73054359~8130033,82098553~16819402,100811185~6742788,117632188~3523956,123786540~13991694,138569964~5721505,145839658~20440219,166383692~20492970,188704016~15709323,206554317~13174349,221546668~19056008,240750794~6163539,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,286842367~2887797,295328539~10049657,310680101~860466,311808142~5347710,317927452~24533792,343898237~9383032,354474286~18045812,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [93218023~5501989,130626673~7151561,155958833~8598020,188704016~7484208,242622323~195129,295328539~10049657,339998690~1680477,354474286~7761279] 2024-09-21T10:21:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:35 smithi119 ceph-mon[101432]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-21T10:21:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:35 smithi119 ceph-mon[105288]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-21T10:21:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:35 smithi145 ceph-mon[89359]: osdmap e1131: 8 total, 8 up, 8 in 2024-09-21T10:21:36.286 INFO:tasks.workunit.client.0.smithi119.stdout: i=16 exists [453390~46018812,49235280~76667,49925057~21508429,73054359~8130033,82098553~25455420,117632188~3523956,123786540~13991694,138569964~5721505,145839658~20440219,166383692~20492970,188704016~15709323,206554317~13174349,221546668~19056008,240750794~6163539,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,286842367~2887797,295328539~10049657,308687752~387021,310680101~860466,311808142~30653102,343898237~9383032,354474286~18045812,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [24402928~3712209,94476115~6860025,149521358~6653734,308687752~387021,312449738~5832798,326705254~11484660,362782022~5032806,375650838~1623501] 2024-09-21T10:21:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:36 smithi145 ceph-mon[89359]: pgmap v1481: 105 pgs: 105 active+clean; 967 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 55 MiB/s wr, 228 op/s 2024-09-21T10:21:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:36 smithi145 ceph-mon[89359]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-21T10:21:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:36 smithi119 ceph-mon[101432]: pgmap v1481: 105 pgs: 105 active+clean; 967 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 55 MiB/s wr, 228 op/s 2024-09-21T10:21:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:36 smithi119 ceph-mon[101432]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-21T10:21:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:36 smithi119 ceph-mon[105288]: pgmap v1481: 105 pgs: 105 active+clean; 967 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 126 KiB/s rd, 55 MiB/s wr, 228 op/s 2024-09-21T10:21:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:36 smithi119 ceph-mon[105288]: osdmap e1132: 8 total, 8 up, 8 in 2024-09-21T10:21:37.214 INFO:tasks.workunit.client.0.smithi119.stdout: i=17 exists [453390~46018812,49235280~76667,49925057~21508429,73054359~8130033,82098553~25455420,115032578~6123566,123786540~13991694,138569964~5721505,145520825~20759052,166383692~20492970,188704016~15709323,206554317~13174349,221546668~19056008,240750794~7871990,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,286842367~4771106,295328539~10049657,308687752~387021,310680101~860466,311808142~30653102,343898237~9383032,354474286~18045812,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [21580119~40605,36108218~2131227,56798181~2951998,99737615~3868557,115032578~5231699,145520825~2856203,217337982~790545,244242120~4380664,289110302~2503171,369530674~171507] 2024-09-21T10:21:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:37 smithi145 ceph-mon[89359]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-21T10:21:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:37 smithi119 ceph-mon[101432]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-21T10:21:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:37 smithi119 ceph-mon[105288]: osdmap e1133: 8 total, 8 up, 8 in 2024-09-21T10:21:38.288 INFO:tasks.workunit.client.0.smithi119.stdout: i=18 exists [453390~46018812,49235280~76667,49925057~21508429,73054359~8130033,82098553~25455420,111320656~999803,115032578~6123566,123786540~13991694,138569964~5721505,145520825~20759052,166383692~20492970,188704016~15709323,206554317~34048359,240750794~7871990,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,286842367~4771106,295328539~10049657,308687752~387021,310680101~860466,311808142~30653102,343898237~9383032,354474286~18045812,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [5283954~379530,22321062~7287402,111320656~999803,158673218~2042874,178074079~578013,219644207~5421076,239752877~622730,327471636~7679356,356333748~12099017] 2024-09-21T10:21:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:38 smithi145 ceph-mon[89359]: pgmap v1484: 105 pgs: 105 active+clean; 976 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 235 KiB/s rd, 53 MiB/s wr, 371 op/s 2024-09-21T10:21:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:38 smithi145 ceph-mon[89359]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-21T10:21:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:38 smithi119 ceph-mon[105288]: pgmap v1484: 105 pgs: 105 active+clean; 976 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 235 KiB/s rd, 53 MiB/s wr, 371 op/s 2024-09-21T10:21:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:38 smithi119 ceph-mon[105288]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-21T10:21:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:38 smithi119 ceph-mon[101432]: pgmap v1484: 105 pgs: 105 active+clean; 976 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 235 KiB/s rd, 53 MiB/s wr, 371 op/s 2024-09-21T10:21:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:38 smithi119 ceph-mon[101432]: osdmap e1134: 8 total, 8 up, 8 in 2024-09-21T10:21:39.270 INFO:tasks.workunit.client.0.smithi119.stdout: i=19 exists [453390~46018812,49235280~76667,49925057~21508429,73054359~8130033,81257092~26296881,111320656~9835488,123786540~13991694,138362421~5929048,145520825~20759052,166383692~20492970,188704016~15709323,206554317~34048359,240750794~7871990,249384618~2637003,252284483~4291897,266551876~6299714,274092665~1398104,279614755~745233,283387878~284859,286842367~4771106,295328539~10049657,308687752~387021,310680101~860466,311808142~30653102,343898237~9383032,354474286~18045812,372977457~18766715,393686301~7562933,401854282~135665,404747422~13241238] wrote [81257092~4373663,112059719~6181785,138362421~4876744,148029827~993101,194480825~5989263,201720598~2601332,283387878~284859,332458135~6409492,366527804~2793741,395326825~2821070] 2024-09-21T10:21:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:39 smithi145 ceph-mon[89359]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-21T10:21:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:39 smithi119 ceph-mon[105288]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-21T10:21:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:39 smithi119 ceph-mon[101432]: osdmap e1135: 8 total, 8 up, 8 in 2024-09-21T10:21:40.220 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 1 diff [16777216~12582912,50331648~4194304,117440512~4194304,138412032~4194304,205520896~8388608,226492416~8388608,318767104~8388608,377487360~4194304,398458880~4194304] 2024-09-21T10:21:40.221 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [16777216~12582912,50331648~4194304,117440512~4194304,138412032~4194304,205520896~8388608,226492416~8388608,318767104~8388608,377487360~4194304,398458880~4194304] 2024-09-21T10:21:40.230 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [16777216~16777216,50331648~8388608,117440512~4194304,138412032~8388608,205520896~12582912,226492416~8388608,318767104~8388608,377487360~8388608,398458880~4194304] 2024-09-21T10:21:40.230 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 2 diff [12582912~16777216,50331648~8388608,88080384~4194304,117440512~4194304,138412032~4194304,167772160~4194304,197132288~4194304,205520896~8388608,226492416~8388608,239075328~4194304,297795584~8388608,318767104~8388608,331350016~8388608,377487360~12582912,398458880~4194304] 2024-09-21T10:21:40.230 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~16777216,50331648~8388608,88080384~4194304,117440512~4194304,138412032~4194304,167772160~4194304,197132288~4194304,205520896~8388608,226492416~8388608,239075328~4194304,297795584~8388608,318767104~8388608,331350016~8388608,377487360~12582912,398458880~4194304] 2024-09-21T10:21:40.232 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~20971520,50331648~8388608,88080384~4194304,117440512~4194304,138412032~8388608,167772160~4194304,197132288~4194304,205520896~12582912,226492416~8388608,239075328~8388608,297795584~8388608,318767104~8388608,331350016~12582912,377487360~16777216,398458880~4194304] 2024-09-21T10:21:40.233 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 3 diff [12582912~16777216,50331648~16777216,83886080~8388608,117440512~4194304,138412032~4194304,167772160~4194304,188743680~12582912,205520896~8388608,226492416~16777216,251658240~8388608,297795584~8388608,318767104~8388608,331350016~8388608,377487360~12582912,394264576~8388608] 2024-09-21T10:21:40.233 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~16777216,50331648~16777216,83886080~8388608,117440512~4194304,138412032~4194304,167772160~4194304,188743680~12582912,205520896~8388608,226492416~16777216,251658240~8388608,297795584~8388608,318767104~8388608,331350016~8388608,377487360~12582912,394264576~8388608] 2024-09-21T10:21:40.236 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~20971520,50331648~16777216,83886080~8388608,117440512~4194304,138412032~8388608,167772160~4194304,188743680~29360128,226492416~20971520,251658240~8388608,297795584~8388608,318767104~8388608,331350016~12582912,377487360~25165824] 2024-09-21T10:21:40.236 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 4 diff [12582912~20971520,50331648~20971520,83886080~8388608,117440512~4194304,138412032~8388608,150994944~4194304,167772160~4194304,180355072~4194304,188743680~12582912,205520896~8388608,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,318767104~8388608,331350016~8388608,352321536~4194304,377487360~12582912,394264576~8388608] 2024-09-21T10:21:40.236 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~20971520,50331648~20971520,83886080~8388608,117440512~4194304,138412032~8388608,150994944~4194304,167772160~4194304,180355072~4194304,188743680~12582912,205520896~8388608,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,318767104~8388608,331350016~8388608,352321536~4194304,377487360~12582912,394264576~8388608] 2024-09-21T10:21:40.239 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~25165824,50331648~25165824,83886080~8388608,117440512~4194304,138412032~8388608,150994944~8388608,167772160~4194304,180355072~37748736,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,318767104~8388608,331350016~12582912,352321536~4194304,377487360~25165824] 2024-09-21T10:21:40.239 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 5 diff [12582912~20971520,50331648~29360128,83886080~8388608,117440512~4194304,138412032~8388608,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,314572800~12582912,331350016~8388608,352321536~4194304,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.240 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~20971520,50331648~29360128,83886080~8388608,117440512~4194304,138412032~8388608,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,314572800~12582912,331350016~8388608,352321536~4194304,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.243 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~25165824,50331648~29360128,83886080~8388608,117440512~4194304,138412032~8388608,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,352321536~4194304,364904448~37748736] 2024-09-21T10:21:40.243 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 6 diff [8388608~25165824,41943040~4194304,50331648~29360128,83886080~8388608,117440512~4194304,138412032~8388608,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,314572800~12582912,331350016~8388608,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.243 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~25165824,41943040~4194304,50331648~29360128,83886080~8388608,117440512~4194304,138412032~8388608,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,314572800~12582912,331350016~8388608,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.247 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~4194304,50331648~29360128,83886080~12582912,117440512~4194304,138412032~8388608,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~37748736] 2024-09-21T10:21:40.247 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 7 diff [8388608~25165824,41943040~50331648,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,314572800~12582912,331350016~12582912,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.247 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~25165824,41943040~50331648,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~4194304,297795584~8388608,314572800~12582912,331350016~12582912,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.251 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~54525952,117440512~4194304,138412032~8388608,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~37748736] 2024-09-21T10:21:40.251 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 8 diff [8388608~29360128,41943040~50331648,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~12582912,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.251 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~29360128,41943040~50331648,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~12582912,205520896~16777216,226492416~20971520,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~12582912,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.256 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~54525952,117440512~4194304,138412032~8388608,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,285212672~20971520,314572800~12582912,331350016~29360128,364904448~46137344] 2024-09-21T10:21:40.256 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 9 diff [4194304~88080384,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~33554432,226492416~20971520,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~12582912,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.256 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~88080384,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~33554432,226492416~20971520,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~12582912,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.261 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~92274688,117440512~4194304,138412032~8388608,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,285212672~20971520,314572800~12582912,331350016~29360128,364904448~46137344] 2024-09-21T10:21:40.262 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 10 diff [4194304~96468992,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~58720256,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~16777216,331350016~25165824,360710144~29360128,394264576~12582912] 2024-09-21T10:21:40.262 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~96468992,117440512~4194304,138412032~8388608,150994944~20971520,176160768~8388608,188743680~58720256,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~16777216,331350016~25165824,360710144~29360128,394264576~12582912] 2024-09-21T10:21:40.267 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~96468992,117440512~4194304,138412032~8388608,150994944~20971520,176160768~71303168,251658240~8388608,268435456~8388608,285212672~20971520,310378496~16777216,331350016~79691776] 2024-09-21T10:21:40.268 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 11 diff [0~100663296,117440512~12582912,138412032~8388608,150994944~20971520,176160768~8388608,188743680~58720256,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.268 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~100663296,117440512~12582912,138412032~8388608,150994944~20971520,176160768~8388608,188743680~58720256,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.273 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~100663296,117440512~12582912,138412032~33554432,176160768~71303168,251658240~8388608,268435456~8388608,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.273 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 12 diff [0~109051904,117440512~12582912,138412032~8388608,150994944~20971520,176160768~8388608,188743680~71303168,264241152~12582912,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.273 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,117440512~12582912,138412032~8388608,150994944~20971520,176160768~8388608,188743680~71303168,264241152~12582912,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.281 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,117440512~12582912,138412032~33554432,176160768~83886080,264241152~12582912,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.281 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 13 diff [0~109051904,117440512~12582912,138412032~33554432,176160768~8388608,188743680~71303168,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.281 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,117440512~12582912,138412032~33554432,176160768~8388608,188743680~71303168,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.288 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,117440512~16777216,138412032~33554432,176160768~83886080,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.288 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 14 diff [0~109051904,117440512~12582912,138412032~46137344,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.289 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,117440512~12582912,138412032~46137344,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.295 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,117440512~16777216,138412032~121634816,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.296 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 15 diff [0~109051904,117440512~83886080,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.296 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,117440512~83886080,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.303 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,117440512~142606336,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.303 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 16 diff [0~109051904,117440512~83886080,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~96468992,394264576~20971520] 2024-09-21T10:21:40.303 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,117440512~83886080,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~96468992,394264576~20971520] 2024-09-21T10:21:40.311 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,117440512~142606336,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.311 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 17 diff [0~109051904,113246208~88080384,205520896~29360128,239075328~20971520,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.311 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,113246208~88080384,205520896~29360128,239075328~20971520,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,113246208~146800640,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 18 diff [0~201326592,205520896~54525952,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.319 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.329 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.329 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to 19 diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:40.329 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:40.338 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:40.338 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 2 diff [12582912~8388608,50331648~8388608,88080384~4194304,167772160~4194304,197132288~4194304,239075328~4194304,297795584~8388608,331350016~8388608,381681664~8388608] 2024-09-21T10:21:40.338 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,50331648~8388608,88080384~4194304,167772160~4194304,197132288~4194304,239075328~4194304,297795584~8388608,331350016~8388608,381681664~8388608] 2024-09-21T10:21:40.340 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~12582912,50331648~8388608,88080384~4194304,167772160~4194304,197132288~4194304,239075328~8388608,297795584~8388608,331350016~12582912,381681664~12582912] 2024-09-21T10:21:40.340 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 3 diff [12582912~8388608,50331648~16777216,83886080~8388608,167772160~4194304,188743680~12582912,205520896~4194304,230686720~12582912,251658240~8388608,297795584~8388608,331350016~8388608,381681664~8388608,394264576~4194304] 2024-09-21T10:21:40.340 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,50331648~16777216,83886080~8388608,167772160~4194304,188743680~12582912,205520896~4194304,230686720~12582912,251658240~8388608,297795584~8388608,331350016~8388608,381681664~8388608,394264576~4194304] 2024-09-21T10:21:40.341 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~12582912,50331648~16777216,83886080~8388608,167772160~4194304,188743680~25165824,230686720~16777216,251658240~8388608,297795584~8388608,331350016~12582912,381681664~16777216] 2024-09-21T10:21:40.341 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 4 diff [12582912~8388608,29360128~4194304,50331648~20971520,83886080~8388608,138412032~4194304,150994944~4194304,167772160~4194304,180355072~4194304,188743680~12582912,205520896~4194304,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,331350016~8388608,352321536~4194304,381681664~8388608,394264576~4194304] 2024-09-21T10:21:40.341 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,29360128~4194304,50331648~20971520,83886080~8388608,138412032~4194304,150994944~4194304,167772160~4194304,180355072~4194304,188743680~12582912,205520896~4194304,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,331350016~8388608,352321536~4194304,381681664~8388608,394264576~4194304] 2024-09-21T10:21:40.343 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~12582912,29360128~8388608,50331648~25165824,83886080~8388608,138412032~4194304,150994944~8388608,167772160~4194304,180355072~33554432,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,331350016~12582912,352321536~4194304,381681664~16777216] 2024-09-21T10:21:40.343 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 5 diff [12582912~8388608,29360128~4194304,50331648~29360128,83886080~8388608,138412032~4194304,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,352321536~4194304,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.343 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,29360128~4194304,50331648~29360128,83886080~8388608,138412032~4194304,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,352321536~4194304,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.346 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~12582912,29360128~8388608,50331648~29360128,83886080~8388608,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,352321536~4194304,364904448~37748736] 2024-09-21T10:21:40.346 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 6 diff [8388608~16777216,29360128~4194304,41943040~4194304,50331648~29360128,83886080~8388608,138412032~4194304,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.346 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~16777216,29360128~4194304,41943040~4194304,50331648~29360128,83886080~8388608,138412032~4194304,150994944~12582912,167772160~4194304,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.350 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~4194304,50331648~29360128,83886080~12582912,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~37748736] 2024-09-21T10:21:40.350 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 7 diff [8388608~16777216,29360128~4194304,41943040~50331648,138412032~4194304,150994944~20971520,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~12582912,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.350 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~16777216,29360128~4194304,41943040~50331648,138412032~4194304,150994944~20971520,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~12582912,348127232~8388608,364904448~25165824,394264576~8388608] 2024-09-21T10:21:40.353 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~54525952,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~37748736] 2024-09-21T10:21:40.353 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 8 diff [8388608~25165824,41943040~50331648,138412032~4194304,150994944~20971520,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.353 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~25165824,41943040~50331648,138412032~4194304,150994944~20971520,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.357 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~54525952,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,285212672~20971520,314572800~12582912,331350016~29360128,364904448~46137344] 2024-09-21T10:21:40.357 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 9 diff [4194304~33554432,41943040~50331648,138412032~4194304,150994944~20971520,176160768~8388608,188743680~29360128,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.357 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~33554432,41943040~50331648,138412032~4194304,150994944~20971520,176160768~8388608,188743680~29360128,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~25165824,394264576~12582912] 2024-09-21T10:21:40.361 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~92274688,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,285212672~20971520,314572800~12582912,331350016~29360128,364904448~46137344] 2024-09-21T10:21:40.361 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 10 diff [4194304~33554432,41943040~58720256,138412032~4194304,150994944~20971520,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~12582912,331350016~25165824,360710144~29360128,394264576~12582912] 2024-09-21T10:21:40.361 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~33554432,41943040~58720256,138412032~4194304,150994944~20971520,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~12582912,331350016~25165824,360710144~29360128,394264576~12582912] 2024-09-21T10:21:40.365 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~96468992,138412032~4194304,150994944~20971520,176160768~71303168,251658240~8388608,268435456~8388608,285212672~20971520,310378496~16777216,331350016~79691776] 2024-09-21T10:21:40.366 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 11 diff [0~37748736,41943040~58720256,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.366 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~37748736,41943040~58720256,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.371 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~100663296,121634816~8388608,138412032~33554432,176160768~71303168,251658240~8388608,268435456~8388608,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.371 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 12 diff [0~37748736,41943040~67108864,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~46137344,239075328~20971520,264241152~12582912,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.371 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~37748736,41943040~67108864,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~46137344,239075328~20971520,264241152~12582912,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.377 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~8388608,138412032~33554432,176160768~83886080,264241152~12582912,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.377 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 13 diff [0~109051904,121634816~8388608,138412032~33554432,176160768~8388608,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.377 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~8388608,138412032~33554432,176160768~8388608,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.383 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~12582912,138412032~33554432,176160768~83886080,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.383 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 14 diff [0~109051904,121634816~8388608,138412032~46137344,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.383 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~8388608,138412032~46137344,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.389 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~12582912,138412032~121634816,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.389 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 15 diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.389 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~79691776,394264576~20971520] 2024-09-21T10:21:40.397 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~138412032,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.397 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 16 diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~96468992,394264576~20971520] 2024-09-21T10:21:40.397 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~96468992,394264576~20971520] 2024-09-21T10:21:40.404 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~138412032,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.404 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 17 diff [0~109051904,113246208~88080384,205520896~29360128,239075328~20971520,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.404 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,113246208~88080384,205520896~29360128,239075328~20971520,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.412 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,113246208~146800640,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.412 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 18 diff [0~201326592,205520896~29360128,239075328~20971520,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.412 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~16777216,285212672~104857600,394264576~20971520] 2024-09-21T10:21:40.420 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.420 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to 19 diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:40.420 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:40.427 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:40.428 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 3 diff [16777216~4194304,54525952~8388608,83886080~8388608,188743680~12582912,205520896~4194304,230686720~8388608,251658240~8388608,297795584~8388608,394264576~4194304] 2024-09-21T10:21:40.428 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [16777216~4194304,54525952~8388608,83886080~8388608,188743680~12582912,205520896~4194304,230686720~8388608,251658240~8388608,297795584~8388608,394264576~4194304] 2024-09-21T10:21:40.428 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [16777216~4194304,54525952~12582912,83886080~8388608,188743680~25165824,230686720~8388608,251658240~8388608,297795584~8388608,394264576~4194304] 2024-09-21T10:21:40.429 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 4 diff [16777216~4194304,29360128~4194304,54525952~16777216,83886080~8388608,138412032~4194304,150994944~4194304,180355072~4194304,188743680~12582912,205520896~4194304,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,331350016~8388608,352321536~4194304,394264576~4194304] 2024-09-21T10:21:40.429 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [16777216~4194304,29360128~4194304,54525952~16777216,83886080~8388608,138412032~4194304,150994944~4194304,180355072~4194304,188743680~12582912,205520896~4194304,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,331350016~8388608,352321536~4194304,394264576~4194304] 2024-09-21T10:21:40.430 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [16777216~4194304,29360128~8388608,54525952~20971520,83886080~8388608,138412032~4194304,150994944~8388608,180355072~33554432,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,331350016~12582912,352321536~4194304,394264576~4194304] 2024-09-21T10:21:40.430 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 5 diff [16777216~4194304,29360128~4194304,54525952~25165824,83886080~8388608,138412032~4194304,150994944~12582912,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,352321536~4194304,364904448~16777216,394264576~8388608] 2024-09-21T10:21:40.430 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [16777216~4194304,29360128~4194304,54525952~25165824,83886080~8388608,138412032~4194304,150994944~12582912,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,352321536~4194304,364904448~16777216,394264576~8388608] 2024-09-21T10:21:40.432 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [16777216~4194304,29360128~8388608,54525952~25165824,83886080~8388608,138412032~4194304,150994944~16777216,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,352321536~4194304,364904448~20971520,394264576~8388608] 2024-09-21T10:21:40.432 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 6 diff [8388608~4194304,16777216~8388608,29360128~4194304,41943040~4194304,50331648~29360128,83886080~8388608,138412032~4194304,150994944~12582912,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,394264576~8388608] 2024-09-21T10:21:40.432 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~4194304,16777216~8388608,29360128~4194304,41943040~4194304,50331648~29360128,83886080~8388608,138412032~4194304,150994944~12582912,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,394264576~8388608] 2024-09-21T10:21:40.434 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~4194304,16777216~20971520,41943040~4194304,50331648~29360128,83886080~12582912,138412032~4194304,150994944~16777216,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~20971520,394264576~8388608] 2024-09-21T10:21:40.435 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 7 diff [8388608~16777216,29360128~4194304,41943040~50331648,138412032~4194304,150994944~16777216,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~12582912,348127232~8388608,364904448~16777216,394264576~8388608] 2024-09-21T10:21:40.435 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~16777216,29360128~4194304,41943040~50331648,138412032~4194304,150994944~16777216,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~4194304,297795584~8388608,314572800~8388608,331350016~12582912,348127232~8388608,364904448~16777216,394264576~8388608] 2024-09-21T10:21:40.437 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~54525952,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,297795584~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~20971520,394264576~8388608] 2024-09-21T10:21:40.438 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 8 diff [8388608~25165824,41943040~50331648,138412032~4194304,150994944~16777216,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~16777216,394264576~12582912] 2024-09-21T10:21:40.438 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~25165824,41943040~50331648,138412032~4194304,150994944~16777216,176160768~8388608,188743680~12582912,205520896~12582912,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~16777216,394264576~12582912] 2024-09-21T10:21:40.440 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~54525952,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,285212672~20971520,314572800~12582912,331350016~29360128,364904448~20971520,394264576~16777216] 2024-09-21T10:21:40.440 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 9 diff [4194304~33554432,41943040~50331648,138412032~4194304,150994944~16777216,176160768~8388608,188743680~29360128,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~16777216,394264576~12582912] 2024-09-21T10:21:40.440 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~33554432,41943040~50331648,138412032~4194304,150994944~16777216,176160768~8388608,188743680~29360128,226492416~8388608,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,314572800~8388608,331350016~25165824,364904448~16777216,394264576~12582912] 2024-09-21T10:21:40.444 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~92274688,138412032~4194304,150994944~20971520,176160768~46137344,226492416~20971520,251658240~8388608,268435456~8388608,285212672~20971520,314572800~12582912,331350016~29360128,364904448~20971520,394264576~16777216] 2024-09-21T10:21:40.444 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 10 diff [4194304~33554432,41943040~58720256,138412032~4194304,150994944~16777216,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~12582912,331350016~25165824,360710144~20971520,394264576~12582912] 2024-09-21T10:21:40.444 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~33554432,41943040~58720256,138412032~4194304,150994944~16777216,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~12582912,331350016~25165824,360710144~20971520,394264576~12582912] 2024-09-21T10:21:40.448 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~96468992,138412032~4194304,150994944~20971520,176160768~71303168,251658240~8388608,268435456~8388608,285212672~20971520,310378496~16777216,331350016~54525952,394264576~16777216] 2024-09-21T10:21:40.449 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 11 diff [0~37748736,41943040~58720256,121634816~8388608,138412032~8388608,150994944~16777216,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~71303168,394264576~20971520] 2024-09-21T10:21:40.449 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~37748736,41943040~58720256,121634816~8388608,138412032~8388608,150994944~16777216,176160768~8388608,188743680~46137344,239075328~8388608,251658240~8388608,268435456~8388608,285212672~4194304,293601280~12582912,310378496~71303168,394264576~20971520] 2024-09-21T10:21:40.452 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~100663296,121634816~8388608,138412032~33554432,176160768~71303168,251658240~8388608,268435456~8388608,285212672~20971520,310378496~75497472,394264576~25165824] 2024-09-21T10:21:40.452 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 12 diff [0~37748736,41943040~67108864,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~46137344,239075328~20971520,264241152~12582912,285212672~4194304,293601280~12582912,310378496~71303168,394264576~20971520] 2024-09-21T10:21:40.452 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~37748736,41943040~67108864,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~46137344,239075328~20971520,264241152~12582912,285212672~4194304,293601280~12582912,310378496~71303168,394264576~20971520] 2024-09-21T10:21:40.457 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~8388608,138412032~33554432,176160768~83886080,264241152~12582912,285212672~20971520,310378496~75497472,394264576~25165824] 2024-09-21T10:21:40.457 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 13 diff [0~109051904,121634816~8388608,138412032~33554432,176160768~8388608,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.457 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~8388608,138412032~33554432,176160768~8388608,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.462 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~12582912,138412032~33554432,176160768~83886080,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.462 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 14 diff [0~109051904,121634816~8388608,138412032~46137344,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.462 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~8388608,138412032~46137344,188743680~46137344,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.468 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~12582912,138412032~121634816,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.468 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 15 diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.468 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.474 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~138412032,264241152~16777216,285212672~20971520,310378496~109051904] 2024-09-21T10:21:40.474 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 16 diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~88080384,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.474 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~79691776,205520896~29360128,239075328~20971520,264241152~16777216,285212672~4194304,293601280~88080384,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.481 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~138412032,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.481 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 17 diff [0~109051904,113246208~88080384,205520896~29360128,239075328~20971520,264241152~16777216,285212672~96468992,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.481 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,113246208~88080384,205520896~29360128,239075328~20971520,264241152~16777216,285212672~96468992,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.489 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,113246208~146800640,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.489 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 18 diff [0~201326592,205520896~29360128,239075328~20971520,264241152~16777216,285212672~96468992,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.489 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~16777216,285212672~96468992,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.496 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~16777216,285212672~134217728] 2024-09-21T10:21:40.496 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to 19 diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.496 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.502 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:40.503 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 4 diff [29360128~4194304,62914560~8388608,138412032~4194304,150994944~4194304,180355072~4194304,226492416~4194304,239075328~8388608,268435456~4194304,331350016~8388608,352321536~4194304] 2024-09-21T10:21:40.503 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [29360128~4194304,62914560~8388608,138412032~4194304,150994944~4194304,180355072~4194304,226492416~4194304,239075328~8388608,268435456~4194304,331350016~8388608,352321536~4194304] 2024-09-21T10:21:40.503 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [29360128~8388608,62914560~12582912,138412032~4194304,150994944~8388608,180355072~8388608,226492416~8388608,239075328~8388608,268435456~8388608,331350016~12582912,352321536~4194304] 2024-09-21T10:21:40.503 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 5 diff [29360128~4194304,62914560~16777216,138412032~4194304,150994944~12582912,176160768~8388608,209715200~8388608,226492416~4194304,239075328~8388608,251658240~4194304,268435456~4194304,314572800~8388608,331350016~8388608,352321536~4194304,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.504 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [29360128~4194304,62914560~16777216,138412032~4194304,150994944~12582912,176160768~8388608,209715200~8388608,226492416~4194304,239075328~8388608,251658240~4194304,268435456~4194304,314572800~8388608,331350016~8388608,352321536~4194304,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.505 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [29360128~8388608,62914560~16777216,138412032~4194304,150994944~16777216,176160768~12582912,209715200~12582912,226492416~8388608,239075328~8388608,251658240~4194304,268435456~8388608,314572800~12582912,331350016~12582912,352321536~4194304,364904448~20971520,398458880~4194304] 2024-09-21T10:21:40.505 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 6 diff [8388608~4194304,20971520~4194304,29360128~4194304,41943040~4194304,50331648~8388608,62914560~16777216,88080384~4194304,138412032~4194304,150994944~12582912,176160768~8388608,209715200~8388608,226492416~4194304,239075328~8388608,251658240~4194304,268435456~4194304,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.505 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~4194304,20971520~4194304,29360128~4194304,41943040~4194304,50331648~8388608,62914560~16777216,88080384~4194304,138412032~4194304,150994944~12582912,176160768~8388608,209715200~8388608,226492416~4194304,239075328~8388608,251658240~4194304,268435456~4194304,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.507 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~4194304,20971520~16777216,41943040~4194304,50331648~8388608,62914560~16777216,88080384~8388608,138412032~4194304,150994944~16777216,176160768~12582912,209715200~12582912,226492416~8388608,239075328~8388608,251658240~4194304,268435456~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~20971520,398458880~4194304] 2024-09-21T10:21:40.507 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 7 diff [8388608~8388608,20971520~4194304,29360128~4194304,41943040~16777216,62914560~20971520,88080384~4194304,138412032~4194304,150994944~16777216,176160768~8388608,192937984~4194304,209715200~8388608,226492416~4194304,234881024~12582912,251658240~4194304,268435456~4194304,314572800~8388608,331350016~12582912,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.507 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~8388608,20971520~4194304,29360128~4194304,41943040~16777216,62914560~20971520,88080384~4194304,138412032~4194304,150994944~16777216,176160768~8388608,192937984~4194304,209715200~8388608,226492416~4194304,234881024~12582912,251658240~4194304,268435456~4194304,314572800~8388608,331350016~12582912,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.509 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~16777216,62914560~33554432,138412032~4194304,150994944~20971520,176160768~12582912,192937984~8388608,209715200~12582912,226492416~20971520,251658240~4194304,268435456~8388608,314572800~12582912,331350016~12582912,348127232~12582912,364904448~20971520,398458880~4194304] 2024-09-21T10:21:40.509 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 8 diff [8388608~8388608,20971520~12582912,41943040~16777216,62914560~29360128,138412032~4194304,150994944~16777216,176160768~8388608,192937984~4194304,209715200~8388608,226492416~4194304,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.509 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~8388608,20971520~12582912,41943040~16777216,62914560~29360128,138412032~4194304,150994944~16777216,176160768~8388608,192937984~4194304,209715200~8388608,226492416~4194304,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.512 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~16777216,62914560~33554432,138412032~4194304,150994944~20971520,176160768~12582912,192937984~8388608,209715200~12582912,226492416~20971520,251658240~4194304,268435456~8388608,285212672~12582912,314572800~12582912,331350016~29360128,364904448~20971520,398458880~12582912] 2024-09-21T10:21:40.512 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 9 diff [4194304~12582912,20971520~16777216,41943040~16777216,62914560~29360128,138412032~4194304,150994944~16777216,176160768~8388608,188743680~16777216,209715200~8388608,226492416~4194304,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.512 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~16777216,62914560~29360128,138412032~4194304,150994944~16777216,176160768~8388608,188743680~16777216,209715200~8388608,226492416~4194304,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.514 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~54525952,62914560~33554432,138412032~4194304,150994944~20971520,176160768~29360128,209715200~12582912,226492416~20971520,251658240~4194304,268435456~8388608,285212672~12582912,314572800~12582912,331350016~29360128,364904448~20971520,398458880~12582912] 2024-09-21T10:21:40.514 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 10 diff [4194304~12582912,20971520~16777216,41943040~16777216,62914560~37748736,138412032~4194304,150994944~16777216,176160768~8388608,188743680~16777216,209715200~20971520,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,310378496~12582912,331350016~25165824,360710144~20971520,398458880~8388608] 2024-09-21T10:21:40.514 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~16777216,41943040~16777216,62914560~37748736,138412032~4194304,150994944~16777216,176160768~8388608,188743680~16777216,209715200~20971520,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,310378496~12582912,331350016~25165824,360710144~20971520,398458880~8388608] 2024-09-21T10:21:40.517 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~54525952,62914560~37748736,138412032~4194304,150994944~20971520,176160768~29360128,209715200~37748736,251658240~4194304,268435456~8388608,285212672~16777216,310378496~16777216,331350016~54525952,398458880~12582912] 2024-09-21T10:21:40.517 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 11 diff [0~37748736,41943040~16777216,62914560~37748736,121634816~8388608,138412032~8388608,150994944~16777216,176160768~8388608,188743680~16777216,209715200~20971520,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.517 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~37748736,41943040~16777216,62914560~37748736,121634816~8388608,138412032~8388608,150994944~16777216,176160768~8388608,188743680~16777216,209715200~20971520,234881024~12582912,251658240~4194304,268435456~8388608,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.521 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,62914560~37748736,121634816~8388608,138412032~33554432,176160768~29360128,209715200~37748736,251658240~4194304,268435456~8388608,285212672~16777216,310378496~75497472,398458880~20971520] 2024-09-21T10:21:40.521 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 12 diff [0~37748736,41943040~16777216,62914560~46137344,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~16777216,209715200~20971520,234881024~20971520,264241152~12582912,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.521 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~37748736,41943040~16777216,62914560~46137344,121634816~8388608,138412032~8388608,150994944~20971520,176160768~8388608,188743680~16777216,209715200~20971520,234881024~20971520,264241152~12582912,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.524 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,62914560~46137344,121634816~8388608,138412032~33554432,176160768~29360128,209715200~46137344,264241152~12582912,285212672~16777216,310378496~75497472,398458880~20971520] 2024-09-21T10:21:40.524 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 13 diff [0~109051904,121634816~8388608,138412032~33554432,176160768~8388608,188743680~41943040,234881024~20971520,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.524 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~8388608,138412032~33554432,176160768~8388608,188743680~41943040,234881024~20971520,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.528 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~12582912,138412032~33554432,176160768~79691776,264241152~16777216,285212672~16777216,310378496~83886080,398458880~20971520] 2024-09-21T10:21:40.528 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 14 diff [0~109051904,121634816~8388608,138412032~46137344,188743680~41943040,234881024~20971520,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.528 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~8388608,138412032~46137344,188743680~41943040,234881024~20971520,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.532 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~12582912,138412032~117440512,264241152~16777216,285212672~16777216,310378496~83886080,398458880~20971520] 2024-09-21T10:21:40.532 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 15 diff [0~109051904,121634816~79691776,205520896~25165824,234881024~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.532 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~79691776,205520896~25165824,234881024~20971520,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.537 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~134217728,264241152~16777216,285212672~20971520,310378496~83886080,398458880~20971520] 2024-09-21T10:21:40.537 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 16 diff [0~109051904,121634816~79691776,205520896~25165824,234881024~20971520,264241152~16777216,285212672~4194304,293601280~88080384,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.537 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,121634816~79691776,205520896~25165824,234881024~20971520,264241152~16777216,285212672~4194304,293601280~88080384,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.542 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,121634816~134217728,264241152~16777216,285212672~109051904,398458880~20971520] 2024-09-21T10:21:40.543 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 17 diff [0~109051904,113246208~88080384,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.543 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~109051904,113246208~88080384,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:40 smithi145 ceph-mon[89359]: pgmap v1487: 105 pgs: 105 active+clean; 1.1 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 222 KiB/s rd, 39 MiB/s wr, 346 op/s 2024-09-21T10:21:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:40 smithi145 ceph-mon[89359]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-21T10:21:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:21:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:21:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:40.548 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~109051904,113246208~142606336,264241152~16777216,285212672~109051904,398458880~20971520] 2024-09-21T10:21:40.548 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 18 diff [0~201326592,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.548 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.553 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~16777216,285212672~109051904,398458880~20971520] 2024-09-21T10:21:40.553 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to 19 diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.553 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.559 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:40.559 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 5 diff [71303168~8388608,155189248~8388608,176160768~4194304,209715200~8388608,251658240~4194304,314572800~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.559 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [71303168~8388608,155189248~8388608,176160768~4194304,209715200~8388608,251658240~4194304,314572800~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.560 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [71303168~8388608,155189248~12582912,176160768~4194304,209715200~12582912,251658240~4194304,314572800~12582912,364904448~20971520,398458880~4194304] 2024-09-21T10:21:40.560 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 6 diff [8388608~4194304,20971520~4194304,41943040~4194304,50331648~8388608,67108864~12582912,88080384~4194304,155189248~8388608,176160768~4194304,209715200~8388608,251658240~4194304,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.560 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~4194304,20971520~4194304,41943040~4194304,50331648~8388608,67108864~12582912,88080384~4194304,155189248~8388608,176160768~4194304,209715200~8388608,251658240~4194304,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.562 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~4194304,20971520~8388608,41943040~4194304,50331648~8388608,67108864~12582912,88080384~8388608,155189248~12582912,176160768~4194304,209715200~12582912,251658240~4194304,314572800~12582912,331350016~12582912,348127232~12582912,364904448~20971520,398458880~4194304] 2024-09-21T10:21:40.562 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 7 diff [8388608~8388608,20971520~4194304,41943040~16777216,67108864~16777216,88080384~4194304,150994944~16777216,176160768~4194304,192937984~4194304,209715200~8388608,234881024~4194304,251658240~4194304,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.562 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~8388608,20971520~4194304,41943040~16777216,67108864~16777216,88080384~4194304,150994944~16777216,176160768~4194304,192937984~4194304,209715200~8388608,234881024~4194304,251658240~4194304,314572800~8388608,331350016~8388608,348127232~8388608,364904448~16777216,398458880~4194304] 2024-09-21T10:21:40.564 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~20971520,41943040~16777216,67108864~29360128,150994944~20971520,176160768~4194304,192937984~8388608,209715200~12582912,234881024~4194304,251658240~4194304,314572800~12582912,331350016~12582912,348127232~12582912,364904448~20971520,398458880~4194304] 2024-09-21T10:21:40.564 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 8 diff [8388608~8388608,20971520~12582912,41943040~16777216,67108864~25165824,150994944~16777216,176160768~4194304,192937984~4194304,209715200~8388608,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.564 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~8388608,20971520~12582912,41943040~16777216,67108864~25165824,150994944~16777216,176160768~4194304,192937984~4194304,209715200~8388608,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.566 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~16777216,67108864~29360128,150994944~20971520,176160768~4194304,192937984~8388608,209715200~12582912,234881024~4194304,251658240~4194304,272629760~4194304,285212672~12582912,314572800~12582912,331350016~29360128,364904448~20971520,398458880~12582912] 2024-09-21T10:21:40.567 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 9 diff [4194304~12582912,20971520~37748736,67108864~25165824,150994944~16777216,176160768~4194304,188743680~16777216,209715200~8388608,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.567 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~37748736,67108864~25165824,150994944~16777216,176160768~4194304,188743680~16777216,209715200~8388608,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,314572800~8388608,331350016~25165824,364904448~16777216,398458880~8388608] 2024-09-21T10:21:40.569 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~54525952,67108864~29360128,150994944~20971520,176160768~4194304,188743680~16777216,209715200~12582912,234881024~4194304,251658240~4194304,272629760~4194304,285212672~12582912,314572800~12582912,331350016~29360128,364904448~20971520,398458880~12582912] 2024-09-21T10:21:40.569 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 10 diff [4194304~12582912,20971520~37748736,67108864~33554432,150994944~16777216,176160768~4194304,188743680~16777216,209715200~16777216,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~12582912,331350016~25165824,360710144~20971520,398458880~8388608] 2024-09-21T10:21:40.569 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~37748736,67108864~33554432,150994944~16777216,176160768~4194304,188743680~16777216,209715200~16777216,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~12582912,331350016~25165824,360710144~20971520,398458880~8388608] 2024-09-21T10:21:40.572 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~54525952,67108864~33554432,150994944~20971520,176160768~4194304,188743680~16777216,209715200~20971520,234881024~4194304,251658240~4194304,272629760~4194304,285212672~16777216,310378496~16777216,331350016~54525952,398458880~12582912] 2024-09-21T10:21:40.572 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 11 diff [0~58720256,67108864~33554432,121634816~8388608,142606336~4194304,150994944~16777216,176160768~4194304,188743680~16777216,209715200~16777216,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.572 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~58720256,67108864~33554432,121634816~8388608,142606336~4194304,150994944~16777216,176160768~4194304,188743680~16777216,209715200~16777216,234881024~4194304,251658240~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.576 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,67108864~33554432,121634816~8388608,142606336~29360128,176160768~4194304,188743680~16777216,209715200~20971520,234881024~4194304,251658240~4194304,272629760~4194304,285212672~16777216,310378496~75497472,398458880~20971520] 2024-09-21T10:21:40.577 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 12 diff [0~58720256,67108864~41943040,121634816~8388608,142606336~4194304,150994944~20971520,176160768~4194304,188743680~16777216,209715200~16777216,234881024~4194304,247463936~8388608,264241152~12582912,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.577 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~58720256,67108864~41943040,121634816~8388608,142606336~4194304,150994944~20971520,176160768~4194304,188743680~16777216,209715200~16777216,234881024~4194304,247463936~8388608,264241152~12582912,285212672~4194304,293601280~4194304,310378496~71303168,398458880~16777216] 2024-09-21T10:21:40.581 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,67108864~41943040,121634816~8388608,142606336~29360128,176160768~4194304,188743680~16777216,209715200~20971520,234881024~4194304,247463936~8388608,264241152~12582912,285212672~16777216,310378496~75497472,398458880~20971520] 2024-09-21T10:21:40.581 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 13 diff [0~62914560,67108864~41943040,121634816~8388608,142606336~29360128,176160768~8388608,188743680~37748736,234881024~4194304,247463936~8388608,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.581 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~8388608,142606336~29360128,176160768~8388608,188743680~37748736,234881024~4194304,247463936~8388608,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.585 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~12582912,142606336~29360128,176160768~54525952,234881024~4194304,247463936~8388608,264241152~16777216,285212672~16777216,310378496~83886080,398458880~20971520] 2024-09-21T10:21:40.585 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 14 diff [0~62914560,67108864~41943040,121634816~8388608,138412032~46137344,188743680~41943040,234881024~4194304,247463936~8388608,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.585 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~8388608,138412032~46137344,188743680~41943040,234881024~4194304,247463936~8388608,264241152~16777216,285212672~4194304,293601280~4194304,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.590 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~12582912,138412032~100663296,247463936~8388608,264241152~16777216,285212672~16777216,310378496~83886080,398458880~20971520] 2024-09-21T10:21:40.590 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 15 diff [0~62914560,67108864~41943040,121634816~79691776,205520896~25165824,234881024~8388608,247463936~8388608,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.590 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~79691776,205520896~25165824,234881024~8388608,247463936~8388608,264241152~16777216,285212672~4194304,293601280~12582912,310378496~71303168,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.596 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~121634816,247463936~8388608,264241152~16777216,285212672~20971520,310378496~83886080,398458880~20971520] 2024-09-21T10:21:40.596 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 16 diff [0~62914560,67108864~41943040,121634816~79691776,205520896~25165824,234881024~8388608,247463936~8388608,264241152~16777216,285212672~4194304,293601280~88080384,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.596 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~79691776,205520896~25165824,234881024~8388608,247463936~8388608,264241152~16777216,285212672~4194304,293601280~88080384,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.602 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~121634816,247463936~8388608,264241152~16777216,285212672~109051904,398458880~20971520] 2024-09-21T10:21:40.602 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 17 diff [0~62914560,67108864~41943040,113246208~88080384,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.602 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,113246208~88080384,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.608 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,113246208~142606336,264241152~16777216,285212672~109051904,398458880~20971520] 2024-09-21T10:21:40.608 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 18 diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.608 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~16777216,285212672~96468992,385875968~4194304,398458880~16777216] 2024-09-21T10:21:40.615 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~16777216,285212672~109051904,398458880~20971520] 2024-09-21T10:21:40.615 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to 19 diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.615 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:40 smithi119 ceph-mon[101432]: pgmap v1487: 105 pgs: 105 active+clean; 1.1 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 222 KiB/s rd, 39 MiB/s wr, 346 op/s 2024-09-21T10:21:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:40 smithi119 ceph-mon[101432]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-21T10:21:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:40 smithi119 ceph-mon[105288]: pgmap v1487: 105 pgs: 105 active+clean; 1.1 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 222 KiB/s rd, 39 MiB/s wr, 346 op/s 2024-09-21T10:21:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:40 smithi119 ceph-mon[105288]: osdmap e1136: 8 total, 8 up, 8 in 2024-09-21T10:21:40.620 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:40.620 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 6 diff [8388608~4194304,20971520~4194304,41943040~4194304,50331648~8388608,67108864~4194304,88080384~4194304,331350016~8388608,348127232~8388608,364904448~8388608] 2024-09-21T10:21:40.621 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~4194304,20971520~4194304,41943040~4194304,50331648~8388608,67108864~4194304,88080384~4194304,331350016~8388608,348127232~8388608,364904448~8388608] 2024-09-21T10:21:40.622 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~4194304,20971520~8388608,41943040~4194304,50331648~8388608,67108864~4194304,88080384~8388608,331350016~12582912,348127232~12582912,364904448~8388608] 2024-09-21T10:21:40.622 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 7 diff [8388608~8388608,20971520~4194304,41943040~16777216,67108864~4194304,79691776~4194304,88080384~4194304,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,331350016~8388608,348127232~8388608,364904448~16777216] 2024-09-21T10:21:40.622 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~8388608,20971520~4194304,41943040~16777216,67108864~4194304,79691776~4194304,88080384~4194304,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,331350016~8388608,348127232~8388608,364904448~16777216] 2024-09-21T10:21:40.624 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~20971520,41943040~16777216,67108864~4194304,79691776~16777216,150994944~8388608,163577856~8388608,192937984~8388608,234881024~4194304,331350016~12582912,348127232~12582912,364904448~20971520] 2024-09-21T10:21:40.624 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 8 diff [8388608~8388608,20971520~12582912,41943040~16777216,67108864~4194304,79691776~12582912,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,331350016~25165824,364904448~16777216,402653184~4194304] 2024-09-21T10:21:40.624 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [8388608~8388608,20971520~12582912,41943040~16777216,67108864~4194304,79691776~12582912,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,331350016~25165824,364904448~16777216,402653184~4194304] 2024-09-21T10:21:40.626 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [8388608~29360128,41943040~16777216,67108864~4194304,79691776~16777216,150994944~8388608,163577856~8388608,192937984~8388608,234881024~4194304,272629760~4194304,285212672~12582912,331350016~29360128,364904448~20971520,402653184~8388608] 2024-09-21T10:21:40.626 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 9 diff [4194304~12582912,20971520~37748736,67108864~25165824,150994944~8388608,163577856~4194304,188743680~16777216,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,331350016~25165824,364904448~16777216,402653184~4194304] 2024-09-21T10:21:40.626 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~37748736,67108864~25165824,150994944~8388608,163577856~4194304,188743680~16777216,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,331350016~25165824,364904448~16777216,402653184~4194304] 2024-09-21T10:21:40.628 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~54525952,67108864~29360128,150994944~8388608,163577856~8388608,188743680~16777216,234881024~4194304,272629760~4194304,285212672~12582912,331350016~29360128,364904448~20971520,402653184~8388608] 2024-09-21T10:21:40.628 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 10 diff [4194304~12582912,20971520~37748736,67108864~33554432,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,331350016~25165824,360710144~20971520,402653184~4194304] 2024-09-21T10:21:40.628 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~37748736,67108864~33554432,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,331350016~25165824,360710144~20971520,402653184~4194304] 2024-09-21T10:21:40.630 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~54525952,67108864~33554432,150994944~8388608,163577856~8388608,188743680~16777216,218103808~12582912,234881024~4194304,272629760~4194304,285212672~16777216,310378496~8388608,331350016~54525952,402653184~8388608] 2024-09-21T10:21:40.630 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 11 diff [0~58720256,67108864~33554432,121634816~8388608,142606336~4194304,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,402653184~12582912] 2024-09-21T10:21:40.630 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~58720256,67108864~33554432,121634816~8388608,142606336~4194304,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,402653184~12582912] 2024-09-21T10:21:40.634 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,67108864~33554432,121634816~8388608,142606336~16777216,163577856~8388608,188743680~16777216,218103808~12582912,234881024~4194304,272629760~4194304,285212672~16777216,310378496~8388608,322961408~62914560,402653184~16777216] 2024-09-21T10:21:40.634 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 12 diff [0~58720256,67108864~41943040,121634816~8388608,142606336~4194304,150994944~8388608,163577856~8388608,188743680~16777216,218103808~8388608,234881024~4194304,247463936~4194304,264241152~12582912,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,402653184~12582912] 2024-09-21T10:21:40.634 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~58720256,67108864~41943040,121634816~8388608,142606336~4194304,150994944~8388608,163577856~8388608,188743680~16777216,218103808~8388608,234881024~4194304,247463936~4194304,264241152~12582912,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,402653184~12582912] 2024-09-21T10:21:40.637 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,67108864~41943040,121634816~8388608,142606336~16777216,163577856~8388608,188743680~16777216,218103808~12582912,234881024~4194304,247463936~8388608,264241152~12582912,285212672~16777216,310378496~8388608,322961408~62914560,402653184~16777216] 2024-09-21T10:21:40.637 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 13 diff [0~62914560,67108864~41943040,121634816~8388608,142606336~16777216,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.638 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~8388608,142606336~16777216,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.641 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~12582912,142606336~16777216,163577856~8388608,176160768~33554432,218103808~12582912,234881024~4194304,247463936~8388608,264241152~16777216,285212672~16777216,310378496~8388608,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.641 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 14 diff [0~62914560,67108864~41943040,121634816~8388608,138412032~20971520,163577856~20971520,188743680~20971520,218103808~12582912,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.641 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~8388608,138412032~20971520,163577856~20971520,188743680~20971520,218103808~12582912,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.646 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~12582912,138412032~20971520,163577856~46137344,218103808~20971520,247463936~8388608,264241152~16777216,285212672~16777216,310378496~8388608,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.646 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 15 diff [0~62914560,67108864~41943040,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~12582912,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.646 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~12582912,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.650 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~88080384,218103808~25165824,247463936~8388608,264241152~16777216,285212672~20971520,310378496~8388608,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.650 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 16 diff [0~62914560,67108864~41943040,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~25165824,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.651 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~25165824,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.655 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,121634816~88080384,218103808~25165824,247463936~8388608,264241152~16777216,285212672~33554432,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.655 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 17 diff [0~62914560,67108864~41943040,113246208~88080384,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.655 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~41943040,113246208~88080384,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.659 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~41943040,113246208~96468992,213909504~41943040,264241152~16777216,285212672~33554432,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.659 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 18 diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.660 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.664 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~16777216,285212672~33554432,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.664 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to 19 diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:40.665 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:40.670 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:40.670 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 7 diff [12582912~4194304,41943040~8388608,79691776~4194304,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,335544320~8388608,373293056~8388608] 2024-09-21T10:21:40.670 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,41943040~8388608,79691776~4194304,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,335544320~8388608,373293056~8388608] 2024-09-21T10:21:40.671 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~8388608,41943040~8388608,79691776~8388608,150994944~8388608,163577856~8388608,192937984~8388608,234881024~4194304,335544320~8388608,373293056~12582912] 2024-09-21T10:21:40.671 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 8 diff [12582912~4194304,20971520~12582912,41943040~8388608,54525952~4194304,79691776~8388608,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,335544320~12582912,373293056~8388608,402653184~4194304] 2024-09-21T10:21:40.671 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,20971520~12582912,41943040~8388608,54525952~4194304,79691776~8388608,150994944~8388608,163577856~4194304,192937984~4194304,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,335544320~12582912,373293056~8388608,402653184~4194304] 2024-09-21T10:21:40.673 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~25165824,41943040~8388608,54525952~4194304,79691776~8388608,150994944~8388608,163577856~8388608,192937984~8388608,234881024~4194304,272629760~4194304,285212672~12582912,335544320~16777216,373293056~12582912,402653184~8388608] 2024-09-21T10:21:40.673 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 9 diff [4194304~12582912,20971520~29360128,54525952~4194304,71303168~20971520,150994944~8388608,163577856~4194304,188743680~16777216,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,335544320~16777216,364904448~16777216,402653184~4194304] 2024-09-21T10:21:40.673 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~29360128,54525952~4194304,71303168~20971520,150994944~8388608,163577856~4194304,188743680~16777216,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,335544320~16777216,364904448~16777216,402653184~4194304] 2024-09-21T10:21:40.675 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~46137344,54525952~4194304,71303168~25165824,150994944~8388608,163577856~8388608,188743680~16777216,234881024~4194304,272629760~4194304,285212672~12582912,335544320~20971520,364904448~20971520,402653184~8388608] 2024-09-21T10:21:40.675 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 10 diff [4194304~12582912,20971520~29360128,54525952~4194304,71303168~25165824,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,335544320~16777216,360710144~20971520,402653184~4194304] 2024-09-21T10:21:40.675 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~29360128,54525952~4194304,71303168~25165824,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,335544320~16777216,360710144~20971520,402653184~4194304] 2024-09-21T10:21:40.679 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~46137344,54525952~4194304,71303168~29360128,150994944~8388608,163577856~8388608,188743680~16777216,218103808~12582912,234881024~4194304,272629760~4194304,285212672~16777216,310378496~8388608,335544320~20971520,360710144~25165824,402653184~8388608] 2024-09-21T10:21:40.679 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 11 diff [0~50331648,54525952~4194304,71303168~25165824,121634816~8388608,142606336~4194304,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,322961408~8388608,335544320~16777216,356515840~25165824,402653184~12582912] 2024-09-21T10:21:40.679 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~50331648,54525952~4194304,71303168~25165824,121634816~8388608,142606336~4194304,150994944~8388608,163577856~4194304,188743680~16777216,218103808~8388608,234881024~4194304,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,322961408~8388608,335544320~16777216,356515840~25165824,402653184~12582912] 2024-09-21T10:21:40.682 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~50331648,54525952~4194304,71303168~29360128,121634816~8388608,142606336~16777216,163577856~8388608,188743680~16777216,218103808~12582912,234881024~4194304,272629760~4194304,285212672~16777216,310378496~8388608,322961408~62914560,402653184~16777216] 2024-09-21T10:21:40.682 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 12 diff [0~58720256,71303168~25165824,100663296~8388608,121634816~8388608,142606336~4194304,150994944~8388608,163577856~8388608,188743680~16777216,218103808~8388608,234881024~4194304,247463936~4194304,264241152~12582912,285212672~4194304,293601280~4194304,310378496~8388608,322961408~29360128,356515840~25165824,402653184~12582912] 2024-09-21T10:21:40.682 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~58720256,71303168~25165824,100663296~8388608,121634816~8388608,142606336~4194304,150994944~8388608,163577856~8388608,188743680~16777216,218103808~8388608,234881024~4194304,247463936~4194304,264241152~12582912,285212672~4194304,293601280~4194304,310378496~8388608,322961408~29360128,356515840~25165824,402653184~12582912] 2024-09-21T10:21:40.685 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,71303168~37748736,121634816~8388608,142606336~16777216,163577856~8388608,188743680~16777216,218103808~12582912,234881024~4194304,247463936~8388608,264241152~12582912,285212672~16777216,310378496~8388608,322961408~62914560,402653184~16777216] 2024-09-21T10:21:40.685 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 13 diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,142606336~16777216,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~29360128,356515840~25165824,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.685 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,142606336~16777216,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~29360128,356515840~25165824,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.688 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~12582912,142606336~16777216,163577856~8388608,176160768~33554432,218103808~12582912,234881024~4194304,247463936~8388608,264241152~16777216,285212672~16777216,310378496~8388608,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.688 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 14 diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,138412032~20971520,163577856~20971520,188743680~20971520,218103808~12582912,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~29360128,356515840~25165824,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.689 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,138412032~20971520,163577856~20971520,188743680~20971520,218103808~12582912,234881024~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~29360128,356515840~25165824,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.692 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~12582912,138412032~20971520,163577856~46137344,218103808~20971520,247463936~8388608,264241152~16777216,285212672~16777216,310378496~8388608,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.692 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 15 diff [0~62914560,71303168~25165824,100663296~8388608,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~12582912,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.692 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~25165824,100663296~8388608,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~12582912,310378496~8388608,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.696 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~88080384,218103808~25165824,247463936~8388608,264241152~16777216,285212672~20971520,310378496~8388608,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.696 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 16 diff [0~62914560,71303168~33554432,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~25165824,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.696 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,121634816~79691776,205520896~4194304,218103808~12582912,234881024~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~25165824,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.701 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~88080384,218103808~25165824,247463936~8388608,264241152~16777216,285212672~33554432,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.701 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 17 diff [0~62914560,71303168~33554432,113246208~88080384,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.701 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,113246208~88080384,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.706 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,113246208~96468992,213909504~41943040,264241152~16777216,285212672~33554432,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.706 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 18 diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.706 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.711 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~16777216,285212672~33554432,322961408~71303168,402653184~16777216] 2024-09-21T10:21:40.711 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to 19 diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:40.711 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:40.716 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:40.716 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 8 diff [20971520~12582912,41943040~4194304,54525952~4194304,83886080~4194304,272629760~4194304,285212672~4194304,293601280~4194304,339738624~8388608,402653184~4194304] 2024-09-21T10:21:40.716 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~12582912,41943040~4194304,54525952~4194304,83886080~4194304,272629760~4194304,285212672~4194304,293601280~4194304,339738624~8388608,402653184~4194304] 2024-09-21T10:21:40.717 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~16777216,41943040~4194304,54525952~4194304,83886080~4194304,272629760~4194304,285212672~12582912,339738624~12582912,402653184~8388608] 2024-09-21T10:21:40.717 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 9 diff [4194304~8388608,20971520~25165824,54525952~4194304,71303168~20971520,188743680~16777216,272629760~4194304,285212672~4194304,293601280~4194304,339738624~12582912,364904448~12582912,402653184~4194304] 2024-09-21T10:21:40.718 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~8388608,20971520~25165824,54525952~4194304,71303168~20971520,188743680~16777216,272629760~4194304,285212672~4194304,293601280~4194304,339738624~12582912,364904448~12582912,402653184~4194304] 2024-09-21T10:21:40.719 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~12582912,20971520~25165824,54525952~4194304,71303168~25165824,188743680~16777216,272629760~4194304,285212672~12582912,339738624~16777216,364904448~16777216,402653184~8388608] 2024-09-21T10:21:40.719 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 10 diff [4194304~12582912,20971520~25165824,54525952~4194304,71303168~25165824,188743680~16777216,218103808~8388608,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,339738624~12582912,360710144~16777216,402653184~4194304] 2024-09-21T10:21:40.719 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,20971520~25165824,54525952~4194304,71303168~25165824,188743680~16777216,218103808~8388608,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,339738624~12582912,360710144~16777216,402653184~4194304] 2024-09-21T10:21:40.721 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~12582912,20971520~25165824,54525952~4194304,71303168~29360128,188743680~16777216,218103808~12582912,272629760~4194304,285212672~16777216,310378496~8388608,339738624~16777216,360710144~20971520,402653184~8388608] 2024-09-21T10:21:40.721 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 11 diff [0~46137344,54525952~4194304,71303168~25165824,121634816~8388608,142606336~4194304,155189248~4194304,188743680~16777216,218103808~8388608,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,322961408~8388608,339738624~12582912,356515840~20971520,402653184~12582912] 2024-09-21T10:21:40.721 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~46137344,54525952~4194304,71303168~25165824,121634816~8388608,142606336~4194304,155189248~4194304,188743680~16777216,218103808~8388608,272629760~4194304,285212672~4194304,293601280~4194304,310378496~8388608,322961408~8388608,339738624~12582912,356515840~20971520,402653184~12582912] 2024-09-21T10:21:40.723 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~46137344,54525952~4194304,71303168~29360128,121634816~8388608,142606336~8388608,155189248~4194304,188743680~16777216,218103808~12582912,272629760~4194304,285212672~16777216,310378496~8388608,322961408~12582912,339738624~41943040,402653184~16777216] 2024-09-21T10:21:40.723 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 12 diff [0~58720256,71303168~25165824,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,188743680~16777216,218103808~8388608,247463936~4194304,264241152~12582912,285212672~4194304,293601280~4194304,310378496~8388608,322961408~12582912,339738624~12582912,356515840~20971520,402653184~12582912] 2024-09-21T10:21:40.723 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~58720256,71303168~25165824,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,188743680~16777216,218103808~8388608,247463936~4194304,264241152~12582912,285212672~4194304,293601280~4194304,310378496~8388608,322961408~12582912,339738624~12582912,356515840~20971520,402653184~12582912] 2024-09-21T10:21:40.726 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~58720256,71303168~37748736,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,188743680~16777216,218103808~12582912,247463936~8388608,264241152~12582912,285212672~16777216,310378496~8388608,322961408~12582912,339738624~41943040,402653184~16777216] 2024-09-21T10:21:40.726 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 13 diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~12582912,339738624~12582912,356515840~20971520,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.726 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~12582912,339738624~12582912,356515840~20971520,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.730 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~12582912,142606336~16777216,163577856~8388608,176160768~33554432,218103808~12582912,247463936~8388608,264241152~16777216,285212672~16777216,310378496~8388608,322961408~12582912,339738624~41943040,385875968~8388608,402653184~16777216] 2024-09-21T10:21:40.730 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 14 diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,188743680~20971520,218103808~12582912,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~12582912,339738624~12582912,356515840~20971520,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.730 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~25165824,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,188743680~20971520,218103808~12582912,247463936~4194304,264241152~16777216,285212672~4194304,293601280~4194304,310378496~8388608,322961408~12582912,339738624~12582912,356515840~20971520,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.733 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~12582912,138412032~20971520,163577856~46137344,218103808~16777216,247463936~8388608,264241152~16777216,285212672~16777216,310378496~8388608,322961408~12582912,339738624~41943040,385875968~8388608,402653184~16777216] 2024-09-21T10:21:40.733 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 15 diff [0~62914560,71303168~25165824,100663296~8388608,121634816~29360128,155189248~54525952,218103808~12582912,239075328~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~37748736,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.733 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~25165824,100663296~8388608,121634816~29360128,155189248~54525952,218103808~12582912,239075328~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~37748736,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.737 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~88080384,218103808~16777216,239075328~4194304,247463936~8388608,264241152~16777216,285212672~20971520,310378496~8388608,322961408~12582912,339738624~41943040,385875968~8388608,402653184~16777216] 2024-09-21T10:21:40.737 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 16 diff [0~62914560,71303168~33554432,121634816~88080384,218103808~12582912,239075328~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~25165824,322961408~54525952,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.737 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,121634816~88080384,218103808~12582912,239075328~4194304,247463936~4194304,264241152~16777216,285212672~4194304,293601280~25165824,322961408~54525952,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.741 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~88080384,218103808~16777216,239075328~4194304,247463936~8388608,264241152~16777216,285212672~33554432,322961408~58720256,385875968~8388608,402653184~16777216] 2024-09-21T10:21:40.741 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 17 diff [0~62914560,71303168~33554432,113246208~96468992,213909504~16777216,239075328~12582912,264241152~16777216,285212672~33554432,322961408~54525952,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.741 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,113246208~96468992,213909504~16777216,239075328~12582912,264241152~16777216,285212672~33554432,322961408~54525952,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.745 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,113246208~96468992,213909504~20971520,239075328~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~8388608,402653184~16777216] 2024-09-21T10:21:40.745 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 18 diff [0~62914560,71303168~33554432,109051904~100663296,213909504~16777216,239075328~12582912,264241152~16777216,285212672~33554432,322961408~54525952,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.745 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~100663296,213909504~16777216,239075328~12582912,264241152~16777216,285212672~33554432,322961408~54525952,385875968~4194304,402653184~12582912] 2024-09-21T10:21:40.750 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~16777216,285212672~33554432,322961408~58720256,385875968~8388608,402653184~16777216] 2024-09-21T10:21:40.750 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to 19 diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:40.750 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:40.755 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:40.755 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 9 diff [4194304~8388608,33554432~8388608,71303168~12582912,88080384~4194304,188743680~16777216,343932928~8388608,364904448~12582912] 2024-09-21T10:21:40.755 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~8388608,33554432~8388608,71303168~12582912,88080384~4194304,188743680~16777216,343932928~8388608,364904448~12582912] 2024-09-21T10:21:40.756 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~12582912,33554432~8388608,71303168~25165824,188743680~16777216,343932928~12582912,364904448~16777216] 2024-09-21T10:21:40.756 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 10 diff [4194304~12582912,33554432~8388608,71303168~12582912,88080384~8388608,188743680~16777216,218103808~8388608,293601280~4194304,310378496~8388608,343932928~8388608,360710144~16777216] 2024-09-21T10:21:40.756 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,33554432~8388608,71303168~12582912,88080384~8388608,188743680~16777216,218103808~8388608,293601280~4194304,310378496~8388608,343932928~8388608,360710144~16777216] 2024-09-21T10:21:40.758 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~12582912,33554432~8388608,71303168~29360128,188743680~16777216,218103808~12582912,293601280~8388608,310378496~8388608,343932928~12582912,360710144~20971520] 2024-09-21T10:21:40.758 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 11 diff [0~20971520,33554432~8388608,71303168~12582912,88080384~8388608,121634816~8388608,142606336~4194304,155189248~4194304,188743680~16777216,218103808~8388608,293601280~4194304,310378496~8388608,322961408~8388608,343932928~8388608,356515840~20971520,406847488~8388608] 2024-09-21T10:21:40.758 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~20971520,33554432~8388608,71303168~12582912,88080384~8388608,121634816~8388608,142606336~4194304,155189248~4194304,188743680~16777216,218103808~8388608,293601280~4194304,310378496~8388608,322961408~8388608,343932928~8388608,356515840~20971520,406847488~8388608] 2024-09-21T10:21:40.760 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~20971520,33554432~8388608,71303168~29360128,121634816~8388608,142606336~8388608,155189248~4194304,188743680~16777216,218103808~12582912,293601280~8388608,310378496~8388608,322961408~12582912,343932928~37748736,406847488~12582912] 2024-09-21T10:21:40.760 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 12 diff [0~25165824,33554432~8388608,46137344~8388608,71303168~12582912,88080384~8388608,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,188743680~16777216,218103808~8388608,247463936~4194304,264241152~8388608,293601280~4194304,310378496~8388608,322961408~12582912,343932928~8388608,356515840~20971520,406847488~8388608] 2024-09-21T10:21:40.760 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~25165824,33554432~8388608,46137344~8388608,71303168~12582912,88080384~8388608,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,188743680~16777216,218103808~8388608,247463936~4194304,264241152~8388608,293601280~4194304,310378496~8388608,322961408~12582912,343932928~8388608,356515840~20971520,406847488~8388608] 2024-09-21T10:21:40.762 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~25165824,33554432~8388608,46137344~12582912,71303168~37748736,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,188743680~16777216,218103808~12582912,247463936~8388608,264241152~8388608,293601280~8388608,310378496~8388608,322961408~12582912,343932928~37748736,406847488~12582912] 2024-09-21T10:21:40.762 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 13 diff [0~41943040,46137344~8388608,58720256~4194304,71303168~12582912,88080384~8388608,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,343932928~8388608,356515840~20971520,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.762 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~41943040,46137344~8388608,58720256~4194304,71303168~12582912,88080384~8388608,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,188743680~20971520,218103808~8388608,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,343932928~8388608,356515840~20971520,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.765 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~41943040,46137344~16777216,71303168~37748736,121634816~12582912,142606336~16777216,163577856~8388608,176160768~33554432,218103808~12582912,247463936~8388608,264241152~8388608,276824064~4194304,293601280~8388608,310378496~8388608,322961408~12582912,343932928~37748736,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.765 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 14 diff [0~54525952,58720256~4194304,71303168~12582912,88080384~8388608,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,188743680~20971520,218103808~12582912,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,343932928~8388608,356515840~20971520,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.765 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~54525952,58720256~4194304,71303168~12582912,88080384~8388608,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,188743680~20971520,218103808~12582912,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,343932928~8388608,356515840~20971520,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.767 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~12582912,138412032~20971520,163577856~46137344,218103808~16777216,247463936~8388608,264241152~8388608,276824064~4194304,293601280~8388608,310378496~8388608,322961408~12582912,343932928~37748736,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.768 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 15 diff [0~54525952,58720256~4194304,71303168~12582912,88080384~8388608,100663296~8388608,121634816~29360128,155189248~54525952,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~37748736,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.768 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~54525952,58720256~4194304,71303168~12582912,88080384~8388608,100663296~8388608,121634816~29360128,155189248~54525952,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~37748736,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.771 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~88080384,218103808~16777216,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~41943040,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.771 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 16 diff [0~54525952,58720256~4194304,71303168~12582912,88080384~16777216,121634816~88080384,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~54525952,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.771 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~54525952,58720256~4194304,71303168~12582912,88080384~16777216,121634816~88080384,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~54525952,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.775 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,121634816~88080384,218103808~16777216,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~25165824,322961408~58720256,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.775 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 17 diff [0~62914560,71303168~12582912,88080384~16777216,113246208~96468992,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~54525952,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.775 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~12582912,88080384~16777216,113246208~96468992,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~54525952,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.778 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~37748736,113246208~96468992,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~58720256,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.778 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 18 diff [0~62914560,71303168~12582912,88080384~16777216,109051904~100663296,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~54525952,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.778 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~12582912,88080384~16777216,109051904~100663296,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~54525952,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.782 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~58720256,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.782 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to 19 diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:40.783 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:40.786 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:40.786 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 10 diff [4194304~12582912,88080384~8388608,218103808~8388608,293601280~4194304,310378496~8388608,348127232~4194304,360710144~8388608] 2024-09-21T10:21:40.786 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~12582912,88080384~8388608,218103808~8388608,293601280~4194304,310378496~8388608,348127232~4194304,360710144~8388608] 2024-09-21T10:21:40.787 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~12582912,88080384~12582912,218103808~12582912,293601280~8388608,310378496~8388608,348127232~4194304,360710144~12582912] 2024-09-21T10:21:40.788 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 11 diff [0~20971520,88080384~8388608,121634816~8388608,142606336~4194304,155189248~4194304,218103808~8388608,293601280~4194304,310378496~8388608,322961408~8388608,348127232~4194304,356515840~12582912,406847488~8388608] 2024-09-21T10:21:40.788 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~20971520,88080384~8388608,121634816~8388608,142606336~4194304,155189248~4194304,218103808~8388608,293601280~4194304,310378496~8388608,322961408~8388608,348127232~4194304,356515840~12582912,406847488~8388608] 2024-09-21T10:21:40.789 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~20971520,88080384~12582912,121634816~8388608,142606336~8388608,155189248~4194304,218103808~12582912,293601280~8388608,310378496~8388608,322961408~12582912,348127232~4194304,356515840~16777216,406847488~12582912] 2024-09-21T10:21:40.789 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 12 diff [0~25165824,46137344~8388608,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,218103808~8388608,247463936~4194304,264241152~8388608,293601280~4194304,310378496~8388608,322961408~12582912,348127232~4194304,356515840~12582912,406847488~8388608] 2024-09-21T10:21:40.789 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~25165824,46137344~8388608,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,218103808~8388608,247463936~4194304,264241152~8388608,293601280~4194304,310378496~8388608,322961408~12582912,348127232~4194304,356515840~12582912,406847488~8388608] 2024-09-21T10:21:40.791 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~25165824,46137344~12582912,79691776~29360128,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,218103808~12582912,247463936~8388608,264241152~8388608,293601280~8388608,310378496~8388608,322961408~12582912,348127232~4194304,356515840~16777216,406847488~12582912] 2024-09-21T10:21:40.791 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 13 diff [0~33554432,46137344~8388608,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,205520896~4194304,218103808~8388608,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,348127232~4194304,356515840~12582912,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.791 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,46137344~8388608,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,205520896~4194304,218103808~8388608,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,348127232~4194304,356515840~12582912,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.793 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~37748736,46137344~16777216,79691776~29360128,121634816~12582912,142606336~16777216,163577856~8388608,176160768~12582912,205520896~4194304,218103808~12582912,247463936~8388608,264241152~8388608,276824064~4194304,293601280~8388608,310378496~8388608,322961408~12582912,348127232~4194304,356515840~16777216,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.793 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 14 diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,205520896~4194304,218103808~12582912,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,348127232~4194304,356515840~12582912,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.793 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,205520896~4194304,218103808~12582912,247463936~4194304,264241152~8388608,276824064~4194304,293601280~4194304,310378496~8388608,322961408~12582912,348127232~4194304,356515840~12582912,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.795 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,121634816~12582912,138412032~20971520,163577856~25165824,205520896~4194304,218103808~16777216,247463936~8388608,264241152~8388608,276824064~4194304,293601280~8388608,310378496~8388608,322961408~12582912,348127232~4194304,356515840~16777216,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.795 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 15 diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~29360128,155189248~37748736,205520896~4194304,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~4194304,348127232~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.795 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~29360128,155189248~37748736,205520896~4194304,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~4194304,348127232~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.798 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,121634816~75497472,205520896~4194304,218103808~16777216,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~12582912,310378496~8388608,322961408~12582912,339738624~4194304,348127232~25165824,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.798 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 16 diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~16777216,121634816~71303168,205520896~4194304,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.798 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~16777216,121634816~71303168,205520896~4194304,218103808~12582912,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.801 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,121634816~75497472,205520896~4194304,218103808~16777216,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.801 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 17 diff [0~62914560,79691776~4194304,88080384~16777216,113246208~79691776,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.801 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~4194304,88080384~16777216,113246208~79691776,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.804 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,113246208~83886080,205520896~4194304,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~20971520,348127232~29360128,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.804 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 18 diff [0~62914560,79691776~4194304,88080384~16777216,109051904~83886080,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.804 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~4194304,88080384~16777216,109051904~83886080,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.808 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~117440512,205520896~4194304,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~20971520,348127232~29360128,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.808 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to 19 diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:40.808 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:40.812 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:40.812 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 11 diff [0~20971520,88080384~8388608,121634816~8388608,142606336~4194304,155189248~4194304,322961408~8388608,356515840~8388608,406847488~8388608] 2024-09-21T10:21:40.812 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~20971520,88080384~8388608,121634816~8388608,142606336~4194304,155189248~4194304,322961408~8388608,356515840~8388608,406847488~8388608] 2024-09-21T10:21:40.813 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~20971520,88080384~12582912,121634816~8388608,142606336~8388608,155189248~4194304,322961408~12582912,356515840~12582912,406847488~12582912] 2024-09-21T10:21:40.813 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 12 diff [0~25165824,46137344~8388608,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,247463936~4194304,264241152~8388608,322961408~12582912,356515840~8388608,406847488~8388608] 2024-09-21T10:21:40.813 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~25165824,46137344~8388608,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~4194304,155189248~4194304,163577856~8388608,247463936~4194304,264241152~8388608,322961408~12582912,356515840~8388608,406847488~8388608] 2024-09-21T10:21:40.815 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~25165824,46137344~12582912,79691776~29360128,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,247463936~8388608,264241152~8388608,322961408~12582912,356515840~12582912,406847488~12582912] 2024-09-21T10:21:40.815 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 13 diff [0~33554432,46137344~8388608,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,205520896~4194304,247463936~4194304,264241152~8388608,276824064~4194304,322961408~12582912,356515840~8388608,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.815 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,46137344~8388608,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,142606336~8388608,155189248~4194304,163577856~8388608,176160768~8388608,205520896~4194304,247463936~4194304,264241152~8388608,276824064~4194304,322961408~12582912,356515840~8388608,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.817 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~37748736,46137344~16777216,79691776~29360128,121634816~12582912,142606336~16777216,163577856~8388608,176160768~12582912,205520896~4194304,247463936~8388608,264241152~8388608,276824064~4194304,322961408~12582912,356515840~12582912,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.817 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 14 diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,205520896~4194304,222298112~8388608,247463936~4194304,264241152~8388608,276824064~4194304,322961408~12582912,356515840~12582912,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.817 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~8388608,138412032~12582912,155189248~4194304,163577856~20971520,205520896~4194304,222298112~8388608,247463936~4194304,264241152~8388608,276824064~4194304,322961408~12582912,356515840~12582912,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.819 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,121634816~12582912,138412032~20971520,163577856~25165824,205520896~4194304,222298112~12582912,247463936~8388608,264241152~8388608,276824064~4194304,322961408~12582912,356515840~16777216,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.819 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 15 diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~29360128,155189248~37748736,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,322961408~12582912,339738624~4194304,352321536~20971520,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.819 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~8388608,100663296~8388608,121634816~29360128,155189248~37748736,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,322961408~12582912,339738624~4194304,352321536~20971520,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.821 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,121634816~75497472,205520896~4194304,222298112~12582912,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~12582912,322961408~12582912,339738624~4194304,352321536~20971520,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.821 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 16 diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~16777216,121634816~71303168,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.822 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~33554432,37748736~16777216,58720256~4194304,79691776~4194304,88080384~16777216,121634816~71303168,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.824 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,121634816~75497472,205520896~4194304,222298112~12582912,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.824 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 17 diff [0~62914560,79691776~4194304,88080384~16777216,113246208~79691776,205520896~4194304,213909504~4194304,222298112~8388608,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.824 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~4194304,88080384~16777216,113246208~79691776,205520896~4194304,213909504~4194304,222298112~8388608,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.827 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~29360128,113246208~83886080,205520896~4194304,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~20971520,352321536~25165824,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.828 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 18 diff [0~62914560,79691776~4194304,88080384~16777216,109051904~83886080,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.828 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~4194304,88080384~16777216,109051904~83886080,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,406847488~8388608] 2024-09-21T10:21:40.831 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~117440512,205520896~4194304,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~20971520,352321536~25165824,385875968~8388608,406847488~12582912] 2024-09-21T10:21:40.831 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to 19 diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:40.831 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:40.835 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:40.835 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 12 diff [12582912~8388608,46137344~8388608,79691776~4194304,92274688~4194304,100663296~8388608,163577856~8388608,247463936~4194304,264241152~8388608,322961408~8388608] 2024-09-21T10:21:40.835 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,46137344~8388608,79691776~4194304,92274688~4194304,100663296~8388608,163577856~8388608,247463936~4194304,264241152~8388608,322961408~8388608] 2024-09-21T10:21:40.836 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~12582912,46137344~12582912,79691776~8388608,92274688~4194304,100663296~8388608,163577856~8388608,247463936~8388608,264241152~8388608,322961408~8388608] 2024-09-21T10:21:40.836 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 13 diff [12582912~8388608,25165824~8388608,46137344~8388608,58720256~4194304,79691776~4194304,92274688~4194304,100663296~8388608,125829120~8388608,146800640~4194304,163577856~8388608,176160768~8388608,205520896~4194304,247463936~4194304,264241152~8388608,276824064~4194304,322961408~8388608,385875968~4194304] 2024-09-21T10:21:40.836 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,25165824~8388608,46137344~8388608,58720256~4194304,79691776~4194304,92274688~4194304,100663296~8388608,125829120~8388608,146800640~4194304,163577856~8388608,176160768~8388608,205520896~4194304,247463936~4194304,264241152~8388608,276824064~4194304,322961408~8388608,385875968~4194304] 2024-09-21T10:21:40.838 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~25165824,46137344~16777216,79691776~8388608,92274688~4194304,100663296~8388608,125829120~8388608,146800640~8388608,163577856~8388608,176160768~12582912,205520896~4194304,247463936~8388608,264241152~8388608,276824064~4194304,322961408~8388608,385875968~8388608] 2024-09-21T10:21:40.838 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 14 diff [12582912~8388608,25165824~8388608,37748736~16777216,58720256~4194304,79691776~4194304,92274688~4194304,100663296~8388608,125829120~8388608,138412032~4194304,146800640~4194304,163577856~20971520,205520896~4194304,222298112~8388608,247463936~4194304,264241152~8388608,276824064~4194304,322961408~8388608,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.838 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,25165824~8388608,37748736~16777216,58720256~4194304,79691776~4194304,92274688~4194304,100663296~8388608,125829120~8388608,138412032~4194304,146800640~4194304,163577856~20971520,205520896~4194304,222298112~8388608,247463936~4194304,264241152~8388608,276824064~4194304,322961408~8388608,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.840 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~50331648,79691776~8388608,92274688~4194304,100663296~8388608,125829120~8388608,138412032~16777216,163577856~25165824,205520896~4194304,222298112~12582912,247463936~8388608,264241152~8388608,276824064~4194304,322961408~8388608,364904448~8388608,385875968~8388608] 2024-09-21T10:21:40.840 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 15 diff [12582912~8388608,25165824~8388608,37748736~16777216,58720256~4194304,79691776~4194304,92274688~16777216,125829120~16777216,146800640~4194304,155189248~37748736,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,322961408~8388608,339738624~4194304,352321536~8388608,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.840 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~8388608,25165824~8388608,37748736~16777216,58720256~4194304,79691776~4194304,92274688~16777216,125829120~16777216,146800640~4194304,155189248~37748736,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~12582912,322961408~8388608,339738624~4194304,352321536~8388608,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.842 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~50331648,79691776~8388608,92274688~16777216,125829120~71303168,205520896~4194304,222298112~12582912,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~12582912,322961408~8388608,339738624~4194304,352321536~20971520,385875968~8388608] 2024-09-21T10:21:40.842 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 16 diff [12582912~20971520,37748736~16777216,58720256~4194304,79691776~4194304,92274688~16777216,125829120~16777216,146800640~46137344,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.842 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~20971520,37748736~16777216,58720256~4194304,79691776~4194304,92274688~16777216,125829120~16777216,146800640~46137344,205520896~4194304,222298112~8388608,239075328~4194304,247463936~4194304,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.845 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~50331648,79691776~8388608,92274688~16777216,125829120~71303168,205520896~4194304,222298112~12582912,239075328~4194304,247463936~8388608,264241152~8388608,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~8388608] 2024-09-21T10:21:40.845 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 17 diff [12582912~50331648,79691776~4194304,92274688~16777216,113246208~8388608,125829120~67108864,205520896~4194304,213909504~4194304,222298112~8388608,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.845 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~50331648,79691776~4194304,92274688~16777216,113246208~8388608,125829120~67108864,205520896~4194304,213909504~4194304,222298112~8388608,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.849 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~50331648,79691776~8388608,92274688~16777216,113246208~8388608,125829120~71303168,205520896~4194304,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~20971520,352321536~25165824,385875968~8388608] 2024-09-21T10:21:40.850 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 18 diff [4194304~4194304,12582912~50331648,79691776~4194304,92274688~29360128,125829120~67108864,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.850 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~4194304,92274688~29360128,125829120~67108864,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~4194304,285212672~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.852 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,205520896~4194304,213909504~20971520,239075328~16777216,264241152~8388608,276824064~4194304,285212672~33554432,322961408~20971520,352321536~25165824,385875968~8388608] 2024-09-21T10:21:40.852 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to 19 diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:40.852 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:40.856 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:40.856 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 13 diff [12582912~4194304,25165824~8388608,58720256~4194304,125829120~8388608,146800640~4194304,176160768~8388608,205520896~4194304,276824064~4194304,385875968~4194304] 2024-09-21T10:21:40.856 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,25165824~8388608,58720256~4194304,125829120~8388608,146800640~4194304,176160768~8388608,205520896~4194304,276824064~4194304,385875968~4194304] 2024-09-21T10:21:40.857 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~8388608,25165824~12582912,58720256~4194304,125829120~8388608,146800640~8388608,176160768~12582912,205520896~4194304,276824064~4194304,385875968~8388608] 2024-09-21T10:21:40.857 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 14 diff [12582912~4194304,25165824~8388608,37748736~8388608,58720256~4194304,100663296~4194304,125829120~8388608,138412032~4194304,146800640~4194304,167772160~16777216,205520896~4194304,222298112~8388608,276824064~4194304,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.857 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,25165824~8388608,37748736~8388608,58720256~4194304,100663296~4194304,125829120~8388608,138412032~4194304,146800640~4194304,167772160~16777216,205520896~4194304,222298112~8388608,276824064~4194304,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.858 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~8388608,25165824~25165824,58720256~4194304,100663296~8388608,125829120~8388608,138412032~16777216,167772160~20971520,205520896~4194304,222298112~12582912,276824064~4194304,364904448~8388608,385875968~8388608] 2024-09-21T10:21:40.858 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 15 diff [12582912~4194304,25165824~8388608,37748736~8388608,58720256~4194304,92274688~12582912,125829120~16777216,146800640~4194304,155189248~37748736,205520896~4194304,222298112~8388608,239075328~4194304,276824064~4194304,293601280~12582912,339738624~4194304,352321536~8388608,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.858 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,25165824~8388608,37748736~8388608,58720256~4194304,92274688~12582912,125829120~16777216,146800640~4194304,155189248~37748736,205520896~4194304,222298112~8388608,239075328~4194304,276824064~4194304,293601280~12582912,339738624~4194304,352321536~8388608,364904448~8388608,385875968~4194304] 2024-09-21T10:21:40.861 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~8388608,25165824~25165824,58720256~4194304,92274688~16777216,125829120~71303168,205520896~4194304,222298112~12582912,239075328~4194304,276824064~4194304,293601280~12582912,339738624~4194304,352321536~20971520,385875968~8388608] 2024-09-21T10:21:40.861 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 16 diff [12582912~4194304,20971520~12582912,37748736~8388608,58720256~4194304,92274688~12582912,125829120~16777216,146800640~16777216,167772160~25165824,205520896~4194304,222298112~8388608,239075328~4194304,276824064~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.861 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,20971520~12582912,37748736~8388608,58720256~4194304,92274688~12582912,125829120~16777216,146800640~16777216,167772160~25165824,205520896~4194304,222298112~8388608,239075328~4194304,276824064~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.863 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~37748736,58720256~4194304,92274688~16777216,125829120~71303168,205520896~4194304,222298112~12582912,239075328~4194304,276824064~4194304,293601280~25165824,322961408~20971520,352321536~25165824,385875968~8388608] 2024-09-21T10:21:40.863 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 17 diff [12582912~4194304,20971520~25165824,54525952~8388608,92274688~12582912,113246208~8388608,125829120~37748736,167772160~25165824,205520896~4194304,213909504~4194304,222298112~8388608,239075328~12582912,276824064~4194304,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.863 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [12582912~4194304,20971520~25165824,54525952~8388608,92274688~12582912,113246208~8388608,125829120~37748736,167772160~25165824,205520896~4194304,213909504~4194304,222298112~8388608,239075328~12582912,276824064~4194304,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.865 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [12582912~37748736,54525952~8388608,92274688~16777216,113246208~8388608,125829120~71303168,205520896~4194304,213909504~20971520,239075328~12582912,276824064~4194304,285212672~33554432,322961408~20971520,352321536~25165824,385875968~8388608] 2024-09-21T10:21:40.865 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 18 diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~25165824,205520896~4194304,213909504~16777216,239075328~12582912,276824064~4194304,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.865 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~25165824,205520896~4194304,213909504~16777216,239075328~12582912,276824064~4194304,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304] 2024-09-21T10:21:40.868 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,92274688~29360128,125829120~71303168,205520896~4194304,213909504~20971520,239075328~12582912,276824064~4194304,285212672~33554432,322961408~20971520,352321536~25165824,385875968~8388608] 2024-09-21T10:21:40.868 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to 19 diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:40.868 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:40.872 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:40.872 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 14 diff [37748736~8388608,100663296~4194304,125829120~4194304,138412032~4194304,167772160~12582912,222298112~8388608,364904448~8388608] 2024-09-21T10:21:40.872 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [37748736~8388608,100663296~4194304,125829120~4194304,138412032~4194304,167772160~12582912,222298112~8388608,364904448~8388608] 2024-09-21T10:21:40.873 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [37748736~12582912,100663296~8388608,125829120~4194304,138412032~8388608,167772160~12582912,222298112~12582912,364904448~8388608] 2024-09-21T10:21:40.873 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 15 diff [37748736~8388608,92274688~12582912,125829120~16777216,155189248~25165824,184549376~8388608,222298112~8388608,239075328~4194304,293601280~12582912,339738624~4194304,352321536~8388608,364904448~8388608] 2024-09-21T10:21:40.873 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [37748736~8388608,92274688~12582912,125829120~16777216,155189248~25165824,184549376~8388608,222298112~8388608,239075328~4194304,293601280~12582912,339738624~4194304,352321536~8388608,364904448~8388608] 2024-09-21T10:21:40.874 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [37748736~12582912,92274688~16777216,125829120~20971520,155189248~25165824,184549376~12582912,222298112~12582912,239075328~4194304,293601280~12582912,339738624~4194304,352321536~20971520] 2024-09-21T10:21:40.874 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 16 diff [20971520~4194304,37748736~8388608,92274688~12582912,125829120~16777216,146800640~16777216,167772160~12582912,184549376~8388608,222298112~8388608,239075328~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.874 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,37748736~8388608,92274688~12582912,125829120~16777216,146800640~16777216,167772160~12582912,184549376~8388608,222298112~8388608,239075328~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.877 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~8388608,37748736~12582912,92274688~16777216,125829120~54525952,184549376~12582912,222298112~12582912,239075328~4194304,293601280~25165824,322961408~20971520,352321536~25165824] 2024-09-21T10:21:40.877 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 17 diff [20971520~4194304,33554432~12582912,54525952~4194304,92274688~12582912,113246208~8388608,125829120~37748736,167772160~12582912,184549376~8388608,213909504~4194304,222298112~8388608,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.877 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,33554432~12582912,54525952~4194304,92274688~12582912,113246208~8388608,125829120~37748736,167772160~12582912,184549376~8388608,213909504~4194304,222298112~8388608,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.879 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~8388608,33554432~16777216,54525952~8388608,92274688~16777216,113246208~8388608,125829120~54525952,184549376~12582912,213909504~20971520,239075328~12582912,285212672~33554432,322961408~20971520,352321536~25165824] 2024-09-21T10:21:40.879 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 18 diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~8388608,213909504~16777216,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.879 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~8388608,213909504~16777216,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.881 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,92274688~29360128,125829120~54525952,184549376~12582912,213909504~20971520,239075328~12582912,285212672~33554432,322961408~20971520,352321536~25165824] 2024-09-21T10:21:40.881 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to 19 diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.882 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.884 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,79691776~8388608,92274688~29360128,125829120~54525952,184549376~20971520,213909504~20971520,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.884 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 15 diff [92274688~8388608,130023424~8388608,155189248~12582912,184549376~8388608,239075328~4194304,293601280~12582912,339738624~4194304,352321536~8388608] 2024-09-21T10:21:40.884 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [92274688~8388608,130023424~8388608,155189248~12582912,184549376~8388608,239075328~4194304,293601280~12582912,339738624~4194304,352321536~8388608] 2024-09-21T10:21:40.886 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [92274688~8388608,130023424~8388608,155189248~12582912,184549376~12582912,239075328~4194304,293601280~12582912,339738624~4194304,352321536~12582912] 2024-09-21T10:21:40.886 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 16 diff [20971520~4194304,92274688~8388608,130023424~8388608,146800640~16777216,184549376~8388608,239075328~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~16777216,373293056~4194304] 2024-09-21T10:21:40.886 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,92274688~8388608,130023424~8388608,146800640~16777216,184549376~8388608,239075328~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~16777216,373293056~4194304] 2024-09-21T10:21:40.887 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~8388608,92274688~12582912,130023424~8388608,146800640~20971520,184549376~12582912,239075328~4194304,293601280~25165824,322961408~20971520,352321536~16777216,373293056~4194304] 2024-09-21T10:21:40.888 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 17 diff [20971520~4194304,33554432~4194304,54525952~4194304,92274688~12582912,113246208~8388608,130023424~8388608,142606336~20971520,184549376~8388608,213909504~4194304,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.888 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,33554432~4194304,54525952~4194304,92274688~12582912,113246208~8388608,130023424~8388608,142606336~20971520,184549376~8388608,213909504~4194304,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.889 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~8388608,33554432~8388608,54525952~8388608,92274688~12582912,113246208~8388608,130023424~8388608,142606336~25165824,184549376~12582912,213909504~8388608,239075328~12582912,285212672~33554432,322961408~20971520,352321536~25165824] 2024-09-21T10:21:40.890 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 18 diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,92274688~12582912,109051904~12582912,130023424~8388608,142606336~20971520,176160768~4194304,184549376~8388608,213909504~12582912,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.890 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,92274688~12582912,109051904~12582912,130023424~8388608,142606336~20971520,176160768~4194304,184549376~8388608,213909504~12582912,239075328~12582912,285212672~4194304,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824] 2024-09-21T10:21:40.891 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,92274688~12582912,109051904~12582912,130023424~8388608,142606336~25165824,176160768~4194304,184549376~12582912,213909504~12582912,239075328~12582912,285212672~33554432,322961408~20971520,352321536~25165824] 2024-09-21T10:21:40.891 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to 19 diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.892 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.894 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,130023424~37748736,176160768~4194304,184549376~20971520,213909504~12582912,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.894 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 16 diff [20971520~4194304,92274688~8388608,146800640~8388608,306184192~12582912,322961408~12582912,360710144~8388608,373293056~4194304] 2024-09-21T10:21:40.894 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,92274688~8388608,146800640~8388608,306184192~12582912,322961408~12582912,360710144~8388608,373293056~4194304] 2024-09-21T10:21:40.895 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~8388608,92274688~12582912,146800640~12582912,306184192~12582912,322961408~16777216,360710144~8388608,373293056~4194304] 2024-09-21T10:21:40.895 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 17 diff [20971520~4194304,33554432~4194304,54525952~4194304,92274688~12582912,113246208~8388608,142606336~12582912,213909504~4194304,243269632~8388608,285212672~4194304,306184192~12582912,322961408~12582912,360710144~16777216] 2024-09-21T10:21:40.895 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,33554432~4194304,54525952~4194304,92274688~12582912,113246208~8388608,142606336~12582912,213909504~4194304,243269632~8388608,285212672~4194304,306184192~12582912,322961408~12582912,360710144~16777216] 2024-09-21T10:21:40.896 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~8388608,33554432~8388608,54525952~8388608,92274688~12582912,113246208~8388608,142606336~16777216,213909504~8388608,243269632~8388608,285212672~8388608,306184192~12582912,322961408~16777216,360710144~16777216] 2024-09-21T10:21:40.896 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 18 diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,92274688~12582912,109051904~12582912,142606336~16777216,176160768~4194304,213909504~12582912,239075328~12582912,285212672~4194304,306184192~12582912,322961408~12582912,352321536~12582912,369098752~8388608] 2024-09-21T10:21:40.896 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,92274688~12582912,109051904~12582912,142606336~16777216,176160768~4194304,213909504~12582912,239075328~12582912,285212672~4194304,306184192~12582912,322961408~12582912,352321536~12582912,369098752~8388608] 2024-09-21T10:21:40.898 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,92274688~12582912,109051904~12582912,142606336~20971520,176160768~4194304,213909504~12582912,239075328~12582912,285212672~8388608,306184192~12582912,322961408~16777216,352321536~25165824] 2024-09-21T10:21:40.898 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to 19 diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.899 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.900 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,134217728~29360128,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,306184192~12582912,322961408~16777216,352321536~25165824,394264576~4194304] 2024-09-21T10:21:40.901 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to 17 diff [20971520~4194304,33554432~4194304,54525952~4194304,96468992~4194304,113246208~8388608,142606336~4194304,213909504~4194304,243269632~8388608,285212672~4194304,369098752~4194304] 2024-09-21T10:21:40.901 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [20971520~4194304,33554432~4194304,54525952~4194304,96468992~4194304,113246208~8388608,142606336~4194304,213909504~4194304,243269632~8388608,285212672~4194304,369098752~4194304] 2024-09-21T10:21:40.902 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [20971520~4194304,33554432~8388608,54525952~8388608,96468992~8388608,113246208~8388608,142606336~8388608,213909504~8388608,243269632~8388608,285212672~8388608,369098752~4194304] 2024-09-21T10:21:40.902 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to 18 diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,96468992~4194304,109051904~12582912,142606336~4194304,155189248~4194304,176160768~4194304,213909504~12582912,239075328~12582912,285212672~4194304,327155712~8388608,352321536~12582912,369098752~4194304] 2024-09-21T10:21:40.902 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,96468992~4194304,109051904~12582912,142606336~4194304,155189248~4194304,176160768~4194304,213909504~12582912,239075328~12582912,285212672~4194304,327155712~8388608,352321536~12582912,369098752~4194304] 2024-09-21T10:21:40.903 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,96468992~8388608,109051904~12582912,142606336~8388608,155189248~8388608,176160768~4194304,213909504~12582912,239075328~12582912,285212672~8388608,327155712~8388608,352321536~20971520] 2024-09-21T10:21:40.904 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to 19 diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,96468992~4194304,109051904~12582912,134217728~12582912,155189248~4194304,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:40.904 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,96468992~4194304,109051904~12582912,134217728~12582912,155189248~4194304,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:40.905 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,96468992~8388608,109051904~12582912,134217728~16777216,155189248~8388608,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:40.905 INFO:tasks.workunit.client.0.smithi119.stdout:from 17 to 18 diff [4194304~4194304,20971520~8388608,109051904~4194304,155189248~4194304,176160768~4194304,218103808~8388608,239075328~4194304,327155712~8388608,352321536~12582912] 2024-09-21T10:21:40.905 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,109051904~4194304,155189248~4194304,176160768~4194304,218103808~8388608,239075328~4194304,327155712~8388608,352321536~12582912] 2024-09-21T10:21:40.906 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~12582912,109051904~4194304,155189248~8388608,176160768~4194304,218103808~8388608,239075328~4194304,327155712~8388608,352321536~16777216] 2024-09-21T10:21:40.906 INFO:tasks.workunit.client.0.smithi119.stdout:from 17 to 19 diff [4194304~4194304,20971520~8388608,79691776~8388608,109051904~8388608,134217728~8388608,146800640~4194304,155189248~4194304,176160768~4194304,192937984~12582912,218103808~8388608,239075328~4194304,281018368~4194304,327155712~12582912,352321536~16777216,394264576~4194304] 2024-09-21T10:21:40.906 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,79691776~8388608,109051904~8388608,134217728~8388608,146800640~4194304,155189248~4194304,176160768~4194304,192937984~12582912,218103808~8388608,239075328~4194304,281018368~4194304,327155712~12582912,352321536~16777216,394264576~4194304] 2024-09-21T10:21:40.907 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~12582912,79691776~8388608,109051904~12582912,134217728~16777216,155189248~8388608,176160768~4194304,192937984~12582912,218103808~8388608,239075328~4194304,281018368~4194304,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:40.908 INFO:tasks.workunit.client.0.smithi119.stdout:from 18 to 19 diff [79691776~8388608,109051904~8388608,134217728~8388608,146800640~4194304,192937984~12582912,281018368~4194304,331350016~8388608,364904448~4194304,394264576~4194304] 2024-09-21T10:21:40.908 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [79691776~8388608,109051904~8388608,134217728~8388608,146800640~4194304,192937984~12582912,281018368~4194304,331350016~8388608,364904448~4194304,394264576~4194304] 2024-09-21T10:21:40.908 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [79691776~8388608,109051904~12582912,134217728~16777216,192937984~12582912,281018368~4194304,331350016~8388608,364904448~8388608,394264576~4194304] 2024-09-21T10:21:41.222 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:41.222 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:41.240 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:41.240 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:41.240 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:41.248 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:41.248 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:41.248 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:41.256 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:41.256 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:41.256 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:41.261 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:41.261 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:41.262 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:41.268 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:41.268 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:41.268 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:41.273 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:41.273 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:41.274 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:41.279 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:41.279 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:41.279 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:41.284 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:41.284 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:41.284 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:41.289 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:41.289 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:41.289 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:41.293 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:41.293 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:41.293 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:41.297 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:41.297 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:41.297 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:41.300 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:41.300 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:41.301 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:41.303 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:41.303 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.303 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.306 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,79691776~8388608,92274688~29360128,125829120~54525952,184549376~20971520,213909504~20971520,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.306 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.306 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.308 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,130023424~37748736,176160768~4194304,184549376~20971520,213909504~12582912,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.308 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.308 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.310 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,134217728~29360128,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,306184192~12582912,322961408~16777216,352321536~25165824,394264576~4194304] 2024-09-21T10:21:41.310 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,96468992~4194304,109051904~12582912,134217728~12582912,155189248~4194304,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:41.310 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,96468992~4194304,109051904~12582912,134217728~12582912,155189248~4194304,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:41.312 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,96468992~8388608,109051904~12582912,134217728~16777216,155189248~8388608,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:41.312 INFO:tasks.workunit.client.0.smithi119.stdout:from 17 to HEAD diff [4194304~4194304,20971520~8388608,79691776~8388608,109051904~8388608,134217728~8388608,146800640~4194304,155189248~4194304,176160768~4194304,192937984~12582912,218103808~8388608,239075328~4194304,281018368~4194304,327155712~12582912,352321536~16777216,394264576~4194304] 2024-09-21T10:21:41.312 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,79691776~8388608,109051904~8388608,134217728~8388608,146800640~4194304,155189248~4194304,176160768~4194304,192937984~12582912,218103808~8388608,239075328~4194304,281018368~4194304,327155712~12582912,352321536~16777216,394264576~4194304] 2024-09-21T10:21:41.313 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~12582912,79691776~8388608,109051904~12582912,134217728~16777216,155189248~8388608,176160768~4194304,192937984~12582912,218103808~8388608,239075328~4194304,281018368~4194304,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:41 smithi145 ceph-mon[89359]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-21T10:21:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:41 smithi119 ceph-mon[105288]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-21T10:21:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:41 smithi119 ceph-mon[101432]: osdmap e1137: 8 total, 8 up, 8 in 2024-09-21T10:21:42.233 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:42.233 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:42.247 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:42.247 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:42.247 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:42.255 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:42.255 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:42.255 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:42.262 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:42.262 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:42.262 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:42.269 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:42.269 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:42.269 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:42.274 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:42.274 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:42.274 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:42.280 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:42.280 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:42.280 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:42.284 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:42.285 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:42.285 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:42.289 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:42.289 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:42.289 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:42.293 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:42.293 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:42.293 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:42.296 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:42.296 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:42.296 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:42.300 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:42.300 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:42.300 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:42.305 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:42.305 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:42.305 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:42.307 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:42.308 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.308 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.310 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,79691776~8388608,92274688~29360128,125829120~54525952,184549376~20971520,213909504~20971520,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.310 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.310 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.312 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,130023424~37748736,176160768~4194304,184549376~20971520,213909504~12582912,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.312 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.312 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.313 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,134217728~29360128,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,306184192~12582912,322961408~16777216,352321536~25165824,394264576~4194304] 2024-09-21T10:21:42.313 INFO:tasks.workunit.client.0.smithi119.stdout:from 16 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,96468992~4194304,109051904~12582912,134217728~12582912,155189248~4194304,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:42.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,96468992~4194304,109051904~12582912,134217728~12582912,155189248~4194304,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:42.314 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,96468992~8388608,109051904~12582912,134217728~16777216,155189248~8388608,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,327155712~12582912,352321536~20971520,394264576~4194304] 2024-09-21T10:21:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:42 smithi145 ceph-mon[89359]: pgmap v1490: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 256 KiB/s rd, 47 MiB/s wr, 397 op/s 2024-09-21T10:21:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:42 smithi145 ceph-mon[89359]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-21T10:21:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:42 smithi119 ceph-mon[105288]: pgmap v1490: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 256 KiB/s rd, 47 MiB/s wr, 397 op/s 2024-09-21T10:21:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:42 smithi119 ceph-mon[105288]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-21T10:21:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:42 smithi119 ceph-mon[101432]: pgmap v1490: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 256 KiB/s rd, 47 MiB/s wr, 397 op/s 2024-09-21T10:21:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:42 smithi119 ceph-mon[101432]: osdmap e1138: 8 total, 8 up, 8 in 2024-09-21T10:21:43.244 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:43.245 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:43.260 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:43.261 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:43.261 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:43.269 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:43.269 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:43.269 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:43.276 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:43.276 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:43.276 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:43.283 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:43.283 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:43.283 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:43.289 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:43.289 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:43.289 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:43.294 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:43.294 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:43.294 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:43.299 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:43.299 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:43.299 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:43.303 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:43.303 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:43.303 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:43.307 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:43.307 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:43.307 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:43.311 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:43.311 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:43.311 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:43.314 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:43.314 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:43.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:43.317 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:43.317 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:43.317 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:43.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:43.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.319 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.321 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,79691776~8388608,92274688~29360128,125829120~54525952,184549376~20971520,213909504~20971520,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.321 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.321 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.323 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,130023424~37748736,176160768~4194304,184549376~20971520,213909504~12582912,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.323 INFO:tasks.workunit.client.0.smithi119.stdout:from 15 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.323 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,134217728~25165824,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~8388608,306184192~12582912,322961408~12582912,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.323 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,134217728~29360128,176160768~4194304,192937984~12582912,213909504~12582912,239075328~12582912,281018368~12582912,306184192~12582912,322961408~16777216,352321536~25165824,394264576~4194304] 2024-09-21T10:21:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:43 smithi145 ceph-mon[89359]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-21T10:21:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:43 smithi119 ceph-mon[105288]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-21T10:21:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:43 smithi119 ceph-mon[101432]: osdmap e1139: 8 total, 8 up, 8 in 2024-09-21T10:21:44.250 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:44.250 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:44.265 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:44.265 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:44.265 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:44.274 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:44.274 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:44.274 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:44.278 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:44.278 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:44.278 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:44.283 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:44.283 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:44.283 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:44.288 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:44.288 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:44.289 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:44.293 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:44.293 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:44.293 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:44.297 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:44.297 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:44.297 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:44.301 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:44.301 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:44.301 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:44.305 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:44.305 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:44.305 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:44.308 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:44.308 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:44.308 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:44.311 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:44.312 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:44.312 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:44.314 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:44.314 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:44.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:44.316 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:44.316 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:44.316 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:44.318 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,79691776~8388608,92274688~29360128,125829120~54525952,184549376~20971520,213909504~20971520,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:44.318 INFO:tasks.workunit.client.0.smithi119.stdout:from 14 to HEAD diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:44.318 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~4194304,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,130023424~33554432,176160768~4194304,184549376~12582912,201326592~4194304,213909504~12582912,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:44.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~20971520,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,130023424~37748736,176160768~4194304,184549376~20971520,213909504~12582912,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:44 smithi119 ceph-mon[105288]: pgmap v1493: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 221 KiB/s rd, 12 MiB/s wr, 356 op/s 2024-09-21T10:21:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:44 smithi119 ceph-mon[105288]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-21T10:21:44.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:44 smithi119 ceph-mon[101432]: pgmap v1493: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 221 KiB/s rd, 12 MiB/s wr, 356 op/s 2024-09-21T10:21:44.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:44 smithi119 ceph-mon[101432]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-21T10:21:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:44 smithi145 ceph-mon[89359]: pgmap v1493: 105 pgs: 105 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 221 KiB/s rd, 12 MiB/s wr, 356 op/s 2024-09-21T10:21:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:44 smithi145 ceph-mon[89359]: osdmap e1140: 8 total, 8 up, 8 in 2024-09-21T10:21:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:21:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:21:44] "GET /metrics HTTP/1.1" 200 36714 "" "Prometheus/2.51.0" 2024-09-21T10:21:45.266 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:45.266 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:45.280 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:45.280 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:45.280 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:45.286 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:45.286 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:45.286 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:45.291 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:45.291 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:45.291 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:45.296 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:45.296 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:45.296 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:45.300 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:45.300 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:45.300 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:45.304 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:45.304 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:45.305 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:45.308 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:45.308 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:45.308 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:45.311 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:45.311 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:45.311 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:45.314 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:45.314 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:45.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:45.316 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:45.316 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:45.316 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:45.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:45.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:45.319 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:45.321 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:45.321 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:45.321 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:45.322 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:45.322 INFO:tasks.workunit.client.0.smithi119.stdout:from 13 to HEAD diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:45.322 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,20971520~8388608,33554432~12582912,54525952~4194304,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~12582912,184549376~12582912,201326592~4194304,213909504~16777216,239075328~12582912,281018368~8388608,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,394264576~4194304] 2024-09-21T10:21:45.323 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,20971520~29360128,54525952~8388608,79691776~8388608,92274688~29360128,125829120~54525952,184549376~20971520,213909504~20971520,239075328~12582912,281018368~37748736,322961408~20971520,352321536~25165824,394264576~4194304] 2024-09-21T10:21:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:45 smithi145 ceph-mon[89359]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-21T10:21:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:45 smithi145 ceph-mon[89359]: pgmap v1496: 105 pgs: 105 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 549 KiB/s rd, 7.8 MiB/s wr, 1.09k op/s 2024-09-21T10:21:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:45 smithi119 ceph-mon[105288]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-21T10:21:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:45 smithi119 ceph-mon[105288]: pgmap v1496: 105 pgs: 105 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 549 KiB/s rd, 7.8 MiB/s wr, 1.09k op/s 2024-09-21T10:21:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:45 smithi119 ceph-mon[101432]: osdmap e1141: 8 total, 8 up, 8 in 2024-09-21T10:21:45.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:45 smithi119 ceph-mon[101432]: pgmap v1496: 105 pgs: 105 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 549 KiB/s rd, 7.8 MiB/s wr, 1.09k op/s 2024-09-21T10:21:46.265 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:46.266 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:46.281 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:46.281 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:46.281 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:46.286 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:46.286 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:46.286 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:46.291 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:46.291 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:46.291 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:46.296 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:46.296 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:46.296 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:46.300 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:46.300 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:46.300 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:46.304 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:46.305 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:46.305 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:46.309 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:46.309 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:46.309 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:46.312 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:46.312 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:46.312 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:46.315 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:46.315 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:46.315 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:46.317 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:46.317 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:46.317 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:46.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:46.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:46.319 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:46.321 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:46.321 INFO:tasks.workunit.client.0.smithi119.stdout:from 12 to HEAD diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:46.321 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~4194304,20971520~25165824,54525952~8388608,79691776~8388608,92274688~12582912,109051904~12582912,125829120~37748736,167772160~29360128,201326592~8388608,213909504~16777216,239075328~12582912,276824064~12582912,293601280~25165824,322961408~12582912,339738624~4194304,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:46.322 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~37748736,54525952~8388608,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~12582912,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:46 smithi145 ceph-mon[89359]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-21T10:21:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:46 smithi119 ceph-mon[105288]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-21T10:21:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:46 smithi119 ceph-mon[101432]: osdmap e1142: 8 total, 8 up, 8 in 2024-09-21T10:21:47.279 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:47.279 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:47.291 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:47.291 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:47.291 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:47.295 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:47.296 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:47.296 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:47.301 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:47.301 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:47.301 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:47.305 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:47.305 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:47.306 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:47.310 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:47.310 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:47.310 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:47.313 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:47.313 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:47.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:47.317 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:47.317 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:47.317 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:47.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:47.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:47.319 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:47.322 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:47.322 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:47.322 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:47.324 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:47.324 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:47.324 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:47.325 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:47.325 INFO:tasks.workunit.client.0.smithi119.stdout:from 11 to HEAD diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:47.325 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~71303168,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304] 2024-09-21T10:21:47.326 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [4194304~4194304,12582912~50331648,79691776~8388608,92274688~29360128,125829120~83886080,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912] 2024-09-21T10:21:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:47 smithi145 ceph-mon[89359]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-21T10:21:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:47 smithi145 ceph-mon[89359]: pgmap v1499: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 628 KiB/s rd, 6.8 MiB/s wr, 1.18k op/s 2024-09-21T10:21:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:47 smithi119 ceph-mon[105288]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-21T10:21:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:47 smithi119 ceph-mon[105288]: pgmap v1499: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 628 KiB/s rd, 6.8 MiB/s wr, 1.18k op/s 2024-09-21T10:21:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:47 smithi119 ceph-mon[101432]: osdmap e1143: 8 total, 8 up, 8 in 2024-09-21T10:21:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:47 smithi119 ceph-mon[101432]: pgmap v1499: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.2 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 628 KiB/s rd, 6.8 MiB/s wr, 1.18k op/s 2024-09-21T10:21:48.279 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:48.279 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:48.297 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:48.297 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:48.297 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:48.302 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:48.303 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:48.303 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:48.307 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:48.307 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:48.307 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:48.312 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:48.312 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:48.312 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:48.316 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:48.316 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:48.316 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:48.320 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:48.320 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:48.320 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:48.323 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:48.323 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:48.323 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:48.325 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:48.326 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:48.326 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:48.328 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:48.328 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:48.328 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:48.329 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:48.329 INFO:tasks.workunit.client.0.smithi119.stdout:from 10 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:48.329 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,352321536~25165824,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:48.331 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,352321536~25165824,385875968~12582912,406847488~12582912] 2024-09-21T10:21:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:48 smithi145 ceph-mon[89359]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-21T10:21:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:48 smithi119 ceph-mon[105288]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-21T10:21:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:48 smithi119 ceph-mon[101432]: osdmap e1144: 8 total, 8 up, 8 in 2024-09-21T10:21:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:21:49.269 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:49.269 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:49.280 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:49.280 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:49.280 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:49.284 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:49.284 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:49.284 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:49.288 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:49.288 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:49.288 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:49.292 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:49.293 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:49.293 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:49.296 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:49.296 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:49.296 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:49.299 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:49.299 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:49.299 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:49.302 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:49.303 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:49.303 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:49.304 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:49.305 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:49.305 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:49.307 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:49.307 INFO:tasks.workunit.client.0.smithi119.stdout:from 9 to HEAD diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:49.307 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,79691776~25165824,109051904~88080384,201326592~8388608,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~20971520,348127232~29360128,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:49.308 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,79691776~130023424,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~20971520,348127232~29360128,385875968~12582912,406847488~12582912] 2024-09-21T10:21:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:49 smithi145 ceph-mon[89359]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-21T10:21:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:49 smithi145 ceph-mon[89359]: pgmap v1502: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 308 KiB/s rd, 6.2 KiB/s wr, 543 op/s 2024-09-21T10:21:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:49 smithi145 ceph-mon[89359]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-21T10:21:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:49 smithi119 ceph-mon[105288]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-21T10:21:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:49 smithi119 ceph-mon[105288]: pgmap v1502: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 308 KiB/s rd, 6.2 KiB/s wr, 543 op/s 2024-09-21T10:21:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:49 smithi119 ceph-mon[105288]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-21T10:21:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:49 smithi119 ceph-mon[101432]: osdmap e1145: 8 total, 8 up, 8 in 2024-09-21T10:21:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:49 smithi119 ceph-mon[101432]: pgmap v1502: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 308 KiB/s rd, 6.2 KiB/s wr, 543 op/s 2024-09-21T10:21:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:49 smithi119 ceph-mon[101432]: osdmap e1146: 8 total, 8 up, 8 in 2024-09-21T10:21:50.297 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:50.297 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:50.307 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:50.307 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:50.307 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:50.312 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:50.312 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:50.312 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:50.316 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:50.316 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:50.316 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:50.321 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:50.321 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:50.321 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:50.324 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:50.324 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:50.324 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:50.327 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:50.327 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:50.327 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:50.329 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:50.329 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:50.330 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:50.331 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:50.331 INFO:tasks.workunit.client.0.smithi119.stdout:from 8 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:50.331 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~8388608,276824064~12582912,293601280~25165824,322961408~54525952,385875968~4194304,394264576~4194304,406847488~8388608] 2024-09-21T10:21:50.333 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~8388608,276824064~41943040,322961408~58720256,385875968~12582912,406847488~12582912] 2024-09-21T10:21:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:21:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:21:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:21:51.291 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:51.291 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:51.310 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:51.311 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:51.311 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:51.314 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:51.314 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:51.314 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:51.317 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:51.318 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:51.318 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:51.321 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:51.321 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:51.321 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:51.323 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:51.323 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:51.324 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:51.326 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:51.326 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:51.326 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:51.328 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:51.328 INFO:tasks.workunit.client.0.smithi119.stdout:from 7 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:51.328 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,239075328~12582912,264241152~54525952,322961408~54525952,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:51.329 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~20971520,239075328~16777216,264241152~54525952,322961408~58720256,385875968~12582912,402653184~16777216] 2024-09-21T10:21:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:51 smithi145 ceph-mon[89359]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-21T10:21:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:51 smithi145 ceph-mon[89359]: pgmap v1505: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 267 KiB/s rd, 6.0 KiB/s wr, 462 op/s 2024-09-21T10:21:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:51 smithi119 ceph-mon[105288]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-21T10:21:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:51 smithi119 ceph-mon[105288]: pgmap v1505: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 267 KiB/s rd, 6.0 KiB/s wr, 462 op/s 2024-09-21T10:21:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:51 smithi119 ceph-mon[101432]: osdmap e1147: 8 total, 8 up, 8 in 2024-09-21T10:21:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:51 smithi119 ceph-mon[101432]: pgmap v1505: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 1.1 GiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 267 KiB/s rd, 6.0 KiB/s wr, 462 op/s 2024-09-21T10:21:52.289 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:52.290 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:52.312 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:52.313 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:52.313 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:52.316 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:52.316 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:52.316 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:52.319 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:52.319 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:52.320 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:52.323 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:52.323 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:52.323 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:52.325 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:52.325 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:52.325 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:52.327 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:52.327 INFO:tasks.workunit.client.0.smithi119.stdout:from 6 to HEAD diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:52.327 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,71303168~33554432,109051904~92274688,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:52.328 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,71303168~138412032,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:52 smithi145 ceph-mon[89359]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-21T10:21:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:52 smithi119 ceph-mon[101432]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-21T10:21:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:52 smithi119 ceph-mon[105288]: osdmap e1148: 8 total, 8 up, 8 in 2024-09-21T10:21:53.305 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:53.306 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:53.324 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:53.324 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:53.324 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:53.327 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:53.327 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:53.327 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:53.329 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:53.329 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:53.329 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:53.331 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:53.331 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:53.331 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:53.333 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:53.333 INFO:tasks.workunit.client.0.smithi119.stdout:from 5 to HEAD diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:53.333 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~4194304,213909504~16777216,234881024~16777216,264241152~54525952,322961408~58720256,385875968~4194304,394264576~4194304,402653184~12582912] 2024-09-21T10:21:53.334 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~142606336,213909504~41943040,264241152~54525952,322961408~75497472,402653184~16777216] 2024-09-21T10:21:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:53 smithi145 ceph-mon[89359]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-21T10:21:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:53 smithi145 ceph-mon[89359]: pgmap v1508: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 1.1 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 122 KiB/s rd, 4.7 KiB/s wr, 172 op/s 2024-09-21T10:21:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:53 smithi119 ceph-mon[101432]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-21T10:21:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:53 smithi119 ceph-mon[101432]: pgmap v1508: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 1.1 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 122 KiB/s rd, 4.7 KiB/s wr, 172 op/s 2024-09-21T10:21:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:53 smithi119 ceph-mon[105288]: osdmap e1149: 8 total, 8 up, 8 in 2024-09-21T10:21:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:53 smithi119 ceph-mon[105288]: pgmap v1508: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 1.1 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 122 KiB/s rd, 4.7 KiB/s wr, 172 op/s 2024-09-21T10:21:54.265 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:54.265 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:54.275 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:54.275 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:54.275 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:54.277 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:54.278 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:54.278 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:54.280 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:54.280 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:54.280 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:54.281 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:54.281 INFO:tasks.workunit.client.0.smithi119.stdout:from 4 to HEAD diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:54.281 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~62914560,67108864~134217728,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:54.282 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~62914560,67108864~188743680,264241152~155189248] 2024-09-21T10:21:54.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:54 smithi119 ceph-mon[101432]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-21T10:21:54.541 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:54 smithi119 ceph-mon[101432]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-21T10:21:54.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:54 smithi119 ceph-mon[105288]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-21T10:21:54.541 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:54 smithi119 ceph-mon[105288]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-21T10:21:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:54 smithi145 ceph-mon[89359]: osdmap e1150: 8 total, 8 up, 8 in 2024-09-21T10:21:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:54 smithi145 ceph-mon[89359]: osdmap e1151: 8 total, 8 up, 8 in 2024-09-21T10:21:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:21:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:21:54] "GET /metrics HTTP/1.1" 200 36718 "" "Prometheus/2.51.0" 2024-09-21T10:21:55.267 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:55.267 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:55.280 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:55.280 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:55.280 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:55.282 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:55.282 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:55.282 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:55.284 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:55.284 INFO:tasks.workunit.client.0.smithi119.stdout:from 3 to HEAD diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:55.284 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~25165824,234881024~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:55.285 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~255852544,264241152~155189248] 2024-09-21T10:21:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:55 smithi119 ceph-mon[105288]: pgmap v1511: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 961 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 1.2 KiB/s wr, 129 op/s 2024-09-21T10:21:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:55 smithi119 ceph-mon[105288]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-21T10:21:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:55 smithi119 ceph-mon[101432]: pgmap v1511: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 961 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 1.2 KiB/s wr, 129 op/s 2024-09-21T10:21:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:55 smithi119 ceph-mon[101432]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-21T10:21:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:55 smithi145 ceph-mon[89359]: pgmap v1511: 105 pgs: 1 active+clean+snaptrim, 104 active+clean; 961 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 1.2 KiB/s wr, 129 op/s 2024-09-21T10:21:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:55 smithi145 ceph-mon[89359]: osdmap e1152: 8 total, 8 up, 8 in 2024-09-21T10:21:56.320 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:56.320 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:56.328 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:56.328 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:56.328 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:56.329 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:56.330 INFO:tasks.workunit.client.0.smithi119.stdout:from 2 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:56.330 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~117440512,385875968~4194304,394264576~20971520] 2024-09-21T10:21:56.331 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:57.330 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:57.330 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:57.337 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:57.337 INFO:tasks.workunit.client.0.smithi119.stdout:from 1 to HEAD diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:57.338 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~29360128,239075328~20971520,264241152~125829120,394264576~20971520] 2024-09-21T10:21:57.339 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:57 smithi145 ceph-mon[89359]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-21T10:21:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:57 smithi145 ceph-mon[89359]: pgmap v1514: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 953 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 6.5 KiB/s wr, 244 op/s 2024-09-21T10:21:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:57 smithi119 ceph-mon[105288]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-21T10:21:57.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:57 smithi119 ceph-mon[105288]: pgmap v1514: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 953 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 6.5 KiB/s wr, 244 op/s 2024-09-21T10:21:57.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:57 smithi119 ceph-mon[101432]: osdmap e1153: 8 total, 8 up, 8 in 2024-09-21T10:21:57.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:57 smithi119 ceph-mon[101432]: pgmap v1514: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 953 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 148 KiB/s rd, 6.5 KiB/s wr, 244 op/s 2024-09-21T10:21:58.326 INFO:tasks.workunit.client.0.smithi119.stdout:from 0 to HEAD diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:58.326 INFO:tasks.workunit.client.0.smithi119.stdout: limited diff [0~201326592,205520896~54525952,264241152~125829120,394264576~20971520] 2024-09-21T10:21:58.332 INFO:tasks.workunit.client.0.smithi119.stdout: actual was [0~260046848,264241152~155189248] 2024-09-21T10:21:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:58 smithi119 ceph-mon[105288]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-21T10:21:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:58 smithi119 ceph-mon[101432]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-21T10:21:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:58 smithi145 ceph-mon[89359]: osdmap e1154: 8 total, 8 up, 8 in 2024-09-21T10:21:59.270 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateStress (39153 ms) 2024-09-21T10:21:59.270 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParent 2024-09-21T10:21:59.270 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:21:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:59 smithi119 ceph-mon[105288]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-21T10:21:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:59 smithi119 ceph-mon[105288]: pgmap v1517: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 836 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 6.0 KiB/s wr, 163 op/s 2024-09-21T10:21:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:21:59 smithi119 ceph-mon[105288]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-21T10:21:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:59 smithi119 ceph-mon[101432]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-21T10:21:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:59 smithi119 ceph-mon[101432]: pgmap v1517: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 836 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 6.0 KiB/s wr, 163 op/s 2024-09-21T10:21:59.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:21:59 smithi119 ceph-mon[101432]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-21T10:21:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:59 smithi145 ceph-mon[89359]: osdmap e1155: 8 total, 8 up, 8 in 2024-09-21T10:21:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:59 smithi145 ceph-mon[89359]: pgmap v1517: 105 pgs: 2 active+clean+snaptrim, 103 active+clean; 836 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 108 KiB/s rd, 6.0 KiB/s wr, 163 op/s 2024-09-21T10:21:59.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:21:59 smithi145 ceph-mon[89359]: osdmap e1156: 8 total, 8 up, 8 in 2024-09-21T10:22:00.466 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateParent (1197 ms) 2024-09-21T10:22:00.467 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateIgnoreParent 2024-09-21T10:22:00.467 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:22:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:22:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:22:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:01.479 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [0~20971520] to clone 2024-09-21T10:22:01.479 INFO:tasks.workunit.client.0.smithi119.stdout: diff was [0~20971520] 2024-09-21T10:22:01.483 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateIgnoreParent (1017 ms) 2024-09-21T10:22:01.483 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateCallbackError 2024-09-21T10:22:01.483 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:22:01.502 INFO:tasks.workunit.client.0.smithi119.stdout: wrote [255809~70911,1696269~35366,2450636~62149,3719346~24188,4267961~12439,4584383~63608,9392692~48558,11011566~54931,17112331~38548,20868840~93127] 2024-09-21T10:22:01.511 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateCallbackError (28 ms) 2024-09-21T10:22:01.511 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateParentDiscard 2024-09-21T10:22:01.511 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:22:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:01 smithi119 ceph-mon[105288]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-21T10:22:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:01 smithi119 ceph-mon[105288]: pgmap v1520: 105 pgs: 105 active+clean; 778 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 4.7 KiB/s wr, 101 op/s 2024-09-21T10:22:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:01 smithi119 ceph-mon[101432]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-21T10:22:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:01 smithi119 ceph-mon[101432]: pgmap v1520: 105 pgs: 105 active+clean; 778 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 4.7 KiB/s wr, 101 op/s 2024-09-21T10:22:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:01 smithi145 ceph-mon[89359]: osdmap e1157: 8 total, 8 up, 8 in 2024-09-21T10:22:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:01 smithi145 ceph-mon[89359]: pgmap v1520: 105 pgs: 105 active+clean; 778 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 65 KiB/s rd, 4.7 KiB/s wr, 101 op/s 2024-09-21T10:22:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:02 smithi119 ceph-mon[101432]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-21T10:22:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:02 smithi119 ceph-mon[105288]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-21T10:22:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:02 smithi145 ceph-mon[89359]: osdmap e1158: 8 total, 8 up, 8 in 2024-09-21T10:22:03.475 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateParentDiscard (1964 ms) 2024-09-21T10:22:03.476 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnalignedSmall 2024-09-21T10:22:03.476 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:22:03.523 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnalignedSmall (48 ms) 2024-09-21T10:22:03.523 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateUnaligned 2024-09-21T10:22:03.524 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:22:03.612 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateUnaligned (89 ms) 2024-09-21T10:22:03.612 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateTryAcquireLock 2024-09-21T10:22:03.612 INFO:tasks.workunit.client.0.smithi119.stdout:using new format! 2024-09-21T10:22:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:03 smithi145 ceph-mon[89359]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-21T10:22:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:03 smithi145 ceph-mon[89359]: pgmap v1523: 105 pgs: 105 active+clean; 789 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-09-21T10:22:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:03 smithi119 ceph-mon[105288]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-21T10:22:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:03 smithi119 ceph-mon[105288]: pgmap v1523: 105 pgs: 105 active+clean; 789 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-09-21T10:22:03.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:03 smithi119 ceph-mon[101432]: osdmap e1159: 8 total, 8 up, 8 in 2024-09-21T10:22:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:03 smithi119 ceph-mon[101432]: pgmap v1523: 105 pgs: 105 active+clean; 789 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.1 MiB/s rd, 2.1 MiB/s wr, 127 op/s 2024-09-21T10:22:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:04 smithi145 ceph-mon[89359]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-21T10:22:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:04 smithi145 ceph-mon[89359]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-21T10:22:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:22:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:22:04] "GET /metrics HTTP/1.1" 200 36716 "" "Prometheus/2.51.0" 2024-09-21T10:22:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:04 smithi119 ceph-mon[101432]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-21T10:22:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:04 smithi119 ceph-mon[101432]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-21T10:22:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:04 smithi119 ceph-mon[105288]: osdmap e1160: 8 total, 8 up, 8 in 2024-09-21T10:22:04.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:04 smithi119 ceph-mon[105288]: osdmap e1161: 8 total, 8 up, 8 in 2024-09-21T10:22:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:05 smithi145 ceph-mon[89359]: pgmap v1526: 105 pgs: 105 active+clean; 839 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s rd, 10 MiB/s wr, 330 op/s 2024-09-21T10:22:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:05 smithi119 ceph-mon[101432]: pgmap v1526: 105 pgs: 105 active+clean; 839 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s rd, 10 MiB/s wr, 330 op/s 2024-09-21T10:22:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:05 smithi119 ceph-mon[105288]: pgmap v1526: 105 pgs: 105 active+clean; 839 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.3 MiB/s rd, 10 MiB/s wr, 330 op/s 2024-09-21T10:22:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:07 smithi119 ceph-mon[101432]: pgmap v1527: 105 pgs: 105 active+clean; 857 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 10 MiB/s wr, 261 op/s 2024-09-21T10:22:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:07 smithi119 ceph-mon[105288]: pgmap v1527: 105 pgs: 105 active+clean; 857 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 10 MiB/s wr, 261 op/s 2024-09-21T10:22:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:07 smithi145 ceph-mon[89359]: pgmap v1527: 105 pgs: 105 active+clean; 857 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.6 MiB/s rd, 10 MiB/s wr, 261 op/s 2024-09-21T10:22:08.653 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateTryAcquireLock (5040 ms) 2024-09-21T10:22:08.653 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] DiffIterateTest/1.DiffIterateStriping 2024-09-21T10:22:08.653 INFO:tasks.workunit.client.0.smithi119.stdout:SKIPPING 2024-09-21T10:22:08.653 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] DiffIterateTest/1.DiffIterateStriping (0 ms) 2024-09-21T10:22:08.858 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 19 tests from DiffIterateTest/1 (249832 ms total) 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] Global test environment tear-down 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout:[==========] 425 tests from 16 test suites ran. (1433152 ms total) 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout:[ PASSED ] 425 tests. 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout: YOU HAVE 2 DISABLED TESTS 2024-09-21T10:22:08.859 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:22:08.866 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T10:22:08.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T10:22:08.939 INFO:tasks.workunit:Stopping ['rbd/test_librbd.sh'] on client.0... 2024-09-21T10:22:08.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:22:09.192 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:09.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T10:22:09.216 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T10:22:09.216 DEBUG:teuthology.orchestra.run.smithi119:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:22:09.277 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T10:22:09.277 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T10:22:09.278 INFO:teuthology.task.print:**** done rbd/test_librbd.sh 7-workload 2024-09-21T10:22:09.278 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:09.278 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:22:09.287 INFO:teuthology.task.print:**** done upgrading first half of cluster **** 2024-09-21T10:22:09.287 INFO:teuthology.run_tasks:Running task ceph.healthy... 2024-09-21T10:22:09.295 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T10:22:09.295 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T10:22:09.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T10:22:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:09.977 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:09 smithi119 ceph-mon[105288]: pgmap v1528: 105 pgs: 105 active+clean; 876 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 9.9 MiB/s wr, 212 op/s 2024-09-21T10:22:09.977 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:09 smithi119 ceph-mon[105288]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-21T10:22:09.977 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:09 smithi119 ceph-mon[101432]: pgmap v1528: 105 pgs: 105 active+clean; 876 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 9.9 MiB/s wr, 212 op/s 2024-09-21T10:22:09.978 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:09 smithi119 ceph-mon[101432]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-21T10:22:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:09 smithi145 ceph-mon[89359]: pgmap v1528: 105 pgs: 105 active+clean; 876 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.7 MiB/s rd, 9.9 MiB/s wr, 212 op/s 2024-09-21T10:22:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:09 smithi145 ceph-mon[89359]: osdmap e1162: 8 total, 8 up, 8 in 2024-09-21T10:22:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:22:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:22:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:11.433 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:22:12.097 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:11 smithi119 ceph-mon[105288]: pgmap v1530: 73 pgs: 73 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:22:12.097 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3644525577' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:22:12.097 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:11 smithi119 ceph-mon[101432]: pgmap v1530: 73 pgs: 73 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:22:12.097 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3644525577' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:22:12.122 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":38,"flags":0,"active_gid":34104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":3126057112},{"type":"v1","addr":"172.21.15.119:6801","nonce":3126057112}]},"active_addr":"172.21.15.119:6801/3126057112","active_change":"2024-09-21T10:01:32.365206+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":25594,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.119:8443/","prometheus":"http://172.21.15.119:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":254,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":3375048793}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":1273954712}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":3694613141}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":1389600505}]}]} 2024-09-21T10:22:12.126 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T10:22:12.126 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T10:22:12.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:22:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:11 smithi145 ceph-mon[89359]: pgmap v1530: 73 pgs: 73 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:22:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3644525577' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-21T10:22:12.369 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:14.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:13 smithi119 ceph-mon[105288]: pgmap v1531: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:22:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:13 smithi119 ceph-mon[101432]: pgmap v1531: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:22:14.159 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:22:14.160 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":1162,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T10:22:08.837947+0000","last_up_change":"2024-09-21T10:08:54.045519+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":44,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:56:58.923308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:56:59.224014+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":496,"up_from":509,"up_thru":1014,"down_at":497,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6803","nonce":2607764328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6805","nonce":2607764328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6809","nonce":2607764328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6807","nonce":2607764328}]},"public_addr":"172.21.15.119:6803/2607764328","cluster_addr":"172.21.15.119:6805/2607764328","heartbeat_back_addr":"172.21.15.119:6809/2607764328","heartbeat_front_addr":"172.21.15.119:6807/2607764328","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":529,"up_from":542,"up_thru":1014,"down_at":530,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6811","nonce":2769715805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6813","nonce":2769715805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6817","nonce":2769715805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6815","nonce":2769715805}]},"public_addr":"172.21.15.119:6811/2769715805","cluster_addr":"172.21.15.119:6813/2769715805","heartbeat_back_addr":"172.21.15.119:6817/2769715805","heartbeat_front_addr":"172.21.15.119:6815/2769715805","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":613,"up_from":626,"up_thru":1014,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6819","nonce":3806024048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6821","nonce":3806024048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6825","nonce":3806024048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6823","nonce":3806024048}]},"public_addr":"172.21.15.119:6819/3806024048","cluster_addr":"172.21.15.119:6821/3806024048","heartbeat_back_addr":"172.21.15.119:6825/3806024048","heartbeat_front_addr":"172.21.15.119:6823/3806024048","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":644,"up_from":653,"up_thru":1014,"down_at":644,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6827","nonce":1249776352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6829","nonce":1249776352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6833","nonce":1249776352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6831","nonce":1249776352}]},"public_addr":"172.21.15.119:6827/1249776352","cluster_addr":"172.21.15.119:6829/1249776352","heartbeat_back_addr":"172.21.15.119:6833/1249776352","heartbeat_front_addr":"172.21.15.119:6831/1249776352","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":496},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":529},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":613},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":644},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1","mappings":[{"from":4,"to":1}]},{"pgid":"3.b","mappings":[{"from":4,"to":2},{"from":3,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":1}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":4,"to":1}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/1139856442":"2024-09-21T11:11:38.888722+0000","172.21.15.119:0/2535372813":"2024-09-21T11:06:31.109355+0000","172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.145:0/2808539890":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/49791021":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/576401189":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3992357764":"2024-09-21T11:06:32.113588+0000","172.21.15.119:0/194249182":"2024-09-22T10:01:32.364993+0000","172.21.15.145:6835/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/2781454362":"2024-09-22T09:59:43.810176+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1608923788":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.145:0/974232973":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/547636249":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/460015937":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.145:0/2571529276":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.145:6834/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2566533967":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/2715091572":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/923321224":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/391955164":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/3401809855":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/3153987795":"2024-09-22T10:01:32.364993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":4,"length":5}]},{"pool":16,"snaps":[{"begin":30,"length":8},{"begin":40,"length":8}]},{"pool":17,"snaps":[{"begin":17,"length":2},{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":43,"snaps":[{"begin":104,"length":1}]},{"pool":44,"snaps":[{"begin":104,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:22:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:13 smithi145 ceph-mon[89359]: pgmap v1531: 73 pgs: 73 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail 2024-09-21T10:22:14.787 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:22:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:22:14] "GET /metrics HTTP/1.1" 200 36716 "" "Prometheus/2.51.0" 2024-09-21T10:22:14.812 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T10:22:14.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:22:15.057 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1188132771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:22:15.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1188132771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:22:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1188132771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:22:16.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:15 smithi119 ceph-mon[101432]: pgmap v1532: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:16.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:15 smithi119 ceph-mon[105288]: pgmap v1532: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:16.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:15 smithi145 ceph-mon[89359]: pgmap v1532: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:16.841 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:22:16.841 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":1162,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T10:22:08.837947+0000","last_up_change":"2024-09-21T10:08:54.045519+0000","last_in_change":"2024-09-21T09:54:54.573647+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":22,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":44,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:56:58.923308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:56:59.224014+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":496,"up_from":509,"up_thru":1014,"down_at":497,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6803","nonce":2607764328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6805","nonce":2607764328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6809","nonce":2607764328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6807","nonce":2607764328}]},"public_addr":"172.21.15.119:6803/2607764328","cluster_addr":"172.21.15.119:6805/2607764328","heartbeat_back_addr":"172.21.15.119:6809/2607764328","heartbeat_front_addr":"172.21.15.119:6807/2607764328","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":529,"up_from":542,"up_thru":1014,"down_at":530,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6811","nonce":2769715805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6813","nonce":2769715805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6817","nonce":2769715805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6815","nonce":2769715805}]},"public_addr":"172.21.15.119:6811/2769715805","cluster_addr":"172.21.15.119:6813/2769715805","heartbeat_back_addr":"172.21.15.119:6817/2769715805","heartbeat_front_addr":"172.21.15.119:6815/2769715805","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":613,"up_from":626,"up_thru":1014,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6819","nonce":3806024048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6821","nonce":3806024048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6825","nonce":3806024048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6823","nonce":3806024048}]},"public_addr":"172.21.15.119:6819/3806024048","cluster_addr":"172.21.15.119:6821/3806024048","heartbeat_back_addr":"172.21.15.119:6825/3806024048","heartbeat_front_addr":"172.21.15.119:6823/3806024048","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":644,"up_from":653,"up_thru":1014,"down_at":644,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6827","nonce":1249776352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6829","nonce":1249776352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6833","nonce":1249776352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6831","nonce":1249776352}]},"public_addr":"172.21.15.119:6827/1249776352","cluster_addr":"172.21.15.119:6829/1249776352","heartbeat_back_addr":"172.21.15.119:6833/1249776352","heartbeat_front_addr":"172.21.15.119:6831/1249776352","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6801","nonce":2801631610}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6803","nonce":2801631610}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6807","nonce":2801631610}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":2801631610},{"type":"v1","addr":"172.21.15.145:6805","nonce":2801631610}]},"public_addr":"172.21.15.145:6801/2801631610","cluster_addr":"172.21.15.145:6803/2801631610","heartbeat_back_addr":"172.21.15.145:6807/2801631610","heartbeat_front_addr":"172.21.15.145:6805/2801631610","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6817","nonce":3533380366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6819","nonce":3533380366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6823","nonce":3533380366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3533380366},{"type":"v1","addr":"172.21.15.145:6821","nonce":3533380366}]},"public_addr":"172.21.15.145:6817/3533380366","cluster_addr":"172.21.15.145:6819/3533380366","heartbeat_back_addr":"172.21.15.145:6823/3533380366","heartbeat_front_addr":"172.21.15.145:6821/3533380366","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1014,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":496},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":529},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":613},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":644},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1","mappings":[{"from":4,"to":1}]},{"pgid":"3.b","mappings":[{"from":4,"to":2},{"from":3,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":1}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.1b","mappings":[{"from":4,"to":1}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1c","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/1139856442":"2024-09-21T11:11:38.888722+0000","172.21.15.119:0/2535372813":"2024-09-21T11:06:31.109355+0000","172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.145:0/2808539890":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/49791021":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/576401189":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3992357764":"2024-09-21T11:06:32.113588+0000","172.21.15.119:0/194249182":"2024-09-22T10:01:32.364993+0000","172.21.15.145:6835/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/2781454362":"2024-09-22T09:59:43.810176+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1608923788":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.145:0/974232973":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/547636249":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/460015937":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.145:0/2571529276":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.145:6834/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2566533967":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/2715091572":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/923321224":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/391955164":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/3401809855":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/3153987795":"2024-09-22T10:01:32.364993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":4,"length":5}]},{"pool":16,"snaps":[{"begin":30,"length":8},{"begin":40,"length":8}]},{"pool":17,"snaps":[{"begin":17,"length":2},{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":43,"snaps":[{"begin":104,"length":1}]},{"pool":44,"snaps":[{"begin":104,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:22:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2322241118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:22:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2322241118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:22:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2322241118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:22:17.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T10:22:17.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T10:22:17.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T10:22:17.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T10:22:17.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T10:22:17.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T10:22:17.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T10:22:17.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T10:22:17.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:17 smithi119 ceph-mon[105288]: pgmap v1533: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:17.873 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:17.931 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.932 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.932 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.932 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.935 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.935 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.939 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:17.939 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:18.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:17 smithi119 ceph-mon[101432]: pgmap v1533: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:18.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:17 smithi145 ceph-mon[89359]: pgmap v1533: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:19 smithi119 ceph-mon[105288]: pgmap v1534: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:19 smithi119 ceph-mon[101432]: pgmap v1534: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:20.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:19 smithi145 ceph-mon[89359]: pgmap v1534: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:22:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:22:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:21 smithi119 ceph-mon[105288]: pgmap v1535: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:21 smithi119 ceph-mon[101432]: pgmap v1535: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:21 smithi145 ceph-mon[89359]: pgmap v1535: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:23 smithi119 ceph-mon[105288]: pgmap v1536: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:23 smithi119 ceph-mon[101432]: pgmap v1536: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:23 smithi145 ceph-mon[89359]: pgmap v1536: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:24.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:22:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:22:24] "GET /metrics HTTP/1.1" 200 34285 "" "Prometheus/2.51.0" 2024-09-21T10:22:26.130 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[101432]: pgmap v1537: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:26.130 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:22:26.130 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:26.131 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:26.131 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[105288]: pgmap v1537: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:26.131 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:22:26.131 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:26.131 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:25 smithi145 ceph-mon[89359]: pgmap v1537: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:22:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:27.749 INFO:teuthology.orchestra.run.smithi119.stdout:210453397834 2024-09-21T10:22:27.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T10:22:28.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:27 smithi119 ceph-mon[105288]: pgmap v1538: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:28.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:27 smithi119 ceph-mon[101432]: pgmap v1538: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:28.255 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:27 smithi145 ceph-mon[89359]: pgmap v1538: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:29.450 INFO:teuthology.orchestra.run.smithi119.stdout:184683594064 2024-09-21T10:22:29.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T10:22:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:29 smithi119 ceph-mon[105288]: pgmap v1539: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:29 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:30.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:29 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:30.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:29 smithi119 ceph-mon[101432]: pgmap v1539: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:30.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:29 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:30.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:29 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:30.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:29 smithi145 ceph-mon[89359]: pgmap v1539: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:30.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:29 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:30.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:29 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:30.427 INFO:teuthology.orchestra.run.smithi119.stdout:2327872274623 2024-09-21T10:22:30.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T10:22:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:22:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:22:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:32.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:31 smithi119 ceph-mon[105288]: pgmap v1540: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:32.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:31 smithi119 ceph-mon[101432]: pgmap v1540: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:31 smithi145 ceph-mon[89359]: pgmap v1540: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:33.003 INFO:teuthology.orchestra.run.smithi119.stdout:2688649527470 2024-09-21T10:22:33.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T10:22:33.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:33.242 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:33.669 INFO:teuthology.orchestra.run.smithi119.stdout:133143986521 2024-09-21T10:22:33.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T10:22:33.962 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:33 smithi119 ceph-mon[105288]: pgmap v1541: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:33.962 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:33 smithi119 ceph-mon[101432]: pgmap v1541: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:33.984 INFO:teuthology.orchestra.run.smithi119.stdout:2804613644453 2024-09-21T10:22:33.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T10:22:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:33 smithi145 ceph-mon[89359]: pgmap v1541: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:34.361 INFO:teuthology.orchestra.run.smithi119.stdout:158913790293 2024-09-21T10:22:34.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T10:22:34.645 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:22:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:22:34] "GET /metrics HTTP/1.1" 200 34283 "" "Prometheus/2.51.0" 2024-09-21T10:22:35.070 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:35.071 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:35.548 INFO:teuthology.orchestra.run.smithi119.stdout:2186138353863 2024-09-21T10:22:35.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T10:22:36.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:35 smithi119 ceph-mon[105288]: pgmap v1542: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:36.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:35 smithi119 ceph-mon[101432]: pgmap v1542: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:35 smithi145 ceph-mon[89359]: pgmap v1542: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:38.161 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:37 smithi119 ceph-mon[105288]: pgmap v1543: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:38.162 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:37 smithi119 ceph-mon[101432]: pgmap v1543: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:37 smithi145 ceph-mon[89359]: pgmap v1543: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:38.560 INFO:teuthology.orchestra.run.smithi119.stdout:210453397836 2024-09-21T10:22:39.164 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2935397417' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:22:39.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2935397417' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:22:39.188 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397834 got 210453397836 for osd.7 2024-09-21T10:22:39.188 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2935397417' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:22:39.549 INFO:teuthology.orchestra.run.smithi119.stdout:2327872274625 2024-09-21T10:22:39.747 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:39.748 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:39.748 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:39 smithi119 ceph-mon[105288]: pgmap v1544: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3108765331' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:22:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:39 smithi119 ceph-mon[101432]: pgmap v1544: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3108765331' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:22:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:39 smithi145 ceph-mon[89359]: pgmap v1544: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3108765331' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:22:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:22:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:22:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:42.174 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:41 smithi119 ceph-mon[101432]: pgmap v1545: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:42.174 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:41 smithi119 ceph-mon[105288]: pgmap v1545: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:41 smithi145 ceph-mon[89359]: pgmap v1545: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:43.643 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2327872274623 got 2327872274625 for osd.1 2024-09-21T10:22:43.643 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:44.215 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:43 smithi119 ceph-mon[105288]: pgmap v1546: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:44.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:43 smithi119 ceph-mon[101432]: pgmap v1546: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:43 smithi145 ceph-mon[89359]: pgmap v1546: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:44.777 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:22:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:22:44] "GET /metrics HTTP/1.1" 200 34283 "" "Prometheus/2.51.0" 2024-09-21T10:22:45.059 INFO:teuthology.orchestra.run.smithi119.stdout:2688649527474 2024-09-21T10:22:45.401 INFO:teuthology.orchestra.run.smithi119.stdout:133143986524 2024-09-21T10:22:45.966 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986521 got 133143986524 for osd.4 2024-09-21T10:22:45.966 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:46.061 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:45 smithi119 ceph-mon[101432]: pgmap v1547: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:46.061 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/559072551' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:22:46.061 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2336732019' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:22:46.062 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:45 smithi119 ceph-mon[105288]: pgmap v1547: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:46.062 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/559072551' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:22:46.062 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2336732019' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:22:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:45 smithi145 ceph-mon[89359]: pgmap v1547: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/559072551' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:22:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2336732019' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:22:46.576 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:46.576 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:47.457 INFO:teuthology.orchestra.run.smithi119.stdout:184683594067 2024-09-21T10:22:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:47 smithi119 ceph-mon[101432]: pgmap v1548: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3543421990' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:22:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:47 smithi119 ceph-mon[105288]: pgmap v1548: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3543421990' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:22:48.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:47 smithi145 ceph-mon[89359]: pgmap v1548: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3543421990' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:22:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:22:48.952 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2688649527470 got 2688649527474 for osd.2 2024-09-21T10:22:48.952 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:49.363 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683594064 got 184683594067 for osd.6 2024-09-21T10:22:49.363 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:50.005 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:49 smithi119 ceph-mon[101432]: pgmap v1549: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:50.005 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:49 smithi119 ceph-mon[105288]: pgmap v1549: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:50.088 INFO:teuthology.orchestra.run.smithi119.stdout:2804613644457 2024-09-21T10:22:50.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:49 smithi145 ceph-mon[89359]: pgmap v1549: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:22:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:22:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:22:50.944 INFO:teuthology.orchestra.run.smithi119.stdout:2186138353866 2024-09-21T10:22:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2437156742' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:22:51.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2437156742' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:22:51.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2437156742' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:22:51.664 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2804613644453 got 2804613644457 for osd.3 2024-09-21T10:22:51.664 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:52.193 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:51 smithi119 ceph-mon[105288]: pgmap v1550: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:52.193 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:51 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2702072034' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:22:52.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:51 smithi119 ceph-mon[101432]: pgmap v1550: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:52.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:51 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2702072034' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:22:52.216 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2186138353863 got 2186138353866 for osd.0 2024-09-21T10:22:52.216 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:51 smithi145 ceph-mon[89359]: pgmap v1550: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:51 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2702072034' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:22:52.569 INFO:teuthology.orchestra.run.smithi119.stdout:158913790297 2024-09-21T10:22:53.267 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1625232375' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:22:53.267 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1625232375' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:22:53.290 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790293 got 158913790297 for osd.5 2024-09-21T10:22:53.290 DEBUG:teuthology.parallel:result is None 2024-09-21T10:22:53.290 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:22:53.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:22:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1625232375' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:22:54.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:53 smithi119 ceph-mon[101432]: pgmap v1551: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:54.223 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:53 smithi119 ceph-mon[105288]: pgmap v1551: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:53 smithi145 ceph-mon[89359]: pgmap v1551: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:54.549 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:54.800 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:22:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:22:54] "GET /metrics HTTP/1.1" 200 34286 "" "Prometheus/2.51.0" 2024-09-21T10:22:56.170 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:55 smithi119 ceph-mon[105288]: pgmap v1552: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:56.171 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:55 smithi119 ceph-mon[101432]: pgmap v1552: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:55 smithi145 ceph-mon[89359]: pgmap v1552: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:56.928 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:22:56.929 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:22:57.570 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1553,"stamp":"2024-09-21T10:22:56.793114+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590492,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1547316,"kb_used_data":98392,"kb_used_omap":268,"kb_used_meta":1448563,"kb_avail":748413900,"statfs":{"total":767960285184,"available":766375833600,"internally_reserved":0,"allocated":100753408,"data_stored":65583767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274448,"internal_metadata":1483329520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002814"},"pg_stats":[{"pgid":"3.1f","version":"64'4","reported_seq":1758,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435395+0000","last_change":"2024-09-21T10:08:06.538833+0000","last_active":"2024-09-21T10:22:09.435395+0000","last_peered":"2024-09-21T10:22:09.435395+0000","last_clean":"2024-09-21T10:22:09.435395+0000","last_became_active":"2024-09-21T10:08:06.538526+0000","last_became_peered":"2024-09-21T10:08:06.538526+0000","last_unstale":"2024-09-21T10:22:09.435395+0000","last_undegraded":"2024-09-21T10:22:09.435395+0000","last_fullsized":"2024-09-21T10:22:09.435395+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:59.194474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733525+0000","last_change":"2024-09-21T10:06:42.626454+0000","last_active":"2024-09-21T10:22:04.733525+0000","last_peered":"2024-09-21T10:22:04.733525+0000","last_clean":"2024-09-21T10:22:04.733525+0000","last_became_active":"2024-09-21T10:06:42.626116+0000","last_became_peered":"2024-09-21T10:06:42.626116+0000","last_unstale":"2024-09-21T10:22:04.733525+0000","last_undegraded":"2024-09-21T10:22:04.733525+0000","last_fullsized":"2024-09-21T10:22:04.733525+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:01.968111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1625,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002741+0000","last_change":"2024-09-21T10:08:55.142962+0000","last_active":"2024-09-21T10:22:09.002741+0000","last_peered":"2024-09-21T10:22:09.002741+0000","last_clean":"2024-09-21T10:22:09.002741+0000","last_became_active":"2024-09-21T10:08:55.139681+0000","last_became_peered":"2024-09-21T10:08:55.139681+0000","last_unstale":"2024-09-21T10:22:09.002741+0000","last_undegraded":"2024-09-21T10:22:09.002741+0000","last_fullsized":"2024-09-21T10:22:09.002741+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:52.577104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1656,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002658+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:22:09.002658+0000","last_peered":"2024-09-21T10:22:09.002658+0000","last_clean":"2024-09-21T10:22:09.002658+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:22:09.002658+0000","last_undegraded":"2024-09-21T10:22:09.002658+0000","last_fullsized":"2024-09-21T10:22:09.002658+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329357+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T10:22:04.329357+0000","last_peered":"2024-09-21T10:22:04.329357+0000","last_clean":"2024-09-21T10:22:04.329357+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T10:22:04.329357+0000","last_undegraded":"2024-09-21T10:22:04.329357+0000","last_fullsized":"2024-09-21T10:22:04.329357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734008+0000","last_change":"2024-09-21T10:08:55.108381+0000","last_active":"2024-09-21T10:22:04.734008+0000","last_peered":"2024-09-21T10:22:04.734008+0000","last_clean":"2024-09-21T10:22:04.734008+0000","last_became_active":"2024-09-21T10:08:55.108074+0000","last_became_peered":"2024-09-21T10:08:55.108074+0000","last_unstale":"2024-09-21T10:22:04.734008+0000","last_undegraded":"2024-09-21T10:22:04.734008+0000","last_fullsized":"2024-09-21T10:22:04.734008+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.846660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329020+0000","last_change":"2024-09-21T10:06:42.622644+0000","last_active":"2024-09-21T10:22:04.329020+0000","last_peered":"2024-09-21T10:22:04.329020+0000","last_clean":"2024-09-21T10:22:04.329020+0000","last_became_active":"2024-09-21T10:06:42.621612+0000","last_became_peered":"2024-09-21T10:06:42.621612+0000","last_unstale":"2024-09-21T10:22:04.329020+0000","last_undegraded":"2024-09-21T10:22:04.329020+0000","last_fullsized":"2024-09-21T10:22:04.329020+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:21.301285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733451+0000","last_change":"2024-09-21T10:08:55.109486+0000","last_active":"2024-09-21T10:22:04.733451+0000","last_peered":"2024-09-21T10:22:04.733451+0000","last_clean":"2024-09-21T10:22:04.733451+0000","last_became_active":"2024-09-21T10:08:55.109050+0000","last_became_peered":"2024-09-21T10:08:55.109050+0000","last_unstale":"2024-09-21T10:22:04.733451+0000","last_undegraded":"2024-09-21T10:22:04.733451+0000","last_fullsized":"2024-09-21T10:22:04.733451+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:13.242342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433323+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:22:09.433323+0000","last_peered":"2024-09-21T10:22:09.433323+0000","last_clean":"2024-09-21T10:22:09.433323+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:22:09.433323+0000","last_undegraded":"2024-09-21T10:22:09.433323+0000","last_fullsized":"2024-09-21T10:22:09.433323+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1654,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.852970+0000","last_change":"2024-09-21T10:08:55.117054+0000","last_active":"2024-09-21T10:22:08.852970+0000","last_peered":"2024-09-21T10:22:08.852970+0000","last_clean":"2024-09-21T10:22:08.852970+0000","last_became_active":"2024-09-21T10:08:55.116767+0000","last_became_peered":"2024-09-21T10:08:55.116767+0000","last_unstale":"2024-09-21T10:22:08.852970+0000","last_undegraded":"2024-09-21T10:22:08.852970+0000","last_fullsized":"2024-09-21T10:22:08.852970+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:13.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1138,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734095+0000","last_change":"2024-09-21T10:08:06.539436+0000","last_active":"2024-09-21T10:22:04.734095+0000","last_peered":"2024-09-21T10:22:04.734095+0000","last_clean":"2024-09-21T10:22:04.734095+0000","last_became_active":"2024-09-21T10:08:06.538856+0000","last_became_peered":"2024-09-21T10:08:06.538856+0000","last_unstale":"2024-09-21T10:22:04.734095+0000","last_undegraded":"2024-09-21T10:22:04.734095+0000","last_fullsized":"2024-09-21T10:22:04.734095+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:43.502357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1621,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002896+0000","last_change":"2024-09-21T10:08:55.116807+0000","last_active":"2024-09-21T10:22:09.002896+0000","last_peered":"2024-09-21T10:22:09.002896+0000","last_clean":"2024-09-21T10:22:09.002896+0000","last_became_active":"2024-09-21T10:08:55.115948+0000","last_became_peered":"2024-09-21T10:08:55.115948+0000","last_unstale":"2024-09-21T10:22:09.002896+0000","last_undegraded":"2024-09-21T10:22:09.002896+0000","last_fullsized":"2024-09-21T10:22:09.002896+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:21.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433192+0000","last_change":"2024-09-21T10:08:55.109470+0000","last_active":"2024-09-21T10:22:09.433192+0000","last_peered":"2024-09-21T10:22:09.433192+0000","last_clean":"2024-09-21T10:22:09.433192+0000","last_became_active":"2024-09-21T10:08:55.106885+0000","last_became_peered":"2024-09-21T10:08:55.106885+0000","last_unstale":"2024-09-21T10:22:09.433192+0000","last_undegraded":"2024-09-21T10:22:09.433192+0000","last_fullsized":"2024-09-21T10:22:09.433192+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:51.866025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1752,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434498+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:22:09.434498+0000","last_peered":"2024-09-21T10:22:09.434498+0000","last_clean":"2024-09-21T10:22:09.434498+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:22:09.434498+0000","last_undegraded":"2024-09-21T10:22:09.434498+0000","last_fullsized":"2024-09-21T10:22:09.434498+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1657,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003291+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:22:09.003291+0000","last_peered":"2024-09-21T10:22:09.003291+0000","last_clean":"2024-09-21T10:22:09.003291+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:22:09.003291+0000","last_undegraded":"2024-09-21T10:22:09.003291+0000","last_fullsized":"2024-09-21T10:22:09.003291+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733941+0000","last_change":"2024-09-21T10:06:42.631648+0000","last_active":"2024-09-21T10:22:04.733941+0000","last_peered":"2024-09-21T10:22:04.733941+0000","last_clean":"2024-09-21T10:22:04.733941+0000","last_became_active":"2024-09-21T10:06:42.631301+0000","last_became_peered":"2024-09-21T10:06:42.631301+0000","last_unstale":"2024-09-21T10:22:04.733941+0000","last_undegraded":"2024-09-21T10:22:04.733941+0000","last_fullsized":"2024-09-21T10:22:04.733941+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:52.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1639,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003592+0000","last_change":"2024-09-21T10:08:55.134315+0000","last_active":"2024-09-21T10:22:09.003592+0000","last_peered":"2024-09-21T10:22:09.003592+0000","last_clean":"2024-09-21T10:22:09.003592+0000","last_became_active":"2024-09-21T10:08:55.132999+0000","last_became_peered":"2024-09-21T10:08:55.132999+0000","last_unstale":"2024-09-21T10:22:09.003592+0000","last_undegraded":"2024-09-21T10:22:09.003592+0000","last_fullsized":"2024-09-21T10:22:09.003592+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:24.870318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1749,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435447+0000","last_change":"2024-09-21T10:08:55.105723+0000","last_active":"2024-09-21T10:22:09.435447+0000","last_peered":"2024-09-21T10:22:09.435447+0000","last_clean":"2024-09-21T10:22:09.435447+0000","last_became_active":"2024-09-21T10:08:55.105283+0000","last_became_peered":"2024-09-21T10:08:55.105283+0000","last_unstale":"2024-09-21T10:22:09.435447+0000","last_undegraded":"2024-09-21T10:22:09.435447+0000","last_fullsized":"2024-09-21T10:22:09.435447+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:25.633107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1622,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003537+0000","last_change":"2024-09-21T10:08:55.131455+0000","last_active":"2024-09-21T10:22:09.003537+0000","last_peered":"2024-09-21T10:22:09.003537+0000","last_clean":"2024-09-21T10:22:09.003537+0000","last_became_active":"2024-09-21T10:08:55.129639+0000","last_became_peered":"2024-09-21T10:08:55.129639+0000","last_unstale":"2024-09-21T10:22:09.003537+0000","last_undegraded":"2024-09-21T10:22:09.003537+0000","last_fullsized":"2024-09-21T10:22:09.003537+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:50.934309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329289+0000","last_change":"2024-09-21T10:06:42.623974+0000","last_active":"2024-09-21T10:22:04.329289+0000","last_peered":"2024-09-21T10:22:04.329289+0000","last_clean":"2024-09-21T10:22:04.329289+0000","last_became_active":"2024-09-21T10:06:42.623598+0000","last_became_peered":"2024-09-21T10:06:42.623598+0000","last_unstale":"2024-09-21T10:22:04.329289+0000","last_undegraded":"2024-09-21T10:22:04.329289+0000","last_fullsized":"2024-09-21T10:22:04.329289+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.261148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1270,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.436983+0000","last_change":"2024-09-21T10:08:55.106181+0000","last_active":"2024-09-21T10:22:09.436983+0000","last_peered":"2024-09-21T10:22:09.436983+0000","last_clean":"2024-09-21T10:22:09.436983+0000","last_became_active":"2024-09-21T10:08:55.105882+0000","last_became_peered":"2024-09-21T10:08:55.105882+0000","last_unstale":"2024-09-21T10:22:09.436983+0000","last_undegraded":"2024-09-21T10:22:09.436983+0000","last_fullsized":"2024-09-21T10:22:09.436983+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:30.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1725,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435243+0000","last_change":"2024-09-21T10:08:06.538126+0000","last_active":"2024-09-21T10:22:09.435243+0000","last_peered":"2024-09-21T10:22:09.435243+0000","last_clean":"2024-09-21T10:22:09.435243+0000","last_became_active":"2024-09-21T10:08:06.537779+0000","last_became_peered":"2024-09-21T10:08:06.537779+0000","last_unstale":"2024-09-21T10:22:09.435243+0000","last_undegraded":"2024-09-21T10:22:09.435243+0000","last_fullsized":"2024-09-21T10:22:09.435243+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:34.731581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1125,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733857+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T10:22:04.733857+0000","last_peered":"2024-09-21T10:22:04.733857+0000","last_clean":"2024-09-21T10:22:04.733857+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T10:22:04.733857+0000","last_undegraded":"2024-09-21T10:22:04.733857+0000","last_fullsized":"2024-09-21T10:22:04.733857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733533+0000","last_change":"2024-09-21T10:06:42.633523+0000","last_active":"2024-09-21T10:22:04.733533+0000","last_peered":"2024-09-21T10:22:04.733533+0000","last_clean":"2024-09-21T10:22:04.733533+0000","last_became_active":"2024-09-21T10:06:42.633019+0000","last_became_peered":"2024-09-21T10:06:42.633019+0000","last_unstale":"2024-09-21T10:22:04.733533+0000","last_undegraded":"2024-09-21T10:22:04.733533+0000","last_fullsized":"2024-09-21T10:22:04.733533+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:54.272637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735047+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T10:22:04.735047+0000","last_peered":"2024-09-21T10:22:04.735047+0000","last_clean":"2024-09-21T10:22:04.735047+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T10:22:04.735047+0000","last_undegraded":"2024-09-21T10:22:04.735047+0000","last_fullsized":"2024-09-21T10:22:04.735047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734189+0000","last_change":"2024-09-21T10:08:06.551874+0000","last_active":"2024-09-21T10:22:04.734189+0000","last_peered":"2024-09-21T10:22:04.734189+0000","last_clean":"2024-09-21T10:22:04.734189+0000","last_became_active":"2024-09-21T10:08:06.551572+0000","last_became_peered":"2024-09-21T10:08:06.551572+0000","last_unstale":"2024-09-21T10:22:04.734189+0000","last_undegraded":"2024-09-21T10:22:04.734189+0000","last_fullsized":"2024-09-21T10:22:04.734189+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:56.324479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434303+0000","last_change":"2024-09-21T10:06:42.626709+0000","last_active":"2024-09-21T10:22:09.434303+0000","last_peered":"2024-09-21T10:22:09.434303+0000","last_clean":"2024-09-21T10:22:09.434303+0000","last_became_active":"2024-09-21T10:06:42.626389+0000","last_became_peered":"2024-09-21T10:06:42.626389+0000","last_unstale":"2024-09-21T10:22:09.434303+0000","last_undegraded":"2024-09-21T10:22:09.434303+0000","last_fullsized":"2024-09-21T10:22:09.434303+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:39.634788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003391+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:22:09.003391+0000","last_peered":"2024-09-21T10:22:09.003391+0000","last_clean":"2024-09-21T10:22:09.003391+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:22:09.003391+0000","last_undegraded":"2024-09-21T10:22:09.003391+0000","last_fullsized":"2024-09-21T10:22:09.003391+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1724,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.438211+0000","last_change":"2024-09-21T10:06:42.630257+0000","last_active":"2024-09-21T10:22:09.438211+0000","last_peered":"2024-09-21T10:22:09.438211+0000","last_clean":"2024-09-21T10:22:09.438211+0000","last_became_active":"2024-09-21T10:06:42.628412+0000","last_became_peered":"2024-09-21T10:06:42.628412+0000","last_unstale":"2024-09-21T10:22:09.438211+0000","last_undegraded":"2024-09-21T10:22:09.438211+0000","last_fullsized":"2024-09-21T10:22:09.438211+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:30.166565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1632,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.004029+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:22:09.004029+0000","last_peered":"2024-09-21T10:22:09.004029+0000","last_clean":"2024-09-21T10:22:09.004029+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:22:09.004029+0000","last_undegraded":"2024-09-21T10:22:09.004029+0000","last_fullsized":"2024-09-21T10:22:09.004029+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1741,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432598+0000","last_change":"2024-09-21T10:06:09.039187+0000","last_active":"2024-09-21T10:22:09.432598+0000","last_peered":"2024-09-21T10:22:09.432598+0000","last_clean":"2024-09-21T10:22:09.432598+0000","last_became_active":"2024-09-21T10:06:09.038741+0000","last_became_peered":"2024-09-21T10:06:09.038741+0000","last_unstale":"2024-09-21T10:22:09.432598+0000","last_undegraded":"2024-09-21T10:22:09.432598+0000","last_fullsized":"2024-09-21T10:22:09.432598+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:40.920128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498646+0000","last_change":"2024-09-21T10:06:09.028808+0000","last_active":"2024-09-21T10:22:04.498646+0000","last_peered":"2024-09-21T10:22:04.498646+0000","last_clean":"2024-09-21T10:22:04.498646+0000","last_became_active":"2024-09-21T10:06:09.028492+0000","last_became_peered":"2024-09-21T10:06:09.028492+0000","last_unstale":"2024-09-21T10:22:04.498646+0000","last_undegraded":"2024-09-21T10:22:04.498646+0000","last_fullsized":"2024-09-21T10:22:04.498646+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:41.134185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435081+0000","last_change":"2024-09-21T10:08:06.539915+0000","last_active":"2024-09-21T10:22:09.435081+0000","last_peered":"2024-09-21T10:22:09.435081+0000","last_clean":"2024-09-21T10:22:09.435081+0000","last_became_active":"2024-09-21T10:08:06.539121+0000","last_became_peered":"2024-09-21T10:08:06.539121+0000","last_unstale":"2024-09-21T10:22:09.435081+0000","last_undegraded":"2024-09-21T10:22:09.435081+0000","last_fullsized":"2024-09-21T10:22:09.435081+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:23.549392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1715,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435365+0000","last_change":"2024-09-21T10:06:42.625569+0000","last_active":"2024-09-21T10:22:09.435365+0000","last_peered":"2024-09-21T10:22:09.435365+0000","last_clean":"2024-09-21T10:22:09.435365+0000","last_became_active":"2024-09-21T10:06:42.625259+0000","last_became_peered":"2024-09-21T10:06:42.625259+0000","last_unstale":"2024-09-21T10:22:09.435365+0000","last_undegraded":"2024-09-21T10:22:09.435365+0000","last_fullsized":"2024-09-21T10:22:09.435365+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:45.833707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1740,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435255+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:22:09.435255+0000","last_peered":"2024-09-21T10:22:09.435255+0000","last_clean":"2024-09-21T10:22:09.435255+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:22:09.435255+0000","last_undegraded":"2024-09-21T10:22:09.435255+0000","last_fullsized":"2024-09-21T10:22:09.435255+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329383+0000","last_change":"2024-09-21T10:08:55.102678+0000","last_active":"2024-09-21T10:22:04.329383+0000","last_peered":"2024-09-21T10:22:04.329383+0000","last_clean":"2024-09-21T10:22:04.329383+0000","last_became_active":"2024-09-21T10:08:55.102022+0000","last_became_peered":"2024-09-21T10:08:55.102022+0000","last_unstale":"2024-09-21T10:22:04.329383+0000","last_undegraded":"2024-09-21T10:22:04.329383+0000","last_fullsized":"2024-09-21T10:22:04.329383+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:03.178484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1737,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435170+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:22:09.435170+0000","last_peered":"2024-09-21T10:22:09.435170+0000","last_clean":"2024-09-21T10:22:09.435170+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:22:09.435170+0000","last_undegraded":"2024-09-21T10:22:09.435170+0000","last_fullsized":"2024-09-21T10:22:09.435170+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733623+0000","last_change":"2024-09-21T10:06:09.043111+0000","last_active":"2024-09-21T10:22:04.733623+0000","last_peered":"2024-09-21T10:22:04.733623+0000","last_clean":"2024-09-21T10:22:04.733623+0000","last_became_active":"2024-09-21T10:06:09.042813+0000","last_became_peered":"2024-09-21T10:06:09.042813+0000","last_unstale":"2024-09-21T10:22:04.733623+0000","last_undegraded":"2024-09-21T10:22:04.733623+0000","last_fullsized":"2024-09-21T10:22:04.733623+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:50.398531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328788+0000","last_change":"2024-09-21T10:06:42.622375+0000","last_active":"2024-09-21T10:22:04.328788+0000","last_peered":"2024-09-21T10:22:04.328788+0000","last_clean":"2024-09-21T10:22:04.328788+0000","last_became_active":"2024-09-21T10:06:42.622019+0000","last_became_peered":"2024-09-21T10:06:42.622019+0000","last_unstale":"2024-09-21T10:22:04.328788+0000","last_undegraded":"2024-09-21T10:22:04.328788+0000","last_fullsized":"2024-09-21T10:22:04.328788+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:22.917477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1728,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.437050+0000","last_change":"2024-09-21T10:06:42.627252+0000","last_active":"2024-09-21T10:22:09.437050+0000","last_peered":"2024-09-21T10:22:09.437050+0000","last_clean":"2024-09-21T10:22:09.437050+0000","last_became_active":"2024-09-21T10:06:42.626942+0000","last_became_peered":"2024-09-21T10:06:42.626942+0000","last_unstale":"2024-09-21T10:22:09.437050+0000","last_undegraded":"2024-09-21T10:22:09.437050+0000","last_fullsized":"2024-09-21T10:22:09.437050+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:21.844784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1727,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434134+0000","last_change":"2024-09-21T10:08:06.532136+0000","last_active":"2024-09-21T10:22:09.434134+0000","last_peered":"2024-09-21T10:22:09.434134+0000","last_clean":"2024-09-21T10:22:09.434134+0000","last_became_active":"2024-09-21T10:08:06.531195+0000","last_became_peered":"2024-09-21T10:08:06.531195+0000","last_unstale":"2024-09-21T10:22:09.434134+0000","last_undegraded":"2024-09-21T10:22:09.434134+0000","last_fullsized":"2024-09-21T10:22:09.434134+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:02.094677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1670,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.851840+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:22:08.851840+0000","last_peered":"2024-09-21T10:22:08.851840+0000","last_clean":"2024-09-21T10:22:08.851840+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:22:08.851840+0000","last_undegraded":"2024-09-21T10:22:08.851840+0000","last_fullsized":"2024-09-21T10:22:08.851840+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1762,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434579+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:22:09.434579+0000","last_peered":"2024-09-21T10:22:09.434579+0000","last_clean":"2024-09-21T10:22:09.434579+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:22:09.434579+0000","last_undegraded":"2024-09-21T10:22:09.434579+0000","last_fullsized":"2024-09-21T10:22:09.434579+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1763,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432709+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:22:09.432709+0000","last_peered":"2024-09-21T10:22:09.432709+0000","last_clean":"2024-09-21T10:22:09.432709+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:22:09.432709+0000","last_undegraded":"2024-09-21T10:22:09.432709+0000","last_fullsized":"2024-09-21T10:22:09.432709+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1149,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497998+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:22:04.497998+0000","last_peered":"2024-09-21T10:22:04.497998+0000","last_clean":"2024-09-21T10:22:04.497998+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:22:04.497998+0000","last_undegraded":"2024-09-21T10:22:04.497998+0000","last_fullsized":"2024-09-21T10:22:04.497998+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733686+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T10:22:04.733686+0000","last_peered":"2024-09-21T10:22:04.733686+0000","last_clean":"2024-09-21T10:22:04.733686+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T10:22:04.733686+0000","last_undegraded":"2024-09-21T10:22:04.733686+0000","last_fullsized":"2024-09-21T10:22:04.733686+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1624,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002576+0000","last_change":"2024-09-21T10:08:55.129563+0000","last_active":"2024-09-21T10:22:09.002576+0000","last_peered":"2024-09-21T10:22:09.002576+0000","last_clean":"2024-09-21T10:22:09.002576+0000","last_became_active":"2024-09-21T10:08:55.129268+0000","last_became_peered":"2024-09-21T10:08:55.129268+0000","last_unstale":"2024-09-21T10:22:09.002576+0000","last_undegraded":"2024-09-21T10:22:09.002576+0000","last_fullsized":"2024-09-21T10:22:09.002576+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:32.246505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328879+0000","last_change":"2024-09-21T10:08:06.540841+0000","last_active":"2024-09-21T10:22:04.328879+0000","last_peered":"2024-09-21T10:22:04.328879+0000","last_clean":"2024-09-21T10:22:04.328879+0000","last_became_active":"2024-09-21T10:08:06.539359+0000","last_became_peered":"2024-09-21T10:08:06.539359+0000","last_unstale":"2024-09-21T10:22:04.328879+0000","last_undegraded":"2024-09-21T10:22:04.328879+0000","last_fullsized":"2024-09-21T10:22:04.328879+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:53.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1236,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497868+0000","last_change":"2024-09-21T10:06:09.040194+0000","last_active":"2024-09-21T10:22:04.497868+0000","last_peered":"2024-09-21T10:22:04.497868+0000","last_clean":"2024-09-21T10:22:04.497868+0000","last_became_active":"2024-09-21T10:06:09.039512+0000","last_became_peered":"2024-09-21T10:06:09.039512+0000","last_unstale":"2024-09-21T10:22:04.497868+0000","last_undegraded":"2024-09-21T10:22:04.497868+0000","last_fullsized":"2024-09-21T10:22:04.497868+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:37.194722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435992+0000","last_change":"2024-09-21T10:08:55.112951+0000","last_active":"2024-09-21T10:22:09.435992+0000","last_peered":"2024-09-21T10:22:09.435992+0000","last_clean":"2024-09-21T10:22:09.435992+0000","last_became_active":"2024-09-21T10:08:55.112577+0000","last_became_peered":"2024-09-21T10:08:55.112577+0000","last_unstale":"2024-09-21T10:22:09.435992+0000","last_undegraded":"2024-09-21T10:22:09.435992+0000","last_fullsized":"2024-09-21T10:22:09.435992+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:06.470913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003480+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:22:09.003480+0000","last_peered":"2024-09-21T10:22:09.003480+0000","last_clean":"2024-09-21T10:22:09.003480+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:22:09.003480+0000","last_undegraded":"2024-09-21T10:22:09.003480+0000","last_fullsized":"2024-09-21T10:22:09.003480+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733924+0000","last_change":"2024-09-21T10:06:42.634333+0000","last_active":"2024-09-21T10:22:04.733924+0000","last_peered":"2024-09-21T10:22:04.733924+0000","last_clean":"2024-09-21T10:22:04.733924+0000","last_became_active":"2024-09-21T10:06:42.634037+0000","last_became_peered":"2024-09-21T10:06:42.634037+0000","last_unstale":"2024-09-21T10:22:04.733924+0000","last_undegraded":"2024-09-21T10:22:04.733924+0000","last_fullsized":"2024-09-21T10:22:04.733924+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:27.609590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735108+0000","last_change":"2024-09-21T10:08:06.537347+0000","last_active":"2024-09-21T10:22:04.735108+0000","last_peered":"2024-09-21T10:22:04.735108+0000","last_clean":"2024-09-21T10:22:04.735108+0000","last_became_active":"2024-09-21T10:08:06.537034+0000","last_became_peered":"2024-09-21T10:08:06.537034+0000","last_unstale":"2024-09-21T10:22:04.735108+0000","last_undegraded":"2024-09-21T10:22:04.735108+0000","last_fullsized":"2024-09-21T10:22:04.735108+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:26.325664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1129,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734203+0000","last_change":"2024-09-21T10:08:06.541436+0000","last_active":"2024-09-21T10:22:04.734203+0000","last_peered":"2024-09-21T10:22:04.734203+0000","last_clean":"2024-09-21T10:22:04.734203+0000","last_became_active":"2024-09-21T10:08:06.540616+0000","last_became_peered":"2024-09-21T10:08:06.540616+0000","last_unstale":"2024-09-21T10:22:04.734203+0000","last_undegraded":"2024-09-21T10:22:04.734203+0000","last_fullsized":"2024-09-21T10:22:04.734203+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:18.252764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733503+0000","last_change":"2024-09-21T10:06:42.631470+0000","last_active":"2024-09-21T10:22:04.733503+0000","last_peered":"2024-09-21T10:22:04.733503+0000","last_clean":"2024-09-21T10:22:04.733503+0000","last_became_active":"2024-09-21T10:06:42.631081+0000","last_became_peered":"2024-09-21T10:06:42.631081+0000","last_unstale":"2024-09-21T10:22:04.733503+0000","last_undegraded":"2024-09-21T10:22:04.733503+0000","last_fullsized":"2024-09-21T10:22:04.733503+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:16.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733727+0000","last_change":"2024-09-21T10:08:55.113216+0000","last_active":"2024-09-21T10:22:04.733727+0000","last_peered":"2024-09-21T10:22:04.733727+0000","last_clean":"2024-09-21T10:22:04.733727+0000","last_became_active":"2024-09-21T10:08:55.111808+0000","last_became_peered":"2024-09-21T10:08:55.111808+0000","last_unstale":"2024-09-21T10:22:04.733727+0000","last_undegraded":"2024-09-21T10:22:04.733727+0000","last_fullsized":"2024-09-21T10:22:04.733727+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:24:21.526156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1269,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435511+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:22:09.435511+0000","last_peered":"2024-09-21T10:22:09.435511+0000","last_clean":"2024-09-21T10:22:09.435511+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:22:09.435511+0000","last_undegraded":"2024-09-21T10:22:09.435511+0000","last_fullsized":"2024-09-21T10:22:09.435511+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1751,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433268+0000","last_change":"2024-09-21T10:06:42.626688+0000","last_active":"2024-09-21T10:22:09.433268+0000","last_peered":"2024-09-21T10:22:09.433268+0000","last_clean":"2024-09-21T10:22:09.433268+0000","last_became_active":"2024-09-21T10:06:42.626326+0000","last_became_peered":"2024-09-21T10:06:42.626326+0000","last_unstale":"2024-09-21T10:22:09.433268+0000","last_undegraded":"2024-09-21T10:22:09.433268+0000","last_fullsized":"2024-09-21T10:22:09.433268+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:00.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329061+0000","last_change":"2024-09-21T10:08:55.099610+0000","last_active":"2024-09-21T10:22:04.329061+0000","last_peered":"2024-09-21T10:22:04.329061+0000","last_clean":"2024-09-21T10:22:04.329061+0000","last_became_active":"2024-09-21T10:08:55.099313+0000","last_became_peered":"2024-09-21T10:08:55.099313+0000","last_unstale":"2024-09-21T10:22:04.329061+0000","last_undegraded":"2024-09-21T10:22:04.329061+0000","last_fullsized":"2024-09-21T10:22:04.329061+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:01.266798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733287+0000","last_change":"2024-09-21T10:06:42.631675+0000","last_active":"2024-09-21T10:22:04.733287+0000","last_peered":"2024-09-21T10:22:04.733287+0000","last_clean":"2024-09-21T10:22:04.733287+0000","last_became_active":"2024-09-21T10:06:42.631098+0000","last_became_peered":"2024-09-21T10:06:42.631098+0000","last_unstale":"2024-09-21T10:22:04.733287+0000","last_undegraded":"2024-09-21T10:22:04.733287+0000","last_fullsized":"2024-09-21T10:22:04.733287+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:46.855597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498604+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T10:22:04.498604+0000","last_peered":"2024-09-21T10:22:04.498604+0000","last_clean":"2024-09-21T10:22:04.498604+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T10:22:04.498604+0000","last_undegraded":"2024-09-21T10:22:04.498604+0000","last_fullsized":"2024-09-21T10:22:04.498604+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734962+0000","last_change":"2024-09-21T10:08:55.107939+0000","last_active":"2024-09-21T10:22:04.734962+0000","last_peered":"2024-09-21T10:22:04.734962+0000","last_clean":"2024-09-21T10:22:04.734962+0000","last_became_active":"2024-09-21T10:08:55.107640+0000","last_became_peered":"2024-09-21T10:08:55.107640+0000","last_unstale":"2024-09-21T10:22:04.734962+0000","last_undegraded":"2024-09-21T10:22:04.734962+0000","last_fullsized":"2024-09-21T10:22:04.734962+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:11.287664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1122,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498791+0000","last_change":"2024-09-21T10:06:42.632153+0000","last_active":"2024-09-21T10:22:04.498791+0000","last_peered":"2024-09-21T10:22:04.498791+0000","last_clean":"2024-09-21T10:22:04.498791+0000","last_became_active":"2024-09-21T10:06:42.631728+0000","last_became_peered":"2024-09-21T10:06:42.631728+0000","last_unstale":"2024-09-21T10:22:04.498791+0000","last_undegraded":"2024-09-21T10:22:04.498791+0000","last_fullsized":"2024-09-21T10:22:04.498791+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:53.412364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1101,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328844+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T10:22:04.328844+0000","last_peered":"2024-09-21T10:22:04.328844+0000","last_clean":"2024-09-21T10:22:04.328844+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T10:22:04.328844+0000","last_undegraded":"2024-09-21T10:22:04.328844+0000","last_fullsized":"2024-09-21T10:22:04.328844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.499006+0000","last_change":"2024-09-21T10:06:09.030444+0000","last_active":"2024-09-21T10:22:04.499006+0000","last_peered":"2024-09-21T10:22:04.499006+0000","last_clean":"2024-09-21T10:22:04.499006+0000","last_became_active":"2024-09-21T10:06:09.030054+0000","last_became_peered":"2024-09-21T10:06:09.030054+0000","last_unstale":"2024-09-21T10:22:04.499006+0000","last_undegraded":"2024-09-21T10:22:04.499006+0000","last_fullsized":"2024-09-21T10:22:04.499006+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:27.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1128,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735035+0000","last_change":"2024-09-21T10:06:09.032476+0000","last_active":"2024-09-21T10:22:04.735035+0000","last_peered":"2024-09-21T10:22:04.735035+0000","last_clean":"2024-09-21T10:22:04.735035+0000","last_became_active":"2024-09-21T10:06:09.032121+0000","last_became_peered":"2024-09-21T10:06:09.032121+0000","last_unstale":"2024-09-21T10:22:04.735035+0000","last_undegraded":"2024-09-21T10:22:04.735035+0000","last_fullsized":"2024-09-21T10:22:04.735035+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:41.862129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1631,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003838+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:22:09.003838+0000","last_peered":"2024-09-21T10:22:09.003838+0000","last_clean":"2024-09-21T10:22:09.003838+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:22:09.003838+0000","last_undegraded":"2024-09-21T10:22:09.003838+0000","last_fullsized":"2024-09-21T10:22:09.003838+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1130,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498375+0000","last_change":"2024-09-21T10:08:06.546911+0000","last_active":"2024-09-21T10:22:04.498375+0000","last_peered":"2024-09-21T10:22:04.498375+0000","last_clean":"2024-09-21T10:22:04.498375+0000","last_became_active":"2024-09-21T10:08:06.546417+0000","last_became_peered":"2024-09-21T10:08:06.546417+0000","last_unstale":"2024-09-21T10:22:04.498375+0000","last_undegraded":"2024-09-21T10:22:04.498375+0000","last_fullsized":"2024-09-21T10:22:04.498375+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:46.544701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434412+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:22:09.434412+0000","last_peered":"2024-09-21T10:22:09.434412+0000","last_clean":"2024-09-21T10:22:09.434412+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:22:09.434412+0000","last_undegraded":"2024-09-21T10:22:09.434412+0000","last_fullsized":"2024-09-21T10:22:09.434412+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432873+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:22:09.432873+0000","last_peered":"2024-09-21T10:22:09.432873+0000","last_clean":"2024-09-21T10:22:09.432873+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:22:09.432873+0000","last_undegraded":"2024-09-21T10:22:09.432873+0000","last_fullsized":"2024-09-21T10:22:09.432873+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329295+0000","last_change":"2024-09-21T10:08:55.102502+0000","last_active":"2024-09-21T10:22:04.329295+0000","last_peered":"2024-09-21T10:22:04.329295+0000","last_clean":"2024-09-21T10:22:04.329295+0000","last_became_active":"2024-09-21T10:08:55.101777+0000","last_became_peered":"2024-09-21T10:08:55.101777+0000","last_unstale":"2024-09-21T10:22:04.329295+0000","last_undegraded":"2024-09-21T10:22:04.329295+0000","last_fullsized":"2024-09-21T10:22:04.329295+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:51.293401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1152,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498158+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:22:04.498158+0000","last_peered":"2024-09-21T10:22:04.498158+0000","last_clean":"2024-09-21T10:22:04.498158+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:22:04.498158+0000","last_undegraded":"2024-09-21T10:22:04.498158+0000","last_fullsized":"2024-09-21T10:22:04.498158+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1639,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003615+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:22:09.003615+0000","last_peered":"2024-09-21T10:22:09.003615+0000","last_clean":"2024-09-21T10:22:09.003615+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:22:09.003615+0000","last_undegraded":"2024-09-21T10:22:09.003615+0000","last_fullsized":"2024-09-21T10:22:09.003615+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":124,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397840,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314188,"kb_used_data":12672,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93430964,"statfs":{"total":95995035648,"available":95673307136,"internally_reserved":0,"allocated":12976128,"data_stored":8569725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594069,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322504,"kb_used_data":12668,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93422648,"statfs":{"total":95995035648,"available":95664791552,"internally_reserved":0,"allocated":12972032,"data_stored":8569696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790298,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313596,"kb_used_data":12080,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93431556,"statfs":{"total":95995035648,"available":95673913344,"internally_reserved":0,"allocated":12369920,"data_stored":7975146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986526,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321920,"kb_used_data":12088,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93423232,"statfs":{"total":95995035648,"available":95665389568,"internally_reserved":0,"allocated":12378112,"data_stored":7979343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644458,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58100,"kb_used_data":12072,"kb_used_omap":67,"kb_used_meta":45948,"kb_avail":93687052,"statfs":{"total":95995035648,"available":95935541248,"internally_reserved":0,"allocated":12361728,"data_stored":7974597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":47051613},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66232,"kb_used_data":12076,"kb_used_omap":75,"kb_used_meta":54068,"kb_avail":93678920,"statfs":{"total":95995035648,"available":95927214080,"internally_reserved":0,"allocated":12365824,"data_stored":7975119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":55365786},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274629,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88188,"kb_used_data":12080,"kb_used_omap":51,"kb_used_meta":76044,"kb_avail":93656964,"statfs":{"total":95995035648,"available":95904731136,"internally_reserved":0,"allocated":12369920,"data_stored":7974658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":77869397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353867,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62588,"kb_used_data":12656,"kb_used_omap":66,"kb_used_meta":49853,"kb_avail":93682564,"statfs":{"total":95995035648,"available":95930945536,"internally_reserved":0,"allocated":12959744,"data_stored":8565483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":51049796},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:22:57.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:22:57.817 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: pgmap v1553: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.830 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:57 smithi145 ceph-mon[89359]: from='client.35883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: pgmap v1553: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[101432]: from='client.35883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: pgmap v1553: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:22:57.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:57 smithi119 ceph-mon[105288]: from='client.35883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:22:59.611 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:22:59.612 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:22:59.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:22:59 smithi119 ceph-mon[105288]: pgmap v1554: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:22:59.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:22:59 smithi119 ceph-mon[101432]: pgmap v1554: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:00.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:22:59 smithi145 ceph-mon[89359]: pgmap v1554: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:00.243 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2024-09-21T10:22:58.793483+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590492,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1547316,"kb_used_data":98392,"kb_used_omap":268,"kb_used_meta":1448563,"kb_avail":748413900,"statfs":{"total":767960285184,"available":766375833600,"internally_reserved":0,"allocated":100753408,"data_stored":65583767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274448,"internal_metadata":1483329520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002717"},"pg_stats":[{"pgid":"3.1f","version":"64'4","reported_seq":1758,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435395+0000","last_change":"2024-09-21T10:08:06.538833+0000","last_active":"2024-09-21T10:22:09.435395+0000","last_peered":"2024-09-21T10:22:09.435395+0000","last_clean":"2024-09-21T10:22:09.435395+0000","last_became_active":"2024-09-21T10:08:06.538526+0000","last_became_peered":"2024-09-21T10:08:06.538526+0000","last_unstale":"2024-09-21T10:22:09.435395+0000","last_undegraded":"2024-09-21T10:22:09.435395+0000","last_fullsized":"2024-09-21T10:22:09.435395+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:59.194474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733525+0000","last_change":"2024-09-21T10:06:42.626454+0000","last_active":"2024-09-21T10:22:04.733525+0000","last_peered":"2024-09-21T10:22:04.733525+0000","last_clean":"2024-09-21T10:22:04.733525+0000","last_became_active":"2024-09-21T10:06:42.626116+0000","last_became_peered":"2024-09-21T10:06:42.626116+0000","last_unstale":"2024-09-21T10:22:04.733525+0000","last_undegraded":"2024-09-21T10:22:04.733525+0000","last_fullsized":"2024-09-21T10:22:04.733525+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:01.968111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1626,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002789+0000","last_change":"2024-09-21T10:08:55.142962+0000","last_active":"2024-09-21T10:22:09.002789+0000","last_peered":"2024-09-21T10:22:09.002789+0000","last_clean":"2024-09-21T10:22:09.002789+0000","last_became_active":"2024-09-21T10:08:55.139681+0000","last_became_peered":"2024-09-21T10:08:55.139681+0000","last_unstale":"2024-09-21T10:22:09.002789+0000","last_undegraded":"2024-09-21T10:22:09.002789+0000","last_fullsized":"2024-09-21T10:22:09.002789+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:52.577104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1656,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002658+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:22:09.002658+0000","last_peered":"2024-09-21T10:22:09.002658+0000","last_clean":"2024-09-21T10:22:09.002658+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:22:09.002658+0000","last_undegraded":"2024-09-21T10:22:09.002658+0000","last_fullsized":"2024-09-21T10:22:09.002658+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329357+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T10:22:04.329357+0000","last_peered":"2024-09-21T10:22:04.329357+0000","last_clean":"2024-09-21T10:22:04.329357+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T10:22:04.329357+0000","last_undegraded":"2024-09-21T10:22:04.329357+0000","last_fullsized":"2024-09-21T10:22:04.329357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734008+0000","last_change":"2024-09-21T10:08:55.108381+0000","last_active":"2024-09-21T10:22:04.734008+0000","last_peered":"2024-09-21T10:22:04.734008+0000","last_clean":"2024-09-21T10:22:04.734008+0000","last_became_active":"2024-09-21T10:08:55.108074+0000","last_became_peered":"2024-09-21T10:08:55.108074+0000","last_unstale":"2024-09-21T10:22:04.734008+0000","last_undegraded":"2024-09-21T10:22:04.734008+0000","last_fullsized":"2024-09-21T10:22:04.734008+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.846660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329020+0000","last_change":"2024-09-21T10:06:42.622644+0000","last_active":"2024-09-21T10:22:04.329020+0000","last_peered":"2024-09-21T10:22:04.329020+0000","last_clean":"2024-09-21T10:22:04.329020+0000","last_became_active":"2024-09-21T10:06:42.621612+0000","last_became_peered":"2024-09-21T10:06:42.621612+0000","last_unstale":"2024-09-21T10:22:04.329020+0000","last_undegraded":"2024-09-21T10:22:04.329020+0000","last_fullsized":"2024-09-21T10:22:04.329020+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:21.301285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733451+0000","last_change":"2024-09-21T10:08:55.109486+0000","last_active":"2024-09-21T10:22:04.733451+0000","last_peered":"2024-09-21T10:22:04.733451+0000","last_clean":"2024-09-21T10:22:04.733451+0000","last_became_active":"2024-09-21T10:08:55.109050+0000","last_became_peered":"2024-09-21T10:08:55.109050+0000","last_unstale":"2024-09-21T10:22:04.733451+0000","last_undegraded":"2024-09-21T10:22:04.733451+0000","last_fullsized":"2024-09-21T10:22:04.733451+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:13.242342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433323+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:22:09.433323+0000","last_peered":"2024-09-21T10:22:09.433323+0000","last_clean":"2024-09-21T10:22:09.433323+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:22:09.433323+0000","last_undegraded":"2024-09-21T10:22:09.433323+0000","last_fullsized":"2024-09-21T10:22:09.433323+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1654,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.852970+0000","last_change":"2024-09-21T10:08:55.117054+0000","last_active":"2024-09-21T10:22:08.852970+0000","last_peered":"2024-09-21T10:22:08.852970+0000","last_clean":"2024-09-21T10:22:08.852970+0000","last_became_active":"2024-09-21T10:08:55.116767+0000","last_became_peered":"2024-09-21T10:08:55.116767+0000","last_unstale":"2024-09-21T10:22:08.852970+0000","last_undegraded":"2024-09-21T10:22:08.852970+0000","last_fullsized":"2024-09-21T10:22:08.852970+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:13.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1138,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734095+0000","last_change":"2024-09-21T10:08:06.539436+0000","last_active":"2024-09-21T10:22:04.734095+0000","last_peered":"2024-09-21T10:22:04.734095+0000","last_clean":"2024-09-21T10:22:04.734095+0000","last_became_active":"2024-09-21T10:08:06.538856+0000","last_became_peered":"2024-09-21T10:08:06.538856+0000","last_unstale":"2024-09-21T10:22:04.734095+0000","last_undegraded":"2024-09-21T10:22:04.734095+0000","last_fullsized":"2024-09-21T10:22:04.734095+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:43.502357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1621,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002896+0000","last_change":"2024-09-21T10:08:55.116807+0000","last_active":"2024-09-21T10:22:09.002896+0000","last_peered":"2024-09-21T10:22:09.002896+0000","last_clean":"2024-09-21T10:22:09.002896+0000","last_became_active":"2024-09-21T10:08:55.115948+0000","last_became_peered":"2024-09-21T10:08:55.115948+0000","last_unstale":"2024-09-21T10:22:09.002896+0000","last_undegraded":"2024-09-21T10:22:09.002896+0000","last_fullsized":"2024-09-21T10:22:09.002896+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:21.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433192+0000","last_change":"2024-09-21T10:08:55.109470+0000","last_active":"2024-09-21T10:22:09.433192+0000","last_peered":"2024-09-21T10:22:09.433192+0000","last_clean":"2024-09-21T10:22:09.433192+0000","last_became_active":"2024-09-21T10:08:55.106885+0000","last_became_peered":"2024-09-21T10:08:55.106885+0000","last_unstale":"2024-09-21T10:22:09.433192+0000","last_undegraded":"2024-09-21T10:22:09.433192+0000","last_fullsized":"2024-09-21T10:22:09.433192+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:51.866025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1752,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434498+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:22:09.434498+0000","last_peered":"2024-09-21T10:22:09.434498+0000","last_clean":"2024-09-21T10:22:09.434498+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:22:09.434498+0000","last_undegraded":"2024-09-21T10:22:09.434498+0000","last_fullsized":"2024-09-21T10:22:09.434498+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1657,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003291+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:22:09.003291+0000","last_peered":"2024-09-21T10:22:09.003291+0000","last_clean":"2024-09-21T10:22:09.003291+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:22:09.003291+0000","last_undegraded":"2024-09-21T10:22:09.003291+0000","last_fullsized":"2024-09-21T10:22:09.003291+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733941+0000","last_change":"2024-09-21T10:06:42.631648+0000","last_active":"2024-09-21T10:22:04.733941+0000","last_peered":"2024-09-21T10:22:04.733941+0000","last_clean":"2024-09-21T10:22:04.733941+0000","last_became_active":"2024-09-21T10:06:42.631301+0000","last_became_peered":"2024-09-21T10:06:42.631301+0000","last_unstale":"2024-09-21T10:22:04.733941+0000","last_undegraded":"2024-09-21T10:22:04.733941+0000","last_fullsized":"2024-09-21T10:22:04.733941+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:52.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003631+0000","last_change":"2024-09-21T10:08:55.134315+0000","last_active":"2024-09-21T10:22:09.003631+0000","last_peered":"2024-09-21T10:22:09.003631+0000","last_clean":"2024-09-21T10:22:09.003631+0000","last_became_active":"2024-09-21T10:08:55.132999+0000","last_became_peered":"2024-09-21T10:08:55.132999+0000","last_unstale":"2024-09-21T10:22:09.003631+0000","last_undegraded":"2024-09-21T10:22:09.003631+0000","last_fullsized":"2024-09-21T10:22:09.003631+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:24.870318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1749,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435447+0000","last_change":"2024-09-21T10:08:55.105723+0000","last_active":"2024-09-21T10:22:09.435447+0000","last_peered":"2024-09-21T10:22:09.435447+0000","last_clean":"2024-09-21T10:22:09.435447+0000","last_became_active":"2024-09-21T10:08:55.105283+0000","last_became_peered":"2024-09-21T10:08:55.105283+0000","last_unstale":"2024-09-21T10:22:09.435447+0000","last_undegraded":"2024-09-21T10:22:09.435447+0000","last_fullsized":"2024-09-21T10:22:09.435447+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:25.633107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1623,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003595+0000","last_change":"2024-09-21T10:08:55.131455+0000","last_active":"2024-09-21T10:22:09.003595+0000","last_peered":"2024-09-21T10:22:09.003595+0000","last_clean":"2024-09-21T10:22:09.003595+0000","last_became_active":"2024-09-21T10:08:55.129639+0000","last_became_peered":"2024-09-21T10:08:55.129639+0000","last_unstale":"2024-09-21T10:22:09.003595+0000","last_undegraded":"2024-09-21T10:22:09.003595+0000","last_fullsized":"2024-09-21T10:22:09.003595+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:50.934309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329289+0000","last_change":"2024-09-21T10:06:42.623974+0000","last_active":"2024-09-21T10:22:04.329289+0000","last_peered":"2024-09-21T10:22:04.329289+0000","last_clean":"2024-09-21T10:22:04.329289+0000","last_became_active":"2024-09-21T10:06:42.623598+0000","last_became_peered":"2024-09-21T10:06:42.623598+0000","last_unstale":"2024-09-21T10:22:04.329289+0000","last_undegraded":"2024-09-21T10:22:04.329289+0000","last_fullsized":"2024-09-21T10:22:04.329289+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.261148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1270,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.436983+0000","last_change":"2024-09-21T10:08:55.106181+0000","last_active":"2024-09-21T10:22:09.436983+0000","last_peered":"2024-09-21T10:22:09.436983+0000","last_clean":"2024-09-21T10:22:09.436983+0000","last_became_active":"2024-09-21T10:08:55.105882+0000","last_became_peered":"2024-09-21T10:08:55.105882+0000","last_unstale":"2024-09-21T10:22:09.436983+0000","last_undegraded":"2024-09-21T10:22:09.436983+0000","last_fullsized":"2024-09-21T10:22:09.436983+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:30.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1725,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435243+0000","last_change":"2024-09-21T10:08:06.538126+0000","last_active":"2024-09-21T10:22:09.435243+0000","last_peered":"2024-09-21T10:22:09.435243+0000","last_clean":"2024-09-21T10:22:09.435243+0000","last_became_active":"2024-09-21T10:08:06.537779+0000","last_became_peered":"2024-09-21T10:08:06.537779+0000","last_unstale":"2024-09-21T10:22:09.435243+0000","last_undegraded":"2024-09-21T10:22:09.435243+0000","last_fullsized":"2024-09-21T10:22:09.435243+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:34.731581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1125,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733857+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T10:22:04.733857+0000","last_peered":"2024-09-21T10:22:04.733857+0000","last_clean":"2024-09-21T10:22:04.733857+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T10:22:04.733857+0000","last_undegraded":"2024-09-21T10:22:04.733857+0000","last_fullsized":"2024-09-21T10:22:04.733857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733533+0000","last_change":"2024-09-21T10:06:42.633523+0000","last_active":"2024-09-21T10:22:04.733533+0000","last_peered":"2024-09-21T10:22:04.733533+0000","last_clean":"2024-09-21T10:22:04.733533+0000","last_became_active":"2024-09-21T10:06:42.633019+0000","last_became_peered":"2024-09-21T10:06:42.633019+0000","last_unstale":"2024-09-21T10:22:04.733533+0000","last_undegraded":"2024-09-21T10:22:04.733533+0000","last_fullsized":"2024-09-21T10:22:04.733533+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:54.272637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735047+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T10:22:04.735047+0000","last_peered":"2024-09-21T10:22:04.735047+0000","last_clean":"2024-09-21T10:22:04.735047+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T10:22:04.735047+0000","last_undegraded":"2024-09-21T10:22:04.735047+0000","last_fullsized":"2024-09-21T10:22:04.735047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734189+0000","last_change":"2024-09-21T10:08:06.551874+0000","last_active":"2024-09-21T10:22:04.734189+0000","last_peered":"2024-09-21T10:22:04.734189+0000","last_clean":"2024-09-21T10:22:04.734189+0000","last_became_active":"2024-09-21T10:08:06.551572+0000","last_became_peered":"2024-09-21T10:08:06.551572+0000","last_unstale":"2024-09-21T10:22:04.734189+0000","last_undegraded":"2024-09-21T10:22:04.734189+0000","last_fullsized":"2024-09-21T10:22:04.734189+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:56.324479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434303+0000","last_change":"2024-09-21T10:06:42.626709+0000","last_active":"2024-09-21T10:22:09.434303+0000","last_peered":"2024-09-21T10:22:09.434303+0000","last_clean":"2024-09-21T10:22:09.434303+0000","last_became_active":"2024-09-21T10:06:42.626389+0000","last_became_peered":"2024-09-21T10:06:42.626389+0000","last_unstale":"2024-09-21T10:22:09.434303+0000","last_undegraded":"2024-09-21T10:22:09.434303+0000","last_fullsized":"2024-09-21T10:22:09.434303+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:39.634788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003429+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:22:09.003429+0000","last_peered":"2024-09-21T10:22:09.003429+0000","last_clean":"2024-09-21T10:22:09.003429+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:22:09.003429+0000","last_undegraded":"2024-09-21T10:22:09.003429+0000","last_fullsized":"2024-09-21T10:22:09.003429+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1724,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.438211+0000","last_change":"2024-09-21T10:06:42.630257+0000","last_active":"2024-09-21T10:22:09.438211+0000","last_peered":"2024-09-21T10:22:09.438211+0000","last_clean":"2024-09-21T10:22:09.438211+0000","last_became_active":"2024-09-21T10:06:42.628412+0000","last_became_peered":"2024-09-21T10:06:42.628412+0000","last_unstale":"2024-09-21T10:22:09.438211+0000","last_undegraded":"2024-09-21T10:22:09.438211+0000","last_fullsized":"2024-09-21T10:22:09.438211+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:30.166565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1632,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.004029+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:22:09.004029+0000","last_peered":"2024-09-21T10:22:09.004029+0000","last_clean":"2024-09-21T10:22:09.004029+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:22:09.004029+0000","last_undegraded":"2024-09-21T10:22:09.004029+0000","last_fullsized":"2024-09-21T10:22:09.004029+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1741,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432598+0000","last_change":"2024-09-21T10:06:09.039187+0000","last_active":"2024-09-21T10:22:09.432598+0000","last_peered":"2024-09-21T10:22:09.432598+0000","last_clean":"2024-09-21T10:22:09.432598+0000","last_became_active":"2024-09-21T10:06:09.038741+0000","last_became_peered":"2024-09-21T10:06:09.038741+0000","last_unstale":"2024-09-21T10:22:09.432598+0000","last_undegraded":"2024-09-21T10:22:09.432598+0000","last_fullsized":"2024-09-21T10:22:09.432598+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:40.920128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498646+0000","last_change":"2024-09-21T10:06:09.028808+0000","last_active":"2024-09-21T10:22:04.498646+0000","last_peered":"2024-09-21T10:22:04.498646+0000","last_clean":"2024-09-21T10:22:04.498646+0000","last_became_active":"2024-09-21T10:06:09.028492+0000","last_became_peered":"2024-09-21T10:06:09.028492+0000","last_unstale":"2024-09-21T10:22:04.498646+0000","last_undegraded":"2024-09-21T10:22:04.498646+0000","last_fullsized":"2024-09-21T10:22:04.498646+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:41.134185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435081+0000","last_change":"2024-09-21T10:08:06.539915+0000","last_active":"2024-09-21T10:22:09.435081+0000","last_peered":"2024-09-21T10:22:09.435081+0000","last_clean":"2024-09-21T10:22:09.435081+0000","last_became_active":"2024-09-21T10:08:06.539121+0000","last_became_peered":"2024-09-21T10:08:06.539121+0000","last_unstale":"2024-09-21T10:22:09.435081+0000","last_undegraded":"2024-09-21T10:22:09.435081+0000","last_fullsized":"2024-09-21T10:22:09.435081+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:23.549392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1715,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435365+0000","last_change":"2024-09-21T10:06:42.625569+0000","last_active":"2024-09-21T10:22:09.435365+0000","last_peered":"2024-09-21T10:22:09.435365+0000","last_clean":"2024-09-21T10:22:09.435365+0000","last_became_active":"2024-09-21T10:06:42.625259+0000","last_became_peered":"2024-09-21T10:06:42.625259+0000","last_unstale":"2024-09-21T10:22:09.435365+0000","last_undegraded":"2024-09-21T10:22:09.435365+0000","last_fullsized":"2024-09-21T10:22:09.435365+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:45.833707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1740,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435255+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:22:09.435255+0000","last_peered":"2024-09-21T10:22:09.435255+0000","last_clean":"2024-09-21T10:22:09.435255+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:22:09.435255+0000","last_undegraded":"2024-09-21T10:22:09.435255+0000","last_fullsized":"2024-09-21T10:22:09.435255+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329383+0000","last_change":"2024-09-21T10:08:55.102678+0000","last_active":"2024-09-21T10:22:04.329383+0000","last_peered":"2024-09-21T10:22:04.329383+0000","last_clean":"2024-09-21T10:22:04.329383+0000","last_became_active":"2024-09-21T10:08:55.102022+0000","last_became_peered":"2024-09-21T10:08:55.102022+0000","last_unstale":"2024-09-21T10:22:04.329383+0000","last_undegraded":"2024-09-21T10:22:04.329383+0000","last_fullsized":"2024-09-21T10:22:04.329383+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:03.178484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1737,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435170+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:22:09.435170+0000","last_peered":"2024-09-21T10:22:09.435170+0000","last_clean":"2024-09-21T10:22:09.435170+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:22:09.435170+0000","last_undegraded":"2024-09-21T10:22:09.435170+0000","last_fullsized":"2024-09-21T10:22:09.435170+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733623+0000","last_change":"2024-09-21T10:06:09.043111+0000","last_active":"2024-09-21T10:22:04.733623+0000","last_peered":"2024-09-21T10:22:04.733623+0000","last_clean":"2024-09-21T10:22:04.733623+0000","last_became_active":"2024-09-21T10:06:09.042813+0000","last_became_peered":"2024-09-21T10:06:09.042813+0000","last_unstale":"2024-09-21T10:22:04.733623+0000","last_undegraded":"2024-09-21T10:22:04.733623+0000","last_fullsized":"2024-09-21T10:22:04.733623+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:50.398531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328788+0000","last_change":"2024-09-21T10:06:42.622375+0000","last_active":"2024-09-21T10:22:04.328788+0000","last_peered":"2024-09-21T10:22:04.328788+0000","last_clean":"2024-09-21T10:22:04.328788+0000","last_became_active":"2024-09-21T10:06:42.622019+0000","last_became_peered":"2024-09-21T10:06:42.622019+0000","last_unstale":"2024-09-21T10:22:04.328788+0000","last_undegraded":"2024-09-21T10:22:04.328788+0000","last_fullsized":"2024-09-21T10:22:04.328788+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:22.917477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1728,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.437050+0000","last_change":"2024-09-21T10:06:42.627252+0000","last_active":"2024-09-21T10:22:09.437050+0000","last_peered":"2024-09-21T10:22:09.437050+0000","last_clean":"2024-09-21T10:22:09.437050+0000","last_became_active":"2024-09-21T10:06:42.626942+0000","last_became_peered":"2024-09-21T10:06:42.626942+0000","last_unstale":"2024-09-21T10:22:09.437050+0000","last_undegraded":"2024-09-21T10:22:09.437050+0000","last_fullsized":"2024-09-21T10:22:09.437050+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:21.844784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1727,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434134+0000","last_change":"2024-09-21T10:08:06.532136+0000","last_active":"2024-09-21T10:22:09.434134+0000","last_peered":"2024-09-21T10:22:09.434134+0000","last_clean":"2024-09-21T10:22:09.434134+0000","last_became_active":"2024-09-21T10:08:06.531195+0000","last_became_peered":"2024-09-21T10:08:06.531195+0000","last_unstale":"2024-09-21T10:22:09.434134+0000","last_undegraded":"2024-09-21T10:22:09.434134+0000","last_fullsized":"2024-09-21T10:22:09.434134+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:02.094677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1670,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.851840+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:22:08.851840+0000","last_peered":"2024-09-21T10:22:08.851840+0000","last_clean":"2024-09-21T10:22:08.851840+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:22:08.851840+0000","last_undegraded":"2024-09-21T10:22:08.851840+0000","last_fullsized":"2024-09-21T10:22:08.851840+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1762,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434579+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:22:09.434579+0000","last_peered":"2024-09-21T10:22:09.434579+0000","last_clean":"2024-09-21T10:22:09.434579+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:22:09.434579+0000","last_undegraded":"2024-09-21T10:22:09.434579+0000","last_fullsized":"2024-09-21T10:22:09.434579+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1763,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432709+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:22:09.432709+0000","last_peered":"2024-09-21T10:22:09.432709+0000","last_clean":"2024-09-21T10:22:09.432709+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:22:09.432709+0000","last_undegraded":"2024-09-21T10:22:09.432709+0000","last_fullsized":"2024-09-21T10:22:09.432709+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1149,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497998+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:22:04.497998+0000","last_peered":"2024-09-21T10:22:04.497998+0000","last_clean":"2024-09-21T10:22:04.497998+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:22:04.497998+0000","last_undegraded":"2024-09-21T10:22:04.497998+0000","last_fullsized":"2024-09-21T10:22:04.497998+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733686+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T10:22:04.733686+0000","last_peered":"2024-09-21T10:22:04.733686+0000","last_clean":"2024-09-21T10:22:04.733686+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T10:22:04.733686+0000","last_undegraded":"2024-09-21T10:22:04.733686+0000","last_fullsized":"2024-09-21T10:22:04.733686+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1624,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002576+0000","last_change":"2024-09-21T10:08:55.129563+0000","last_active":"2024-09-21T10:22:09.002576+0000","last_peered":"2024-09-21T10:22:09.002576+0000","last_clean":"2024-09-21T10:22:09.002576+0000","last_became_active":"2024-09-21T10:08:55.129268+0000","last_became_peered":"2024-09-21T10:08:55.129268+0000","last_unstale":"2024-09-21T10:22:09.002576+0000","last_undegraded":"2024-09-21T10:22:09.002576+0000","last_fullsized":"2024-09-21T10:22:09.002576+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:32.246505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328879+0000","last_change":"2024-09-21T10:08:06.540841+0000","last_active":"2024-09-21T10:22:04.328879+0000","last_peered":"2024-09-21T10:22:04.328879+0000","last_clean":"2024-09-21T10:22:04.328879+0000","last_became_active":"2024-09-21T10:08:06.539359+0000","last_became_peered":"2024-09-21T10:08:06.539359+0000","last_unstale":"2024-09-21T10:22:04.328879+0000","last_undegraded":"2024-09-21T10:22:04.328879+0000","last_fullsized":"2024-09-21T10:22:04.328879+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:53.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1236,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497868+0000","last_change":"2024-09-21T10:06:09.040194+0000","last_active":"2024-09-21T10:22:04.497868+0000","last_peered":"2024-09-21T10:22:04.497868+0000","last_clean":"2024-09-21T10:22:04.497868+0000","last_became_active":"2024-09-21T10:06:09.039512+0000","last_became_peered":"2024-09-21T10:06:09.039512+0000","last_unstale":"2024-09-21T10:22:04.497868+0000","last_undegraded":"2024-09-21T10:22:04.497868+0000","last_fullsized":"2024-09-21T10:22:04.497868+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:37.194722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435992+0000","last_change":"2024-09-21T10:08:55.112951+0000","last_active":"2024-09-21T10:22:09.435992+0000","last_peered":"2024-09-21T10:22:09.435992+0000","last_clean":"2024-09-21T10:22:09.435992+0000","last_became_active":"2024-09-21T10:08:55.112577+0000","last_became_peered":"2024-09-21T10:08:55.112577+0000","last_unstale":"2024-09-21T10:22:09.435992+0000","last_undegraded":"2024-09-21T10:22:09.435992+0000","last_fullsized":"2024-09-21T10:22:09.435992+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:06.470913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003480+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:22:09.003480+0000","last_peered":"2024-09-21T10:22:09.003480+0000","last_clean":"2024-09-21T10:22:09.003480+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:22:09.003480+0000","last_undegraded":"2024-09-21T10:22:09.003480+0000","last_fullsized":"2024-09-21T10:22:09.003480+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733924+0000","last_change":"2024-09-21T10:06:42.634333+0000","last_active":"2024-09-21T10:22:04.733924+0000","last_peered":"2024-09-21T10:22:04.733924+0000","last_clean":"2024-09-21T10:22:04.733924+0000","last_became_active":"2024-09-21T10:06:42.634037+0000","last_became_peered":"2024-09-21T10:06:42.634037+0000","last_unstale":"2024-09-21T10:22:04.733924+0000","last_undegraded":"2024-09-21T10:22:04.733924+0000","last_fullsized":"2024-09-21T10:22:04.733924+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:27.609590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735108+0000","last_change":"2024-09-21T10:08:06.537347+0000","last_active":"2024-09-21T10:22:04.735108+0000","last_peered":"2024-09-21T10:22:04.735108+0000","last_clean":"2024-09-21T10:22:04.735108+0000","last_became_active":"2024-09-21T10:08:06.537034+0000","last_became_peered":"2024-09-21T10:08:06.537034+0000","last_unstale":"2024-09-21T10:22:04.735108+0000","last_undegraded":"2024-09-21T10:22:04.735108+0000","last_fullsized":"2024-09-21T10:22:04.735108+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:26.325664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1129,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734203+0000","last_change":"2024-09-21T10:08:06.541436+0000","last_active":"2024-09-21T10:22:04.734203+0000","last_peered":"2024-09-21T10:22:04.734203+0000","last_clean":"2024-09-21T10:22:04.734203+0000","last_became_active":"2024-09-21T10:08:06.540616+0000","last_became_peered":"2024-09-21T10:08:06.540616+0000","last_unstale":"2024-09-21T10:22:04.734203+0000","last_undegraded":"2024-09-21T10:22:04.734203+0000","last_fullsized":"2024-09-21T10:22:04.734203+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:18.252764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733503+0000","last_change":"2024-09-21T10:06:42.631470+0000","last_active":"2024-09-21T10:22:04.733503+0000","last_peered":"2024-09-21T10:22:04.733503+0000","last_clean":"2024-09-21T10:22:04.733503+0000","last_became_active":"2024-09-21T10:06:42.631081+0000","last_became_peered":"2024-09-21T10:06:42.631081+0000","last_unstale":"2024-09-21T10:22:04.733503+0000","last_undegraded":"2024-09-21T10:22:04.733503+0000","last_fullsized":"2024-09-21T10:22:04.733503+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:16.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733727+0000","last_change":"2024-09-21T10:08:55.113216+0000","last_active":"2024-09-21T10:22:04.733727+0000","last_peered":"2024-09-21T10:22:04.733727+0000","last_clean":"2024-09-21T10:22:04.733727+0000","last_became_active":"2024-09-21T10:08:55.111808+0000","last_became_peered":"2024-09-21T10:08:55.111808+0000","last_unstale":"2024-09-21T10:22:04.733727+0000","last_undegraded":"2024-09-21T10:22:04.733727+0000","last_fullsized":"2024-09-21T10:22:04.733727+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:24:21.526156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1269,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435511+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:22:09.435511+0000","last_peered":"2024-09-21T10:22:09.435511+0000","last_clean":"2024-09-21T10:22:09.435511+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:22:09.435511+0000","last_undegraded":"2024-09-21T10:22:09.435511+0000","last_fullsized":"2024-09-21T10:22:09.435511+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1751,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433268+0000","last_change":"2024-09-21T10:06:42.626688+0000","last_active":"2024-09-21T10:22:09.433268+0000","last_peered":"2024-09-21T10:22:09.433268+0000","last_clean":"2024-09-21T10:22:09.433268+0000","last_became_active":"2024-09-21T10:06:42.626326+0000","last_became_peered":"2024-09-21T10:06:42.626326+0000","last_unstale":"2024-09-21T10:22:09.433268+0000","last_undegraded":"2024-09-21T10:22:09.433268+0000","last_fullsized":"2024-09-21T10:22:09.433268+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:00.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329061+0000","last_change":"2024-09-21T10:08:55.099610+0000","last_active":"2024-09-21T10:22:04.329061+0000","last_peered":"2024-09-21T10:22:04.329061+0000","last_clean":"2024-09-21T10:22:04.329061+0000","last_became_active":"2024-09-21T10:08:55.099313+0000","last_became_peered":"2024-09-21T10:08:55.099313+0000","last_unstale":"2024-09-21T10:22:04.329061+0000","last_undegraded":"2024-09-21T10:22:04.329061+0000","last_fullsized":"2024-09-21T10:22:04.329061+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:01.266798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733287+0000","last_change":"2024-09-21T10:06:42.631675+0000","last_active":"2024-09-21T10:22:04.733287+0000","last_peered":"2024-09-21T10:22:04.733287+0000","last_clean":"2024-09-21T10:22:04.733287+0000","last_became_active":"2024-09-21T10:06:42.631098+0000","last_became_peered":"2024-09-21T10:06:42.631098+0000","last_unstale":"2024-09-21T10:22:04.733287+0000","last_undegraded":"2024-09-21T10:22:04.733287+0000","last_fullsized":"2024-09-21T10:22:04.733287+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:46.855597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498604+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T10:22:04.498604+0000","last_peered":"2024-09-21T10:22:04.498604+0000","last_clean":"2024-09-21T10:22:04.498604+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T10:22:04.498604+0000","last_undegraded":"2024-09-21T10:22:04.498604+0000","last_fullsized":"2024-09-21T10:22:04.498604+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734962+0000","last_change":"2024-09-21T10:08:55.107939+0000","last_active":"2024-09-21T10:22:04.734962+0000","last_peered":"2024-09-21T10:22:04.734962+0000","last_clean":"2024-09-21T10:22:04.734962+0000","last_became_active":"2024-09-21T10:08:55.107640+0000","last_became_peered":"2024-09-21T10:08:55.107640+0000","last_unstale":"2024-09-21T10:22:04.734962+0000","last_undegraded":"2024-09-21T10:22:04.734962+0000","last_fullsized":"2024-09-21T10:22:04.734962+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:11.287664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1122,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498791+0000","last_change":"2024-09-21T10:06:42.632153+0000","last_active":"2024-09-21T10:22:04.498791+0000","last_peered":"2024-09-21T10:22:04.498791+0000","last_clean":"2024-09-21T10:22:04.498791+0000","last_became_active":"2024-09-21T10:06:42.631728+0000","last_became_peered":"2024-09-21T10:06:42.631728+0000","last_unstale":"2024-09-21T10:22:04.498791+0000","last_undegraded":"2024-09-21T10:22:04.498791+0000","last_fullsized":"2024-09-21T10:22:04.498791+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:53.412364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1101,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328844+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T10:22:04.328844+0000","last_peered":"2024-09-21T10:22:04.328844+0000","last_clean":"2024-09-21T10:22:04.328844+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T10:22:04.328844+0000","last_undegraded":"2024-09-21T10:22:04.328844+0000","last_fullsized":"2024-09-21T10:22:04.328844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.499006+0000","last_change":"2024-09-21T10:06:09.030444+0000","last_active":"2024-09-21T10:22:04.499006+0000","last_peered":"2024-09-21T10:22:04.499006+0000","last_clean":"2024-09-21T10:22:04.499006+0000","last_became_active":"2024-09-21T10:06:09.030054+0000","last_became_peered":"2024-09-21T10:06:09.030054+0000","last_unstale":"2024-09-21T10:22:04.499006+0000","last_undegraded":"2024-09-21T10:22:04.499006+0000","last_fullsized":"2024-09-21T10:22:04.499006+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:27.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1128,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735035+0000","last_change":"2024-09-21T10:06:09.032476+0000","last_active":"2024-09-21T10:22:04.735035+0000","last_peered":"2024-09-21T10:22:04.735035+0000","last_clean":"2024-09-21T10:22:04.735035+0000","last_became_active":"2024-09-21T10:06:09.032121+0000","last_became_peered":"2024-09-21T10:06:09.032121+0000","last_unstale":"2024-09-21T10:22:04.735035+0000","last_undegraded":"2024-09-21T10:22:04.735035+0000","last_fullsized":"2024-09-21T10:22:04.735035+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:41.862129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1631,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003838+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:22:09.003838+0000","last_peered":"2024-09-21T10:22:09.003838+0000","last_clean":"2024-09-21T10:22:09.003838+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:22:09.003838+0000","last_undegraded":"2024-09-21T10:22:09.003838+0000","last_fullsized":"2024-09-21T10:22:09.003838+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1130,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498375+0000","last_change":"2024-09-21T10:08:06.546911+0000","last_active":"2024-09-21T10:22:04.498375+0000","last_peered":"2024-09-21T10:22:04.498375+0000","last_clean":"2024-09-21T10:22:04.498375+0000","last_became_active":"2024-09-21T10:08:06.546417+0000","last_became_peered":"2024-09-21T10:08:06.546417+0000","last_unstale":"2024-09-21T10:22:04.498375+0000","last_undegraded":"2024-09-21T10:22:04.498375+0000","last_fullsized":"2024-09-21T10:22:04.498375+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:46.544701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434412+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:22:09.434412+0000","last_peered":"2024-09-21T10:22:09.434412+0000","last_clean":"2024-09-21T10:22:09.434412+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:22:09.434412+0000","last_undegraded":"2024-09-21T10:22:09.434412+0000","last_fullsized":"2024-09-21T10:22:09.434412+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432873+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:22:09.432873+0000","last_peered":"2024-09-21T10:22:09.432873+0000","last_clean":"2024-09-21T10:22:09.432873+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:22:09.432873+0000","last_undegraded":"2024-09-21T10:22:09.432873+0000","last_fullsized":"2024-09-21T10:22:09.432873+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329295+0000","last_change":"2024-09-21T10:08:55.102502+0000","last_active":"2024-09-21T10:22:04.329295+0000","last_peered":"2024-09-21T10:22:04.329295+0000","last_clean":"2024-09-21T10:22:04.329295+0000","last_became_active":"2024-09-21T10:08:55.101777+0000","last_became_peered":"2024-09-21T10:08:55.101777+0000","last_unstale":"2024-09-21T10:22:04.329295+0000","last_undegraded":"2024-09-21T10:22:04.329295+0000","last_fullsized":"2024-09-21T10:22:04.329295+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:51.293401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1152,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498158+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:22:04.498158+0000","last_peered":"2024-09-21T10:22:04.498158+0000","last_clean":"2024-09-21T10:22:04.498158+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:22:04.498158+0000","last_undegraded":"2024-09-21T10:22:04.498158+0000","last_fullsized":"2024-09-21T10:22:04.498158+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003671+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:22:09.003671+0000","last_peered":"2024-09-21T10:22:09.003671+0000","last_clean":"2024-09-21T10:22:09.003671+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:22:09.003671+0000","last_undegraded":"2024-09-21T10:22:09.003671+0000","last_fullsized":"2024-09-21T10:22:09.003671+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":124,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397841,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314188,"kb_used_data":12672,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93430964,"statfs":{"total":95995035648,"available":95673307136,"internally_reserved":0,"allocated":12976128,"data_stored":8569725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594070,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322504,"kb_used_data":12668,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93422648,"statfs":{"total":95995035648,"available":95664791552,"internally_reserved":0,"allocated":12972032,"data_stored":8569696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790299,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313596,"kb_used_data":12080,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93431556,"statfs":{"total":95995035648,"available":95673913344,"internally_reserved":0,"allocated":12369920,"data_stored":7975146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986527,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321920,"kb_used_data":12088,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93423232,"statfs":{"total":95995035648,"available":95665389568,"internally_reserved":0,"allocated":12378112,"data_stored":7979343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644459,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58100,"kb_used_data":12072,"kb_used_omap":67,"kb_used_meta":45948,"kb_avail":93687052,"statfs":{"total":95995035648,"available":95935541248,"internally_reserved":0,"allocated":12361728,"data_stored":7974597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":47051613},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527476,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66232,"kb_used_data":12076,"kb_used_omap":75,"kb_used_meta":54068,"kb_avail":93678920,"statfs":{"total":95995035648,"available":95927214080,"internally_reserved":0,"allocated":12365824,"data_stored":7975119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":55365786},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274629,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88188,"kb_used_data":12080,"kb_used_omap":51,"kb_used_meta":76044,"kb_avail":93656964,"statfs":{"total":95995035648,"available":95904731136,"internally_reserved":0,"allocated":12369920,"data_stored":7974658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":77869397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353868,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62588,"kb_used_data":12656,"kb_used_omap":66,"kb_used_meta":49853,"kb_avail":93682564,"statfs":{"total":95995035648,"available":95930945536,"internally_reserved":0,"allocated":12959744,"data_stored":8565483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":51049796},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:23:00.248 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:23:00.248 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T10:23:00.249 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T10:23:00.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T10:23:00.495 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:23:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:23:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:00 smithi119 ceph-mon[101432]: from='client.35889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:00 smithi119 ceph-mon[105288]: from='client.35889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:00 smithi145 ceph-mon[89359]: from='client.35889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:01.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:01 smithi119 ceph-mon[105288]: pgmap v1555: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:01.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:01 smithi119 ceph-mon[101432]: pgmap v1555: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:01 smithi145 ceph-mon[89359]: pgmap v1555: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:02.413 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:23:02.413 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T10:23:03.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2326665495' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:23:03.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:03.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2326665495' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:23:03.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:03.057 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T10:23:03.057 INFO:teuthology.run_tasks:Running task print... 2024-09-21T10:23:03.067 INFO:teuthology.task.print:**** applying stress + thrashing to mixed-version cluster **** 2024-09-21T10:23:03.067 INFO:teuthology.run_tasks:Running task parallel... 2024-09-21T10:23:03.076 INFO:teuthology.task.parallel:starting parallel... 2024-09-21T10:23:03.076 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-21T10:23:03.076 INFO:teuthology.task.sequential:In sequential, running task thrashosds... 2024-09-21T10:23:03.078 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:23:03.078 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'create_rbd_pool': True, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'OSD_DOWN', 'mons down', 'mon down', 'MON_DOWN', 'out of quorum', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'OSDMAP_FLAGS', 'PG_AVAILABILITY'], 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '62da54b4b83a96ca9976b7e1d0c8272a564a2208'}} 2024-09-21T10:23:03.078 INFO:tasks.thrashosds:config is {'aggressive_pg_num_changes': False, 'chance_force_recovery': 0, 'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_thrash_pg_upmap_items': 0, 'disable_objectstore_tool_tests': True, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T10:23:03.078 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T10:23:03.078 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T10:23:03.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:23:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2326665495' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-21T10:23:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:03.325 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:03 smithi119 ceph-mon[105288]: pgmap v1556: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:03 smithi119 ceph-mon[101432]: pgmap v1556: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:03 smithi145 ceph-mon[89359]: pgmap v1556: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:23:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:23:04] "GET /metrics HTTP/1.1" 200 34282 "" "Prometheus/2.51.0" 2024-09-21T10:23:05.177 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:23:05.177 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:23:05.828 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1557,"stamp":"2024-09-21T10:23:04.794913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590492,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1547316,"kb_used_data":98392,"kb_used_omap":268,"kb_used_meta":1448563,"kb_avail":748413900,"statfs":{"total":767960285184,"available":766375833600,"internally_reserved":0,"allocated":100753408,"data_stored":65583767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274448,"internal_metadata":1483329520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002831"},"pg_stats":[{"pgid":"3.1f","version":"64'4","reported_seq":1758,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435395+0000","last_change":"2024-09-21T10:08:06.538833+0000","last_active":"2024-09-21T10:22:09.435395+0000","last_peered":"2024-09-21T10:22:09.435395+0000","last_clean":"2024-09-21T10:22:09.435395+0000","last_became_active":"2024-09-21T10:08:06.538526+0000","last_became_peered":"2024-09-21T10:08:06.538526+0000","last_unstale":"2024-09-21T10:22:09.435395+0000","last_undegraded":"2024-09-21T10:22:09.435395+0000","last_fullsized":"2024-09-21T10:22:09.435395+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:59.194474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733525+0000","last_change":"2024-09-21T10:06:42.626454+0000","last_active":"2024-09-21T10:22:04.733525+0000","last_peered":"2024-09-21T10:22:04.733525+0000","last_clean":"2024-09-21T10:22:04.733525+0000","last_became_active":"2024-09-21T10:06:42.626116+0000","last_became_peered":"2024-09-21T10:06:42.626116+0000","last_unstale":"2024-09-21T10:22:04.733525+0000","last_undegraded":"2024-09-21T10:22:04.733525+0000","last_fullsized":"2024-09-21T10:22:04.733525+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:01.968111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1626,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002789+0000","last_change":"2024-09-21T10:08:55.142962+0000","last_active":"2024-09-21T10:22:09.002789+0000","last_peered":"2024-09-21T10:22:09.002789+0000","last_clean":"2024-09-21T10:22:09.002789+0000","last_became_active":"2024-09-21T10:08:55.139681+0000","last_became_peered":"2024-09-21T10:08:55.139681+0000","last_unstale":"2024-09-21T10:22:09.002789+0000","last_undegraded":"2024-09-21T10:22:09.002789+0000","last_fullsized":"2024-09-21T10:22:09.002789+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:52.577104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1656,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002658+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:22:09.002658+0000","last_peered":"2024-09-21T10:22:09.002658+0000","last_clean":"2024-09-21T10:22:09.002658+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:22:09.002658+0000","last_undegraded":"2024-09-21T10:22:09.002658+0000","last_fullsized":"2024-09-21T10:22:09.002658+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329357+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T10:22:04.329357+0000","last_peered":"2024-09-21T10:22:04.329357+0000","last_clean":"2024-09-21T10:22:04.329357+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T10:22:04.329357+0000","last_undegraded":"2024-09-21T10:22:04.329357+0000","last_fullsized":"2024-09-21T10:22:04.329357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734008+0000","last_change":"2024-09-21T10:08:55.108381+0000","last_active":"2024-09-21T10:22:04.734008+0000","last_peered":"2024-09-21T10:22:04.734008+0000","last_clean":"2024-09-21T10:22:04.734008+0000","last_became_active":"2024-09-21T10:08:55.108074+0000","last_became_peered":"2024-09-21T10:08:55.108074+0000","last_unstale":"2024-09-21T10:22:04.734008+0000","last_undegraded":"2024-09-21T10:22:04.734008+0000","last_fullsized":"2024-09-21T10:22:04.734008+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.846660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329020+0000","last_change":"2024-09-21T10:06:42.622644+0000","last_active":"2024-09-21T10:22:04.329020+0000","last_peered":"2024-09-21T10:22:04.329020+0000","last_clean":"2024-09-21T10:22:04.329020+0000","last_became_active":"2024-09-21T10:06:42.621612+0000","last_became_peered":"2024-09-21T10:06:42.621612+0000","last_unstale":"2024-09-21T10:22:04.329020+0000","last_undegraded":"2024-09-21T10:22:04.329020+0000","last_fullsized":"2024-09-21T10:22:04.329020+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:21.301285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733451+0000","last_change":"2024-09-21T10:08:55.109486+0000","last_active":"2024-09-21T10:22:04.733451+0000","last_peered":"2024-09-21T10:22:04.733451+0000","last_clean":"2024-09-21T10:22:04.733451+0000","last_became_active":"2024-09-21T10:08:55.109050+0000","last_became_peered":"2024-09-21T10:08:55.109050+0000","last_unstale":"2024-09-21T10:22:04.733451+0000","last_undegraded":"2024-09-21T10:22:04.733451+0000","last_fullsized":"2024-09-21T10:22:04.733451+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:13.242342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433323+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:22:09.433323+0000","last_peered":"2024-09-21T10:22:09.433323+0000","last_clean":"2024-09-21T10:22:09.433323+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:22:09.433323+0000","last_undegraded":"2024-09-21T10:22:09.433323+0000","last_fullsized":"2024-09-21T10:22:09.433323+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1655,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.853024+0000","last_change":"2024-09-21T10:08:55.117054+0000","last_active":"2024-09-21T10:22:08.853024+0000","last_peered":"2024-09-21T10:22:08.853024+0000","last_clean":"2024-09-21T10:22:08.853024+0000","last_became_active":"2024-09-21T10:08:55.116767+0000","last_became_peered":"2024-09-21T10:08:55.116767+0000","last_unstale":"2024-09-21T10:22:08.853024+0000","last_undegraded":"2024-09-21T10:22:08.853024+0000","last_fullsized":"2024-09-21T10:22:08.853024+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:13.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1138,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734095+0000","last_change":"2024-09-21T10:08:06.539436+0000","last_active":"2024-09-21T10:22:04.734095+0000","last_peered":"2024-09-21T10:22:04.734095+0000","last_clean":"2024-09-21T10:22:04.734095+0000","last_became_active":"2024-09-21T10:08:06.538856+0000","last_became_peered":"2024-09-21T10:08:06.538856+0000","last_unstale":"2024-09-21T10:22:04.734095+0000","last_undegraded":"2024-09-21T10:22:04.734095+0000","last_fullsized":"2024-09-21T10:22:04.734095+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:43.502357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1621,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002896+0000","last_change":"2024-09-21T10:08:55.116807+0000","last_active":"2024-09-21T10:22:09.002896+0000","last_peered":"2024-09-21T10:22:09.002896+0000","last_clean":"2024-09-21T10:22:09.002896+0000","last_became_active":"2024-09-21T10:08:55.115948+0000","last_became_peered":"2024-09-21T10:08:55.115948+0000","last_unstale":"2024-09-21T10:22:09.002896+0000","last_undegraded":"2024-09-21T10:22:09.002896+0000","last_fullsized":"2024-09-21T10:22:09.002896+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:21.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433192+0000","last_change":"2024-09-21T10:08:55.109470+0000","last_active":"2024-09-21T10:22:09.433192+0000","last_peered":"2024-09-21T10:22:09.433192+0000","last_clean":"2024-09-21T10:22:09.433192+0000","last_became_active":"2024-09-21T10:08:55.106885+0000","last_became_peered":"2024-09-21T10:08:55.106885+0000","last_unstale":"2024-09-21T10:22:09.433192+0000","last_undegraded":"2024-09-21T10:22:09.433192+0000","last_fullsized":"2024-09-21T10:22:09.433192+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:51.866025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1752,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434498+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:22:09.434498+0000","last_peered":"2024-09-21T10:22:09.434498+0000","last_clean":"2024-09-21T10:22:09.434498+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:22:09.434498+0000","last_undegraded":"2024-09-21T10:22:09.434498+0000","last_fullsized":"2024-09-21T10:22:09.434498+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1657,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003291+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:22:09.003291+0000","last_peered":"2024-09-21T10:22:09.003291+0000","last_clean":"2024-09-21T10:22:09.003291+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:22:09.003291+0000","last_undegraded":"2024-09-21T10:22:09.003291+0000","last_fullsized":"2024-09-21T10:22:09.003291+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733941+0000","last_change":"2024-09-21T10:06:42.631648+0000","last_active":"2024-09-21T10:22:04.733941+0000","last_peered":"2024-09-21T10:22:04.733941+0000","last_clean":"2024-09-21T10:22:04.733941+0000","last_became_active":"2024-09-21T10:06:42.631301+0000","last_became_peered":"2024-09-21T10:06:42.631301+0000","last_unstale":"2024-09-21T10:22:04.733941+0000","last_undegraded":"2024-09-21T10:22:04.733941+0000","last_fullsized":"2024-09-21T10:22:04.733941+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:52.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003631+0000","last_change":"2024-09-21T10:08:55.134315+0000","last_active":"2024-09-21T10:22:09.003631+0000","last_peered":"2024-09-21T10:22:09.003631+0000","last_clean":"2024-09-21T10:22:09.003631+0000","last_became_active":"2024-09-21T10:08:55.132999+0000","last_became_peered":"2024-09-21T10:08:55.132999+0000","last_unstale":"2024-09-21T10:22:09.003631+0000","last_undegraded":"2024-09-21T10:22:09.003631+0000","last_fullsized":"2024-09-21T10:22:09.003631+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:24.870318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435488+0000","last_change":"2024-09-21T10:08:55.105723+0000","last_active":"2024-09-21T10:22:09.435488+0000","last_peered":"2024-09-21T10:22:09.435488+0000","last_clean":"2024-09-21T10:22:09.435488+0000","last_became_active":"2024-09-21T10:08:55.105283+0000","last_became_peered":"2024-09-21T10:08:55.105283+0000","last_unstale":"2024-09-21T10:22:09.435488+0000","last_undegraded":"2024-09-21T10:22:09.435488+0000","last_fullsized":"2024-09-21T10:22:09.435488+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:25.633107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1623,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003595+0000","last_change":"2024-09-21T10:08:55.131455+0000","last_active":"2024-09-21T10:22:09.003595+0000","last_peered":"2024-09-21T10:22:09.003595+0000","last_clean":"2024-09-21T10:22:09.003595+0000","last_became_active":"2024-09-21T10:08:55.129639+0000","last_became_peered":"2024-09-21T10:08:55.129639+0000","last_unstale":"2024-09-21T10:22:09.003595+0000","last_undegraded":"2024-09-21T10:22:09.003595+0000","last_fullsized":"2024-09-21T10:22:09.003595+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:50.934309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329289+0000","last_change":"2024-09-21T10:06:42.623974+0000","last_active":"2024-09-21T10:22:04.329289+0000","last_peered":"2024-09-21T10:22:04.329289+0000","last_clean":"2024-09-21T10:22:04.329289+0000","last_became_active":"2024-09-21T10:06:42.623598+0000","last_became_peered":"2024-09-21T10:06:42.623598+0000","last_unstale":"2024-09-21T10:22:04.329289+0000","last_undegraded":"2024-09-21T10:22:04.329289+0000","last_fullsized":"2024-09-21T10:22:04.329289+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.261148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1270,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.436983+0000","last_change":"2024-09-21T10:08:55.106181+0000","last_active":"2024-09-21T10:22:09.436983+0000","last_peered":"2024-09-21T10:22:09.436983+0000","last_clean":"2024-09-21T10:22:09.436983+0000","last_became_active":"2024-09-21T10:08:55.105882+0000","last_became_peered":"2024-09-21T10:08:55.105882+0000","last_unstale":"2024-09-21T10:22:09.436983+0000","last_undegraded":"2024-09-21T10:22:09.436983+0000","last_fullsized":"2024-09-21T10:22:09.436983+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:30.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435278+0000","last_change":"2024-09-21T10:08:06.538126+0000","last_active":"2024-09-21T10:22:09.435278+0000","last_peered":"2024-09-21T10:22:09.435278+0000","last_clean":"2024-09-21T10:22:09.435278+0000","last_became_active":"2024-09-21T10:08:06.537779+0000","last_became_peered":"2024-09-21T10:08:06.537779+0000","last_unstale":"2024-09-21T10:22:09.435278+0000","last_undegraded":"2024-09-21T10:22:09.435278+0000","last_fullsized":"2024-09-21T10:22:09.435278+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:34.731581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1125,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733857+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T10:22:04.733857+0000","last_peered":"2024-09-21T10:22:04.733857+0000","last_clean":"2024-09-21T10:22:04.733857+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T10:22:04.733857+0000","last_undegraded":"2024-09-21T10:22:04.733857+0000","last_fullsized":"2024-09-21T10:22:04.733857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733533+0000","last_change":"2024-09-21T10:06:42.633523+0000","last_active":"2024-09-21T10:22:04.733533+0000","last_peered":"2024-09-21T10:22:04.733533+0000","last_clean":"2024-09-21T10:22:04.733533+0000","last_became_active":"2024-09-21T10:06:42.633019+0000","last_became_peered":"2024-09-21T10:06:42.633019+0000","last_unstale":"2024-09-21T10:22:04.733533+0000","last_undegraded":"2024-09-21T10:22:04.733533+0000","last_fullsized":"2024-09-21T10:22:04.733533+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:54.272637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735047+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T10:22:04.735047+0000","last_peered":"2024-09-21T10:22:04.735047+0000","last_clean":"2024-09-21T10:22:04.735047+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T10:22:04.735047+0000","last_undegraded":"2024-09-21T10:22:04.735047+0000","last_fullsized":"2024-09-21T10:22:04.735047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734189+0000","last_change":"2024-09-21T10:08:06.551874+0000","last_active":"2024-09-21T10:22:04.734189+0000","last_peered":"2024-09-21T10:22:04.734189+0000","last_clean":"2024-09-21T10:22:04.734189+0000","last_became_active":"2024-09-21T10:08:06.551572+0000","last_became_peered":"2024-09-21T10:08:06.551572+0000","last_unstale":"2024-09-21T10:22:04.734189+0000","last_undegraded":"2024-09-21T10:22:04.734189+0000","last_fullsized":"2024-09-21T10:22:04.734189+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:56.324479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434303+0000","last_change":"2024-09-21T10:06:42.626709+0000","last_active":"2024-09-21T10:22:09.434303+0000","last_peered":"2024-09-21T10:22:09.434303+0000","last_clean":"2024-09-21T10:22:09.434303+0000","last_became_active":"2024-09-21T10:06:42.626389+0000","last_became_peered":"2024-09-21T10:06:42.626389+0000","last_unstale":"2024-09-21T10:22:09.434303+0000","last_undegraded":"2024-09-21T10:22:09.434303+0000","last_fullsized":"2024-09-21T10:22:09.434303+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:39.634788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003429+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:22:09.003429+0000","last_peered":"2024-09-21T10:22:09.003429+0000","last_clean":"2024-09-21T10:22:09.003429+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:22:09.003429+0000","last_undegraded":"2024-09-21T10:22:09.003429+0000","last_fullsized":"2024-09-21T10:22:09.003429+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1724,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.438211+0000","last_change":"2024-09-21T10:06:42.630257+0000","last_active":"2024-09-21T10:22:09.438211+0000","last_peered":"2024-09-21T10:22:09.438211+0000","last_clean":"2024-09-21T10:22:09.438211+0000","last_became_active":"2024-09-21T10:06:42.628412+0000","last_became_peered":"2024-09-21T10:06:42.628412+0000","last_unstale":"2024-09-21T10:22:09.438211+0000","last_undegraded":"2024-09-21T10:22:09.438211+0000","last_fullsized":"2024-09-21T10:22:09.438211+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:30.166565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1632,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.004029+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:22:09.004029+0000","last_peered":"2024-09-21T10:22:09.004029+0000","last_clean":"2024-09-21T10:22:09.004029+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:22:09.004029+0000","last_undegraded":"2024-09-21T10:22:09.004029+0000","last_fullsized":"2024-09-21T10:22:09.004029+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1741,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432598+0000","last_change":"2024-09-21T10:06:09.039187+0000","last_active":"2024-09-21T10:22:09.432598+0000","last_peered":"2024-09-21T10:22:09.432598+0000","last_clean":"2024-09-21T10:22:09.432598+0000","last_became_active":"2024-09-21T10:06:09.038741+0000","last_became_peered":"2024-09-21T10:06:09.038741+0000","last_unstale":"2024-09-21T10:22:09.432598+0000","last_undegraded":"2024-09-21T10:22:09.432598+0000","last_fullsized":"2024-09-21T10:22:09.432598+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:40.920128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498646+0000","last_change":"2024-09-21T10:06:09.028808+0000","last_active":"2024-09-21T10:22:04.498646+0000","last_peered":"2024-09-21T10:22:04.498646+0000","last_clean":"2024-09-21T10:22:04.498646+0000","last_became_active":"2024-09-21T10:06:09.028492+0000","last_became_peered":"2024-09-21T10:06:09.028492+0000","last_unstale":"2024-09-21T10:22:04.498646+0000","last_undegraded":"2024-09-21T10:22:04.498646+0000","last_fullsized":"2024-09-21T10:22:04.498646+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:41.134185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435081+0000","last_change":"2024-09-21T10:08:06.539915+0000","last_active":"2024-09-21T10:22:09.435081+0000","last_peered":"2024-09-21T10:22:09.435081+0000","last_clean":"2024-09-21T10:22:09.435081+0000","last_became_active":"2024-09-21T10:08:06.539121+0000","last_became_peered":"2024-09-21T10:08:06.539121+0000","last_unstale":"2024-09-21T10:22:09.435081+0000","last_undegraded":"2024-09-21T10:22:09.435081+0000","last_fullsized":"2024-09-21T10:22:09.435081+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:23.549392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1715,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435365+0000","last_change":"2024-09-21T10:06:42.625569+0000","last_active":"2024-09-21T10:22:09.435365+0000","last_peered":"2024-09-21T10:22:09.435365+0000","last_clean":"2024-09-21T10:22:09.435365+0000","last_became_active":"2024-09-21T10:06:42.625259+0000","last_became_peered":"2024-09-21T10:06:42.625259+0000","last_unstale":"2024-09-21T10:22:09.435365+0000","last_undegraded":"2024-09-21T10:22:09.435365+0000","last_fullsized":"2024-09-21T10:22:09.435365+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:45.833707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1740,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435255+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:22:09.435255+0000","last_peered":"2024-09-21T10:22:09.435255+0000","last_clean":"2024-09-21T10:22:09.435255+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:22:09.435255+0000","last_undegraded":"2024-09-21T10:22:09.435255+0000","last_fullsized":"2024-09-21T10:22:09.435255+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329383+0000","last_change":"2024-09-21T10:08:55.102678+0000","last_active":"2024-09-21T10:22:04.329383+0000","last_peered":"2024-09-21T10:22:04.329383+0000","last_clean":"2024-09-21T10:22:04.329383+0000","last_became_active":"2024-09-21T10:08:55.102022+0000","last_became_peered":"2024-09-21T10:08:55.102022+0000","last_unstale":"2024-09-21T10:22:04.329383+0000","last_undegraded":"2024-09-21T10:22:04.329383+0000","last_fullsized":"2024-09-21T10:22:04.329383+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:03.178484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1737,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435170+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:22:09.435170+0000","last_peered":"2024-09-21T10:22:09.435170+0000","last_clean":"2024-09-21T10:22:09.435170+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:22:09.435170+0000","last_undegraded":"2024-09-21T10:22:09.435170+0000","last_fullsized":"2024-09-21T10:22:09.435170+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733623+0000","last_change":"2024-09-21T10:06:09.043111+0000","last_active":"2024-09-21T10:22:04.733623+0000","last_peered":"2024-09-21T10:22:04.733623+0000","last_clean":"2024-09-21T10:22:04.733623+0000","last_became_active":"2024-09-21T10:06:09.042813+0000","last_became_peered":"2024-09-21T10:06:09.042813+0000","last_unstale":"2024-09-21T10:22:04.733623+0000","last_undegraded":"2024-09-21T10:22:04.733623+0000","last_fullsized":"2024-09-21T10:22:04.733623+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:50.398531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328788+0000","last_change":"2024-09-21T10:06:42.622375+0000","last_active":"2024-09-21T10:22:04.328788+0000","last_peered":"2024-09-21T10:22:04.328788+0000","last_clean":"2024-09-21T10:22:04.328788+0000","last_became_active":"2024-09-21T10:06:42.622019+0000","last_became_peered":"2024-09-21T10:06:42.622019+0000","last_unstale":"2024-09-21T10:22:04.328788+0000","last_undegraded":"2024-09-21T10:22:04.328788+0000","last_fullsized":"2024-09-21T10:22:04.328788+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:22.917477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1728,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.437050+0000","last_change":"2024-09-21T10:06:42.627252+0000","last_active":"2024-09-21T10:22:09.437050+0000","last_peered":"2024-09-21T10:22:09.437050+0000","last_clean":"2024-09-21T10:22:09.437050+0000","last_became_active":"2024-09-21T10:06:42.626942+0000","last_became_peered":"2024-09-21T10:06:42.626942+0000","last_unstale":"2024-09-21T10:22:09.437050+0000","last_undegraded":"2024-09-21T10:22:09.437050+0000","last_fullsized":"2024-09-21T10:22:09.437050+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:21.844784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1727,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434134+0000","last_change":"2024-09-21T10:08:06.532136+0000","last_active":"2024-09-21T10:22:09.434134+0000","last_peered":"2024-09-21T10:22:09.434134+0000","last_clean":"2024-09-21T10:22:09.434134+0000","last_became_active":"2024-09-21T10:08:06.531195+0000","last_became_peered":"2024-09-21T10:08:06.531195+0000","last_unstale":"2024-09-21T10:22:09.434134+0000","last_undegraded":"2024-09-21T10:22:09.434134+0000","last_fullsized":"2024-09-21T10:22:09.434134+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:02.094677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1670,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.851840+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:22:08.851840+0000","last_peered":"2024-09-21T10:22:08.851840+0000","last_clean":"2024-09-21T10:22:08.851840+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:22:08.851840+0000","last_undegraded":"2024-09-21T10:22:08.851840+0000","last_fullsized":"2024-09-21T10:22:08.851840+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1762,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434579+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:22:09.434579+0000","last_peered":"2024-09-21T10:22:09.434579+0000","last_clean":"2024-09-21T10:22:09.434579+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:22:09.434579+0000","last_undegraded":"2024-09-21T10:22:09.434579+0000","last_fullsized":"2024-09-21T10:22:09.434579+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1763,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432709+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:22:09.432709+0000","last_peered":"2024-09-21T10:22:09.432709+0000","last_clean":"2024-09-21T10:22:09.432709+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:22:09.432709+0000","last_undegraded":"2024-09-21T10:22:09.432709+0000","last_fullsized":"2024-09-21T10:22:09.432709+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1149,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497998+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:22:04.497998+0000","last_peered":"2024-09-21T10:22:04.497998+0000","last_clean":"2024-09-21T10:22:04.497998+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:22:04.497998+0000","last_undegraded":"2024-09-21T10:22:04.497998+0000","last_fullsized":"2024-09-21T10:22:04.497998+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733686+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T10:22:04.733686+0000","last_peered":"2024-09-21T10:22:04.733686+0000","last_clean":"2024-09-21T10:22:04.733686+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T10:22:04.733686+0000","last_undegraded":"2024-09-21T10:22:04.733686+0000","last_fullsized":"2024-09-21T10:22:04.733686+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1624,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002576+0000","last_change":"2024-09-21T10:08:55.129563+0000","last_active":"2024-09-21T10:22:09.002576+0000","last_peered":"2024-09-21T10:22:09.002576+0000","last_clean":"2024-09-21T10:22:09.002576+0000","last_became_active":"2024-09-21T10:08:55.129268+0000","last_became_peered":"2024-09-21T10:08:55.129268+0000","last_unstale":"2024-09-21T10:22:09.002576+0000","last_undegraded":"2024-09-21T10:22:09.002576+0000","last_fullsized":"2024-09-21T10:22:09.002576+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:32.246505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328879+0000","last_change":"2024-09-21T10:08:06.540841+0000","last_active":"2024-09-21T10:22:04.328879+0000","last_peered":"2024-09-21T10:22:04.328879+0000","last_clean":"2024-09-21T10:22:04.328879+0000","last_became_active":"2024-09-21T10:08:06.539359+0000","last_became_peered":"2024-09-21T10:08:06.539359+0000","last_unstale":"2024-09-21T10:22:04.328879+0000","last_undegraded":"2024-09-21T10:22:04.328879+0000","last_fullsized":"2024-09-21T10:22:04.328879+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:53.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1236,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497868+0000","last_change":"2024-09-21T10:06:09.040194+0000","last_active":"2024-09-21T10:22:04.497868+0000","last_peered":"2024-09-21T10:22:04.497868+0000","last_clean":"2024-09-21T10:22:04.497868+0000","last_became_active":"2024-09-21T10:06:09.039512+0000","last_became_peered":"2024-09-21T10:06:09.039512+0000","last_unstale":"2024-09-21T10:22:04.497868+0000","last_undegraded":"2024-09-21T10:22:04.497868+0000","last_fullsized":"2024-09-21T10:22:04.497868+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:37.194722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435992+0000","last_change":"2024-09-21T10:08:55.112951+0000","last_active":"2024-09-21T10:22:09.435992+0000","last_peered":"2024-09-21T10:22:09.435992+0000","last_clean":"2024-09-21T10:22:09.435992+0000","last_became_active":"2024-09-21T10:08:55.112577+0000","last_became_peered":"2024-09-21T10:08:55.112577+0000","last_unstale":"2024-09-21T10:22:09.435992+0000","last_undegraded":"2024-09-21T10:22:09.435992+0000","last_fullsized":"2024-09-21T10:22:09.435992+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:06.470913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003480+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:22:09.003480+0000","last_peered":"2024-09-21T10:22:09.003480+0000","last_clean":"2024-09-21T10:22:09.003480+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:22:09.003480+0000","last_undegraded":"2024-09-21T10:22:09.003480+0000","last_fullsized":"2024-09-21T10:22:09.003480+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733924+0000","last_change":"2024-09-21T10:06:42.634333+0000","last_active":"2024-09-21T10:22:04.733924+0000","last_peered":"2024-09-21T10:22:04.733924+0000","last_clean":"2024-09-21T10:22:04.733924+0000","last_became_active":"2024-09-21T10:06:42.634037+0000","last_became_peered":"2024-09-21T10:06:42.634037+0000","last_unstale":"2024-09-21T10:22:04.733924+0000","last_undegraded":"2024-09-21T10:22:04.733924+0000","last_fullsized":"2024-09-21T10:22:04.733924+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:27.609590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735108+0000","last_change":"2024-09-21T10:08:06.537347+0000","last_active":"2024-09-21T10:22:04.735108+0000","last_peered":"2024-09-21T10:22:04.735108+0000","last_clean":"2024-09-21T10:22:04.735108+0000","last_became_active":"2024-09-21T10:08:06.537034+0000","last_became_peered":"2024-09-21T10:08:06.537034+0000","last_unstale":"2024-09-21T10:22:04.735108+0000","last_undegraded":"2024-09-21T10:22:04.735108+0000","last_fullsized":"2024-09-21T10:22:04.735108+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:26.325664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1129,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734203+0000","last_change":"2024-09-21T10:08:06.541436+0000","last_active":"2024-09-21T10:22:04.734203+0000","last_peered":"2024-09-21T10:22:04.734203+0000","last_clean":"2024-09-21T10:22:04.734203+0000","last_became_active":"2024-09-21T10:08:06.540616+0000","last_became_peered":"2024-09-21T10:08:06.540616+0000","last_unstale":"2024-09-21T10:22:04.734203+0000","last_undegraded":"2024-09-21T10:22:04.734203+0000","last_fullsized":"2024-09-21T10:22:04.734203+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:18.252764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733503+0000","last_change":"2024-09-21T10:06:42.631470+0000","last_active":"2024-09-21T10:22:04.733503+0000","last_peered":"2024-09-21T10:22:04.733503+0000","last_clean":"2024-09-21T10:22:04.733503+0000","last_became_active":"2024-09-21T10:06:42.631081+0000","last_became_peered":"2024-09-21T10:06:42.631081+0000","last_unstale":"2024-09-21T10:22:04.733503+0000","last_undegraded":"2024-09-21T10:22:04.733503+0000","last_fullsized":"2024-09-21T10:22:04.733503+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:16.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733727+0000","last_change":"2024-09-21T10:08:55.113216+0000","last_active":"2024-09-21T10:22:04.733727+0000","last_peered":"2024-09-21T10:22:04.733727+0000","last_clean":"2024-09-21T10:22:04.733727+0000","last_became_active":"2024-09-21T10:08:55.111808+0000","last_became_peered":"2024-09-21T10:08:55.111808+0000","last_unstale":"2024-09-21T10:22:04.733727+0000","last_undegraded":"2024-09-21T10:22:04.733727+0000","last_fullsized":"2024-09-21T10:22:04.733727+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:24:21.526156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1269,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435511+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:22:09.435511+0000","last_peered":"2024-09-21T10:22:09.435511+0000","last_clean":"2024-09-21T10:22:09.435511+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:22:09.435511+0000","last_undegraded":"2024-09-21T10:22:09.435511+0000","last_fullsized":"2024-09-21T10:22:09.435511+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1751,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433268+0000","last_change":"2024-09-21T10:06:42.626688+0000","last_active":"2024-09-21T10:22:09.433268+0000","last_peered":"2024-09-21T10:22:09.433268+0000","last_clean":"2024-09-21T10:22:09.433268+0000","last_became_active":"2024-09-21T10:06:42.626326+0000","last_became_peered":"2024-09-21T10:06:42.626326+0000","last_unstale":"2024-09-21T10:22:09.433268+0000","last_undegraded":"2024-09-21T10:22:09.433268+0000","last_fullsized":"2024-09-21T10:22:09.433268+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:00.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329061+0000","last_change":"2024-09-21T10:08:55.099610+0000","last_active":"2024-09-21T10:22:04.329061+0000","last_peered":"2024-09-21T10:22:04.329061+0000","last_clean":"2024-09-21T10:22:04.329061+0000","last_became_active":"2024-09-21T10:08:55.099313+0000","last_became_peered":"2024-09-21T10:08:55.099313+0000","last_unstale":"2024-09-21T10:22:04.329061+0000","last_undegraded":"2024-09-21T10:22:04.329061+0000","last_fullsized":"2024-09-21T10:22:04.329061+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:01.266798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733287+0000","last_change":"2024-09-21T10:06:42.631675+0000","last_active":"2024-09-21T10:22:04.733287+0000","last_peered":"2024-09-21T10:22:04.733287+0000","last_clean":"2024-09-21T10:22:04.733287+0000","last_became_active":"2024-09-21T10:06:42.631098+0000","last_became_peered":"2024-09-21T10:06:42.631098+0000","last_unstale":"2024-09-21T10:22:04.733287+0000","last_undegraded":"2024-09-21T10:22:04.733287+0000","last_fullsized":"2024-09-21T10:22:04.733287+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:46.855597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498604+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T10:22:04.498604+0000","last_peered":"2024-09-21T10:22:04.498604+0000","last_clean":"2024-09-21T10:22:04.498604+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T10:22:04.498604+0000","last_undegraded":"2024-09-21T10:22:04.498604+0000","last_fullsized":"2024-09-21T10:22:04.498604+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734962+0000","last_change":"2024-09-21T10:08:55.107939+0000","last_active":"2024-09-21T10:22:04.734962+0000","last_peered":"2024-09-21T10:22:04.734962+0000","last_clean":"2024-09-21T10:22:04.734962+0000","last_became_active":"2024-09-21T10:08:55.107640+0000","last_became_peered":"2024-09-21T10:08:55.107640+0000","last_unstale":"2024-09-21T10:22:04.734962+0000","last_undegraded":"2024-09-21T10:22:04.734962+0000","last_fullsized":"2024-09-21T10:22:04.734962+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:11.287664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1122,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498791+0000","last_change":"2024-09-21T10:06:42.632153+0000","last_active":"2024-09-21T10:22:04.498791+0000","last_peered":"2024-09-21T10:22:04.498791+0000","last_clean":"2024-09-21T10:22:04.498791+0000","last_became_active":"2024-09-21T10:06:42.631728+0000","last_became_peered":"2024-09-21T10:06:42.631728+0000","last_unstale":"2024-09-21T10:22:04.498791+0000","last_undegraded":"2024-09-21T10:22:04.498791+0000","last_fullsized":"2024-09-21T10:22:04.498791+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:53.412364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1101,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328844+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T10:22:04.328844+0000","last_peered":"2024-09-21T10:22:04.328844+0000","last_clean":"2024-09-21T10:22:04.328844+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T10:22:04.328844+0000","last_undegraded":"2024-09-21T10:22:04.328844+0000","last_fullsized":"2024-09-21T10:22:04.328844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.499006+0000","last_change":"2024-09-21T10:06:09.030444+0000","last_active":"2024-09-21T10:22:04.499006+0000","last_peered":"2024-09-21T10:22:04.499006+0000","last_clean":"2024-09-21T10:22:04.499006+0000","last_became_active":"2024-09-21T10:06:09.030054+0000","last_became_peered":"2024-09-21T10:06:09.030054+0000","last_unstale":"2024-09-21T10:22:04.499006+0000","last_undegraded":"2024-09-21T10:22:04.499006+0000","last_fullsized":"2024-09-21T10:22:04.499006+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:27.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1128,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735035+0000","last_change":"2024-09-21T10:06:09.032476+0000","last_active":"2024-09-21T10:22:04.735035+0000","last_peered":"2024-09-21T10:22:04.735035+0000","last_clean":"2024-09-21T10:22:04.735035+0000","last_became_active":"2024-09-21T10:06:09.032121+0000","last_became_peered":"2024-09-21T10:06:09.032121+0000","last_unstale":"2024-09-21T10:22:04.735035+0000","last_undegraded":"2024-09-21T10:22:04.735035+0000","last_fullsized":"2024-09-21T10:22:04.735035+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:41.862129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1631,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003838+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:22:09.003838+0000","last_peered":"2024-09-21T10:22:09.003838+0000","last_clean":"2024-09-21T10:22:09.003838+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:22:09.003838+0000","last_undegraded":"2024-09-21T10:22:09.003838+0000","last_fullsized":"2024-09-21T10:22:09.003838+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1130,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498375+0000","last_change":"2024-09-21T10:08:06.546911+0000","last_active":"2024-09-21T10:22:04.498375+0000","last_peered":"2024-09-21T10:22:04.498375+0000","last_clean":"2024-09-21T10:22:04.498375+0000","last_became_active":"2024-09-21T10:08:06.546417+0000","last_became_peered":"2024-09-21T10:08:06.546417+0000","last_unstale":"2024-09-21T10:22:04.498375+0000","last_undegraded":"2024-09-21T10:22:04.498375+0000","last_fullsized":"2024-09-21T10:22:04.498375+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:46.544701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1751,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434451+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:22:09.434451+0000","last_peered":"2024-09-21T10:22:09.434451+0000","last_clean":"2024-09-21T10:22:09.434451+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:22:09.434451+0000","last_undegraded":"2024-09-21T10:22:09.434451+0000","last_fullsized":"2024-09-21T10:22:09.434451+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432873+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:22:09.432873+0000","last_peered":"2024-09-21T10:22:09.432873+0000","last_clean":"2024-09-21T10:22:09.432873+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:22:09.432873+0000","last_undegraded":"2024-09-21T10:22:09.432873+0000","last_fullsized":"2024-09-21T10:22:09.432873+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329295+0000","last_change":"2024-09-21T10:08:55.102502+0000","last_active":"2024-09-21T10:22:04.329295+0000","last_peered":"2024-09-21T10:22:04.329295+0000","last_clean":"2024-09-21T10:22:04.329295+0000","last_became_active":"2024-09-21T10:08:55.101777+0000","last_became_peered":"2024-09-21T10:08:55.101777+0000","last_unstale":"2024-09-21T10:22:04.329295+0000","last_undegraded":"2024-09-21T10:22:04.329295+0000","last_fullsized":"2024-09-21T10:22:04.329295+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:51.293401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1152,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498158+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:22:04.498158+0000","last_peered":"2024-09-21T10:22:04.498158+0000","last_clean":"2024-09-21T10:22:04.498158+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:22:04.498158+0000","last_undegraded":"2024-09-21T10:22:04.498158+0000","last_fullsized":"2024-09-21T10:22:04.498158+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003671+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:22:09.003671+0000","last_peered":"2024-09-21T10:22:09.003671+0000","last_clean":"2024-09-21T10:22:09.003671+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:22:09.003671+0000","last_undegraded":"2024-09-21T10:22:09.003671+0000","last_fullsized":"2024-09-21T10:22:09.003671+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":124,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397842,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314188,"kb_used_data":12672,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93430964,"statfs":{"total":95995035648,"available":95673307136,"internally_reserved":0,"allocated":12976128,"data_stored":8569725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594071,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322504,"kb_used_data":12668,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93422648,"statfs":{"total":95995035648,"available":95664791552,"internally_reserved":0,"allocated":12972032,"data_stored":8569696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790300,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313596,"kb_used_data":12080,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93431556,"statfs":{"total":95995035648,"available":95673913344,"internally_reserved":0,"allocated":12369920,"data_stored":7975146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986528,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321920,"kb_used_data":12088,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93423232,"statfs":{"total":95995035648,"available":95665389568,"internally_reserved":0,"allocated":12378112,"data_stored":7979343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644460,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58100,"kb_used_data":12072,"kb_used_omap":67,"kb_used_meta":45948,"kb_avail":93687052,"statfs":{"total":95995035648,"available":95935541248,"internally_reserved":0,"allocated":12361728,"data_stored":7974597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":47051613},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527478,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66232,"kb_used_data":12076,"kb_used_omap":75,"kb_used_meta":54068,"kb_avail":93678920,"statfs":{"total":95995035648,"available":95927214080,"internally_reserved":0,"allocated":12365824,"data_stored":7975119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":55365786},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274630,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88188,"kb_used_data":12080,"kb_used_omap":51,"kb_used_meta":76044,"kb_avail":93656964,"statfs":{"total":95995035648,"available":95904731136,"internally_reserved":0,"allocated":12369920,"data_stored":7974658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":77869397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62588,"kb_used_data":12656,"kb_used_omap":66,"kb_used_meta":49853,"kb_avail":93682564,"statfs":{"total":95995035648,"available":95930945536,"internally_reserved":0,"allocated":12959744,"data_stored":8565483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":51049796},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:23:05.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:23:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:05 smithi119 ceph-mon[105288]: pgmap v1557: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:05 smithi119 ceph-mon[105288]: from='client.35901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:05 smithi119 ceph-mon[101432]: pgmap v1557: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:05 smithi119 ceph-mon[101432]: from='client.35901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:05 smithi145 ceph-mon[89359]: pgmap v1557: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:05 smithi145 ceph-mon[89359]: from='client.35901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:07.892 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:23:07.893 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:23:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:07 smithi119 ceph-mon[101432]: pgmap v1558: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:07 smithi119 ceph-mon[105288]: pgmap v1558: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:07 smithi145 ceph-mon[89359]: pgmap v1558: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:08.478 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1558,"stamp":"2024-09-21T10:23:06.795405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":212,"num_write_kb":1659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590492,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":219,"acting":219,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":219,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1547316,"kb_used_data":98392,"kb_used_omap":268,"kb_used_meta":1448563,"kb_avail":748413900,"statfs":{"total":767960285184,"available":766375833600,"internally_reserved":0,"allocated":100753408,"data_stored":65583767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274448,"internal_metadata":1483329520},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002832"},"pg_stats":[{"pgid":"3.1f","version":"64'4","reported_seq":1758,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435395+0000","last_change":"2024-09-21T10:08:06.538833+0000","last_active":"2024-09-21T10:22:09.435395+0000","last_peered":"2024-09-21T10:22:09.435395+0000","last_clean":"2024-09-21T10:22:09.435395+0000","last_became_active":"2024-09-21T10:08:06.538526+0000","last_became_peered":"2024-09-21T10:08:06.538526+0000","last_unstale":"2024-09-21T10:22:09.435395+0000","last_undegraded":"2024-09-21T10:22:09.435395+0000","last_fullsized":"2024-09-21T10:22:09.435395+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:59.194474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733525+0000","last_change":"2024-09-21T10:06:42.626454+0000","last_active":"2024-09-21T10:22:04.733525+0000","last_peered":"2024-09-21T10:22:04.733525+0000","last_clean":"2024-09-21T10:22:04.733525+0000","last_became_active":"2024-09-21T10:06:42.626116+0000","last_became_peered":"2024-09-21T10:06:42.626116+0000","last_unstale":"2024-09-21T10:22:04.733525+0000","last_undegraded":"2024-09-21T10:22:04.733525+0000","last_fullsized":"2024-09-21T10:22:04.733525+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:01.968111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1626,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002789+0000","last_change":"2024-09-21T10:08:55.142962+0000","last_active":"2024-09-21T10:22:09.002789+0000","last_peered":"2024-09-21T10:22:09.002789+0000","last_clean":"2024-09-21T10:22:09.002789+0000","last_became_active":"2024-09-21T10:08:55.139681+0000","last_became_peered":"2024-09-21T10:08:55.139681+0000","last_unstale":"2024-09-21T10:22:09.002789+0000","last_undegraded":"2024-09-21T10:22:09.002789+0000","last_fullsized":"2024-09-21T10:22:09.002789+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:52.577104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1656,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002658+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:22:09.002658+0000","last_peered":"2024-09-21T10:22:09.002658+0000","last_clean":"2024-09-21T10:22:09.002658+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:22:09.002658+0000","last_undegraded":"2024-09-21T10:22:09.002658+0000","last_fullsized":"2024-09-21T10:22:09.002658+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329357+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T10:22:04.329357+0000","last_peered":"2024-09-21T10:22:04.329357+0000","last_clean":"2024-09-21T10:22:04.329357+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T10:22:04.329357+0000","last_undegraded":"2024-09-21T10:22:04.329357+0000","last_fullsized":"2024-09-21T10:22:04.329357+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734008+0000","last_change":"2024-09-21T10:08:55.108381+0000","last_active":"2024-09-21T10:22:04.734008+0000","last_peered":"2024-09-21T10:22:04.734008+0000","last_clean":"2024-09-21T10:22:04.734008+0000","last_became_active":"2024-09-21T10:08:55.108074+0000","last_became_peered":"2024-09-21T10:08:55.108074+0000","last_unstale":"2024-09-21T10:22:04.734008+0000","last_undegraded":"2024-09-21T10:22:04.734008+0000","last_fullsized":"2024-09-21T10:22:04.734008+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.846660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329020+0000","last_change":"2024-09-21T10:06:42.622644+0000","last_active":"2024-09-21T10:22:04.329020+0000","last_peered":"2024-09-21T10:22:04.329020+0000","last_clean":"2024-09-21T10:22:04.329020+0000","last_became_active":"2024-09-21T10:06:42.621612+0000","last_became_peered":"2024-09-21T10:06:42.621612+0000","last_unstale":"2024-09-21T10:22:04.329020+0000","last_undegraded":"2024-09-21T10:22:04.329020+0000","last_fullsized":"2024-09-21T10:22:04.329020+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:21.301285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733451+0000","last_change":"2024-09-21T10:08:55.109486+0000","last_active":"2024-09-21T10:22:04.733451+0000","last_peered":"2024-09-21T10:22:04.733451+0000","last_clean":"2024-09-21T10:22:04.733451+0000","last_became_active":"2024-09-21T10:08:55.109050+0000","last_became_peered":"2024-09-21T10:08:55.109050+0000","last_unstale":"2024-09-21T10:22:04.733451+0000","last_undegraded":"2024-09-21T10:22:04.733451+0000","last_fullsized":"2024-09-21T10:22:04.733451+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:13.242342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433323+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:22:09.433323+0000","last_peered":"2024-09-21T10:22:09.433323+0000","last_clean":"2024-09-21T10:22:09.433323+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:22:09.433323+0000","last_undegraded":"2024-09-21T10:22:09.433323+0000","last_fullsized":"2024-09-21T10:22:09.433323+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1655,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.853024+0000","last_change":"2024-09-21T10:08:55.117054+0000","last_active":"2024-09-21T10:22:08.853024+0000","last_peered":"2024-09-21T10:22:08.853024+0000","last_clean":"2024-09-21T10:22:08.853024+0000","last_became_active":"2024-09-21T10:08:55.116767+0000","last_became_peered":"2024-09-21T10:08:55.116767+0000","last_unstale":"2024-09-21T10:22:08.853024+0000","last_undegraded":"2024-09-21T10:22:08.853024+0000","last_fullsized":"2024-09-21T10:22:08.853024+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:13.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1138,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734095+0000","last_change":"2024-09-21T10:08:06.539436+0000","last_active":"2024-09-21T10:22:04.734095+0000","last_peered":"2024-09-21T10:22:04.734095+0000","last_clean":"2024-09-21T10:22:04.734095+0000","last_became_active":"2024-09-21T10:08:06.538856+0000","last_became_peered":"2024-09-21T10:08:06.538856+0000","last_unstale":"2024-09-21T10:22:04.734095+0000","last_undegraded":"2024-09-21T10:22:04.734095+0000","last_fullsized":"2024-09-21T10:22:04.734095+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:43.502357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1621,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002896+0000","last_change":"2024-09-21T10:08:55.116807+0000","last_active":"2024-09-21T10:22:09.002896+0000","last_peered":"2024-09-21T10:22:09.002896+0000","last_clean":"2024-09-21T10:22:09.002896+0000","last_became_active":"2024-09-21T10:08:55.115948+0000","last_became_peered":"2024-09-21T10:08:55.115948+0000","last_unstale":"2024-09-21T10:22:09.002896+0000","last_undegraded":"2024-09-21T10:22:09.002896+0000","last_fullsized":"2024-09-21T10:22:09.002896+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:21.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433192+0000","last_change":"2024-09-21T10:08:55.109470+0000","last_active":"2024-09-21T10:22:09.433192+0000","last_peered":"2024-09-21T10:22:09.433192+0000","last_clean":"2024-09-21T10:22:09.433192+0000","last_became_active":"2024-09-21T10:08:55.106885+0000","last_became_peered":"2024-09-21T10:08:55.106885+0000","last_unstale":"2024-09-21T10:22:09.433192+0000","last_undegraded":"2024-09-21T10:22:09.433192+0000","last_fullsized":"2024-09-21T10:22:09.433192+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:51.866025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1752,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434498+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:22:09.434498+0000","last_peered":"2024-09-21T10:22:09.434498+0000","last_clean":"2024-09-21T10:22:09.434498+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:22:09.434498+0000","last_undegraded":"2024-09-21T10:22:09.434498+0000","last_fullsized":"2024-09-21T10:22:09.434498+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1657,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003291+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:22:09.003291+0000","last_peered":"2024-09-21T10:22:09.003291+0000","last_clean":"2024-09-21T10:22:09.003291+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:22:09.003291+0000","last_undegraded":"2024-09-21T10:22:09.003291+0000","last_fullsized":"2024-09-21T10:22:09.003291+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733941+0000","last_change":"2024-09-21T10:06:42.631648+0000","last_active":"2024-09-21T10:22:04.733941+0000","last_peered":"2024-09-21T10:22:04.733941+0000","last_clean":"2024-09-21T10:22:04.733941+0000","last_became_active":"2024-09-21T10:06:42.631301+0000","last_became_peered":"2024-09-21T10:06:42.631301+0000","last_unstale":"2024-09-21T10:22:04.733941+0000","last_undegraded":"2024-09-21T10:22:04.733941+0000","last_fullsized":"2024-09-21T10:22:04.733941+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:52.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003631+0000","last_change":"2024-09-21T10:08:55.134315+0000","last_active":"2024-09-21T10:22:09.003631+0000","last_peered":"2024-09-21T10:22:09.003631+0000","last_clean":"2024-09-21T10:22:09.003631+0000","last_became_active":"2024-09-21T10:08:55.132999+0000","last_became_peered":"2024-09-21T10:08:55.132999+0000","last_unstale":"2024-09-21T10:22:09.003631+0000","last_undegraded":"2024-09-21T10:22:09.003631+0000","last_fullsized":"2024-09-21T10:22:09.003631+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:24.870318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435488+0000","last_change":"2024-09-21T10:08:55.105723+0000","last_active":"2024-09-21T10:22:09.435488+0000","last_peered":"2024-09-21T10:22:09.435488+0000","last_clean":"2024-09-21T10:22:09.435488+0000","last_became_active":"2024-09-21T10:08:55.105283+0000","last_became_peered":"2024-09-21T10:08:55.105283+0000","last_unstale":"2024-09-21T10:22:09.435488+0000","last_undegraded":"2024-09-21T10:22:09.435488+0000","last_fullsized":"2024-09-21T10:22:09.435488+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:25.633107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1623,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003595+0000","last_change":"2024-09-21T10:08:55.131455+0000","last_active":"2024-09-21T10:22:09.003595+0000","last_peered":"2024-09-21T10:22:09.003595+0000","last_clean":"2024-09-21T10:22:09.003595+0000","last_became_active":"2024-09-21T10:08:55.129639+0000","last_became_peered":"2024-09-21T10:08:55.129639+0000","last_unstale":"2024-09-21T10:22:09.003595+0000","last_undegraded":"2024-09-21T10:22:09.003595+0000","last_fullsized":"2024-09-21T10:22:09.003595+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:50.934309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329289+0000","last_change":"2024-09-21T10:06:42.623974+0000","last_active":"2024-09-21T10:22:04.329289+0000","last_peered":"2024-09-21T10:22:04.329289+0000","last_clean":"2024-09-21T10:22:04.329289+0000","last_became_active":"2024-09-21T10:06:42.623598+0000","last_became_peered":"2024-09-21T10:06:42.623598+0000","last_unstale":"2024-09-21T10:22:04.329289+0000","last_undegraded":"2024-09-21T10:22:04.329289+0000","last_fullsized":"2024-09-21T10:22:04.329289+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.261148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1270,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.436983+0000","last_change":"2024-09-21T10:08:55.106181+0000","last_active":"2024-09-21T10:22:09.436983+0000","last_peered":"2024-09-21T10:22:09.436983+0000","last_clean":"2024-09-21T10:22:09.436983+0000","last_became_active":"2024-09-21T10:08:55.105882+0000","last_became_peered":"2024-09-21T10:08:55.105882+0000","last_unstale":"2024-09-21T10:22:09.436983+0000","last_undegraded":"2024-09-21T10:22:09.436983+0000","last_fullsized":"2024-09-21T10:22:09.436983+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:30.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435278+0000","last_change":"2024-09-21T10:08:06.538126+0000","last_active":"2024-09-21T10:22:09.435278+0000","last_peered":"2024-09-21T10:22:09.435278+0000","last_clean":"2024-09-21T10:22:09.435278+0000","last_became_active":"2024-09-21T10:08:06.537779+0000","last_became_peered":"2024-09-21T10:08:06.537779+0000","last_unstale":"2024-09-21T10:22:09.435278+0000","last_undegraded":"2024-09-21T10:22:09.435278+0000","last_fullsized":"2024-09-21T10:22:09.435278+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:34.731581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1125,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733857+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T10:22:04.733857+0000","last_peered":"2024-09-21T10:22:04.733857+0000","last_clean":"2024-09-21T10:22:04.733857+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T10:22:04.733857+0000","last_undegraded":"2024-09-21T10:22:04.733857+0000","last_fullsized":"2024-09-21T10:22:04.733857+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733533+0000","last_change":"2024-09-21T10:06:42.633523+0000","last_active":"2024-09-21T10:22:04.733533+0000","last_peered":"2024-09-21T10:22:04.733533+0000","last_clean":"2024-09-21T10:22:04.733533+0000","last_became_active":"2024-09-21T10:06:42.633019+0000","last_became_peered":"2024-09-21T10:06:42.633019+0000","last_unstale":"2024-09-21T10:22:04.733533+0000","last_undegraded":"2024-09-21T10:22:04.733533+0000","last_fullsized":"2024-09-21T10:22:04.733533+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:54.272637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735047+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T10:22:04.735047+0000","last_peered":"2024-09-21T10:22:04.735047+0000","last_clean":"2024-09-21T10:22:04.735047+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T10:22:04.735047+0000","last_undegraded":"2024-09-21T10:22:04.735047+0000","last_fullsized":"2024-09-21T10:22:04.735047+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734189+0000","last_change":"2024-09-21T10:08:06.551874+0000","last_active":"2024-09-21T10:22:04.734189+0000","last_peered":"2024-09-21T10:22:04.734189+0000","last_clean":"2024-09-21T10:22:04.734189+0000","last_became_active":"2024-09-21T10:08:06.551572+0000","last_became_peered":"2024-09-21T10:08:06.551572+0000","last_unstale":"2024-09-21T10:22:04.734189+0000","last_undegraded":"2024-09-21T10:22:04.734189+0000","last_fullsized":"2024-09-21T10:22:04.734189+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:56.324479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1754,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434303+0000","last_change":"2024-09-21T10:06:42.626709+0000","last_active":"2024-09-21T10:22:09.434303+0000","last_peered":"2024-09-21T10:22:09.434303+0000","last_clean":"2024-09-21T10:22:09.434303+0000","last_became_active":"2024-09-21T10:06:42.626389+0000","last_became_peered":"2024-09-21T10:06:42.626389+0000","last_unstale":"2024-09-21T10:22:09.434303+0000","last_undegraded":"2024-09-21T10:22:09.434303+0000","last_fullsized":"2024-09-21T10:22:09.434303+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:39.634788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003429+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:22:09.003429+0000","last_peered":"2024-09-21T10:22:09.003429+0000","last_clean":"2024-09-21T10:22:09.003429+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:22:09.003429+0000","last_undegraded":"2024-09-21T10:22:09.003429+0000","last_fullsized":"2024-09-21T10:22:09.003429+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1724,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.438211+0000","last_change":"2024-09-21T10:06:42.630257+0000","last_active":"2024-09-21T10:22:09.438211+0000","last_peered":"2024-09-21T10:22:09.438211+0000","last_clean":"2024-09-21T10:22:09.438211+0000","last_became_active":"2024-09-21T10:06:42.628412+0000","last_became_peered":"2024-09-21T10:06:42.628412+0000","last_unstale":"2024-09-21T10:22:09.438211+0000","last_undegraded":"2024-09-21T10:22:09.438211+0000","last_fullsized":"2024-09-21T10:22:09.438211+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:30.166565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1632,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.004029+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:22:09.004029+0000","last_peered":"2024-09-21T10:22:09.004029+0000","last_clean":"2024-09-21T10:22:09.004029+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:22:09.004029+0000","last_undegraded":"2024-09-21T10:22:09.004029+0000","last_fullsized":"2024-09-21T10:22:09.004029+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1741,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432598+0000","last_change":"2024-09-21T10:06:09.039187+0000","last_active":"2024-09-21T10:22:09.432598+0000","last_peered":"2024-09-21T10:22:09.432598+0000","last_clean":"2024-09-21T10:22:09.432598+0000","last_became_active":"2024-09-21T10:06:09.038741+0000","last_became_peered":"2024-09-21T10:06:09.038741+0000","last_unstale":"2024-09-21T10:22:09.432598+0000","last_undegraded":"2024-09-21T10:22:09.432598+0000","last_fullsized":"2024-09-21T10:22:09.432598+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:40.920128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498646+0000","last_change":"2024-09-21T10:06:09.028808+0000","last_active":"2024-09-21T10:22:04.498646+0000","last_peered":"2024-09-21T10:22:04.498646+0000","last_clean":"2024-09-21T10:22:04.498646+0000","last_became_active":"2024-09-21T10:06:09.028492+0000","last_became_peered":"2024-09-21T10:06:09.028492+0000","last_unstale":"2024-09-21T10:22:04.498646+0000","last_undegraded":"2024-09-21T10:22:04.498646+0000","last_fullsized":"2024-09-21T10:22:04.498646+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:41.134185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435081+0000","last_change":"2024-09-21T10:08:06.539915+0000","last_active":"2024-09-21T10:22:09.435081+0000","last_peered":"2024-09-21T10:22:09.435081+0000","last_clean":"2024-09-21T10:22:09.435081+0000","last_became_active":"2024-09-21T10:08:06.539121+0000","last_became_peered":"2024-09-21T10:08:06.539121+0000","last_unstale":"2024-09-21T10:22:09.435081+0000","last_undegraded":"2024-09-21T10:22:09.435081+0000","last_fullsized":"2024-09-21T10:22:09.435081+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:23.549392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1715,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435365+0000","last_change":"2024-09-21T10:06:42.625569+0000","last_active":"2024-09-21T10:22:09.435365+0000","last_peered":"2024-09-21T10:22:09.435365+0000","last_clean":"2024-09-21T10:22:09.435365+0000","last_became_active":"2024-09-21T10:06:42.625259+0000","last_became_peered":"2024-09-21T10:06:42.625259+0000","last_unstale":"2024-09-21T10:22:09.435365+0000","last_undegraded":"2024-09-21T10:22:09.435365+0000","last_fullsized":"2024-09-21T10:22:09.435365+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:45.833707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1740,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435255+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:22:09.435255+0000","last_peered":"2024-09-21T10:22:09.435255+0000","last_clean":"2024-09-21T10:22:09.435255+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:22:09.435255+0000","last_undegraded":"2024-09-21T10:22:09.435255+0000","last_fullsized":"2024-09-21T10:22:09.435255+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329383+0000","last_change":"2024-09-21T10:08:55.102678+0000","last_active":"2024-09-21T10:22:04.329383+0000","last_peered":"2024-09-21T10:22:04.329383+0000","last_clean":"2024-09-21T10:22:04.329383+0000","last_became_active":"2024-09-21T10:08:55.102022+0000","last_became_peered":"2024-09-21T10:08:55.102022+0000","last_unstale":"2024-09-21T10:22:04.329383+0000","last_undegraded":"2024-09-21T10:22:04.329383+0000","last_fullsized":"2024-09-21T10:22:04.329383+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:03.178484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1737,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435170+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:22:09.435170+0000","last_peered":"2024-09-21T10:22:09.435170+0000","last_clean":"2024-09-21T10:22:09.435170+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:22:09.435170+0000","last_undegraded":"2024-09-21T10:22:09.435170+0000","last_fullsized":"2024-09-21T10:22:09.435170+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733623+0000","last_change":"2024-09-21T10:06:09.043111+0000","last_active":"2024-09-21T10:22:04.733623+0000","last_peered":"2024-09-21T10:22:04.733623+0000","last_clean":"2024-09-21T10:22:04.733623+0000","last_became_active":"2024-09-21T10:06:09.042813+0000","last_became_peered":"2024-09-21T10:06:09.042813+0000","last_unstale":"2024-09-21T10:22:04.733623+0000","last_undegraded":"2024-09-21T10:22:04.733623+0000","last_fullsized":"2024-09-21T10:22:04.733623+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:50.398531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":1133,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328788+0000","last_change":"2024-09-21T10:06:42.622375+0000","last_active":"2024-09-21T10:22:04.328788+0000","last_peered":"2024-09-21T10:22:04.328788+0000","last_clean":"2024-09-21T10:22:04.328788+0000","last_became_active":"2024-09-21T10:06:42.622019+0000","last_became_peered":"2024-09-21T10:06:42.622019+0000","last_unstale":"2024-09-21T10:22:04.328788+0000","last_undegraded":"2024-09-21T10:22:04.328788+0000","last_fullsized":"2024-09-21T10:22:04.328788+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:22.917477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1728,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.437050+0000","last_change":"2024-09-21T10:06:42.627252+0000","last_active":"2024-09-21T10:22:09.437050+0000","last_peered":"2024-09-21T10:22:09.437050+0000","last_clean":"2024-09-21T10:22:09.437050+0000","last_became_active":"2024-09-21T10:06:42.626942+0000","last_became_peered":"2024-09-21T10:06:42.626942+0000","last_unstale":"2024-09-21T10:22:09.437050+0000","last_undegraded":"2024-09-21T10:22:09.437050+0000","last_fullsized":"2024-09-21T10:22:09.437050+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:21.844784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1727,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434134+0000","last_change":"2024-09-21T10:08:06.532136+0000","last_active":"2024-09-21T10:22:09.434134+0000","last_peered":"2024-09-21T10:22:09.434134+0000","last_clean":"2024-09-21T10:22:09.434134+0000","last_became_active":"2024-09-21T10:08:06.531195+0000","last_became_peered":"2024-09-21T10:08:06.531195+0000","last_unstale":"2024-09-21T10:22:09.434134+0000","last_undegraded":"2024-09-21T10:22:09.434134+0000","last_fullsized":"2024-09-21T10:22:09.434134+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:02.094677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1670,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:08.851840+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:22:08.851840+0000","last_peered":"2024-09-21T10:22:08.851840+0000","last_clean":"2024-09-21T10:22:08.851840+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:22:08.851840+0000","last_undegraded":"2024-09-21T10:22:08.851840+0000","last_fullsized":"2024-09-21T10:22:08.851840+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1762,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434579+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:22:09.434579+0000","last_peered":"2024-09-21T10:22:09.434579+0000","last_clean":"2024-09-21T10:22:09.434579+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:22:09.434579+0000","last_undegraded":"2024-09-21T10:22:09.434579+0000","last_fullsized":"2024-09-21T10:22:09.434579+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1763,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432709+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:22:09.432709+0000","last_peered":"2024-09-21T10:22:09.432709+0000","last_clean":"2024-09-21T10:22:09.432709+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:22:09.432709+0000","last_undegraded":"2024-09-21T10:22:09.432709+0000","last_fullsized":"2024-09-21T10:22:09.432709+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1149,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497998+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:22:04.497998+0000","last_peered":"2024-09-21T10:22:04.497998+0000","last_clean":"2024-09-21T10:22:04.497998+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:22:04.497998+0000","last_undegraded":"2024-09-21T10:22:04.497998+0000","last_fullsized":"2024-09-21T10:22:04.497998+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1102,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733686+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T10:22:04.733686+0000","last_peered":"2024-09-21T10:22:04.733686+0000","last_clean":"2024-09-21T10:22:04.733686+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T10:22:04.733686+0000","last_undegraded":"2024-09-21T10:22:04.733686+0000","last_fullsized":"2024-09-21T10:22:04.733686+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1624,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.002576+0000","last_change":"2024-09-21T10:08:55.129563+0000","last_active":"2024-09-21T10:22:09.002576+0000","last_peered":"2024-09-21T10:22:09.002576+0000","last_clean":"2024-09-21T10:22:09.002576+0000","last_became_active":"2024-09-21T10:08:55.129268+0000","last_became_peered":"2024-09-21T10:08:55.129268+0000","last_unstale":"2024-09-21T10:22:09.002576+0000","last_undegraded":"2024-09-21T10:22:09.002576+0000","last_fullsized":"2024-09-21T10:22:09.002576+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:32.246505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":1132,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328879+0000","last_change":"2024-09-21T10:08:06.540841+0000","last_active":"2024-09-21T10:22:04.328879+0000","last_peered":"2024-09-21T10:22:04.328879+0000","last_clean":"2024-09-21T10:22:04.328879+0000","last_became_active":"2024-09-21T10:08:06.539359+0000","last_became_peered":"2024-09-21T10:08:06.539359+0000","last_unstale":"2024-09-21T10:22:04.328879+0000","last_undegraded":"2024-09-21T10:22:04.328879+0000","last_fullsized":"2024-09-21T10:22:04.328879+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:53.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1236,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.497868+0000","last_change":"2024-09-21T10:06:09.040194+0000","last_active":"2024-09-21T10:22:04.497868+0000","last_peered":"2024-09-21T10:22:04.497868+0000","last_clean":"2024-09-21T10:22:04.497868+0000","last_became_active":"2024-09-21T10:06:09.039512+0000","last_became_peered":"2024-09-21T10:06:09.039512+0000","last_unstale":"2024-09-21T10:22:04.497868+0000","last_undegraded":"2024-09-21T10:22:04.497868+0000","last_fullsized":"2024-09-21T10:22:04.497868+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:37.194722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1726,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435992+0000","last_change":"2024-09-21T10:08:55.112951+0000","last_active":"2024-09-21T10:22:09.435992+0000","last_peered":"2024-09-21T10:22:09.435992+0000","last_clean":"2024-09-21T10:22:09.435992+0000","last_became_active":"2024-09-21T10:08:55.112577+0000","last_became_peered":"2024-09-21T10:08:55.112577+0000","last_unstale":"2024-09-21T10:22:09.435992+0000","last_undegraded":"2024-09-21T10:22:09.435992+0000","last_fullsized":"2024-09-21T10:22:09.435992+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:06.470913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1641,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003480+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:22:09.003480+0000","last_peered":"2024-09-21T10:22:09.003480+0000","last_clean":"2024-09-21T10:22:09.003480+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:22:09.003480+0000","last_undegraded":"2024-09-21T10:22:09.003480+0000","last_fullsized":"2024-09-21T10:22:09.003480+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733924+0000","last_change":"2024-09-21T10:06:42.634333+0000","last_active":"2024-09-21T10:22:04.733924+0000","last_peered":"2024-09-21T10:22:04.733924+0000","last_clean":"2024-09-21T10:22:04.733924+0000","last_became_active":"2024-09-21T10:06:42.634037+0000","last_became_peered":"2024-09-21T10:06:42.634037+0000","last_unstale":"2024-09-21T10:22:04.733924+0000","last_undegraded":"2024-09-21T10:22:04.733924+0000","last_fullsized":"2024-09-21T10:22:04.733924+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:27.609590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735108+0000","last_change":"2024-09-21T10:08:06.537347+0000","last_active":"2024-09-21T10:22:04.735108+0000","last_peered":"2024-09-21T10:22:04.735108+0000","last_clean":"2024-09-21T10:22:04.735108+0000","last_became_active":"2024-09-21T10:08:06.537034+0000","last_became_peered":"2024-09-21T10:08:06.537034+0000","last_unstale":"2024-09-21T10:22:04.735108+0000","last_undegraded":"2024-09-21T10:22:04.735108+0000","last_fullsized":"2024-09-21T10:22:04.735108+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:26.325664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1129,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734203+0000","last_change":"2024-09-21T10:08:06.541436+0000","last_active":"2024-09-21T10:22:04.734203+0000","last_peered":"2024-09-21T10:22:04.734203+0000","last_clean":"2024-09-21T10:22:04.734203+0000","last_became_active":"2024-09-21T10:08:06.540616+0000","last_became_peered":"2024-09-21T10:08:06.540616+0000","last_unstale":"2024-09-21T10:22:04.734203+0000","last_undegraded":"2024-09-21T10:22:04.734203+0000","last_fullsized":"2024-09-21T10:22:04.734203+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:18.252764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1120,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733503+0000","last_change":"2024-09-21T10:06:42.631470+0000","last_active":"2024-09-21T10:22:04.733503+0000","last_peered":"2024-09-21T10:22:04.733503+0000","last_clean":"2024-09-21T10:22:04.733503+0000","last_became_active":"2024-09-21T10:06:42.631081+0000","last_became_peered":"2024-09-21T10:06:42.631081+0000","last_unstale":"2024-09-21T10:22:04.733503+0000","last_undegraded":"2024-09-21T10:22:04.733503+0000","last_fullsized":"2024-09-21T10:22:04.733503+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:16.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733727+0000","last_change":"2024-09-21T10:08:55.113216+0000","last_active":"2024-09-21T10:22:04.733727+0000","last_peered":"2024-09-21T10:22:04.733727+0000","last_clean":"2024-09-21T10:22:04.733727+0000","last_became_active":"2024-09-21T10:08:55.111808+0000","last_became_peered":"2024-09-21T10:08:55.111808+0000","last_unstale":"2024-09-21T10:22:04.733727+0000","last_undegraded":"2024-09-21T10:22:04.733727+0000","last_fullsized":"2024-09-21T10:22:04.733727+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:24:21.526156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1269,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.435511+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:22:09.435511+0000","last_peered":"2024-09-21T10:22:09.435511+0000","last_clean":"2024-09-21T10:22:09.435511+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:22:09.435511+0000","last_undegraded":"2024-09-21T10:22:09.435511+0000","last_fullsized":"2024-09-21T10:22:09.435511+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1751,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.433268+0000","last_change":"2024-09-21T10:06:42.626688+0000","last_active":"2024-09-21T10:22:09.433268+0000","last_peered":"2024-09-21T10:22:09.433268+0000","last_clean":"2024-09-21T10:22:09.433268+0000","last_became_active":"2024-09-21T10:06:42.626326+0000","last_became_peered":"2024-09-21T10:06:42.626326+0000","last_unstale":"2024-09-21T10:22:09.433268+0000","last_undegraded":"2024-09-21T10:22:09.433268+0000","last_fullsized":"2024-09-21T10:22:09.433268+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:00.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329061+0000","last_change":"2024-09-21T10:08:55.099610+0000","last_active":"2024-09-21T10:22:04.329061+0000","last_peered":"2024-09-21T10:22:04.329061+0000","last_clean":"2024-09-21T10:22:04.329061+0000","last_became_active":"2024-09-21T10:08:55.099313+0000","last_became_peered":"2024-09-21T10:08:55.099313+0000","last_unstale":"2024-09-21T10:22:04.329061+0000","last_undegraded":"2024-09-21T10:22:04.329061+0000","last_fullsized":"2024-09-21T10:22:04.329061+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:01.266798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1119,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.733287+0000","last_change":"2024-09-21T10:06:42.631675+0000","last_active":"2024-09-21T10:22:04.733287+0000","last_peered":"2024-09-21T10:22:04.733287+0000","last_clean":"2024-09-21T10:22:04.733287+0000","last_became_active":"2024-09-21T10:06:42.631098+0000","last_became_peered":"2024-09-21T10:06:42.631098+0000","last_unstale":"2024-09-21T10:22:04.733287+0000","last_undegraded":"2024-09-21T10:22:04.733287+0000","last_fullsized":"2024-09-21T10:22:04.733287+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:46.855597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1104,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498604+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T10:22:04.498604+0000","last_peered":"2024-09-21T10:22:04.498604+0000","last_clean":"2024-09-21T10:22:04.498604+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T10:22:04.498604+0000","last_undegraded":"2024-09-21T10:22:04.498604+0000","last_fullsized":"2024-09-21T10:22:04.498604+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1136,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.734962+0000","last_change":"2024-09-21T10:08:55.107939+0000","last_active":"2024-09-21T10:22:04.734962+0000","last_peered":"2024-09-21T10:22:04.734962+0000","last_clean":"2024-09-21T10:22:04.734962+0000","last_became_active":"2024-09-21T10:08:55.107640+0000","last_became_peered":"2024-09-21T10:08:55.107640+0000","last_unstale":"2024-09-21T10:22:04.734962+0000","last_undegraded":"2024-09-21T10:22:04.734962+0000","last_fullsized":"2024-09-21T10:22:04.734962+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:11.287664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1122,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498791+0000","last_change":"2024-09-21T10:06:42.632153+0000","last_active":"2024-09-21T10:22:04.498791+0000","last_peered":"2024-09-21T10:22:04.498791+0000","last_clean":"2024-09-21T10:22:04.498791+0000","last_became_active":"2024-09-21T10:06:42.631728+0000","last_became_peered":"2024-09-21T10:06:42.631728+0000","last_unstale":"2024-09-21T10:22:04.498791+0000","last_undegraded":"2024-09-21T10:22:04.498791+0000","last_fullsized":"2024-09-21T10:22:04.498791+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:53.412364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1101,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.328844+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T10:22:04.328844+0000","last_peered":"2024-09-21T10:22:04.328844+0000","last_clean":"2024-09-21T10:22:04.328844+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T10:22:04.328844+0000","last_undegraded":"2024-09-21T10:22:04.328844+0000","last_fullsized":"2024-09-21T10:22:04.328844+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1121,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.499006+0000","last_change":"2024-09-21T10:06:09.030444+0000","last_active":"2024-09-21T10:22:04.499006+0000","last_peered":"2024-09-21T10:22:04.499006+0000","last_clean":"2024-09-21T10:22:04.499006+0000","last_became_active":"2024-09-21T10:06:09.030054+0000","last_became_peered":"2024-09-21T10:06:09.030054+0000","last_unstale":"2024-09-21T10:22:04.499006+0000","last_undegraded":"2024-09-21T10:22:04.499006+0000","last_fullsized":"2024-09-21T10:22:04.499006+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:27.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1128,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.735035+0000","last_change":"2024-09-21T10:06:09.032476+0000","last_active":"2024-09-21T10:22:04.735035+0000","last_peered":"2024-09-21T10:22:04.735035+0000","last_clean":"2024-09-21T10:22:04.735035+0000","last_became_active":"2024-09-21T10:06:09.032121+0000","last_became_peered":"2024-09-21T10:06:09.032121+0000","last_unstale":"2024-09-21T10:22:04.735035+0000","last_undegraded":"2024-09-21T10:22:04.735035+0000","last_fullsized":"2024-09-21T10:22:04.735035+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:41.862129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1631,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003838+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:22:09.003838+0000","last_peered":"2024-09-21T10:22:09.003838+0000","last_clean":"2024-09-21T10:22:09.003838+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:22:09.003838+0000","last_undegraded":"2024-09-21T10:22:09.003838+0000","last_fullsized":"2024-09-21T10:22:09.003838+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1130,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498375+0000","last_change":"2024-09-21T10:08:06.546911+0000","last_active":"2024-09-21T10:22:04.498375+0000","last_peered":"2024-09-21T10:22:04.498375+0000","last_clean":"2024-09-21T10:22:04.498375+0000","last_became_active":"2024-09-21T10:08:06.546417+0000","last_became_peered":"2024-09-21T10:08:06.546417+0000","last_unstale":"2024-09-21T10:22:04.498375+0000","last_undegraded":"2024-09-21T10:22:04.498375+0000","last_fullsized":"2024-09-21T10:22:04.498375+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:46.544701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1751,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.434451+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:22:09.434451+0000","last_peered":"2024-09-21T10:22:09.434451+0000","last_clean":"2024-09-21T10:22:09.434451+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:22:09.434451+0000","last_undegraded":"2024-09-21T10:22:09.434451+0000","last_fullsized":"2024-09-21T10:22:09.434451+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1750,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.432873+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:22:09.432873+0000","last_peered":"2024-09-21T10:22:09.432873+0000","last_clean":"2024-09-21T10:22:09.432873+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:22:09.432873+0000","last_undegraded":"2024-09-21T10:22:09.432873+0000","last_fullsized":"2024-09-21T10:22:09.432873+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1118,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.329295+0000","last_change":"2024-09-21T10:08:55.102502+0000","last_active":"2024-09-21T10:22:04.329295+0000","last_peered":"2024-09-21T10:22:04.329295+0000","last_clean":"2024-09-21T10:22:04.329295+0000","last_became_active":"2024-09-21T10:08:55.101777+0000","last_became_peered":"2024-09-21T10:08:55.101777+0000","last_unstale":"2024-09-21T10:22:04.329295+0000","last_undegraded":"2024-09-21T10:22:04.329295+0000","last_fullsized":"2024-09-21T10:22:04.329295+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:51.293401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1152,"reported_epoch":1161,"state":"active+clean","last_fresh":"2024-09-21T10:22:04.498158+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:22:04.498158+0000","last_peered":"2024-09-21T10:22:04.498158+0000","last_clean":"2024-09-21T10:22:04.498158+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:22:04.498158+0000","last_undegraded":"2024-09-21T10:22:04.498158+0000","last_fullsized":"2024-09-21T10:22:04.498158+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1640,"reported_epoch":1162,"state":"active+clean","last_fresh":"2024-09-21T10:22:09.003671+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:22:09.003671+0000","last_peered":"2024-09-21T10:22:09.003671+0000","last_clean":"2024-09-21T10:22:09.003671+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:22:09.003671+0000","last_undegraded":"2024-09-21T10:22:09.003671+0000","last_fullsized":"2024-09-21T10:22:09.003671+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":124,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397842,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314188,"kb_used_data":12672,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93430964,"statfs":{"total":95995035648,"available":95673307136,"internally_reserved":0,"allocated":12976128,"data_stored":8569725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594071,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322504,"kb_used_data":12668,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93422648,"statfs":{"total":95995035648,"available":95664791552,"internally_reserved":0,"allocated":12972032,"data_stored":8569696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790300,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313596,"kb_used_data":12080,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93431556,"statfs":{"total":95995035648,"available":95673913344,"internally_reserved":0,"allocated":12369920,"data_stored":7975146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986528,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321920,"kb_used_data":12088,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93423232,"statfs":{"total":95995035648,"available":95665389568,"internally_reserved":0,"allocated":12378112,"data_stored":7979343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644460,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":58100,"kb_used_data":12072,"kb_used_omap":67,"kb_used_meta":45948,"kb_avail":93687052,"statfs":{"total":95995035648,"available":95935541248,"internally_reserved":0,"allocated":12361728,"data_stored":7974597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":47051613},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527478,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66232,"kb_used_data":12076,"kb_used_omap":75,"kb_used_meta":54068,"kb_avail":93678920,"statfs":{"total":95995035648,"available":95927214080,"internally_reserved":0,"allocated":12365824,"data_stored":7975119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":55365786},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274631,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88188,"kb_used_data":12080,"kb_used_omap":51,"kb_used_meta":76044,"kb_avail":93656964,"statfs":{"total":95995035648,"available":95904731136,"internally_reserved":0,"allocated":12369920,"data_stored":7974658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":77869397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62588,"kb_used_data":12656,"kb_used_omap":66,"kb_used_meta":49853,"kb_avail":93682564,"statfs":{"total":95995035648,"available":95930945536,"internally_reserved":0,"allocated":12959744,"data_stored":8565483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":51049796},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:23:08.483 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T10:23:08.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T10:23:08.725 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:08 smithi119 ceph-mon[101432]: from='client.45782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:08 smithi119 ceph-mon[105288]: from='client.45782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:08 smithi145 ceph-mon[89359]: from='client.45782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:23:10.132 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:09 smithi119 ceph-mon[101432]: pgmap v1559: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:10.132 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:09 smithi119 ceph-mon[105288]: pgmap v1559: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:09 smithi145 ceph-mon[89359]: pgmap v1559: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:epoch 1162 2024-09-21T10:23:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:fsid c96f302a-77fe-11ef-baf6-efdc52797490 2024-09-21T10:23:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:created 2024-09-21T09:50:08.133736+0000 2024-09-21T10:23:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:modified 2024-09-21T10:22:08.837947+0000 2024-09-21T10:23:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T10:23:10.520 INFO:teuthology.orchestra.run.smithi119.stdout:crush_version 22 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:full_ratio 0.95 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:backfillfull_ratio 0.9 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:nearfull_ratio 0.85 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:require_min_compat_client luminous 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:min_compat_client luminous 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:require_osd_release quincy 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:stretch_mode_enabled false 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 58 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.00 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:pool 3 'cephfs.foo.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 63 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.50 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:pool 4 'cephfs.foo.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool,bulk stripe_width 0 application cephfs read_balance_score 2.25 2024-09-21T10:23:10.521 INFO:teuthology.orchestra.run.smithi119.stdout:max_osd 8 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 up in weight 1 up_from 509 up_thru 1014 down_at 497 last_clean_interval [8,496) [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328] [v2:172.21.15.119:6804/2607764328,v1:172.21.15.119:6805/2607764328] exists,up cac46a83-240b-41e9-9cbe-6b685ebbb92b 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 up in weight 1 up_from 542 up_thru 1014 down_at 530 last_clean_interval [13,529) [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805] [v2:172.21.15.119:6812/2769715805,v1:172.21.15.119:6813/2769715805] exists,up fe034449-410d-4164-9895-411181da04f2 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 up in weight 1 up_from 626 up_thru 1014 down_at 614 last_clean_interval [18,613) [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048] [v2:172.21.15.119:6820/3806024048,v1:172.21.15.119:6821/3806024048] exists,up db36e41e-f53c-4033-85fe-11dd420f633b 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 up in weight 1 up_from 653 up_thru 1014 down_at 644 last_clean_interval [26,644) [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352] [v2:172.21.15.119:6828/1249776352,v1:172.21.15.119:6829/1249776352] exists,up 2f44491e-2550-4886-88cc-5d1c790f8783 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 up in weight 1 up_from 31 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610] [v2:172.21.15.145:6802/2801631610,v1:172.21.15.145:6803/2801631610] exists,up 2315f8d6-fbe6-4590-8842-d8aba506a0b8 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5 up in weight 1 up_from 37 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320] [v2:172.21.15.145:6810/340169320,v1:172.21.15.145:6811/340169320] exists,up e02fdc71-b568-4361-ab0f-1569f55f6034 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 up in weight 1 up_from 43 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366] [v2:172.21.15.145:6818/3533380366,v1:172.21.15.145:6819/3533380366] exists,up e5bc8e69-00be-456e-b6fd-d6e4e43029a9 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7 up in weight 1 up_from 49 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741] [v2:172.21.15.145:6826/3509158741,v1:172.21.15.145:6827/3509158741] exists,up cfc1495b-a36e-4f6f-8491-781460970f42 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.1 [4,1] 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.b [4,2,3,1] 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.11 [4,2] 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.13 [4,1] 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.15 [7,1] 2024-09-21T10:23:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 3.1b [4,1] 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 4.d [1,5] 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 4.17 [1,2] 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:pg_upmap_items 4.1c [1,5] 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/1139856442 expires 2024-09-21T11:11:38.888722+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2535372813 expires 2024-09-21T11:06:31.109355+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/760350456 expires 2024-09-22T09:55:43.105417+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/2722863607 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:0/2808539890 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/49791021 expires 2024-09-22T09:59:43.810176+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:0/576401189 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/3992357764 expires 2024-09-21T11:06:32.113588+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/194249182 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:6835/2195859977 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.523 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/1747871475 expires 2024-09-22T09:55:43.105417+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2781454362 expires 2024-09-22T09:59:43.810176+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/1747871475 expires 2024-09-22T09:55:43.105417+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/704164355 expires 2024-09-22T09:50:58.568079+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/1608923788 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2211199843 expires 2024-09-22T09:50:36.170901+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:0/974232973 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/3980060910 expires 2024-09-22T09:50:36.170901+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/1780938388 expires 2024-09-22T09:59:43.810176+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/547636249 expires 2024-09-22T09:59:43.810176+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/460015937 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/704164355 expires 2024-09-22T09:50:58.568079+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:0/2571529276 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.524 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/277848857 expires 2024-09-22T09:55:43.105417+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/893592105 expires 2024-09-22T09:50:58.568079+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/1497467670 expires 2024-09-22T09:50:36.170901+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:6834/2195859977 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2566533967 expires 2024-09-22T09:59:43.810176+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:0/2715091572 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/23023314 expires 2024-09-22T09:55:43.105417+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/4205960579 expires 2024-09-22T09:50:36.170901+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/4255735773 expires 2024-09-22T09:50:58.568079+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6801/4205960579 expires 2024-09-22T09:50:36.170901+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/2722863607 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/923321224 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/1218540943 expires 2024-09-22T09:50:58.568079+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/391955164 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:6800/1780938388 expires 2024-09-22T09:59:43.810176+0000 2024-09-21T10:23:10.525 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.145:0/3401809855 expires 2024-09-22T10:00:49.315248+0000 2024-09-21T10:23:10.526 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/2905946815 expires 2024-09-22T09:55:43.105417+0000 2024-09-21T10:23:10.526 INFO:teuthology.orchestra.run.smithi119.stdout:blocklist 172.21.15.119:0/3153987795 expires 2024-09-22T10:01:32.364993+0000 2024-09-21T10:23:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:23:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:23:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:11.141 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3136727364' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:23:11.141 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3136727364' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:23:11.143 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 509 up_thru 1014 down_at 497 last_clean_interval [8,496) [v2:172.21.15.119:6802/2607764328,v1:172.21.15.119:6803/2607764328] [v2:172.21.15.119:6804/2607764328,v1:172.21.15.119:6805/2607764328] exists,up cac46a83-240b-41e9-9cbe-6b685ebbb92b', 'osd.1 up in weight 1 up_from 542 up_thru 1014 down_at 530 last_clean_interval [13,529) [v2:172.21.15.119:6810/2769715805,v1:172.21.15.119:6811/2769715805] [v2:172.21.15.119:6812/2769715805,v1:172.21.15.119:6813/2769715805] exists,up fe034449-410d-4164-9895-411181da04f2', 'osd.2 up in weight 1 up_from 626 up_thru 1014 down_at 614 last_clean_interval [18,613) [v2:172.21.15.119:6818/3806024048,v1:172.21.15.119:6819/3806024048] [v2:172.21.15.119:6820/3806024048,v1:172.21.15.119:6821/3806024048] exists,up db36e41e-f53c-4033-85fe-11dd420f633b', 'osd.3 up in weight 1 up_from 653 up_thru 1014 down_at 644 last_clean_interval [26,644) [v2:172.21.15.119:6826/1249776352,v1:172.21.15.119:6827/1249776352] [v2:172.21.15.119:6828/1249776352,v1:172.21.15.119:6829/1249776352] exists,up 2f44491e-2550-4886-88cc-5d1c790f8783', 'osd.4 up in weight 1 up_from 31 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6800/2801631610,v1:172.21.15.145:6801/2801631610] [v2:172.21.15.145:6802/2801631610,v1:172.21.15.145:6803/2801631610] exists,up 2315f8d6-fbe6-4590-8842-d8aba506a0b8', 'osd.5 up in weight 1 up_from 37 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6808/340169320,v1:172.21.15.145:6809/340169320] [v2:172.21.15.145:6810/340169320,v1:172.21.15.145:6811/340169320] exists,up e02fdc71-b568-4361-ab0f-1569f55f6034', 'osd.6 up in weight 1 up_from 43 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6816/3533380366,v1:172.21.15.145:6817/3533380366] [v2:172.21.15.145:6818/3533380366,v1:172.21.15.145:6819/3533380366] exists,up e5bc8e69-00be-456e-b6fd-d6e4e43029a9', 'osd.7 up in weight 1 up_from 49 up_thru 1014 down_at 0 last_clean_interval [0,0) [v2:172.21.15.145:6824/3509158741,v1:172.21.15.145:6825/3509158741] [v2:172.21.15.145:6826/3509158741,v1:172.21.15.145:6827/3509158741] exists,up cfc1495b-a36e-4f6f-8491-781460970f42'] 2024-09-21T10:23:11.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T10:23:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3136727364' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-21T10:23:11.389 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:11 smithi119 ceph-mon[101432]: pgmap v1560: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:11 smithi119 ceph-mon[105288]: pgmap v1560: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:11 smithi145 ceph-mon[89359]: pgmap v1560: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:13.161 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:23:13.161 INFO:teuthology.orchestra.run.smithi119.stdout: "name": "mon.a", 2024-09-21T10:23:13.161 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster": "ceph", 2024-09-21T10:23:13.161 INFO:teuthology.orchestra.run.smithi119.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T10:23:13.161 INFO:teuthology.orchestra.run.smithi119.stdout: "admin_socket_mode": "", 2024-09-21T10:23:13.161 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_client_required": "cephx, none", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_cluster_required": "cephx", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_debug": "true", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_service_required": "cephx", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_supported": "", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_aio": "true", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T10:23:13.162 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_aio_reap_max": "16", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_async_discard_threads": "0", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_block_size": "4096", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_debug_aio": "false", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_enable_discard": "false", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_flock_retry": "3", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_inject_crash": "0", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_ioring": "false", 2024-09-21T10:23:13.163 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_ioring_hipri": "false", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bdev_type": "", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_allocator": "hybrid", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_buffered_io": "true", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T10:23:13.164 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_replay_recovery": "false", 2024-09-21T10:23:13.165 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluefs_sync_write": "false", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_allocator": "hybrid", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T10:23:13.166 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_create": "true", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_db_create": "false", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_db_path": "", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_db_size": "0", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_path": "", 2024-09-21T10:23:13.167 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_size": "107374182400", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_wal_create": "false", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_wal_path": "", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bluefs": "true", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T10:23:13.168 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_autotune": "true", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_size": "0", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cache_type": "2q", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_clone_cow": "true", 2024-09-21T10:23:13.169 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T10:23:13.170 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T10:23:13.170 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_mode": "none", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_freelist": "false", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T10:23:13.171 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T10:23:13.172 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fail_eio": "false", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T10:23:13.173 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T10:23:13.174 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_max_blob_size": "0", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T10:23:13.175 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T10:23:13.176 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T10:23:13.177 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T10:23:13.177 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T10:23:13.177 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T10:23:13.177 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_spdk_mem": "512", 2024-09-21T10:23:13.177 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T10:23:13.177 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_tracing": "false", 2024-09-21T10:23:13.178 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_write_v2": "false", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_write_v2_random": "false", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T10:23:13.179 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T10:23:13.180 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_cluster_require_version": "2", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_require_signatures": "false", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_require_version": "2", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_service_require_signatures": "false", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_service_require_version": "2", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "cephx_sign_messages": "true", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "chdir": "", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_acl_type": "", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_asio_thread_count": "2", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_cache_mid": "0.750000", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_cache_size": "16384", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_caps_release_delay": "5", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_check_pool_perm": "true", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T10:23:13.181 INFO:teuthology.orchestra.run.smithi119.stdout: "client_debug_force_sync_read": "false", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_debug_getattr_caps": "false", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_debug_inject_features": "", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_die_on_failed_remount": "false", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_dirsize_rbytes": "true", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_force_lazyio": "false", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_fs": "", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_inject_release_failure": "false", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_max_inline_size": "4096", 2024-09-21T10:23:13.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_mds_namespace": "", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_metadata": "", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_mount_gid": "-1", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_mount_timeout": "300", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_mount_uid": "-1", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_mountpoint": "/", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_notify_timeout": "10", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_oc": "true", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_oc_max_objects": "1000", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_oc_size": "209715200", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T10:23:13.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_permissions": "true", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_quota": "true", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_quota_df": "true", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_readahead_max_bytes": "0", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_readahead_max_periods": "4", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_readahead_min": "131072", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_reconnect_stale": "false", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_shutdown_timeout": "30", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_snapdir": ".snap", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_tick_interval": "1", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_trace": "", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_use_faked_inos": "false", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "client_use_random_mds": "false", 2024-09-21T10:23:13.184 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_graylog": "false", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_graylog_port": "12201", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_monitors": "default=true", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_syslog": "false", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "clog_to_syslog_level": "info", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_addr": "-", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_network": "", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_network_interface": "", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "compressor_zlib_isal": "false", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "compressor_zlib_level": "5", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T10:23:13.185 INFO:teuthology.orchestra.run.smithi119.stdout: "compressor_zstd_level": "1", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_location": "", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_location_hook": "", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "crush_location_hook_timeout": "10", 2024-09-21T10:23:13.186 INFO:teuthology.orchestra.run.smithi119.stdout: "daemonize": "false", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "daos_pool": "tank", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_alienstore": "0/5", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_asok": "1/5", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_asok_assert_abort": "false", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_auth": "1/5", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_bdev": "1/3", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_bluefs": "1/5", 2024-09-21T10:23:13.187 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_bluestore": "1/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_buffer": "0/1", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_cephsqlite": "0/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_client": "0/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_compressor": "1/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_context": "0/1", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_crush": "1/1", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_crypto": "1/5", 2024-09-21T10:23:13.188 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_cyanstore": "0/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_dpdk": "1/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_eventtrace": "1/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_filer": "0/1", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_filestore": "1/3", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_finisher": "1/1", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_fuse": "1/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_javaclient": "1/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_journal": "1/3", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_journaler": "0/5", 2024-09-21T10:23:13.189 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_kstore": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_lockdep": "0/1", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mclock": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds_balancer": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds_locker": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds_log": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds_migrator": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_memstore": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mgr": "2/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mgrc": "1/5", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_mon": "20/20", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_monc": "0/10", 2024-09-21T10:23:13.190 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_ms": "1/1", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_none": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_objclass": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_objectcacher": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_objecter": "0/1", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_optracker": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_osd": "1/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_paxos": "20/20", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_perfcounter": "1/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_prioritycache": "1/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rados": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rbd": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T10:23:13.191 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rbd_replay": "0/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_refs": "0/0", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_reserver": "1/1", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_access": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_flight": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_notification": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rgw_sync": "1/5", 2024-09-21T10:23:13.192 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_rocksdb": "4/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_backref": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_cache": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_device": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_epm": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_journal": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_lba": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_odata": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_omap": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_onode": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_t": "0/5", 2024-09-21T10:23:13.193 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_seastore_tm": "0/5", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_striper": "0/1", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_test": "0/5", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_throttle": "1/1", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_timer": "0/1", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_tp": "0/5", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "debug_trace": "1/5", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "device_failure_prediction_mode": "none", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "err_to_graylog": "false", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "err_to_journald": "false", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "err_to_stderr": "true", 2024-09-21T10:23:13.194 INFO:teuthology.orchestra.run.smithi119.stdout: "err_to_syslog": "false", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event_tracing": "false", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_cert_file": "", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_http_port": "9926", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_key_file": "", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_prio_limit": "5", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_sort_metrics": "true", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "exporter_stats_period": "5", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "fake_statfs_for_testing": "0", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "fatal_signal_handlers": "true", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "filer_max_purge_ops": "10", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "filer_max_truncate_ops": "128", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T10:23:13.195 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_blackhole": "false", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_btrfs_snap": "true", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_caller_concurrency": "10", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_debug_omap_check": "false", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_debug_verify_split": "false", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_dump_file": "", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fadvise": "true", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fail_eio": "true", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T10:23:13.196 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fd_cache_size": "128", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fiemap": "false", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_inject_stall": "0", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_journal_parallel": "false", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_journal_trailing": "false", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_journal_writeahead": "false", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_kill_at": "0", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T10:23:13.197 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_merge_threshold": "-10", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_odsync_write": "false", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_omap_backend_path": "", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T10:23:13.198 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_op_threads": "2", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_punch_hole": "false", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_queue_max_ops": "50", 2024-09-21T10:23:13.199 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_seek_data_hole": "false", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_sloppy_crc": "false", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_splice": "false", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_split_multiple": "2", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_split_rand_factor": "20", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_update_to": "1000", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T10:23:13.200 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_xfs_extsize": "false", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "filestore_zfs_snap": "false", 2024-09-21T10:23:13.201 INFO:teuthology.orchestra.run.smithi119.stdout: "fio_dir": "/tmp/fio", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fsid": "c96f302a-77fe-11ef-baf6-efdc52797490", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_allow_other": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_big_writes": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_debug": "false", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_default_permissions": "false", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_disable_pagecache": "false", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_max_write": "0", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_multithreaded": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_require_active_mds": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_set_user_groups": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_splice_move": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_splice_read": "true", 2024-09-21T10:23:13.202 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_splice_write": "true", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "gss_target_name": "ceph", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "heartbeat_file": "", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "heartbeat_inject_failure": "0", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "heartbeat_interval": "5", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "host": "smithi119", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T10:23:13.203 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "inject_early_sigterm": "false", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "jaeger_agent_port": "6799", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "jaeger_tracing_enable": "false", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_aio": "true", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_align_min_size": "65536", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_block_align": "true", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_block_size": "4096", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_dio": "true", 2024-09-21T10:23:13.204 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_discard": "false", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_force_aio": "false", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_ignore_corruption": "false", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_max_write_entries": "100", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_replay_from": "0", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_write_header_frequency": "0", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journal_zero_on_create": "false", 2024-09-21T10:23:13.205 INFO:teuthology.orchestra.run.smithi119.stdout: "journaler_prefetch_periods": "10", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "journaler_prezero_periods": "5", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "journaler_write_head_interval": "15", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "keyfile": "", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_backend": "rocksdb", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_max_bytes": "67108864", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_max_ops": "512", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T10:23:13.206 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_onode_map_size": "1024", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "kstore_sync_transaction": "false", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "librados_thread_count": "2", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "lockdep": "false", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "lockdep_force_backtrace": "false", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_coarse_timestamps": "true", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_flush_on_exit": "false", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_graylog_port": "12201", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_max_new": "1000", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_max_recent": "10000", 2024-09-21T10:23:13.207 INFO:teuthology.orchestra.run.smithi119.stdout: "log_stderr_prefix": "", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_file": "true", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_graylog": "false", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_journald": "false", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_stderr": "false", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "log_to_syslog": "false", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_action_on_write_error": "1", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_alternate_name_max": "8192", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_asio_thread_count": "2", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_export_pin": "true", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T10:23:13.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_interval": "10", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_max": "-1", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_max_until": "-1", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_merge_size": "50", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_mode": "0", 2024-09-21T10:23:13.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_split_bits": "3", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_split_size": "10000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T10:23:13.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_mid": "0.700000", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T10:23:13.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_trim_interval": "1", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T10:23:13.212 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_debug_auth_pins": "false", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_debug_frag": "false", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_debug_scatterstat": "false", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_debug_subtrees": "false", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_default_dir_hash": "2", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_defer_session_stale": "true", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T10:23:13.213 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dir_max_entries": "0", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dir_prefetch": "true", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_early_reply": "true", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_enable_op_tracker": "true", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_enforce_unique_name": "true", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T10:23:13.214 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_inject_health_dummy": "false", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T10:23:13.215 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_join_fs": "", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_journal_format": "1", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_create_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_export_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_import_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_journal_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T10:23:13.216 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_link_at": "0", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_openc_at": "0", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_rename_at": "0", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_max_events": "-1", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_max_segments": "128", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_pause": "false", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_segment_size": "0", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T10:23:13.217 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_trim_threshold": "128", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_completed_requests": "100000", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_export_size": "20971520", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_file_recover": "32", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_purge_files": "64", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_purge_ops": "8192", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T10:23:13.218 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_metrics_update_interval": "2", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_min_caps_per_client": "100", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_numa_node": "-1", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_op_history_duration": "600", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_op_history_size": "20", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_op_log_threshold": "5", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_ping_grace": "15", 2024-09-21T10:23:13.219 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_ping_interval": "5", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_recall_max_caps": "30000", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_replay_interval": "1.000000", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T10:23:13.220 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_root_ino_gid": "0", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_root_ino_uid": "0", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T10:23:13.221 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_shutdown_check": "0", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_skip_ino": "0", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T10:23:13.222 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_snap_min_uid": "0", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_snap_rstat": "false", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_symlink_recovery": "true", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_thrash_exports": "0", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_thrash_fragments": "0", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_tick_interval": "5.000000", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_valgrind_exit": "false", 2024-09-21T10:23:13.223 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_verify_backtrace": "1", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_verify_scatter": "false", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_wipe_sessions": "false", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mempool_debug": "false", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "memstore_page_set": "false", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "memstore_page_size": "65536", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_client_bytes": "134217728", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_client_messages": "512", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T10:23:13.224 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_disabled_modules": "", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_mds_messages": "128", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T10:23:13.225 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_mon_messages": "128", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_op_history_duration": "600", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_op_history_size": "20", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_op_log_threshold": "5", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_osd_messages": "8192", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_pool": "true", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_standby_modules": "true", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_stats_period": "5", 2024-09-21T10:23:13.226 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_stats_threshold": "5", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_tick_period": "2", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_allow_pool_delete": "true", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_bytes": "104857600", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T10:23:13.227 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T10:23:13.228 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_client_target_rank": "-1", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_level": "debug", 2024-09-21T10:23:13.229 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T10:23:13.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T10:23:13.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T10:23:13.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T10:23:13.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T10:23:13.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T10:23:13.230 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T10:23:13.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T10:23:13.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T10:23:13.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_compact_on_start": "false", 2024-09-21T10:23:13.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_compact_on_trim": "true", 2024-09-21T10:23:13.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T10:23:13.231 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_cpu_threads": "4", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_data_avail_crit": "5", 2024-09-21T10:23:13.232 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_data_avail_warn": "5", 2024-09-21T10:23:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T10:23:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T10:23:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T10:23:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_dump_json": "false", 2024-09-21T10:23:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T10:23:13.233 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T10:23:13.234 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_extra_checks": "false", 2024-09-21T10:23:13.234 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T10:23:13.234 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T10:23:13.234 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T10:23:13.234 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T10:23:13.234 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T10:23:13.235 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T10:23:13.235 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T10:23:13.235 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T10:23:13.235 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_down_uptime_grace": "60", 2024-09-21T10:23:13.235 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_election_default_strategy": "1", 2024-09-21T10:23:13.236 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_election_timeout": "5.000000", 2024-09-21T10:23:13.236 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T10:23:13.236 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T10:23:13.236 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T10:23:13.236 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_enable_op_tracker": "true", 2024-09-21T10:23:13.236 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_fake_pool_delete": "false", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_force_quorum_join": "false", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_health_log_update_period": "5", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_health_max_detail": "50", 2024-09-21T10:23:13.237 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_health_to_clog": "true", 2024-09-21T10:23:13.238 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T10:23:13.238 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T10:23:13.238 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_host": "[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]", 2024-09-21T10:23:13.238 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_host_override": "", 2024-09-21T10:23:13.238 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_initial_members": "", 2024-09-21T10:23:13.238 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T10:23:13.239 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T10:23:13.239 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T10:23:13.239 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T10:23:13.239 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T10:23:13.239 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_lease": "5.000000", 2024-09-21T10:23:13.239 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_log_full_interval": "50", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_log_max": "10000", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_log_max_summary": "50", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_log_epochs": "500", 2024-09-21T10:23:13.240 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T10:23:13.241 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T10:23:13.241 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T10:23:13.241 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_osd": "10000", 2024-09-21T10:23:13.241 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T10:23:13.241 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T10:23:13.241 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T10:23:13.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T10:23:13.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T10:23:13.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mds_skip_sanity": "true", 2024-09-21T10:23:13.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_memory_autotune": "true", 2024-09-21T10:23:13.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_memory_target": "2147483648", 2024-09-21T10:23:13.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mgr_digest_period": "5", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T10:23:13.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T10:23:13.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T10:23:13.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_op_complaint_time": "30", 2024-09-21T10:23:13.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_op_history_duration": "600", 2024-09-21T10:23:13.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_op_history_size": "20", 2024-09-21T10:23:13.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T10:23:13.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T10:23:13.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_op_log_threshold": "5", 2024-09-21T10:23:13.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T10:23:13.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T10:23:13.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T10:23:13.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T10:23:13.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T10:23:13.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T10:23:13.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T10:23:13.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_cache_size": "500", 2024-09-21T10:23:13.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T10:23:13.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T10:23:13.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T10:23:13.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T10:23:13.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T10:23:13.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T10:23:13.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T10:23:13.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T10:23:13.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T10:23:13.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T10:23:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T10:23:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T10:23:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T10:23:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T10:23:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T10:23:13.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_report_timeout": "900", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T10:23:13.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T10:23:13.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T10:23:13.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T10:23:13.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T10:23:13.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T10:23:13.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T10:23:13.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_reweight_max_osds": "4", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T10:23:13.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_scrub_interval": "86400", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_scrub_max_keys": "100", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_scrub_timeout": "300", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_session_timeout": "300", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_smart_report_timeout": "5", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T10:23:13.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_stretch_pool_size": "4", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_sync_debug": "false", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_tick_interval": "5", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T10:23:13.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_older_version": "true", 2024-09-21T10:23:13.257 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "monmap": "", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T10:23:13.258 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "motr_tracing_enabled": "false", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_op_threads": "3", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_cm": "false", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_device_name": "", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T10:23:13.259 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_sl": "3", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_rdma_type": "ib", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_async_reap_threshold": "5", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_before_connect": "false", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_ipv4": "true", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_ipv6": "false", 2024-09-21T10:23:13.260 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_msgr1": "true", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_msgr2": "true", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_port_max": "7568", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_port_min": "6800", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_retry_count": "3", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_bind_retry_delay": "5", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_blackhole_client": "false", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_blackhole_mds": "false", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_blackhole_mgr": "false", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_blackhole_mon": "false", 2024-09-21T10:23:13.261 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_blackhole_osd": "false", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_client_mode": "crc secure", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_cluster_type": "", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_compress_secure": "false", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_crc_data": "true", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_crc_header": "true", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_die_on_bug": "true", 2024-09-21T10:23:13.262 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_die_on_old_message": "true", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_hugepages": "", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T10:23:13.263 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_lro": "true", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_pmd": "", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_port_id": "0", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_dump_on_send": "false", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_inject_delay_type": "", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T10:23:13.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_inject_network_congestion": "0", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_inject_socket_failures": "0", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_max_accept_failures": "4", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_max_backoff": "15.000000", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_osd_compress_mode": "none", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_pq_min_cost": "65536", 2024-09-21T10:23:13.265 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_public_type": "", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_service_mode": "crc secure", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_tcp_nodelay": "true", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "ms_type": "async+posix", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "no_config_file": "false", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T10:23:13.266 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_inflight_ops": "1024", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objecter_timeout": "10.000000", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objectstore_blackhole": "false", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "openssl_engine_opts": "", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T10:23:13.267 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_max_ops": "4", 2024-09-21T10:23:13.268 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T10:23:13.268 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T10:23:13.268 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_agent_slop": "0.020000", 2024-09-21T10:23:13.268 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T10:23:13.268 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T10:23:13.268 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_asio_thread_count": "2", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_backfill_scan_max": "512", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_backfill_scan_min": "64", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T10:23:13.269 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_backoff_on_peering": "false", 2024-09-21T10:23:13.270 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T10:23:13.270 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_beacon_report_interval": "300", 2024-09-21T10:23:13.270 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bench_duration": "30", 2024-09-21T10:23:13.270 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T10:23:13.270 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T10:23:13.270 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T10:23:13.271 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_blkin_trace_all": "false", 2024-09-21T10:23:13.271 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T10:23:13.271 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T10:23:13.271 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T10:23:13.271 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T10:23:13.272 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T10:23:13.272 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T10:23:13.272 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:23:13.272 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T10:23:13.272 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T10:23:13.272 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_class_update_on_start": "true", 2024-09-21T10:23:13.273 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_client_message_cap": "256", 2024-09-21T10:23:13.273 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T10:23:13.273 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_client_op_priority": "63", 2024-09-21T10:23:13.273 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_client_watch_timeout": "30", 2024-09-21T10:23:13.273 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_command_max_records": "256", 2024-09-21T10:23:13.274 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_compact_on_start": "false", 2024-09-21T10:23:13.274 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T10:23:13.274 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T10:23:13.274 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T10:23:13.274 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_crush_update_on_start": "true", 2024-09-21T10:23:13.274 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T10:23:13.275 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T10:23:13.275 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T10:23:13.275 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T10:23:13.275 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T10:23:13.275 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T10:23:13.276 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T10:23:13.277 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_op_order": "false", 2024-09-21T10:23:13.277 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T10:23:13.277 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T10:23:13.277 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T10:23:13.277 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T10:23:13.277 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_shutdown": "false", 2024-09-21T10:23:13.278 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T10:23:13.278 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T10:23:13.278 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_trim_objects": "false", 2024-09-21T10:23:13.278 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T10:23:13.278 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T10:23:13.278 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T10:23:13.279 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T10:23:13.279 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T10:23:13.279 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T10:23:13.279 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T10:23:13.279 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T10:23:13.279 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T10:23:13.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T10:23:13.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T10:23:13.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T10:23:13.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_default_notify_timeout": "30", 2024-09-21T10:23:13.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T10:23:13.281 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T10:23:13.281 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T10:23:13.281 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T10:23:13.281 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T10:23:13.281 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_ec_partial_reads": "true", 2024-09-21T10:23:13.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_enable_op_tracker": "true", 2024-09-21T10:23:13.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T10:23:13.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T10:23:13.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T10:23:13.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T10:23:13.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_fast_info": "true", 2024-09-21T10:23:13.283 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_fast_shutdown": "true", 2024-09-21T10:23:13.283 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T10:23:13.283 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T10:23:13.283 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T10:23:13.283 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T10:23:13.284 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T10:23:13.284 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_function_tracing": "false", 2024-09-21T10:23:13.284 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_grace": "20", 2024-09-21T10:23:13.284 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_interval": "6", 2024-09-21T10:23:13.284 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T10:23:13.284 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T10:23:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T10:23:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_stale": "600", 2024-09-21T10:23:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T10:23:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T10:23:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T10:23:13.285 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T10:23:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T10:23:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T10:23:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T10:23:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T10:23:13.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T10:23:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_journal_size": "5120", 2024-09-21T10:23:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_kill_backfill_at": "0", 2024-09-21T10:23:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T10:23:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_map_cache_size": "50", 2024-09-21T10:23:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_map_dedup": "true", 2024-09-21T10:23:13.287 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_map_message_max": "40", 2024-09-21T10:23:13.288 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T10:23:13.288 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T10:23:13.288 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_attr_name_len": "100", 2024-09-21T10:23:13.288 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_attr_size": "0", 2024-09-21T10:23:13.288 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_backfills": "1", 2024-09-21T10:23:13.288 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_markdown_count": "5", 2024-09-21T10:23:13.289 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_markdown_period": "600", 2024-09-21T10:23:13.289 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_object_name_len": "2048", 2024-09-21T10:23:13.289 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T10:23:13.289 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_object_size": "134217728", 2024-09-21T10:23:13.289 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T10:23:13.289 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T10:23:13.290 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T10:23:13.290 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T10:23:13.290 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T10:23:13.290 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_pgls": "1024", 2024-09-21T10:23:13.290 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_push_cost": "8388608", 2024-09-21T10:23:13.291 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_push_objects": "10", 2024-09-21T10:23:13.291 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_scrubs": "3", 2024-09-21T10:23:13.291 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T10:23:13.291 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T10:23:13.291 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T10:23:13.291 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_max_write_size": "90", 2024-09-21T10:23:13.292 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T10:23:13.292 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T10:23:13.292 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T10:23:13.292 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T10:23:13.292 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T10:23:13.292 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T10:23:13.293 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T10:23:13.293 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T10:23:13.293 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_profile": "balanced", 2024-09-21T10:23:13.293 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T10:23:13.293 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T10:23:13.293 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T10:23:13.294 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T10:23:13.294 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T10:23:13.294 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T10:23:13.294 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T10:23:13.294 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_base": "805306368", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_target": "4294967296", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_target_autotune": "false", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T10:23:13.295 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_min_recovery_priority": "0", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mon_report_interval": "5", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_num_cache_shards": "32", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_numa_node": "-1", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_objecter_finishers": "1", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_objectstore": "bluestore", 2024-09-21T10:23:13.296 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_objectstore_fuse": "false", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_objectstore_tracing": "false", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_history_duration": "600", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_history_size": "20", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_log_threshold": "5", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_num_shards": "0", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T10:23:13.297 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_op_thread_timeout": "15", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_open_classes_on_start": "true", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_os_flags": "0", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_peering_op_priority": "255", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_delete_priority": "5", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T10:23:13.298 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T10:23:13.299 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_crimson": "false", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_flags": "0", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_min_size": "0", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T10:23:13.300 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_size": "3", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_default_type": "replicated", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_cost": "20971520", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_max_active": "0", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T10:23:13.301 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_op_priority": "3", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_priority": "5", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_repair_during_recovery": "false", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T10:23:13.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_cost": "52428800", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_end_hour": "0", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T10:23:13.303 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_priority": "5", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T10:23:13.304 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_skip_data_digest": "false", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_smart_report_timeout": "5", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_snap_trim_priority": "5", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T10:23:13.305 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_target_transaction_size": "30", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_tracing": "false", 2024-09-21T10:23:13.306 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_use_stale_snap": "false", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_kill_at": "0", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_max_join_drift": "10", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_min": "500", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_min_wait": "0.050000", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_service_trim_max": "500", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_service_trim_min": "250", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_stash_full_interval": "25", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_trim_max": "500", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "paxos_trim_min": "250", 2024-09-21T10:23:13.307 INFO:teuthology.orchestra.run.smithi119.stdout: "perf": "true", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "pid_file": "", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "public_addr": "-", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "public_addrv": "", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "public_bind_addr": "-", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "public_network": "172.21.0.0/20", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "public_network_interface": "", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "qat_compressor_enabled": "false", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "rados_mon_op_timeout": "0", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "rados_osd_op_timeout": "0", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "rados_tracing": "false", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_atime_update_interval": "60", 2024-09-21T10:23:13.308 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache": "true", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_policy": "writearound", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_size": "33554432", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T10:23:13.309 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_compression_hint": "none", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_config_pool_override_update_timestamp": "1726913501", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_clone_format": "auto", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_data_pool": "", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_features": "61", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_format": "2", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_map_options": "", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_order": "22", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_pool": "rbd", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T10:23:13.310 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_stripe_count": "0", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_io_scheduler": "simple", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T10:23:13.311 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_order": "24", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_pool": "", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_journal_splay_width": "4", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T10:23:13.312 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T10:23:13.313 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T10:23:13.314 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_op_threads": "1", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T10:23:13.315 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_plugins": "", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T10:23:13.316 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T10:23:13.317 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_tracing": "false", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_validate_names": "true", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rbd_validate_pool": "true", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "restapi_base_url": "", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "restapi_log_level": "", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T10:23:13.318 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_admin_entry": "admin", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_backend_store": "rados", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_barbican_url": "", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_beast_enable_async": "true", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T10:23:13.319 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_cache_enabled": "true", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_config_store": "rados", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_content_length_compat": "false", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_cross_domain_policy": "", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T10:23:13.320 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T10:23:13.321 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T10:23:13.322 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T10:23:13.323 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T10:23:13.324 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_log_window": "30", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T10:23:13.325 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_disable_s3select": "false", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T10:23:13.326 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dns_name": "", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dns_s3website_name": "", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_ops_log": "false", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_static_website": "false", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enable_usage_log": "false", 2024-09-21T10:23:13.327 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_expose_bucket": "false", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_extended_http_attrs": "", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_filter": "none", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_max_objs": "32", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T10:23:13.328 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_init_timeout": "300", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T10:23:13.329 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_password": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_project": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_token": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_admin_user": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T10:23:13.330 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_url": "", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_max_objs": "32", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_max_worker": "3", 2024-09-21T10:23:13.331 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ldap_uri": "", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_log_http_headers": "", 2024-09-21T10:23:13.332 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_attr_size": "0", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_listing_results": "5000", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_notify_retries": "10", 2024-09-21T10:23:13.333 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T10:23:13.334 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T10:23:13.335 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_num_control_oids": "8", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_numa_node": "-1", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_op_tracing": "false", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_opa_token": "", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_opa_url": "", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T10:23:13.336 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ops_log_rados": "false", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T10:23:13.337 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_print_continue": "true", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_rados_tracing": "false", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_realm": "", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_realm_id": "", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T10:23:13.338 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_region": "", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_request_uri": "", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T10:23:13.339 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_resolve_cname": "false", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_run_sync_thread": "true", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T10:23:13.340 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_script_uri": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_service_provider_name": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_client_id": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_client_secret": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_entry": "sts", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_key": "sts", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_auth_url": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_custom_header": "", 2024-09-21T10:23:13.341 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_need_stats": "true", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_tenant_name": "", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_url": "", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_lease_period": "120", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T10:23:13.342 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_thread_pool_size": "512", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_comment": "", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_createby": "", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_encoding": "", 2024-09-21T10:23:13.343 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_flag": "false", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_origin": "", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_torrent_tracker": "", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_usage_max_shards": "32", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_use_opa_authz": "false", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_counters_cache": "false", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T10:23:13.344 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_user_unique_email": "true", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_verify_ssl": "true", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_zone": "", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_zone_id": "", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_zonegroup": "", 2024-09-21T10:23:13.345 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_zonegroup_id": "", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_block_size": "4096", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T10:23:13.346 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_partition_filters": "false", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_perf": "false", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "run_dir": "/var/run/ceph", 2024-09-21T10:23:13.347 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_block_create": "true", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_device_size": "53687091200", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_full_integrity_check": "false", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T10:23:13.348 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_main_device_type": "SSD", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "seastore_segment_size": "67108864", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "set_keepcaps": "false", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "setgroup": "ceph", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "setuser": "ceph", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "setuser_match_path": "", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "thp": "false", 2024-09-21T10:23:13.349 INFO:teuthology.orchestra.run.smithi119.stdout: "threadpool_default_timeout": "60", 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout: "throttler_perf_counter": "true", 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout: "tmp_dir": "/tmp", 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout: "uadk_compressor_enabled": "false", 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T10:23:13.350 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:23:13.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T10:23:14.239 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:13 smithi145 ceph-mon[89359]: pgmap v1561: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:14.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:13 smithi119 ceph-mon[101432]: pgmap v1561: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:14.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:13 smithi119 ceph-mon[105288]: pgmap v1561: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:14.612 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:23:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:23:14] "GET /metrics HTTP/1.1" 200 34282 "" "Prometheus/2.51.0" 2024-09-21T10:23:15.836 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a: {} 2024-09-21T10:23:15.836 INFO:teuthology.orchestra.run.smithi119.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:23:15.842 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b: {} 2024-09-21T10:23:15.842 INFO:teuthology.orchestra.run.smithi119.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:23:15.850 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c: {} 2024-09-21T10:23:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:23:16.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:15 smithi119 ceph-mon[101432]: pgmap v1562: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:16.224 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2963348449' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:23:16.224 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:15 smithi119 ceph-mon[105288]: pgmap v1562: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:16.224 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2963348449' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:23:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:15 smithi145 ceph-mon[89359]: pgmap v1562: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2963348449' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:23:16.531 INFO:tasks.thrashosds.thrasher:Unable to test ceph-objectstore-tool, not available on all OSD nodes 2024-09-21T10:23:16.531 INFO:teuthology.task.sequential:In sequential, running task full_sequential... 2024-09-21T10:23:16.532 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:23:16.533 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:23:16.533 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T10:23:16.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T10:23:16.534 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T10:23:16.534 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:23:16.534 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:23:16.534 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T10:23:16.534 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:23:16.535 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:23:16.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T10:23:16.535 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T10:23:16.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:16.536 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T10:23:16.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:23:16.536 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T10:23:16.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:23:16.537 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T10:23:16.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:23:16.565 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:16.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.689 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:16.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:16.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:16.803 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:23:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:16.828 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:16.830 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:16.837 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:16.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:16.944 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:16.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:17.046 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:17.068 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:17.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:17.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:17.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:17.313 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:17.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:17.436 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:17.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:17.560 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:17.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:17.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:17.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:17.806 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:17.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:17.929 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:17.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:18.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:17 smithi145 ceph-mon[89359]: pgmap v1563: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:18.052 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:18.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:18.167 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:17 smithi119 ceph-mon[105288]: pgmap v1563: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:18.167 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:18.167 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:17 smithi119 ceph-mon[101432]: pgmap v1563: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:18.167 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:18.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:18.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:18.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:18.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:18.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:18.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:18.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:18.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:18.651 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:23:18.651 INFO:teuthology.orchestra.run.smithi145.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:23:18.651 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:23:18.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:18.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:18.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:18.916 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:18.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:19.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:19.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:19.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:19.162 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:19.263 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:19.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:19.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:19.407 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:19.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:19.530 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:19.631 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:19.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:19.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:19.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:19.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:19.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:19.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:20.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:20.003 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:19 smithi119 ceph-mon[105288]: pgmap v1564: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:20.003 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:19 smithi119 ceph-mon[101432]: pgmap v1564: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:20.022 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:20.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:20.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:20.146 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:20.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:19 smithi145 ceph-mon[89359]: pgmap v1564: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:20.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:20.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:20.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:20.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:20.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:20.517 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:20.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:23:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:23:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:20.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:20.640 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:20.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:20.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:20.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:20.891 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:20.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:20.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:21.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:21.137 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:21.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:21.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:21.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:21.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:21.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:21.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:21.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:21.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:21.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:21.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:21.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:21.863 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:23:21.863 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:23:21.863 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-21T10:23:21.864 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:23:21.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:21.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:21.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:21.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:22.012 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:22.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:21 smithi119 ceph-mon[105288]: pgmap v1565: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:21 smithi119 ceph-mon[101432]: pgmap v1565: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:22.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:22.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:22.262 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:21 smithi145 ceph-mon[89359]: pgmap v1565: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:22.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:22.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:22.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:22.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:22.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:22.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:22.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:22.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:22.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:22.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:22.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:22.913 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:23:22.983 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:23:22.983 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:22.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:22.994 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:23:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:23.013 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:23:23.013 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:23.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:23.029 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:23:23.029 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:23.029 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3388605289' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:23.030 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:22 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:23.030 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3388605289' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:23.030 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:22 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:23.045 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:23:23.045 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:23.060 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:23:23.061 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:23.076 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:23:23.076 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:23.091 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:23:23.092 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:23.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:23.141 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:23.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:23.269 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3388605289' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:22 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:23:23.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:23.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:23.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:23.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:23.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:23.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:23.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:23.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:23.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:23.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:23.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:23.911 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_0' created 2024-09-21T10:23:23.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:24.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:24.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:24.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:24.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:24.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:24.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:23 smithi145 ceph-mon[89359]: pgmap v1566: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:23 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:23 smithi145 ceph-mon[89359]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-21T10:23:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4008607583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1043159378' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:23:24.346 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:23:24.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[105288]: pgmap v1566: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[105288]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4008607583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1043159378' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[101432]: pgmap v1566: 73 pgs: 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:24.353 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:24.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:23:24.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[101432]: osdmap e1163: 8 total, 8 up, 8 in 2024-09-21T10:23:24.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4008607583' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:24.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1043159378' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-21T10:23:24.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:24.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:24.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:24.487 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:24.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:24.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:24.618 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:23:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:23:24] "GET /metrics HTTP/1.1" 200 34280 "" "Prometheus/2.51.0" 2024-09-21T10:23:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:24.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:24.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:23:24.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:24.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:24.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:24.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:24.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:24.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:24.919 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1043159378' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:23:24.919 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:24 smithi119 ceph-mon[105288]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-21T10:23:24.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:25.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:25.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:25.141 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:25.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:25.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1043159378' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:23:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:24 smithi145 ceph-mon[89359]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-21T10:23:25.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1043159378' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T10:23:25.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:24 smithi119 ceph-mon[101432]: osdmap e1164: 8 total, 8 up, 8 in 2024-09-21T10:23:25.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:25.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:25.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:25.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:25.617 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:25.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T10:23:25.642 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:25.692 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:25.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:25.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:25.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:25.892 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:25.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:25.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:25.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:26.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:26.048 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:23:26.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:26.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:26.200 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:25 smithi119 ceph-mon[105288]: pgmap v1569: 89 pgs: 1 creating+peering, 15 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:26.201 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:25 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:23:26.201 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:25 smithi119 ceph-mon[105288]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-21T10:23:26.201 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:25 smithi119 ceph-mon[101432]: pgmap v1569: 89 pgs: 1 creating+peering, 15 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:26.201 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:25 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:23:26.201 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:25 smithi119 ceph-mon[101432]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-21T10:23:26.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:26.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:26.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:25 smithi145 ceph-mon[89359]: pgmap v1569: 89 pgs: 1 creating+peering, 15 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:25 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:23:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:25 smithi145 ceph-mon[89359]: osdmap e1165: 8 total, 8 up, 8 in 2024-09-21T10:23:26.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:26.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:26.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:26.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:26.617 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:26.643 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:26.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:26.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:26.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:26.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:26.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:26.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:27.016 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:27.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:27.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:27.139 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:27.171 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:26 smithi119 ceph-mon[101432]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-21T10:23:27.171 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:26 smithi119 ceph-mon[105288]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-21T10:23:27.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:27.262 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:27.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:26 smithi145 ceph-mon[89359]: osdmap e1166: 8 total, 8 up, 8 in 2024-09-21T10:23:27.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:27.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:27.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:27.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:27.614 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:27.639 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:27.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:27.766 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:27.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:27.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:27.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:27.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:27.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:28.008 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:28.008 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:28.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:27 smithi119 ceph-mon[101432]: pgmap v1572: 89 pgs: 2 creating+activating, 1 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:27 smithi119 ceph-mon[105288]: pgmap v1572: 89 pgs: 2 creating+activating, 1 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:28.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:28.140 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:28.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:28.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:28.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:27 smithi145 ceph-mon[89359]: pgmap v1572: 89 pgs: 2 creating+activating, 1 creating+peering, 13 unknown, 73 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:28.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:28.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:28.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:28.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:28.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:28.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:28.697 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:28.697 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:28.701 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:28.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:28.760 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:28.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:28.883 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:28.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:28.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:28.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:29.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:29.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:29.134 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:29.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:29.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:29.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:29.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:29.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:29.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:29.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:29.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:29.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.762 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:29.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:29.863 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:29.887 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:29.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:29.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:23:29.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:30.012 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:30.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:30.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:30.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:29 smithi119 ceph-mon[101432]: pgmap v1573: 89 pgs: 2 creating+activating, 1 creating+peering, 86 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:30.178 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:29 smithi119 ceph-mon[105288]: pgmap v1573: 89 pgs: 2 creating+activating, 1 creating+peering, 86 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:30.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:29 smithi145 ceph-mon[89359]: pgmap v1573: 89 pgs: 2 creating+activating, 1 creating+peering, 86 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:30.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.263 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:30.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:30.389 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:30.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:30.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:23:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:23:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:30.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:30.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:30.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:30.766 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:30.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:30.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:30.888 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:30.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:30.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:31.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:31.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:31.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:31.262 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:31.363 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:31.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:31.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:31.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:31.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:31.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:31.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:31.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:31.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:31.888 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:31.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:31.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:31.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:32.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:32.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:32.138 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:32.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:32.260 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:31 smithi145 ceph-mon[89359]: pgmap v1574: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:32.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:31 smithi119 ceph-mon[101432]: pgmap v1574: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:31 smithi119 ceph-mon[105288]: pgmap v1574: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:32.383 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:32.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:32.485 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:32.506 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:32.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:32.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:32.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:32.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:32.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:32.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:32.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:32.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:33.007 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:33.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:33.109 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:33.133 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:33.235 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:33.259 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:33.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:33.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:33.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:33.504 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:33.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:23:33.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:33.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:33.856 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:33.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:33.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:33.885 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:23:33.886 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:33.903 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:23:33.903 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:33.922 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:23:33.922 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:33.939 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:23:33.939 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:33.955 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:23:33.955 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:33.971 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:23:33.971 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[105288]: pgmap v1575: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/720705783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[101432]: pgmap v1575: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:33.972 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/720705783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:23:33.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:33.987 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:23:33.988 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:34.003 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:23:34.003 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.010 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:34.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:33 smithi145 ceph-mon[89359]: pgmap v1575: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/720705783' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:23:34.111 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:34.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:34.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:34.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:34.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:34.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:34.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:34.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:34.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:34.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:23:34.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:34.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:34.734 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T10:23:34.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:34.778 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:34.841 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:23:34] "GET /metrics HTTP/1.1" 200 36697 "" "Prometheus/2.51.0" 2024-09-21T10:23:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/720705783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:23:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[101432]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-21T10:23:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/935356253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:34.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2561442818' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/720705783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:23:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[105288]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-21T10:23:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/935356253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:34.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2561442818' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:34.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:34.901 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:34.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:35.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.024 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/720705783' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:23:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:34 smithi145 ceph-mon[89359]: osdmap e1167: 8 total, 8 up, 8 in 2024-09-21T10:23:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/935356253' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2561442818' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:23:35.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:35.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:35.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:35.274 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:35.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:35.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:35.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:35.525 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:35.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:35.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:35.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:35.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:35.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:35.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:35.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:36.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:36.023 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2561442818' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:23:36.023 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:35 smithi119 ceph-mon[105288]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-21T10:23:36.023 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:35 smithi119 ceph-mon[105288]: pgmap v1578: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:36.023 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2561442818' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:23:36.023 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:35 smithi119 ceph-mon[101432]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-21T10:23:36.023 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:35 smithi119 ceph-mon[101432]: pgmap v1578: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:36.026 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:36.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2561442818' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:23:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:35 smithi145 ceph-mon[89359]: osdmap e1168: 8 total, 8 up, 8 in 2024-09-21T10:23:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:35 smithi145 ceph-mon[89359]: pgmap v1578: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:36.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:36.154 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:36.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:36.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:36.379 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:36.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:36.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:36.532 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:36.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:36.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:36.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:36.739 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:23:36.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:36.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:36.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:36.901 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:36.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:36.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:37.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:36 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:23:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/329151757' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:37.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:36 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:23:37.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/329151757' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:37.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:36 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:23:37.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/329151757' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:23:37.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:37.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:37.248 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:37.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:23:37.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:37.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:37.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:37.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.469 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-21T10:23:37.471 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:23:37.471 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:23:37.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:37.518 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:hints = 1 2024-09-21T10:23:37.518 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:23:37.518 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object prefix: benchmark_data_smithi119_168706 2024-09-21T10:23:37.518 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:37.519 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:23:37.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:37.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:37.667 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:37.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:37.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:37.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:37.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:37.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:37.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:37.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:37.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:38.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:37 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/329151757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:37 smithi145 ceph-mon[89359]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-21T10:23:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:37 smithi145 ceph-mon[89359]: pgmap v1580: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:38.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:38.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/329151757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:38.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[105288]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-21T10:23:38.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[105288]: pgmap v1580: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:38.055 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:38.055 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/329151757' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:23:38.055 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[101432]: osdmap e1169: 8 total, 8 up, 8 in 2024-09-21T10:23:38.055 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:37 smithi119 ceph-mon[101432]: pgmap v1580: 89 pgs: 89 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-21T10:23:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.075 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:38.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:38.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:38.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:38.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:38.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:38.448 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:38.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:38.519 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 1 16 3989 3973 248.235 248.312 0.00439069 0.00401775 2024-09-21T10:23:38.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:38.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:38.649 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:38.692 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:38.715 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:23:38.715 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:23:38.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:23:38.715 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45608.0:1755 43.12 e932/865 43:4835ff5c:::rbd_data.b2288854339b.0000000000000000:head v 932'18, mlcod=921'16)", 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:15:38.011275+0000", 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 480.697670426, 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.032075781999999997, 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.716 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.011275+0000", 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.011275+0000", 2024-09-21T10:23:38.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.011277+0000", 2024-09-21T10:23:38.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8619999999999999e-06 2024-09-21T10:23:38.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.015053+0000", 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037754680000000001 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.015086+0000", 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3387999999999999e-05 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.015104+0000", 2024-09-21T10:23:38.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8454e-05 2024-09-21T10:23:38.721 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.721 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.721 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.721 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.015239+0000", 2024-09-21T10:23:38.721 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013458099999999999 2024-09-21T10:23:38.721 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.722 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.722 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.722 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.015274+0000", 2024-09-21T10:23:38.722 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4604999999999998e-05 2024-09-21T10:23:38.722 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.722 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.043305+0000", 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.028031098000000001 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.723 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:15:38.043351+0000", 2024-09-21T10:23:38.724 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6326000000000003e-05 2024-09-21T10:23:38.724 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.724 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.724 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.724 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45614.0:3151 43.e e970/865 43:742cb3af:::rbd_data.b22e8cd911aa.000000000000004b:head v 970'42, mlcod=961'40)", 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:17:11.297792+0000", 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 387.41115401500002, 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.036741121000000002, 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.725 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.726 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.726 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.297792+0000", 2024-09-21T10:23:38.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.727 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.297792+0000", 2024-09-21T10:23:38.727 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.297792+0000", 2024-09-21T10:23:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3700000000000001e-07 2024-09-21T10:23:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.728 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.306393+0000", 2024-09-21T10:23:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086006069999999997 2024-09-21T10:23:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.729 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.306447+0000", 2024-09-21T10:23:38.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4033000000000003e-05 2024-09-21T10:23:38.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.730 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.306466+0000", 2024-09-21T10:23:38.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9582999999999999e-05 2024-09-21T10:23:38.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.306572+0000", 2024-09-21T10:23:38.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105553 2024-09-21T10:23:38.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.306597+0000", 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5162000000000001e-05 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.334487+0000", 2024-09-21T10:23:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.027889793 2024-09-21T10:23:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.334533+0000", 2024-09-21T10:23:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6053000000000001e-05 2024-09-21T10:23:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45614.0:3169 43.a e970/865 43:5129fa4d:::rbd_data.b22e8cd911aa.000000000000005b:head v 970'29, mlcod=969'26)", 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:17:11.346961+0000", 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 387.36198468100002, 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.054940309, 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.735 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.346961+0000", 2024-09-21T10:23:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.346961+0000", 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.737 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.346963+0000", 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3819999999999998e-06 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.400132+0000", 2024-09-21T10:23:38.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.053169109999999999 2024-09-21T10:23:38.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.400162+0000", 2024-09-21T10:23:38.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.995e-05 2024-09-21T10:23:38.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.400176+0000", 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3636e-05 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.400297+0000", 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120827 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.400312+0000", 2024-09-21T10:23:38.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5012e-05 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.401866+0000", 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015537280000000001 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:11.401901+0000", 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5664000000000003e-05 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.743 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35751.0:1102 44.13 44:c8c4316d:::rbd_data.8ba7ec212ae1.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1078)", 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:19:41.329797+0000", 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 237.379149095, 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.038760032999999999, 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35751", 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1503148914", 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1102 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.329797+0000", 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.329797+0000", 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.329801+0000", 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6909999999999996e-06 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.333487+0000", 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003685841 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.333525+0000", 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8050999999999999e-05 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.333539+0000", 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3570000000000001e-05 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.333649+0000", 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109985 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.333829+0000", 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017997999999999999 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.334008+0000", 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017969500000000001 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.353153+0000", 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.019144985999999999 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.366175+0000", 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013021741999999999 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.368362+0000", 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021868299999999999 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.368462+0000", 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100418 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:19:41.368557+0000", 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4244000000000001e-05 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:1416 44.4 44:273ecde0:::rbd_data.b240926faa55.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1099)", 2024-09-21T10:23:38.749 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:20:48.178343+0000", 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 170.53060244100001, 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.038581004000000002, 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1416 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.178343+0000", 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.178343+0000", 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.178345+0000", 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.691e-06 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.180952+0000", 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026073390000000002 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.180977+0000", 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5256999999999999e-05 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.180989+0000", 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1347e-05 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.181061+0000", 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2522999999999997e-05 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.181180+0000", 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118758 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.181277+0000", 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7102000000000006e-05 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.192810+0000", 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011533175 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216809+0000", 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023999115000000001 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216858+0000", 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8074000000000001e-05 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216896+0000", 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8216999999999997e-05 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216924+0000", 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8405999999999999e-05 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:1417 44.10 44:0e3bb076:::rbd_data.b240926faa55.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1099)", 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:20:48.181019+0000", 2024-09-21T10:23:38.755 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 170.527926248, 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.035897256000000002, 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1417 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.181019+0000", 2024-09-21T10:23:38.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.181019+0000", 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.181021+0000", 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7859999999999999e-06 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.183433+0000", 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024114119999999999 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.183454+0000", 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1319e-05 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.183465+0000", 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0706999999999999e-05 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.183534+0000", 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9193e-05 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.183644+0000", 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110102 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.183744+0000", 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010005099999999999 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.192955+0000", 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092114629999999996 2024-09-21T10:23:38.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.194807+0000", 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018517130000000001 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216833+0000", 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.022026006000000001 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216884+0000", 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0748999999999998e-05 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:48.216917+0000", 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2755000000000001e-05 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:1534 44.10 44:0f1fa67a:::rbd_data.b240b29ae1d4.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1102)", 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:20:58.139781+0000", 2024-09-21T10:23:38.763 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 160.56916494000001, 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.034373090000000002, 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1534 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.139781+0000", 2024-09-21T10:23:38.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.139781+0000", 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.139783+0000", 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4679999999999999e-06 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.765 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.142321+0000", 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025380540000000001 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.142348+0000", 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7035000000000001e-05 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.142359+0000", 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0332999999999999e-05 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.142428+0000", 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9386000000000003e-05 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.142547+0000", 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011955000000000001 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.142675+0000", 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000127384 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.152702+0000", 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010027154 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.153101+0000", 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00039930800000000002 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.174074+0000", 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020972828999999998 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.174118+0000", 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4069000000000001e-05 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:20:58.174154+0000", 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5519999999999999e-05 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:1648 44.f 44:f7d7eab2:::rbd_data.b240cbecc4a7.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-21T10:23:38.769 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:08.208041+0000", 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 150.500904421, 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.049887857000000001, 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1648 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.208041+0000", 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.208041+0000", 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.208044+0000", 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0410000000000002e-06 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210788+0000", 2024-09-21T10:23:38.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027439610000000001 2024-09-21T10:23:38.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210810+0000", 2024-09-21T10:23:38.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1423000000000001e-05 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210820+0000", 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0200999999999999e-05 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210917+0000", 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7424999999999996e-05 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.211094+0000", 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017717899999999999 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.211254+0000", 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015925499999999999 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.220724+0000", 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0094708180000000006 2024-09-21T10:23:38.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257805+0000", 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.037080163999999999 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257842+0000", 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7450000000000002e-05 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257890+0000", 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7547999999999999e-05 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257929+0000", 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9391999999999998e-05 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2156 44.4 44:24b81b71:::rbd_data.b240933cb63.0000000000000008:head [write 0~3145785] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1117)", 2024-09-21T10:23:38.779 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:20.168538+0000", 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 138.54040729799999, 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.044225157000000001, 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.780 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2156 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.168538+0000", 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.168538+0000", 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.168540+0000", 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8679999999999999e-06 2024-09-21T10:23:38.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.170884+0000", 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023441619999999999 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.170908+0000", 2024-09-21T10:23:38.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3813000000000001e-05 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.170920+0000", 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1741e-05 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.171001+0000", 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0963000000000004e-05 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.171172+0000", 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000170996 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.171345+0000", 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017326099999999999 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.176740+0000", 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0053949510000000003 2024-09-21T10:23:38.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.186370+0000", 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0096295619999999995 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.212682+0000", 2024-09-21T10:23:38.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.026312001000000002 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.212721+0000", 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9607999999999998e-05 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:20.212763+0000", 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2231000000000003e-05 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2320 44.10 44:0bbd273a:::rbd_data.b240933cb63.000000000000002d:head [set-alloc-hint object_size 4194304 write_size 4194304,write 380099~3814205] snapc 69=[69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1120)", 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:23.111761+0000", 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 135.59718495999999, 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.043243108000000002, 2024-09-21T10:23:38.789 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2320 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.111761+0000", 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.111761+0000", 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.111763+0000", 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.39e-06 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114381+0000", 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002618418 2024-09-21T10:23:38.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114413+0000", 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1699000000000003e-05 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114424+0000", 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0808999999999999e-05 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114495+0000", 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1407000000000003e-05 2024-09-21T10:23:38.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114502+0000", 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0990000000000001e-06 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119585+0000", 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0050828180000000002 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119718+0000", 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000133072 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119827+0000", 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108393 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.140794+0000", 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020967528999999999 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.146672+0000", 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0058774209999999999 2024-09-21T10:23:38.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.154921+0000", 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082497899999999999 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.154965+0000", 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3294000000000003e-05 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155004+0000", 2024-09-21T10:23:38.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8968999999999999e-05 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2323 44.f 44:f55ffb94:::rbd_data.b240933cb63.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 69=[69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1120)", 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:23.115410+0000", 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 135.59353534100001, 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.039684070000000002, 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2323 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.115410+0000", 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.115410+0000", 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.115412+0000", 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0849999999999999e-06 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119085+0000", 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036723319999999999 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119100+0000", 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5087e-05 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119110+0000", 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0587999999999999e-05 2024-09-21T10:23:38.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119773+0000", 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066215800000000004 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.119892+0000", 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119338 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.120008+0000", 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000115683 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.140619+0000", 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.02061143 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.154433+0000", 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013813570000000001 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155004+0000", 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057192199999999999 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155055+0000", 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0936000000000002e-05 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155094+0000", 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8940999999999999e-05 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.802 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2646 44.10 44:0bbd273a:::rbd_data.b240933cb63.000000000000002d:head [write 2198666~1995638] snapc 6f=[6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1126)", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:29.160600+0000", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 129.54834514000001, 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.032203211000000002, 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2646 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.160600+0000", 2024-09-21T10:23:38.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.160600+0000", 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.160603+0000", 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6529999999999998e-06 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.162292+0000", 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001688625 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.162318+0000", 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5800999999999999e-05 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.162327+0000", 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8530000000000004e-06 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.162396+0000", 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8406999999999997e-05 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.162404+0000", 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5099999999999998e-06 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.164082+0000", 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016777770000000001 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.164176+0000", 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3441000000000002e-05 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.164339+0000", 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016322400000000001 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.170896+0000", 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0065568880000000003 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.170958+0000", 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2805000000000006e-05 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.192727+0000", 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021768852000000002 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.192767+0000", 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0083999999999998e-05 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:29.192804+0000", 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6291000000000002e-05 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2768 44.13 44:cc678c9c:::rbd_data.b240933cb63.000000000000004e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 71=[71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:31.230320+0000", 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 127.478625257, 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.042979406999999997, 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.809 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2768 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.230320+0000", 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.230320+0000", 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.230322+0000", 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1390000000000002e-06 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.232040+0000", 2024-09-21T10:23:38.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001717681 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.232072+0000", 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1962999999999999e-05 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.232088+0000", 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5999e-05 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.232176+0000", 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7392000000000006e-05 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.232360+0000", 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018442099999999999 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.232545+0000", 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000185107 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.252243+0000", 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.019697745999999999 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.254877+0000", 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026340729999999998 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.273178+0000", 2024-09-21T10:23:38.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018301454000000002 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.273248+0000", 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9255e-05 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.273300+0000", 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2176999999999999e-05 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.815 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2843 44.f 44:f197757e:::rbd_data.b240933cb63.000000000000003f:head [set-alloc-hint object_size 4194304 write_size 4194304,write 2310724~1883580] snapc 72=[72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1129)", 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:32.219212+0000", 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 126.48973349000001, 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.034706727999999999, 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.816 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2843 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.219212+0000", 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.219212+0000", 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.219216+0000", 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.63e-06 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.220091+0000", 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00087489399999999998 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.220121+0000", 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0429e-05 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.220140+0000", 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8851e-05 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.220264+0000", 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000124444 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.220450+0000", 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018605499999999999 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.220706+0000", 2024-09-21T10:23:38.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000255677 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.234427+0000", 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013720615 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.252698+0000", 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01827092 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.253809+0000", 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011114619999999999 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.253876+0000", 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7037000000000002e-05 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:32.253919+0000", 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2713999999999998e-05 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.822 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2947 44.f 44:f46d050d:::rbd_data.b240933cb63.0000000000000009:head [write 724909~3469395] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1131)", 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:34.242509+0000", 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 124.466436562, 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.033994824999999999, 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.823 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2947 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.242509+0000", 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.242509+0000", 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.242511+0000", 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0509999999999999e-06 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.244369+0000", 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001857638 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.244408+0000", 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8801999999999999e-05 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.244425+0000", 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7603000000000001e-05 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.244556+0000", 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013129799999999999 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.244670+0000", 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011322799999999999 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:23:38.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.244878+0000", 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020871 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.257407+0000", 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012528170999999999 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.275273+0000", 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017866432000000002 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.276389+0000", 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001116515 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.276451+0000", 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1827000000000003e-05 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:34.276504+0000", 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2549999999999997e-05 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.828 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:3047 44.f 44:f404f2bb:::rbd_data.b240933cb63.000000000000004a:head [write 2071242~2123062] snapc 76=[76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1133)", 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:36.213450+0000", 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 122.49549525, 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.037783337, 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 3047 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.829 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.213450+0000", 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.213450+0000", 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.213454+0000", 2024-09-21T10:23:38.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4510000000000001e-06 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.215897+0000", 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002443481 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.215941+0000", 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3890999999999997e-05 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.215965+0000", 2024-09-21T10:23:38.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3958e-05 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.216080+0000", 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114926 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.216208+0000", 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000128283 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.216457+0000", 2024-09-21T10:23:38.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024825599999999998 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.234688+0000", 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018231859999999999 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.250878+0000", 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016189192000000002 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.251135+0000", 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025747399999999999 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.251177+0000", 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1508999999999998e-05 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:36.251234+0000", 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7055999999999997e-05 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:3158 44.13 44:cc678c9c:::rbd_data.b240933cb63.000000000000004e:head [write 315924~3878380] snapc 78=[78,77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1135)", 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:38.228148+0000", 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 120.48079801599999, 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.058965920999999998, 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 3158 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.228148+0000", 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.228148+0000", 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.228153+0000", 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0390000000000004e-06 2024-09-21T10:23:38.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.231119+0000", 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029659109999999999 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.231143+0000", 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3981999999999998e-05 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.231156+0000", 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3591000000000001e-05 2024-09-21T10:23:38.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.231250+0000", 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4062999999999997e-05 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.231382+0000", 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013138099999999999 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.231588+0000", 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020598099999999999 2024-09-21T10:23:38.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.240530+0000", 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00894292 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.283884+0000", 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.043353105000000003 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.287001+0000", 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031171219999999999 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.287065+0000", 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4344999999999995e-05 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:38.287114+0000", 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8480999999999999e-05 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.840 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:6938 44.4 44:210af00f:::rbd_data.b240bb5bb67f.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:00.492126+0000", 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 98.216819665000003, 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.059416573, 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 6938 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.841 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.492126+0000", 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.492126+0000", 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.842 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.492131+0000", 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1549999999999998e-06 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496520+0000", 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043887630000000004 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496549+0000", 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9312e-05 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496565+0000", 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5889e-05 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496652+0000", 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6938e-05 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496829+0000", 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017715499999999999 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:23:38.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.497024+0000", 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000194954 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.513047+0000", 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016023158999999999 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.528729+0000", 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015681380000000002 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551427+0000", 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.022698156000000001 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551497+0000", 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0636999999999995e-05 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551543+0000", 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5074999999999998e-05 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.846 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:6941 44.13 44:cd8a7da6:::rbd_data.b240bb5bb67f.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:00.496608+0000", 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 98.212337758999993, 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.055006207000000001, 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.847 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 6941 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496608+0000", 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496608+0000", 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496610+0000", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1449999999999998e-06 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.499449+0000", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028388979999999999 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.499478+0000", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9105e-05 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.499490+0000", 2024-09-21T10:23:38.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1647999999999999e-05 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.499567+0000", 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7669999999999993e-05 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.499718+0000", 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000150982 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:23:38.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.499885+0000", 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000166665 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.512852+0000", 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01296711 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551466+0000", 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.038613941999999998 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551519+0000", 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3186000000000002e-05 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551577+0000", 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7757999999999998e-05 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551614+0000", 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7098000000000003e-05 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:7358 44.f 44:f3b23f7a:::rbd_data.b240938dee57.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:03.550928+0000", 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 95.158017373000007, 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.045795776000000003, 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 7358 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.550928+0000", 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.550928+0000", 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.550932+0000", 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5769999999999998e-06 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553439+0000", 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025072430000000001 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553479+0000", 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0283e-05 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553499+0000", 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9843e-05 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553597+0000", 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7440000000000002e-05 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553819+0000", 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00022255899999999999 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,7", 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.554077+0000", 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025800200000000001 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.856 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.570635+0000", 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016557939000000001 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596611+0000", 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.025975669 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596651+0000", 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0092999999999997e-05 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596693+0000", 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1882999999999999e-05 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596724+0000", 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1245000000000003e-05 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:23:38.858 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:23:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:38.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:38.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:38.998 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:38.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:39.100 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:39.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:39.228 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:39.263 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:39.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:39.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:39.401 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:39.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:39.519 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 2 16 7357 7341 229.36 210.5 0.00354889 0.00435235 2024-09-21T10:23:39.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:39.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:39.667 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:39.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:39.803 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:39.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:39.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:39.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:39.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:40.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:40.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:40.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:40.209 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:40.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:23:40.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.358 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:40.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:40.436 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:40 smithi119 ceph-mon[101432]: pgmap v1581: 89 pgs: 89 active+clean; 179 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s 2024-09-21T10:23:40.436 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:40 smithi119 ceph-mon[105288]: pgmap v1581: 89 pgs: 89 active+clean; 179 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s 2024-09-21T10:23:40.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:23:40.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:40.519 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 3 16 10725 10709 223.04 210.5 0.00565101 0.00447175 2024-09-21T10:23:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:23:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:23:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:40 smithi145 ceph-mon[89359]: pgmap v1581: 89 pgs: 89 active+clean; 179 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 30 MiB/s wr, 474 op/s 2024-09-21T10:23:40.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:40.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:40.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:40.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:40.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:40.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:40.818 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:40.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:40.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:41.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:41.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:41.238 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:41.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:41.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:41.475 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:41.505 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:41.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:41.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 4 16 13812 13796 215.512 192.938 0.00284093 0.00463265 2024-09-21T10:23:41.607 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:41.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:41.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:41.773 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:41.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:41.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:42.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:42.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:42.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:42.178 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:42.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.317 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:42.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:42.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:42.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:42.467 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:42 smithi119 ceph-mon[105288]: pgmap v1582: 89 pgs: 89 active+clean; 280 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 39 MiB/s wr, 631 op/s 2024-09-21T10:23:42.467 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:42 smithi119 ceph-mon[101432]: pgmap v1582: 89 pgs: 89 active+clean; 280 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 39 MiB/s wr, 631 op/s 2024-09-21T10:23:42.519 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 5 16 16763 16747 209.293 184.438 0.00781582 0.00477101 2024-09-21T10:23:42.541 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:42 smithi145 ceph-mon[89359]: pgmap v1582: 89 pgs: 89 active+clean; 280 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 39 MiB/s wr, 631 op/s 2024-09-21T10:23:42.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:42.556 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:42.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:42.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:42.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:42.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:42.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:42.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:42.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:42.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:42.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:42.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:43.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:43.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:43.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:43.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:43.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:43.398 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:43.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:43.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 6 16 19592 19576 203.877 176.812 0.00602621 0.00489899 2024-09-21T10:23:43.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:43.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:43.676 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:43.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:43.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:43.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:43.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:43.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:44.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:44.085 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:44.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:44.224 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:44.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:44.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:44.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:44 smithi119 ceph-mon[105288]: pgmap v1583: 89 pgs: 89 active+clean; 406 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s 2024-09-21T10:23:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:44 smithi119 ceph-mon[101432]: pgmap v1583: 89 pgs: 89 active+clean; 406 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s 2024-09-21T10:23:44.421 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:23:44.421 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:23:44.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:44.462 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:23:44.462 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:23:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:44.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:44.506 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:23:44.506 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:23:44.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 7 16 22270 22254 198.662 167.375 0.00904677 0.00502641 2024-09-21T10:23:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:44 smithi145 ceph-mon[89359]: pgmap v1583: 89 pgs: 89 active+clean; 406 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 50 MiB/s wr, 804 op/s 2024-09-21T10:23:44.545 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:23:44.545 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:23:44.577 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:23:44.577 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:23:44.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:44.607 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:23:44.608 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:23:44.647 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:44.648 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:23:44.648 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:23:44.678 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:23:44.679 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:23:44.679 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:23:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:23:44] "GET /metrics HTTP/1.1" 200 36697 "" "Prometheus/2.51.0" 2024-09-21T10:23:44.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:44.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:44.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:44.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:44.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:45.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:45.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:45.160 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:45.179 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:23:45.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:45.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:45.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:45.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:45.442 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/422102061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:45.442 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3091041099' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:23:45.442 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:45 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:23:45.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/422102061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:45.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3091041099' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:23:45.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:45 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:23:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:45.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:45.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 8 15 24666 24651 192.554 149.812 0.00494539 0.00518807 2024-09-21T10:23:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/422102061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:23:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3091041099' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:23:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:45 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:23:45.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:45.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:45.737 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:45.839 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:45.873 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:45.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:45.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:45.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:46.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:46.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:46.149 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:46.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:46.287 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:46.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:46.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:46.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:46 smithi145 ceph-mon[89359]: pgmap v1584: 89 pgs: 89 active+clean; 1.1 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:23:46.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:46 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:23:46.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:46 smithi145 ceph-mon[89359]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-21T10:23:46.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 9 16 27183 27167 188.63 157.25 0.0063221 0.00529585 2024-09-21T10:23:46.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:46.559 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:46 smithi119 ceph-mon[105288]: pgmap v1584: 89 pgs: 89 active+clean; 1.1 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:23:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:46 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:23:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:46 smithi119 ceph-mon[105288]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-21T10:23:46.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:46 smithi119 ceph-mon[101432]: pgmap v1584: 89 pgs: 89 active+clean; 1.1 GiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 114 MiB/s wr, 1.82k op/s 2024-09-21T10:23:46.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:46 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:23:46.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:46 smithi119 ceph-mon[101432]: osdmap e1170: 8 total, 8 up, 8 in 2024-09-21T10:23:46.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:46.700 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:46.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:46.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:46.824 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:46.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:46.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:46.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:47.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:47.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:47.227 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:47.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.368 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:47.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.469 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:47.506 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:47.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 10 16 29379 29363 183.492 137.25 0.00317077 0.00544451 2024-09-21T10:23:47.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:47.645 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:47.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.779 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:47.881 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:47.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:47.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:48.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:48.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:48.154 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:48.187 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:48.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:48.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:48.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:48.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:48.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 11 16 31360 31344 178.066 123.812 0.00558122 0.00561127 2024-09-21T10:23:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:48 smithi145 ceph-mon[89359]: pgmap v1586: 89 pgs: 89 active+clean; 1.2 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T10:23:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:48.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:48.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:48 smithi119 ceph-mon[105288]: pgmap v1586: 89 pgs: 89 active+clean; 1.2 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T10:23:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:48 smithi119 ceph-mon[101432]: pgmap v1586: 89 pgs: 89 active+clean; 1.2 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 124 MiB/s wr, 1.99k op/s 2024-09-21T10:23:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:23:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:23:48.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:48.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:48.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:48.845 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:48.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:48.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:48.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:48.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:49.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:49.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:49.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:49.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:49.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:49.370 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:49.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:49.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:49.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 12 15 32675 32660 170.081 82.25 0.00861062 0.00587246 2024-09-21T10:23:49.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:49.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:49.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:49.758 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:49.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:49.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:49.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:49.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:50.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:50.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:50.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:50.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:50.285 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:50.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:50.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:50.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:50.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 13 16 34613 34597 166.309 121.062 0.0030597 0.00600766 2024-09-21T10:23:50.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:23:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:23:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:23:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:50 smithi145 ceph-mon[89359]: pgmap v1587: 89 pgs: 89 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-21T10:23:50.559 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:50.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:50 smithi119 ceph-mon[105288]: pgmap v1587: 89 pgs: 89 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-21T10:23:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:50 smithi119 ceph-mon[101432]: pgmap v1587: 89 pgs: 89 active+clean; 1.8 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 167 MiB/s wr, 2.67k op/s 2024-09-21T10:23:50.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:50.698 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:50.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:50.832 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:50.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:50.965 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:50.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:51.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:51.102 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:51.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:51.235 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:51.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:51.374 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:51.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:51.507 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:51.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 14 16 36827 36811 164.313 138.375 0.00810152 0.00608108 2024-09-21T10:23:51.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:51.640 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:51.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:51.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:51.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:51.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:51.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:52.004 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:52.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:52.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.302 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:52.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:52.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:52.441 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:52.499 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:52 smithi119 ceph-mon[105288]: pgmap v1588: 89 pgs: 89 active+clean; 1.9 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-09-21T10:23:52.499 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:52 smithi119 ceph-mon[101432]: pgmap v1588: 89 pgs: 89 active+clean; 1.9 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-09-21T10:23:52.520 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 15 16 38700 38684 161.163 117.062 0.0120676 0.00619941 2024-09-21T10:23:52.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:52 smithi145 ceph-mon[89359]: pgmap v1588: 89 pgs: 89 active+clean; 1.9 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 169 MiB/s wr, 2.70k op/s 2024-09-21T10:23:52.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:52.681 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:52.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:52.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:52.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:52.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:52.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:53.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:23:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:53.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:53.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:53.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:53.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:53.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:53.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:53.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:53.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 16 16 40717 40701 158.966 126.062 0.00793723 0.00628441 2024-09-21T10:23:53.552 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.585 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:53.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:53.686 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.722 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:53.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:53.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:53.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:53.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:53.986 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:53.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:54.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:54.137 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:54.221 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:54.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:54.279 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:54.351 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:54 smithi119 ceph-mon[105288]: pgmap v1589: 89 pgs: 89 active+clean; 2.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T10:23:54.351 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:54 smithi119 ceph-mon[101432]: pgmap v1589: 89 pgs: 89 active+clean; 2.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T10:23:54.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:54.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:23:54.416 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:54.518 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:54.522 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 17 16 42716 42700 156.964 124.938 0.00684542 0.00636523 2024-09-21T10:23:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:54 smithi145 ceph-mon[89359]: pgmap v1589: 89 pgs: 89 active+clean; 2.0 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 163 MiB/s wr, 2.61k op/s 2024-09-21T10:23:54.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:54.612 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:23:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:23:54] "GET /metrics HTTP/1.1" 200 36759 "" "Prometheus/2.51.0" 2024-09-21T10:23:54.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.767 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:54.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:54.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:54.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:54.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:54.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:55.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:55.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.051 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:55.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:55.051 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:23:55.052 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:23:55.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:55.191 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:55.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:55.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:55.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:55.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:55.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 18 16 44484 44468 154.382 110.5 0.00640745 0.00647161 2024-09-21T10:23:55.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:55.598 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:55.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:55.732 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:55.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:55.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:55.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:55.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:55.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:56.000 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:56.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:56.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:56.218 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:23:56.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:56.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:56.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:56.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:56.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 19 16 46460 46444 152.756 123.5 0.0104316 0.00654194 2024-09-21T10:23:56.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:56 smithi145 ceph-mon[89359]: pgmap v1590: 89 pgs: 89 active+clean; 2.4 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-21T10:23:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3355214038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:56.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:56.564 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:56 smithi119 ceph-mon[101432]: pgmap v1590: 89 pgs: 89 active+clean; 2.4 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-21T10:23:56.564 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3355214038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:56.564 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:56 smithi119 ceph-mon[105288]: pgmap v1590: 89 pgs: 89 active+clean; 2.4 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 135 MiB/s wr, 2.16k op/s 2024-09-21T10:23:56.564 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3355214038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:23:56.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:23:56.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:56.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:56.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:56.840 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:56.941 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:23:56.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:56.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:23:57.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:57.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:57.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.250 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:57.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:57.350 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:57.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3355214038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:23:57.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[101432]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-21T10:23:57.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3355214038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:23:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[105288]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-21T10:23:57.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:57 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:57.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:57.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:57.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:23:57.520410+0000 min lat: 0.00116449 max lat: 0.167858 avg lat: 0.00663804 2024-09-21T10:23:57.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:23:57.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 20 16 48191 48175 150.526 108.188 0.00621441 0.00663804 2024-09-21T10:23:57.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:57 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:23:57.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3355214038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:23:57.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:57 smithi145 ceph-mon[89359]: osdmap e1171: 8 total, 8 up, 8 in 2024-09-21T10:23:57.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:57 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:23:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.546 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:57.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:57.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.686 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:57.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:57.745 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:23:57.787 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:57.818 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:57.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:57.966 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:57.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:58.067 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:58.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:58.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:58.228 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:58.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:58.332 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:58 smithi119 ceph-mon[105288]: pgmap v1592: 89 pgs: 89 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-21T10:23:58.333 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:58 smithi119 ceph-mon[101432]: pgmap v1592: 89 pgs: 89 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-21T10:23:58.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:58.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:58.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:23:58.487 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:58 smithi145 ceph-mon[89359]: pgmap v1592: 89 pgs: 89 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 132 MiB/s wr, 2.11k op/s 2024-09-21T10:23:58.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:23:58.521 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 21 16 49885 49869 148.399 105.875 0.00872034 0.00673345 2024-09-21T10:23:58.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:23:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:23:58.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.790 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:58.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:58.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:23:58.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:58.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:23:59.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:59.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:59.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:23:59.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:23:59.258 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:23:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2623566789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:23:59 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:59.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:59.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:59.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:23:59.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:59.458 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2623566789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:59.458 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:23:59 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:59.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2623566789' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:59.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:23:59 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:23:59.464 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:23:59.522 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 22 16 51613 51597 146.562 108 0.00570381 0.00681771 2024-09-21T10:23:59.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:23:59.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:23:59.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:23:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:23:59.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:23:59.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:23:59.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:23:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:23:59.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:00.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:00.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:00.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:00.201 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:00.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:24:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:00 smithi145 ceph-mon[89359]: pgmap v1593: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:24:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:00 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:24:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:00 smithi145 ceph-mon[89359]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-21T10:24:00.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:24:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:24:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:00.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:00.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:00.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:00.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:00.492 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:00 smithi119 ceph-mon[101432]: pgmap v1593: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:24:00.492 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:00 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:24:00.492 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:00 smithi119 ceph-mon[101432]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-21T10:24:00.492 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:00 smithi119 ceph-mon[105288]: pgmap v1593: 89 pgs: 89 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:24:00.492 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:00 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:24:00.492 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:00 smithi119 ceph-mon[105288]: osdmap e1172: 8 total, 8 up, 8 in 2024-09-21T10:24:00.525 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 23 16 53284 53268 144.711 104.438 0.00956029 0.0069024 2024-09-21T10:24:00.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.631 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:00.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:00.732 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:00.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:00.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:00.866 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:24:00.867 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:00.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:00.912 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:00.912 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:00.914 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:24:00.914 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:00.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:00.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:00.957 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:24:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:00.994 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:24:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:01.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:01.032 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:24:01.032 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:01.066 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:24:01.066 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:01.074 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:01.096 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:24:01.096 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:01.128 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:24:01.129 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:01.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:01.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:01.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:01.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:01 smithi119 ceph-mon[105288]: pgmap v1595: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:24:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/904772530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:01 smithi119 ceph-mon[101432]: pgmap v1595: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:24:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/904772530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:01.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:01.494 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:01.525 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 24 16 54822 54806 142.686 96.125 0.0184075 0.0070033 2024-09-21T10:24:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:01 smithi145 ceph-mon[89359]: pgmap v1595: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:24:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/904772530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:01.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:01.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:01.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:01.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:01.874 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:01.903 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:01.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:02.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:02.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:02.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:02.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:02.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:02.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:02.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:02.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:02.525 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 25 16 56530 56514 141.248 106.75 0.00340701 0.00707434 2024-09-21T10:24:02.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:02.583 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:02.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:02.723 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:02.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:02.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:02.961 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:02.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:02.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:03.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:03.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:03.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:03.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:03.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:03.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:03.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:03.413 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:03.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:03.525 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 26 16 58208 58192 139.849 104.875 0.00927048 0.00714521 2024-09-21T10:24:03.543 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:03.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:03.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:03.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:03.823 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:03.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:03.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:03.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:03.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:24:04.039 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:03 smithi119 ceph-mon[105288]: pgmap v1596: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:24:04.040 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:03 smithi119 ceph-mon[101432]: pgmap v1596: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:24:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:03 smithi145 ceph-mon[89359]: pgmap v1596: 89 pgs: 89 active+clean; 3.1 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:24:04.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:04.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:04.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:04.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:04.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:04.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:04.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:04.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:04.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:04.528 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 27 16 59651 59635 138.007 90.1875 0.00842615 0.00724048 2024-09-21T10:24:04.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:04.599 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:04.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:04.719 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:24:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:24:04] "GET /metrics HTTP/1.1" 200 36760 "" "Prometheus/2.51.0" 2024-09-21T10:24:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:04.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:04.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:04.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:04.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:04.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.039 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:05.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:05.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:05.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:05.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:05.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:05.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:05.442 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:05.526 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 28 16 61018 61002 136.13 85.4375 0.00555714 0.00733997 2024-09-21T10:24:05.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:05.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:05.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:05.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:05.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:05.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:06.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:05 smithi119 ceph-mon[101432]: pgmap v1597: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-21T10:24:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:05 smithi119 ceph-mon[105288]: pgmap v1597: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-21T10:24:06.134 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:06.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:06.269 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:06.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:06.283 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:24:06.283 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:24:06.283 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:24:06.283 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:24:06.283 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:24:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:05 smithi145 ceph-mon[89359]: pgmap v1597: 89 pgs: 89 active+clean; 3.5 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 120 MiB/s wr, 1.91k op/s 2024-09-21T10:24:06.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:06.411 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:06.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:06.526 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 29 16 62439 62423 134.498 88.8125 0.00913785 0.00743049 2024-09-21T10:24:06.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:06.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:06.677 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:06.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:06.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:06.814 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:06.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:06.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:06.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:07.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:07.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:07.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:07.225 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:07.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:07.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:07.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:07.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:07.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:07.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:07.526 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 30 16 64009 63993 133.285 98.125 0.0112589 0.00749783 2024-09-21T10:24:07.598 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:07.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:07.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:07.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:07.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:07.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:07.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:08.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:08.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:08.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:08.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:07 smithi145 ceph-mon[89359]: pgmap v1598: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-21T10:24:08.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:07 smithi119 ceph-mon[101432]: pgmap v1598: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-21T10:24:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:07 smithi119 ceph-mon[105288]: pgmap v1598: 89 pgs: 89 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-21T10:24:08.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:08.419 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:08.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:08.526 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 31 16 65260 65244 131.508 78.1875 0.00701528 0.00759951 2024-09-21T10:24:08.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:08.655 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:08.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:08.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:08.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:08.958 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:08.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:09.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:09.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:09.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:09.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:09.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:09.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:09.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:09.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:09.466 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:09.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:09.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:24:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:09.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 32 16 66736 66720 130.279 92.25 0.00899068 0.00767107 2024-09-21T10:24:09.618 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:09.643 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:09.745 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:09.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:09.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:09.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:09.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:09.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:10.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:09 smithi119 ceph-mon[101432]: pgmap v1599: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:24:10.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:09 smithi119 ceph-mon[105288]: pgmap v1599: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:24:10.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:10.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:09 smithi145 ceph-mon[89359]: pgmap v1599: 89 pgs: 89 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:24:10.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:10.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:10.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:10.418 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:10.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:10.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 33 16 68074 68058 128.865 83.625 0.0484985 0.00775383 2024-09-21T10:24:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:24:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:24:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:10.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:10.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:10.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:10.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:10.829 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:10.902 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:24:10.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:10.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:10.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:11.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.093 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:11.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:11.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:11.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3720447045' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:11.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3720447045' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3720447045' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.368 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:11.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:11.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:11.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:11.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 34 16 69480 69464 127.66 87.875 0.00938884 0.00782802 2024-09-21T10:24:11.600 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.622 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:11.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:11.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.755 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:11.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:11.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:11.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:12.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:12.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:12.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[105288]: pgmap v1600: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:24:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:24:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[105288]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-21T10:24:12.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[101432]: pgmap v1600: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:24:12.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:12.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:24:12.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:12.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:11 smithi119 ceph-mon[101432]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-21T10:24:12.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:12.178 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:12.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:11 smithi145 ceph-mon[89359]: pgmap v1600: 89 pgs: 89 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-21T10:24:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:11 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:24:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:11 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:24:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:11 smithi145 ceph-mon[89359]: osdmap e1173: 8 total, 8 up, 8 in 2024-09-21T10:24:12.317 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:12.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:12.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:12.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 35 16 70857 70841 126.471 86.0625 0.00775814 0.0079022 2024-09-21T10:24:12.555 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:12.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:12.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:12.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:12.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:12.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:12.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:12.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:12.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:12.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:13.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:13.134 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:13.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:13.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:13.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:13.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:13.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:13.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:13.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 36 16 72349 72333 125.548 93.25 0.00956485 0.00796031 2024-09-21T10:24:13.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:13.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:13.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:13.799 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:13.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:13.930 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:13.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:14.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:14.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:14.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:13 smithi119 ceph-mon[105288]: pgmap v1602: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-09-21T10:24:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:13 smithi119 ceph-mon[101432]: pgmap v1602: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-09-21T10:24:14.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:14.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:13 smithi145 ceph-mon[89359]: pgmap v1602: 89 pgs: 89 active+clean; 4.1 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 95 MiB/s wr, 1.53k op/s 2024-09-21T10:24:14.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:14.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:14.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:14.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:14.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 37 16 73782 73766 124.575 89.5625 0.00978711 0.00802186 2024-09-21T10:24:14.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:14.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:14.688 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:14.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:14.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.778 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:14.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:14.815 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:24:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:24:14] "GET /metrics HTTP/1.1" 200 36760 "" "Prometheus/2.51.0" 2024-09-21T10:24:14.879 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:14.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:14.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:15.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:15.042 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:15.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:15.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:15.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:15.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:15.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:15.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:15.425 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:15.470 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:15.527 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 38 16 75068 75052 123.412 80.375 0.0211597 0.00809821 2024-09-21T10:24:15.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:15.607 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:15.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:15.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:24:15.749 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:15.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:15.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:15.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:15.913 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:24:15.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:16.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:16.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:16.275 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:15 smithi119 ceph-mon[105288]: pgmap v1603: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:24:16.275 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1490259253' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:24:16.275 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:15 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:24:16.275 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:15 smithi119 ceph-mon[101432]: pgmap v1603: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:24:16.275 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1490259253' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:24:16.275 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:15 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:24:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:15 smithi145 ceph-mon[89359]: pgmap v1603: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-21T10:24:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1490259253' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:24:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:15 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:24:16.297 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:16.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:16.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:16.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:16.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 39 16 76302 76286 122.222 77.125 0.0212967 0.00817634 2024-09-21T10:24:16.576 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.610 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:16.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:16.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:16.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:24:16.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:16.877 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:16.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:16.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:16.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:16 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:24:16.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:16 smithi145 ceph-mon[89359]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-21T10:24:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:16 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:24:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:16 smithi119 ceph-mon[101432]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-21T10:24:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:16 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:24:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:16 smithi119 ceph-mon[105288]: osdmap e1174: 8 total, 8 up, 8 in 2024-09-21T10:24:17.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:17.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:17.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:17.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:17.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:17.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:17.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:24:17.527843+0000 min lat: 0.00116449 max lat: 0.167858 avg lat: 0.00824434 2024-09-21T10:24:17.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:24:17.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 40 16 77615 77599 121.218 82.0625 0.00759318 0.00824434 2024-09-21T10:24:17.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:17.573 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:17.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:17.692 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:17.700 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:17.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:17.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:17.836 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:17.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:17.991 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:17.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:18.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:18.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.133 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:18.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:18.185 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:17 smithi119 ceph-mon[105288]: pgmap v1605: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:24:18.185 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:18.185 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:18.185 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:17 smithi119 ceph-mon[101432]: pgmap v1605: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:24:18.185 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:18.185 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:18.235 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:18.267 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:17 smithi145 ceph-mon[89359]: pgmap v1605: 89 pgs: 89 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:24:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:18.369 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:18.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:18.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:18.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 41 16 78875 78859 120.181 78.75 0.00408892 0.00831544 2024-09-21T10:24:18.558 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:18.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:18.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:18.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:18.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:18.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:18.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:18.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:19.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.112 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:19.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:19.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.347 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:24:19.347 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45614.0:3352 43.13 43:cfb4d045:::rbd_object_map.b22e8cd911aa:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:17:14.314808+0000", 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 424.97998621800002, 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.050931070000000002, 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45614", 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:24:19.348 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 3352 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.314808+0000", 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.314808+0000", 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.314809+0000", 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000001e-07 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.314813+0000", 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6610000000000002e-06 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.314820+0000", 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1149999999999996e-06 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.314825+0000", 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.558e-06 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.321175+0000", 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063493949999999999 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.321281+0000", 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0001066 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.321603+0000", 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032203199999999998 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.323595+0000", 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019917569999999998 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.323791+0000", 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000196095 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.365664+0000", 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.041872580999999999 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.365706+0000", 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2150999999999997e-05 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:17:14.365739+0000", 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3447e-05 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.353 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:1649 44.e 44:746c80ec:::rbd_data.b240cbecc4a7.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:08.208094+0000", 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 191.086700681, 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.049813002000000002, 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.354 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 1649 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.208094+0000", 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.208094+0000", 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.208096+0000", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.855e-06 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210720+0000", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026242129999999998 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210749+0000", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9234000000000002e-05 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210761+0000", 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1990999999999999e-05 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.210843+0000", 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1471000000000007e-05 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.211001+0000", 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000158481 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.211099+0000", 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7584999999999995e-05 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.220590+0000", 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0094909339999999995 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257797+0000", 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.037207100999999999 2024-09-21T10:24:19.358 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257825+0000", 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8581999999999998e-05 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257874+0000", 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8550000000000001e-05 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:08.257907+0000", 2024-09-21T10:24:19.359 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3005e-05 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45632.0:2208 44.b e1118/1014 44:d7c5b9e3:::rbd_data.b240933cb63.0000000000000007:head v 1118'8, mlcod=1117'5)", 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:21.111007+0000", 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 178.18378712699999, 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.031592189999999999, 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.360 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.111007+0000", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.111007+0000", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.111009+0000", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.159e-06 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.127056+0000", 2024-09-21T10:24:19.361 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016047049000000001 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.127078+0000", 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2634e-05 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.127089+0000", 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0747000000000001e-05 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.127198+0000", 2024-09-21T10:24:19.362 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010856799999999999 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.127225+0000", 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7815999999999999e-05 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.142554+0000", 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015328728 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.363 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:21.142600+0000", 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5488999999999998e-05 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2317 44.1f 44:f9215db3:::rbd_data.b240933cb63.0000000000000015:head [write 0~3551253] snapc 69=[69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1120)", 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:23.109281+0000", 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 176.18551327599999, 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.046347053999999999, 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.364 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2317 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.109281+0000", 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.109281+0000", 2024-09-21T10:24:19.365 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.109283+0000", 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9750000000000001e-06 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.112538+0000", 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032548210000000002 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.366 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.112554+0000", 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.552e-05 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.112561+0000", 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6699999999999994e-06 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.112612+0000", 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1167e-05 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.367 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.112690+0000", 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7642e-05 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.112816+0000", 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000125785 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.144341+0000", 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.031525125000000001 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.368 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.146532+0000", 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002190703 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155545+0000", 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090130669999999996 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155584+0000", 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9183000000000001e-05 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.369 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.155628+0000", 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4396e-05 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45632.0:2316 44.16 e1120/1014 44:696e36f3:::rbd_data.b240933cb63.0000000000000014:head v 1120'50, mlcod=1114'48)", 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:23.114945+0000", 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 176.17984910000001, 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.031481914999999999, 2024-09-21T10:24:19.370 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114945+0000", 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114945+0000", 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.371 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.114946+0000", 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5499999999999999e-07 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.122093+0000", 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071472619999999997 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.122131+0000", 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7667999999999999e-05 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.372 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.122147+0000", 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5919000000000001e-05 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.122263+0000", 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000115945 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.122289+0000", 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6092999999999999e-05 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.373 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.146386+0000", 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.024097604000000002 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:23.146427+0000", 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1069000000000003e-05 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.374 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2441 44.1f 44:fc9e02d0:::rbd_data.b240933cb63.0000000000000027:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~2702021] snapc 6b=[6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1122)", 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:25.150053+0000", 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 174.14474167700001, 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.037315554000000001, 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2441 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.375 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.150053+0000", 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.150053+0000", 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.150062+0000", 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2359999999999993e-06 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.376 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.151646+0000", 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015833889999999999 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.151675+0000", 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9322000000000001e-05 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.151683+0000", 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9829999999999996e-06 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.377 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.151764+0000", 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0916999999999998e-05 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.151933+0000", 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016957699999999999 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.152092+0000", 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015874500000000001 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.378 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.156016+0000", 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039237279999999996 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.172005+0000", 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015988974999999999 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.187245+0000", 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015240541999999999 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.379 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.187328+0000", 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3010999999999995e-05 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:25.187368+0000", 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0129000000000002e-05 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.380 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2549 44.e 44:72456f42:::rbd_data.b240933cb63.000000000000000a:head [write 94828~2750187] snapc 6d=[6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1124)", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:27.156795+0000", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 172.138000003, 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.042726527, 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2549 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.156795+0000", 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.381 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.156795+0000", 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.156796+0000", 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6279999999999999e-06 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.158508+0000", 2024-09-21T10:24:19.382 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017118890000000001 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.158530+0000", 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2059999999999999e-05 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.158540+0000", 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6350000000000001e-06 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.158621+0000", 2024-09-21T10:24:19.383 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0745e-05 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.158701+0000", 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0743999999999998e-05 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.158890+0000", 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018824 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.166886+0000", 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079965639999999994 2024-09-21T10:24:19.384 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.199317+0000", 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.032431004999999999 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.199440+0000", 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012338900000000001 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.199491+0000", 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0151999999999997e-05 2024-09-21T10:24:19.385 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:27.199521+0000", 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0476000000000001e-05 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45632.0:2587 44.b e1125/1014 44:d7c5b9e3:::rbd_data.b240933cb63.0000000000000007:head v 1125'19, mlcod=1121'15)", 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:28.150897+0000", 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 171.14389770400001, 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.041990909, 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.386 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.150897+0000", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.150897+0000", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.150898+0000", 2024-09-21T10:24:19.387 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6899999999999996e-07 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.169040+0000", 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018142128 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.169055+0000", 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5057e-05 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.169065+0000", 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.944e-06 2024-09-21T10:24:19.388 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.169138+0000", 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2914e-05 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.169162+0000", 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3838000000000001e-05 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.192825+0000", 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023663087999999999 2024-09-21T10:24:19.389 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.192888+0000", 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3170999999999995e-05 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45632.0:2595 44.17 e1125/1014 44:eefa1367:::rbd_data.b240933cb63.0000000000000061:head v 1125'52, mlcod=1121'49)", 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:28.156978+0000", 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 171.13781657000001, 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.034591027000000003, 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.390 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.156978+0000", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.156978+0000", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.156978+0000", 2024-09-21T10:24:19.391 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4299999999999998e-07 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.165597+0000", 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086183939999999997 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.165620+0000", 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3088e-05 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.392 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.165632+0000", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1654000000000001e-05 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.165703+0000", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1796000000000002e-05 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.165726+0000", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2467e-05 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.191515+0000", 2024-09-21T10:24:19.393 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.025789632 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:28.191569+0000", 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3553e-05 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:2776 44.e 44:70302697:::rbd_data.b240933cb63.0000000000000063:head [write 0~2752564] snapc 71=[71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:31.237278+0000", 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 168.057516443, 2024-09-21T10:24:19.394 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.036391834999999997, 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 2776 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.237278+0000", 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.395 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.237278+0000", 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.237282+0000", 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6619999999999998e-06 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.239998+0000", 2024-09-21T10:24:19.396 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002716333 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.240010+0000", 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2211999999999999e-05 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.240017+0000", 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9249999999999998e-06 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.240684+0000", 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066717800000000004 2024-09-21T10:24:19.397 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.240808+0000", 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012370799999999999 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.241024+0000", 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021604799999999999 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.255217+0000", 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014192776000000001 2024-09-21T10:24:19.398 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.270096+0000", 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014879172 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.273575+0000", 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034787849999999999 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.273615+0000", 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9748e-05 2024-09-21T10:24:19.399 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.273670+0000", 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5288000000000003e-05 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45632.0:2779 44.1d e1128/1014 44:bfc92ec7:::rbd_data.b240933cb63.0000000000000062:head v 1128'41, mlcod=1117'38)", 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:21:31.265606+0000", 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 168.02918887600001, 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.033537347000000002, 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.400 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.265606+0000", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.265606+0000", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.265606+0000", 2024-09-21T10:24:19.401 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4900000000000002e-07 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.274955+0000", 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093494919999999992 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.274966+0000", 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0681e-05 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.402 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.274971+0000", 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.339e-06 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.275040+0000", 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8034000000000006e-05 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.275063+0000", 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3278999999999999e-05 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.403 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.299024+0000", 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023961495999999999 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:21:31.299143+0000", 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118777 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:6937 44.e 44:727bac97:::rbd_data.b240bb5bb67f.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:00.490396+0000", 2024-09-21T10:24:19.404 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 138.80439835000001, 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.061086670000000003, 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 6937 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.490396+0000", 2024-09-21T10:24:19.405 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.490396+0000", 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.490398+0000", 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0279999999999999e-06 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.406 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493208+0000", 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028101210000000001 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493251+0000", 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2781999999999998e-05 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493267+0000", 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5418e-05 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.407 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493347+0000", 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0110000000000004e-05 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493478+0000", 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000131097 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493594+0000", 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011593499999999999 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.408 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.501564+0000", 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079704230000000008 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.515522+0000", 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013957815 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551379+0000", 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.035856768999999997 2024-09-21T10:24:19.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551430+0000", 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0877000000000001e-05 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551483+0000", 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3294999999999997e-05 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:6940 44.e 44:75ac8403:::rbd_data.b240bb5bb67f.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1157)", 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:00.493304+0000", 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 138.801490938, 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.058358503999999999, 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 6940 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.411 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493304+0000", 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493304+0000", 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.493305+0000", 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.835e-06 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.412 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496401+0000", 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003095399 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496433+0000", 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2317999999999998e-05 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.496446+0000", 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2723e-05 2024-09-21T10:24:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.498563+0000", 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021170189999999999 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.498751+0000", 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018825600000000001 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.498964+0000", 2024-09-21T10:24:19.414 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021264799999999999 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.512841+0000", 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013877161000000001 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551544+0000", 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.038703134 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.415 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551592+0000", 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7786e-05 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551622+0000", 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0215000000000001e-05 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:00.551662+0000", 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0009999999999998e-05 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.416 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.45632.0:7361 44.2 44:45957388:::rbd_data.b240938dee57.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:03.553419+0000", 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 135.74137525399999, 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.043288105, 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.45632", 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 7361 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.417 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553419+0000", 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553419+0000", 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.553422+0000", 2024-09-21T10:24:19.418 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.01e-06 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.555855+0000", 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024324849999999999 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.555883+0000", 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8581e-05 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.555894+0000", 2024-09-21T10:24:19.419 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0607e-05 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.555979+0000", 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4988999999999996e-05 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.556167+0000", 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000187744 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:24:19.420 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.556407+0000", 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00023982800000000001 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.575925+0000", 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.019518921000000002 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.576296+0000", 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037019199999999998 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.421 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596593+0000", 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020296957000000001 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596652+0000", 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9423999999999999e-05 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.596707+0000", 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5367e-05 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.422 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.45632.0:7359 44.b e1160/1014 44:d0c4b63b:::rbd_data.b240938dee57.0000000000000002:head v 1160'99, mlcod=1156'97)", 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:22:03.572567+0000", 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 135.72222795900001, 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.033625865999999997, 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.572567+0000", 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.423 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.572567+0000", 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.572568+0000", 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1379999999999999e-06 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.587648+0000", 2024-09-21T10:24:19.424 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015080236 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.587670+0000", 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2130999999999999e-05 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.587681+0000", 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0579e-05 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.425 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.587751+0000", 2024-09-21T10:24:19.426 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0788000000000001e-05 2024-09-21T10:24:19.426 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.426 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.426 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.426 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.587782+0000", 2024-09-21T10:24:19.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:19.427 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0536e-05 2024-09-21T10:24:19.427 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.606147+0000", 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018364513999999998 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:22:03.606192+0000", 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5943999999999999e-05 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.428 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:55632 45.8 45:11c6a68d:::benchmark_data_smithi119_168706_object55631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:01.989369+0000", 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 17.305425340999999, 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.032133667999999997, 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55632 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.429 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989369+0000", 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989369+0000", 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989370+0000", 2024-09-21T10:24:19.430 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.19e-06 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989393+0000", 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2625000000000001e-05 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989400+0000", 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0729999999999999e-06 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.431 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989405+0000", 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9169999999999997e-06 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989448+0000", 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2629999999999997e-05 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989536+0000", 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8574999999999998e-05 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:19.432 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.989619+0000", 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2818000000000006e-05 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.991494+0000", 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001875236 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:01.991523+0000", 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8935e-05 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.433 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:02.021386+0000", 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.029862265999999998 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:02.021453+0000", 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7161000000000003e-05 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:02.021503+0000", 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0241999999999998e-05 2024-09-21T10:24:19.434 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:60028 45.b 45:d0b86dbf:::benchmark_data_smithi119_168706_object60027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:04.800249+0000", 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 14.494545792, 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.033159431000000003, 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:24:19.435 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 60028 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.800249+0000", 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.800249+0000", 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.436 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.800250+0000", 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4610000000000001e-06 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.800271+0000", 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0326000000000002e-05 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.800280+0000", 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0969999999999995e-06 2024-09-21T10:24:19.437 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.800286+0000", 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9560000000000002e-06 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.802831+0000", 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025453339999999998 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.806214+0000", 2024-09-21T10:24:19.438 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033832160000000001 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.806344+0000", 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012954000000000001 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.814971+0000", 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0086270879999999998 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.439 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.815475+0000", 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050428799999999998 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.833089+0000", 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017614191000000001 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.833262+0000", 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000173096 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.440 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:04.833408+0000", 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000145838 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:65090 45.8 45:12e92f69:::benchmark_data_smithi119_168706_object65089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:08.390726+0000", 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.904068326999999, 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.032074671999999999, 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.441 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 65090 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390726+0000", 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.442 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390726+0000", 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390728+0000", 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.55e-06 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390747+0000", 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9470000000000002e-05 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.443 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390759+0000", 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2095e-05 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390766+0000", 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2360000000000001e-06 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390812+0000", 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6805000000000003e-05 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.444 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390913+0000", 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100968 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.390996+0000", 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2287999999999995e-05 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.394104+0000", 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031079860000000001 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.445 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.394142+0000", 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8340999999999998e-05 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.422534+0000", 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.028391704 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.422683+0000", 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014892200000000001 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.446 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.422801+0000", 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118307 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:65091 45.8 45:190662ef:::benchmark_data_smithi119_168706_object65090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:08.391325+0000", 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 10.903469772999999, 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.033436127000000003, 2024-09-21T10:24:19.447 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 65091 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.391325+0000", 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.448 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.391325+0000", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.391325+0000", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0900000000000001e-07 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.391342+0000", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6612999999999998e-05 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.391349+0000", 2024-09-21T10:24:19.449 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.204e-06 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.391354+0000", 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.454e-06 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.392888+0000", 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001534026 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.450 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.392968+0000", 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0748999999999995e-05 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.393048+0000", 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9794000000000002e-05 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.422842+0000", 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.029793353000000002 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.423495+0000", 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00065316400000000002 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.424442+0000", 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00094686 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.424640+0000", 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019786599999999999 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.452 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.424761+0000", 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012143500000000001 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69543 45.8 45:1997f656:::benchmark_data_smithi119_168706_object69542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.566721+0000", 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.728073545, 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.062711218999999999, 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:24:19.453 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69543 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566721+0000", 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:24:19.454 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566721+0000", 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566728+0000", 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7129999999999999e-06 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566746+0000", 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7801000000000001e-05 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:24:19.455 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566755+0000", 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4930000000000004e-06 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566760+0000", 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0980000000000001e-06 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.566804+0000", 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3748000000000002e-05 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.456 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.626826+0000", 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.060022363000000002 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.626991+0000", 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000164456 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.629278+0000", 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022877840000000002 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.457 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.629326+0000", 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7471e-05 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.629340+0000", 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3718e-05 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.629387+0000", 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6978000000000003e-05 2024-09-21T10:24:19.458 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.629432+0000", 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5596000000000002e-05 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:24:19.459 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:24:19.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:19.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 42 16 80111 80095 119.159 77.25 0.00678052 0.00838668 2024-09-21T10:24:19.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.594 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:19.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:19.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:19.699 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:19.700 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:19.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:19.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:19.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:19.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:19.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:19.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:19.999 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:20.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.137 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:20.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:20.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:19 smithi119 ceph-mon[101432]: pgmap v1606: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:24:20.193 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/605203850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:20.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:19 smithi119 ceph-mon[105288]: pgmap v1606: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:24:20.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/605203850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:19 smithi145 ceph-mon[89359]: pgmap v1606: 89 pgs: 89 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:24:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/605203850' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:20.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:20.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:20.379 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:20.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:20.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:20.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 43 16 81405 81389 118.268 80.875 0.0254652 0.00845032 2024-09-21T10:24:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:24:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:24:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:20.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:20.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.692 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:20.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:20.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:20.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:20.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:20.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:20.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:21.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:21.149 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:21.251 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:21.287 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:21.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:21.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:21.529 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 44 16 82755 82739 117.497 84.375 0.00669674 0.00850604 2024-09-21T10:24:21.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:21.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:21.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:21.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:21.825 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:21.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:21.957 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:21.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:22.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:22.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:22.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:22.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:24:22.193 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:21 smithi119 ceph-mon[105288]: pgmap v1607: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:24:22.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:21 smithi119 ceph-mon[101432]: pgmap v1607: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:24:22.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:22.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:21 smithi145 ceph-mon[89359]: pgmap v1607: 89 pgs: 89 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-21T10:24:22.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:22.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:22.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:22.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:22.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 45 16 83998 83982 116.612 77.6875 0.0181487 0.00857055 2024-09-21T10:24:22.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:22.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:22.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:22.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:22.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:24:22.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:22.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:22.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:23.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:23.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:23.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:23.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:23.272 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:23.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:23.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:23.454 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:23.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 46 16 85261 85245 115.793 78.9375 0.00688005 0.00863086 2024-09-21T10:24:23.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:23.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:23.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.732 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:23.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:23.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:23.864 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:23.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:23.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:23.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:24.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:24.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:24.027 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:23 smithi119 ceph-mon[101432]: pgmap v1608: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:24:24.027 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:23 smithi119 ceph-mon[105288]: pgmap v1608: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:24:24.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:24.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:24.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:24.290 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:23 smithi145 ceph-mon[89359]: pgmap v1608: 89 pgs: 89 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-21T10:24:24.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:24.429 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:24.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 47 16 86470 86454 114.937 75.5625 0.0133547 0.00869462 2024-09-21T10:24:24.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:24.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:24.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:24.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:24.757 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:24:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:24:24] "GET /metrics HTTP/1.1" 200 36765 "" "Prometheus/2.51.0" 2024-09-21T10:24:24.806 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:24.832 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:24.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:24.963 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:24:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:24.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:24.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:25.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:25.113 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:25.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3463808475' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:24 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3463808475' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:24 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:25.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.250 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:25.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3463808475' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:24 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:24:25.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:25.418 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:25.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:25.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 48 15 87542 87527 113.94 67.0625 0.00745313 0.00877169 2024-09-21T10:24:25.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:25.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:25.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:25.684 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:25.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:25.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:25.818 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:25.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:25.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:26.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.085 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:26.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.223 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:26.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[105288]: pgmap v1609: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[105288]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[101432]: pgmap v1609: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:26.267 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:25 smithi119 ceph-mon[101432]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-21T10:24:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:25 smithi145 ceph-mon[89359]: pgmap v1609: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-21T10:24:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:25 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:25 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:24:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:25 smithi145 ceph-mon[89359]: osdmap e1175: 8 total, 8 up, 8 in 2024-09-21T10:24:26.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.461 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:26.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:26.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 49 16 88752 88736 113.157 75.5625 0.0114721 0.00883201 2024-09-21T10:24:26.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:26.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:26.714 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:26.746 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:26.847 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:26.880 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:26.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:26.981 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:27.016 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:27.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:27.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:27.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:27.283 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:27.357 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:27.358 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:27.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:27.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.530 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 50 16 89936 89920 112.373 74 0.00644829 0.00889345 2024-09-21T10:24:27.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:27.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:27.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:27.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:27.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:27.818 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:27.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:27.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:27.981 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:27.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:27 smithi119 ceph-mon[105288]: pgmap v1611: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:24:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:27 smithi119 ceph-mon[101432]: pgmap v1611: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:24:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:28.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:28.224 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:28.252 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:27 smithi145 ceph-mon[89359]: pgmap v1611: 89 pgs: 89 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-21T10:24:28.353 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.387 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:28.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:28.532 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 51 16 91142 91126 111.646 75.375 0.0144848 0.00895173 2024-09-21T10:24:28.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:28.658 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:28.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:28.791 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:28.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:28.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:28.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:29.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:29.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:29.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:29.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:29.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:29.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:29.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:29.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:29.426 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:29.468 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:29.532 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 52 16 92364 92348 110.967 76.375 0.0172581 0.00900662 2024-09-21T10:24:29.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:29.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:29.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:29.733 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:29.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:29.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:29.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:29.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:24:29.976 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:30.004 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:30.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:30.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:30.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:30.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:30.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:29 smithi145 ceph-mon[89359]: pgmap v1612: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:24:30.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:24:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:24:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:30.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:29 smithi119 ceph-mon[101432]: pgmap v1612: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:24:30.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:29 smithi119 ceph-mon[105288]: pgmap v1612: 89 pgs: 89 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:24:30.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:30.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:30.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:30.532 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 53 16 93475 93459 110.184 69.4375 0.00624703 0.00906989 2024-09-21T10:24:30.547 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:30.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:30.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:30.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:30.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:30.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:30.984 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:30.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:31.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:31.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:31.225 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:31.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:31.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:31.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:31.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:31.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:31.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 54 16 94679 94663 109.535 75.25 0.00960079 0.00912422 2024-09-21T10:24:31.533 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:31.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:31.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:31.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:31.801 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:31.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:31.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:31.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:32.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:32.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:32.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:32.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:32.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:31 smithi145 ceph-mon[89359]: pgmap v1613: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:24:32.324 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:31 smithi119 ceph-mon[101432]: pgmap v1613: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:24:32.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:31 smithi119 ceph-mon[105288]: pgmap v1613: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:24:32.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:32.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:32.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:32.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 55 16 95755 95739 108.766 67.25 0.011133 0.00918846 2024-09-21T10:24:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:32.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:32.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:32.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:32.832 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:32.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:32.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:32.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:33.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:33.085 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:33.105 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:33.106 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:33.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:33.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:24:33.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2537586049' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2537586049' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:33.228 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:33.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:33.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:33.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:33.478 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:33.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 56 16 96793 96777 107.982 64.875 0.0135175 0.00925571 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:24:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:24:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2537586049' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:33.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:24:33.580 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:33.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:33.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.745 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:33.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:33.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:33.850 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:24:33.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:33.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:33.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:34.037 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:34.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:34.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:34.271 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:34.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:34.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-mon[101432]: pgmap v1614: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T10:24:34.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:24:34.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-mon[101432]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-21T10:24:34.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-mon[105288]: pgmap v1614: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T10:24:34.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:24:34.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-mon[105288]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-21T10:24:34.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:34.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:34.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:34.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:34.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 57 15 97864 97849 107.263 67 0.00842379 0.00931804 2024-09-21T10:24:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:34 smithi145 ceph-mon[89359]: pgmap v1614: 89 pgs: 89 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-21T10:24:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:34 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:24:34.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:34 smithi145 ceph-mon[89359]: osdmap e1176: 8 total, 8 up, 8 in 2024-09-21T10:24:34.558 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:34.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:24:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:24:34] "GET /metrics HTTP/1.1" 200 36763 "" "Prometheus/2.51.0" 2024-09-21T10:24:34.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:34.709 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:24:34.709 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:24:34.709 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:24:34.709 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:24:34.709 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:34.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:34.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:34.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:34.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:34.972 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:34.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:35.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:35.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:35.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:35.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:35.252 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:35.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2857074042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2857074042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:35.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:35.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:35.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:35.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 58 16 98966 98950 106.601 68.8125 0.0115079 0.00937484 2024-09-21T10:24:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2857074042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:35.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:35.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:35.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:35.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:35.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:35.914 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:35.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:36.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:36.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:36.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:36.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:36.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:36.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:36.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:36 smithi119 ceph-mon[105288]: pgmap v1616: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:24:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:36 smithi119 ceph-mon[101432]: pgmap v1616: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:24:36.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.473 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 59 15 99945 99930 105.832 61.25 0.00826475 0.00944434 2024-09-21T10:24:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:36 smithi145 ceph-mon[89359]: pgmap v1616: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:24:36.574 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:36.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:36.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:36.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:36.878 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:36.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:36.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:37.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:37.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:37.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:37.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:37.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:37.304 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:37.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:37.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:24:37.532267+0000 min lat: 0.00116449 max lat: 0.167858 avg lat: 0.0095044 2024-09-21T10:24:37.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:24:37.533 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 60 16 100995 100979 105.161 65.5625 0.0128755 0.0095044 2024-09-21T10:24:37.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:37.584 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:37.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:37.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:37.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:37.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:37.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:37.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:24:37.994 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:37.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:38.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:38.120 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:38.222 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:38.253 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:38.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:38 smithi119 ceph-mon[101432]: pgmap v1617: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-21T10:24:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:38 smithi119 ceph-mon[105288]: pgmap v1617: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-21T10:24:38.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:38.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:24:38.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:38.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:38.535 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 61 16 102086 102070 104.551 68.1875 0.0118222 0.00955917 2024-09-21T10:24:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:38 smithi145 ceph-mon[89359]: pgmap v1617: 89 pgs: 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-21T10:24:38.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:38.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:38.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:38.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:38.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:38.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:38.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:38.966 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:38.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:38.990 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:39.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:39.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:39.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:39.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:39.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:39.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:39.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:24:39.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:39.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:39.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:39.502 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:39.535 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 62 16 103098 103082 103.885 63.25 0.00315527 0.00961909 2024-09-21T10:24:39.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:39.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:39.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:39.760 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:39.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:39.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:39.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:39.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:40.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:40.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:40.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:40.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:40.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:40.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:40 smithi119 ceph-mon[101432]: pgmap v1618: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:24:40.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:40 smithi119 ceph-mon[105288]: pgmap v1618: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:24:40.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:40.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:40.445 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:40.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:40.540 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 63 16 104099 104083 103.224 62.5625 0.0156442 0.00968247 2024-09-21T10:24:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:40 smithi145 ceph-mon[89359]: pgmap v1618: 89 pgs: 89 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:24:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:24:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:24:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:40.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:40.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:40.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:40.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:40.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:40.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:40.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:40.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:41.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:41.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.123 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:41.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:41.134 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:24:41.190 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/925769182' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:41.190 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/925769182' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:41.224 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:41.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:41.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:41.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:41.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:41.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:41.538 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 64 16 105163 105147 102.651 66.5 0.0141411 0.00973674 2024-09-21T10:24:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/925769182' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:24:41.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:41.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:41.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:41.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:41.775 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.808 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:41.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:41.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:41.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:41.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:42.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:42.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:42.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:42.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[101432]: pgmap v1619: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:24:42.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:42.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/925769182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:42.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[101432]: osdmap e1177: 8 total, 8 up, 8 in 2024-09-21T10:24:42.288 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[105288]: pgmap v1619: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:24:42.288 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:42.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/925769182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:42.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:42 smithi119 ceph-mon[105288]: osdmap e1177: 8 total, 8 up, 8 in 2024-09-21T10:24:42.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:42.370 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:42.472 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:42.492 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:42.538 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 65 16 106164 106148 102.034 62.5625 0.0106427 0.00979571 2024-09-21T10:24:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:42 smithi145 ceph-mon[89359]: pgmap v1619: 89 pgs: 89 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:24:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:42 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:42 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/925769182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:24:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:42 smithi145 ceph-mon[89359]: osdmap e1177: 8 total, 8 up, 8 in 2024-09-21T10:24:42.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:42.622 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:42.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:42.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:42.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:42.897 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:42.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:42.998 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:43.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:43.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:43.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:43.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:43.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:43.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:43.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:43.423 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:43.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:43.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 66 16 107238 107222 101.505 67.125 0.0154535 0.00984641 2024-09-21T10:24:43.555 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:43.638 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:24:43.639 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:24:43.639 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:24:43.639 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:24:43.639 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:24:43.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:43.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:43.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:43.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:43.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:43.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:43.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:44.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:24:44.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.203 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.231 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:44.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:44.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:44 smithi119 ceph-mon[105288]: pgmap v1621: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-21T10:24:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:44 smithi119 ceph-mon[101432]: pgmap v1621: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-21T10:24:44.373 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:44.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:44.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.524 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:44.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 67 16 108269 108253 100.952 64.4375 0.0205569 0.00990031 2024-09-21T10:24:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:44 smithi145 ceph-mon[89359]: pgmap v1621: 89 pgs: 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 58 MiB/s wr, 926 op/s 2024-09-21T10:24:44.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:44.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:44.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:44.813 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:24:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:24:44] "GET /metrics HTTP/1.1" 200 36763 "" "Prometheus/2.51.0" 2024-09-21T10:24:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.814 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:44.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:44.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:44.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:44.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:45.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:45.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:45.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:45.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:45.324 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:45.359 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:45.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:45.504 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:45.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 68 16 109302 109286 100.417 64.5625 0.0172274 0.00995335 2024-09-21T10:24:45.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:45.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:45.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:45.769 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:45.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:45.896 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:45.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:45.998 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.035 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:46.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:46.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:46.178 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:46.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:46 smithi119 ceph-mon[105288]: pgmap v1622: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T10:24:46.241 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:46 smithi119 ceph-mon[101432]: pgmap v1622: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T10:24:46.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:46.312 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:46.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:46.441 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:46.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 69 16 110365 110349 99.9243 66.4375 0.0161892 0.0100022 2024-09-21T10:24:46.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:46 smithi145 ceph-mon[89359]: pgmap v1622: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T10:24:46.559 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:46.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:24:46.598 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:46.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:46.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:46.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:46.862 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:46.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:46.995 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:46.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:47.097 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:47.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:47.226 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:47.274 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:47.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:47.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:47.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:24:47.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:24:47.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:24:47.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:24:47.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:24:47.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:24:47.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:47.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 70 16 111301 111285 99.3325 58.5 0.017447 0.0100619 2024-09-21T10:24:47.564 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:47.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:47.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:47.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:47.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:47.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:47.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:47.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:47.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:48.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:48.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:48.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.243 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:48.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:48.344 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:48 smithi119 ceph-mon[105288]: pgmap v1623: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3285778602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:48 smithi119 ceph-mon[101432]: pgmap v1623: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3285778602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:48.376 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:48.478 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:48.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:48.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 71 16 112266 112250 98.783 60.3125 0.0129763 0.0101182 2024-09-21T10:24:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:48 smithi145 ceph-mon[89359]: pgmap v1623: 89 pgs: 89 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-21T10:24:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3285778602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:24:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:24:48.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:48.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:48.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:48.875 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:48.896 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:48.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:48.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:49.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:49.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:49.159 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:24:49.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:49.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:49.300 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:49.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:49.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:49.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:49.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 72 16 113260 113244 98.2739 62.125 0.01636 0.0101705 2024-09-21T10:24:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2809238859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:24:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:49 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:24:49.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:49.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2809238859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:24:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:49 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:24:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2809238859' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:24:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:49 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:24:49.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:49.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:49.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:49.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:49.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:49.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:49.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:50.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:50.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:50.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:50.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:50.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:50.389 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:50.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:50 smithi119 ceph-mon[101432]: pgmap v1624: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:24:50.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:50 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:24:50.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:50 smithi119 ceph-mon[101432]: osdmap e1178: 8 total, 8 up, 8 in 2024-09-21T10:24:50.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:50 smithi119 ceph-mon[105288]: pgmap v1624: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:24:50.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:50 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:24:50.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:50 smithi119 ceph-mon[105288]: osdmap e1178: 8 total, 8 up, 8 in 2024-09-21T10:24:50.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:50.518 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:50.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 73 16 114312 114296 97.8284 65.75 0.0163517 0.0102167 2024-09-21T10:24:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:24:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:24:50.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:24:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:50 smithi145 ceph-mon[89359]: pgmap v1624: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:24:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:50 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:24:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:50 smithi145 ceph-mon[89359]: osdmap e1178: 8 total, 8 up, 8 in 2024-09-21T10:24:50.620 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:50.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:50.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:50.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:50.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:50.923 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:50.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:51.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:51.052 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:51.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:51.196 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:51.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:51.247 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:51.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:24:51.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:51.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:51.485 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:51.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:51.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 74 16 115315 115299 97.3535 62.6875 0.0155136 0.010267 2024-09-21T10:24:51.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:51.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:51.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:51.772 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:51.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:51.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:24:51.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:51.910 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:51.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:52.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:52.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:52.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:52.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:52.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:52.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:52.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:52.358 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:52.452 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:52 smithi119 ceph-mon[101432]: pgmap v1626: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T10:24:52.453 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:52 smithi119 ceph-mon[105288]: pgmap v1626: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T10:24:52.459 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:52.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:52.539 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 75 16 116224 116208 96.813 56.8125 0.0177247 0.0103246 2024-09-21T10:24:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:52 smithi145 ceph-mon[89359]: pgmap v1626: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-21T10:24:52.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:52.605 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:52.613 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:52.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.757 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:52.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:52.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:52.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:52.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:52.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:53.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:53.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:53.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:53.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:53.299 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:53.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:53.428 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:53.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:53.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:53.540 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 76 16 117248 117232 96.3805 64 0.010722 0.0103704 2024-09-21T10:24:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:53.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:53.709 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:53.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:53.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:53.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:53.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:53.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:54.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:54.115 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:54.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:54.256 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:54.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:54 smithi119 ceph-mon[105288]: pgmap v1627: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:24:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:54 smithi119 ceph-mon[101432]: pgmap v1627: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:24:54.389 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:54.492 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:54.520 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:54.540 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 77 16 118179 118163 95.8846 58.1875 0.0141061 0.0104239 2024-09-21T10:24:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:54 smithi145 ceph-mon[89359]: pgmap v1627: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-21T10:24:54.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:54.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:54.699 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:24:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:24:54] "GET /metrics HTTP/1.1" 200 36765 "" "Prometheus/2.51.0" 2024-09-21T10:24:54.715 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:24:54.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:54.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:54.950 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:54.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:55.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:55.084 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:55.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:55.173 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:24:55.186 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:55.216 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:55.318 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:55.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:55.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:55.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:55.540 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 78 16 119188 119172 95.4638 63.0625 0.0109333 0.0104701 2024-09-21T10:24:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4110474470' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:55.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:55.613 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:55.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4110474470' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4110474470' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:24:55.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:55.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:55.850 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:55.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:55.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:55.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:55.989 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:56.027 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:56.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:56.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:56.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:56.254 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[105288]: pgmap v1628: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T10:24:56.255 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:56.255 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4110474470' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:56.255 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[105288]: osdmap e1179: 8 total, 8 up, 8 in 2024-09-21T10:24:56.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[101432]: pgmap v1628: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T10:24:56.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:56.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4110474470' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:56.257 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:56 smithi119 ceph-mon[101432]: osdmap e1179: 8 total, 8 up, 8 in 2024-09-21T10:24:56.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:56.305 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:56.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:56.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:56.458 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:56.540 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 79 16 120190 120174 95.0481 62.625 0.00482475 0.0105159 2024-09-21T10:24:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:56 smithi145 ceph-mon[89359]: pgmap v1628: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-21T10:24:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:56 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:24:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4110474470' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:24:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:56 smithi145 ceph-mon[89359]: osdmap e1179: 8 total, 8 up, 8 in 2024-09-21T10:24:56.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:56.598 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:24:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:24:56.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:56.728 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:56.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:56.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:56.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:56.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:56.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:57.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:57.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:57.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:57.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:57.284 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:57.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:57.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:57.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:24:57.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:57.540 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:24:57.539679+0000 min lat: 0.00116449 max lat: 0.167858 avg lat: 0.0105623 2024-09-21T10:24:57.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:24:57.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 80 16 121170 121154 94.6255 61.25 0.038546 0.0105623 2024-09-21T10:24:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:57.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:57.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:57.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:57.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:57.805 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:57.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:57.930 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:57.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:58.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:24:58.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:24:58.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:58.196 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:24:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.221 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:58.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:58.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:24:58.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:58.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:58.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:58.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:58.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 81 16 122047 122031 94.134 54.8125 0.0253938 0.0106179 2024-09-21T10:24:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:58 smithi145 ceph-mon[89359]: pgmap v1630: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-21T10:24:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:58 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2828549431' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:58 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:58.584 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:24:58.604 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:58 smithi119 ceph-mon[101432]: pgmap v1630: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-21T10:24:58.604 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:58 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2828549431' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:58.604 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:58 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:58.604 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:58 smithi119 ceph-mon[105288]: pgmap v1630: 89 pgs: 89 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-21T10:24:58.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:58 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2828549431' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:58.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:58 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:24:58.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:24:58.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:58.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:58.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:24:58.910 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:58.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:24:59.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:59.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:59.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:24:59.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:24:59.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:59.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:59 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:24:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:24:59 smithi119 ceph-mon[101432]: osdmap e1180: 8 total, 8 up, 8 in 2024-09-21T10:24:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:59 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:24:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:24:59 smithi119 ceph-mon[105288]: osdmap e1180: 8 total, 8 up, 8 in 2024-09-21T10:24:59.423 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:24:59.450 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:24:59.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 82 16 123017 123001 93.7254 60.625 0.0133409 0.0106643 2024-09-21T10:24:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:59 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:24:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:24:59 smithi145 ceph-mon[89359]: osdmap e1180: 8 total, 8 up, 8 in 2024-09-21T10:24:59.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:24:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.588 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:59.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:24:59.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:24:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.724 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:24:59.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:24:59.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:24:59.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:24:59.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:24:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:24:59.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:24:59.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.010 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:00.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:00.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:00.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:00.395 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:00.425 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:00.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:00.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 83 16 123962 123946 93.3077 59.0625 0.00735479 0.0107122 2024-09-21T10:25:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:25:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:25:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:00 smithi145 ceph-mon[89359]: pgmap v1632: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-09-21T10:25:00.559 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:00 smithi119 ceph-mon[101432]: pgmap v1632: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-09-21T10:25:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:00 smithi119 ceph-mon[105288]: pgmap v1632: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 957 op/s 2024-09-21T10:25:00.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:00.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.824 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:00.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:00.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:00.964 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:00.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:01.065 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:01.096 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:01.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:01.350 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:01.451 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:01.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 84 16 124873 124857 92.8747 56.9375 0.0135827 0.0107622 2024-09-21T10:25:01.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:01.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.744 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.751 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:01.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:01.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:01.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:01.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:02.016 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:02.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:02.144 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:02.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:02.279 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:02.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:02.414 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:02.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:02.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 85 16 125784 125768 92.4519 56.9375 0.0150875 0.0108108 2024-09-21T10:25:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:02 smithi145 ceph-mon[89359]: pgmap v1633: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:25:02.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:02 smithi119 ceph-mon[101432]: pgmap v1633: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:25:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:02 smithi119 ceph-mon[105288]: pgmap v1633: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-21T10:25:02.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:02.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:02.811 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:02.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:02.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:02.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:03.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:03.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:03.174 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:03.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:03.219 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:25:03.219 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:03.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:03.252 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:25:03.253 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:03.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:03.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1376663560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:03.284 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:03.284 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1376663560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:03.290 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:25:03.290 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:03.327 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:25:03.327 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:03.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:03.357 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:25:03.357 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:03.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:03.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:03.395 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:25:03.395 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:03.430 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:25:03.430 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:03.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:03.465 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:25:03.465 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:03.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 86 16 126696 126680 92.0397 57 0.0111621 0.0108596 2024-09-21T10:25:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1376663560' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:03.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:03.647 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:03.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:03.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:03.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:03.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:03.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:04.010 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:04.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:04.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.198 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:04.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:04.298 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:04.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:04.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:04.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:04.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 87 16 127605 127589 91.6348 56.8125 0.0152622 0.0109077 2024-09-21T10:25:04.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:04 smithi119 ceph-mon[105288]: pgmap v1634: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s 2024-09-21T10:25:04.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:04 smithi119 ceph-mon[101432]: pgmap v1634: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s 2024-09-21T10:25:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:04 smithi145 ceph-mon[89359]: pgmap v1634: 89 pgs: 89 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 47 MiB/s wr, 755 op/s 2024-09-21T10:25:04.555 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:04.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:04.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:04.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:04.812 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:25:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:25:04] "GET /metrics HTTP/1.1" 200 36763 "" "Prometheus/2.51.0" 2024-09-21T10:25:04.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:04.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:04.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:04.988 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:04.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:04.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:05.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.140 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:05.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:05.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.287 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:05.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:05.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:05.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:05.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:05.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 88 16 128370 128354 91.1369 47.8125 0.0396352 0.0109672 2024-09-21T10:25:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:05.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:05.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:05.675 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:05.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:05.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:05.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:05.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:05.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:05.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:06.046 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:06.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:06.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:06.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:25:06.319 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:06.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:06 smithi119 ceph-mon[105288]: pgmap v1635: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 988 op/s 2024-09-21T10:25:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2535114201' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:06.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:06 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:06.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:06 smithi119 ceph-mon[101432]: pgmap v1635: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 988 op/s 2024-09-21T10:25:06.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2535114201' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:06.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:06 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:06.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:06.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:06.541 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 89 16 129233 129217 90.7189 53.9375 0.00880789 0.0110184 2024-09-21T10:25:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:06 smithi145 ceph-mon[89359]: pgmap v1635: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 62 MiB/s wr, 988 op/s 2024-09-21T10:25:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2535114201' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:06 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:06.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.623 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:06.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:06.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:06.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:06.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:06.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:06.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:07.010 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:07.041 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:07.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:07.179 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:07.280 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:07.316 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:07.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:07.434 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:07.434 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:25:07.434 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:07.434 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[105288]: osdmap e1181: 8 total, 8 up, 8 in 2024-09-21T10:25:07.435 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:07.435 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:25:07.435 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:07.435 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:07 smithi119 ceph-mon[101432]: osdmap e1181: 8 total, 8 up, 8 in 2024-09-21T10:25:07.460 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:07.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total time run: 90.0085 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total writes made: 130167 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Write size: 65536 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object size: 65536 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Bandwidth (MB/sec): 90.3852 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Bandwidth: 39.9852 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max bandwidth (MB/sec): 248.312 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min bandwidth (MB/sec): 47.8125 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average IOPS: 1446 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev IOPS: 639.763 2024-09-21T10:25:07.542 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max IOPS: 3973 2024-09-21T10:25:07.543 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min IOPS: 765 2024-09-21T10:25:07.543 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average Latency(s): 0.0110596 2024-09-21T10:25:07.543 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Latency(s): 0.00831969 2024-09-21T10:25:07.543 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max latency(s): 0.167858 2024-09-21T10:25:07.543 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min latency(s): 0.00116449 2024-09-21T10:25:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:07 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:07 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:25:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:07 smithi145 ceph-mon[89359]: osdmap e1181: 8 total, 8 up, 8 in 2024-09-21T10:25:07.547 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:25:07.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:07.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:07.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.733 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:07.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:07.834 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:07.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:07.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:07.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:07.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:08.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:08.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:08.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:08.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:08.402 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:08.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.537 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:08.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:08 smithi145 ceph-mon[89359]: pgmap v1637: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:25:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:08 smithi119 ceph-mon[105288]: pgmap v1637: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:25:08.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:08 smithi119 ceph-mon[101432]: pgmap v1637: 89 pgs: 89 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:25:08.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:08.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:08.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:08.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:08.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:08.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:08.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:08.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:08.941 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:08.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:09.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:09.068 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:09.170 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:09.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:09.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:09.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:09.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:09.478 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:09.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:09.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:09.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:09.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:09.722 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:09.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:09.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:09.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:09.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:10.004 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:10.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:10.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:10.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:10.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:10.190 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:25:10.190 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:25:10.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:132600 45.e e1181/1164 45:77877f65:::benchmark_data_smithi119_168706_object2430:head v 1181'8215, mlcod=1181'8213)", 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.165515+0000", 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.021456354, 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021482919999999999, 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.165515+0000", 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.165515+0000", 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.165515+0000", 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9500000000000001e-07 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.165521+0000", 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5199999999999997e-06 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.165532+0000", 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1564e-05 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.165539+0000", 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9249999999999998e-06 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.166072+0000", 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00053291299999999998 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.166094+0000", 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1698000000000001e-05 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:132604 45.e e1181/1164 45:7dafe99f:::benchmark_data_smithi119_168706_object2434:head v 1181'8216, mlcod=1181'8213)", 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.170487+0000", 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.016483508000000001, 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016567281, 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.170487+0000", 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.170487+0000", 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.170488+0000", 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4900000000000001e-07 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.170492+0000", 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2400000000000001e-06 2024-09-21T10:25:10.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.170500+0000", 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5020000000000003e-06 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.170505+0000", 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2780000000000001e-06 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174704+0000", 2024-09-21T10:25:10.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041986510000000003 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174759+0000", 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5467999999999998e-05 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:132605 45.e e1181/1164 45:74b7a25c:::benchmark_data_smithi119_168706_object2435:head v 1181'8217, mlcod=1181'8213)", 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.174526+0000", 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012445303, 2024-09-21T10:25:10.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012580104999999999, 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174526+0000", 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174526+0000", 2024-09-21T10:25:10.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174526+0000", 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7899999999999998e-07 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174532+0000", 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7849999999999997e-06 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174541+0000", 2024-09-21T10:25:10.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1800000000000002e-06 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174547+0000", 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2740000000000004e-06 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.179009+0000", 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044611599999999996 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:10.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.179035+0000", 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6364e-05 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:132609 45.6 45.665735b6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1181)", 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.174712+0000", 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.01225909, 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012442155999999999, 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:10.202 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 132609 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174712+0000", 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174712+0000", 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174713+0000", 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3300000000000001e-07 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174715+0000", 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8739999999999999e-06 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174722+0000", 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6889999999999998e-06 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.174727+0000", 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2490000000000001e-06 2024-09-21T10:25:10.205 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:132607 45.e e1181/1164 45:7a5e7d6a:::benchmark_data_smithi119_168706_object2437:head v 1181'8218, mlcod=1181'8213)", 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.175966+0000", 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.011005242, 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011240827, 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.175966+0000", 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.175966+0000", 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.175967+0000", 2024-09-21T10:25:10.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5000000000000001e-07 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.175972+0000", 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4790000000000004e-06 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.175982+0000", 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0135e-05 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.175989+0000", 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0820000000000001e-06 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185910+0000", 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0099203099999999999 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185930+0000", 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9959e-05 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:132610 45.e e1181/1164)", 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.181726+0000", 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0052450930000000002, 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005528186, 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.181726+0000", 2024-09-21T10:25:10.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.181726+0000", 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.181726+0000", 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3999999999999997e-07 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.181731+0000", 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.622e-06 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.181737+0000", 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6189999999999996e-06 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.181742+0000", 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5979999999999999e-06 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.212 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:132615 45.5 e1181/1164 45:acb67e11:::benchmark_data_smithi119_168706_object2445:head v 1181'8000, mlcod=1181'7998)", 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:10.185362+0000", 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.001609412, 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019349, 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185362+0000", 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185362+0000", 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185362+0000", 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4799999999999997e-07 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:10.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185369+0000", 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4110000000000001e-06 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185383+0000", 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3685e-05 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185389+0000", 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7349999999999999e-06 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:10.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185463+0000", 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3373000000000004e-05 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:10.185480+0000", 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7634000000000001e-05 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 7 2024-09-21T10:25:10.216 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:25:10.217 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:10.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:10.270 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:25:10.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:10.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:10.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:10.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:10 smithi119 ceph-mon[101432]: pgmap v1638: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 MiB/s wr, 923 op/s 2024-09-21T10:25:10.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/901264157' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:25:10.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:10 smithi119 ceph-mon[105288]: pgmap v1638: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 MiB/s wr, 923 op/s 2024-09-21T10:25:10.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/901264157' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:25:10.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:25:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:25:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:10 smithi145 ceph-mon[89359]: pgmap v1638: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 MiB/s wr, 923 op/s 2024-09-21T10:25:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/901264157' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:25:10.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:10.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:10.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:10.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:10.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:25:10.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:10.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:10.949 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:10.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:11.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:11.097 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:11.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:11.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:11.240 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:11.342 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:11.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:11.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:11.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:11.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/901264157' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:25:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:11 smithi145 ceph-mon[89359]: osdmap e1182: 8 total, 8 up, 8 in 2024-09-21T10:25:11.606 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/901264157' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:25:11.606 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:11 smithi119 ceph-mon[101432]: osdmap e1182: 8 total, 8 up, 8 in 2024-09-21T10:25:11.607 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/901264157' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:25:11.607 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:11 smithi119 ceph-mon[105288]: osdmap e1182: 8 total, 8 up, 8 in 2024-09-21T10:25:11.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:11.641 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:11.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:11.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:11.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:11.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:11.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:11.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:12.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:12.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:12.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:12.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:12.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:12.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:25:12.334 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:12.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:12.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:12.464 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:12 smithi145 ceph-mon[89359]: pgmap v1640: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 57 MiB/s wr, 1.06k op/s 2024-09-21T10:25:12.551 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:12 smithi119 ceph-mon[101432]: pgmap v1640: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 57 MiB/s wr, 1.06k op/s 2024-09-21T10:25:12.551 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:12 smithi119 ceph-mon[105288]: pgmap v1640: 89 pgs: 89 active+clean; 7.9 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 57 MiB/s wr, 1.06k op/s 2024-09-21T10:25:12.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:12.606 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:25:12.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:25:12.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:12.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:12.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:12.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:12.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:12.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:12.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:13.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:13.045 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:13.112 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:13.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:13.162 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:13.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:13.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:13.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:13.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:13.488 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:13 smithi145 ceph-mon[89359]: pgmap v1641: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 30 MiB/s wr, 699 op/s 2024-09-21T10:25:13.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:13 smithi119 ceph-mon[101432]: pgmap v1641: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 30 MiB/s wr, 699 op/s 2024-09-21T10:25:13.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:13 smithi119 ceph-mon[105288]: pgmap v1641: 89 pgs: 89 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 127 B/s rd, 30 MiB/s wr, 699 op/s 2024-09-21T10:25:13.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:13.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:13.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:13.762 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:13.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:13.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:13.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:13.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:13.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:14.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:14.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.174 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.276 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:14.313 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:14.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:14.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:14.575 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:14.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:14.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:14.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:14.847 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:25:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:25:14] "GET /metrics HTTP/1.1" 200 36763 "" "Prometheus/2.51.0" 2024-09-21T10:25:14.847 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:14.949 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:14.982 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:14.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:15.032 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:25:15.032 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:15.075 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:25:15.076 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:15.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:15.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:15.115 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:25:15.115 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:15.153 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:25:15.154 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:15.189 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:25:15.189 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:15.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:15.222 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:25:15.222 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:15.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:15.263 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:25:15.263 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:15.306 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:25:15.306 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:15.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:15.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:15 smithi119 ceph-mon[101432]: pgmap v1642: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 119 B/s rd, 24 MiB/s wr, 988 op/s 2024-09-21T10:25:15.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/328632169' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:15 smithi119 ceph-mon[105288]: pgmap v1642: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 119 B/s rd, 24 MiB/s wr, 988 op/s 2024-09-21T10:25:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/328632169' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:15.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:15.488 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:15.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:15.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:15.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:15.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:15.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:15 smithi145 ceph-mon[89359]: pgmap v1642: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 119 B/s rd, 24 MiB/s wr, 988 op/s 2024-09-21T10:25:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/328632169' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:15.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:15.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:15.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:16.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:16.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:16.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:16.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:16.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:16.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:16.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.590 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:16.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:16.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:16.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:16.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:16.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:16.994 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:16.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:25:17.039 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:25:17.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.123 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:17.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:17.224 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:17.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:17.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:17.329 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:17.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:17.394 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:17.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:17.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:17.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:17.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:17.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:17.801 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:17.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:17.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:17.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:18.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:18.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:18.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:25:18.207 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:18.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[105288]: pgmap v1643: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 904 op/s 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[105288]: from='client.45947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[101432]: pgmap v1643: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 904 op/s 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[101432]: from='client.45947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:18.253 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:17 smithi145 ceph-mon[89359]: pgmap v1643: 89 pgs: 89 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 20 MiB/s wr, 904 op/s 2024-09-21T10:25:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:17 smithi145 ceph-mon[89359]: from='client.45947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:25:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:18.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:18.321 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2024-09-21T10:25:16.836911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8151635941,"num_objects":124402,"num_object_clones":0,"num_object_copies":373206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124402,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":136172,"num_write_kb":8332348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590492,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131118,"ondisk_log_size":131118,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":25816292,"kb_used_data":24086564,"kb_used_omap":268,"kb_used_meta":1728883,"kb_avail":724144924,"statfs":{"total":767960285184,"available":741524402176,"internally_reserved":0,"allocated":24664641536,"data_stored":24628285615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274448,"internal_metadata":1770377200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,2,6,9,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-166330344,"num_objects":-2537,"num_object_clones":0,"num_object_copies":-7611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2537,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9047,"num_write_kb":208257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002878"},"pg_stats":[{"pgid":"45.f","version":"1182'8375","reported_seq":8422,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:12.423662+0000","last_change":"2024-09-21T10:23:24.928314+0000","last_active":"2024-09-21T10:25:12.423662+0000","last_peered":"2024-09-21T10:25:12.423662+0000","last_clean":"2024-09-21T10:25:12.423662+0000","last_became_active":"2024-09-21T10:23:24.928005+0000","last_became_peered":"2024-09-21T10:23:24.928005+0000","last_unstale":"2024-09-21T10:25:12.423662+0000","last_undegraded":"2024-09-21T10:25:12.423662+0000","last_fullsized":"2024-09-21T10:25:12.423662+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8375,"log_dups_size":0,"ondisk_log_size":8375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:15.220461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511770624,"num_objects":7809,"num_object_clones":0,"num_object_copies":23427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8375,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.e","version":"1182'8420","reported_seq":8446,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.513171+0000","last_change":"2024-09-21T10:23:25.544908+0000","last_active":"2024-09-21T10:25:13.513171+0000","last_peered":"2024-09-21T10:25:13.513171+0000","last_clean":"2024-09-21T10:25:13.513171+0000","last_became_active":"2024-09-21T10:23:25.544575+0000","last_became_peered":"2024-09-21T10:23:25.544575+0000","last_unstale":"2024-09-21T10:25:13.513171+0000","last_undegraded":"2024-09-21T10:25:13.513171+0000","last_fullsized":"2024-09-21T10:25:13.513171+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8420,"log_dups_size":0,"ondisk_log_size":8420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:16.735472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506855424,"num_objects":7734,"num_object_clones":0,"num_object_copies":23202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8420,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.d","version":"1182'8538","reported_seq":8586,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.923145+0000","last_change":"2024-09-21T10:23:24.927661+0000","last_active":"2024-09-21T10:25:13.923145+0000","last_peered":"2024-09-21T10:25:13.923145+0000","last_clean":"2024-09-21T10:25:13.923145+0000","last_became_active":"2024-09-21T10:23:24.927398+0000","last_became_peered":"2024-09-21T10:23:24.927398+0000","last_unstale":"2024-09-21T10:25:13.923145+0000","last_undegraded":"2024-09-21T10:25:13.923145+0000","last_fullsized":"2024-09-21T10:25:13.923145+0000","mapping_epoch":1164,"log_start":"1169'600","ondisk_log_start":"1169'600","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7938,"log_dups_size":0,"ondisk_log_size":7938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:19.561056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507052056,"num_objects":7738,"num_object_clones":0,"num_object_copies":23214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7738,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8538,"num_write_kb":520769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.c","version":"1182'8560","reported_seq":8586,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.560994+0000","last_change":"2024-09-21T10:23:24.924772+0000","last_active":"2024-09-21T10:25:13.560994+0000","last_peered":"2024-09-21T10:25:13.560994+0000","last_clean":"2024-09-21T10:25:13.560994+0000","last_became_active":"2024-09-21T10:23:24.924461+0000","last_became_peered":"2024-09-21T10:23:24.924461+0000","last_unstale":"2024-09-21T10:25:13.560994+0000","last_undegraded":"2024-09-21T10:25:13.560994+0000","last_fullsized":"2024-09-21T10:25:13.560994+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8560,"log_dups_size":0,"ondisk_log_size":8560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:25:04.341096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512360448,"num_objects":7818,"num_object_clones":0,"num_object_copies":23454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8560,"num_write_kb":524096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.b","version":"1182'8583","reported_seq":8630,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:15.350235+0000","last_change":"2024-09-21T10:23:25.544535+0000","last_active":"2024-09-21T10:25:15.350235+0000","last_peered":"2024-09-21T10:25:15.350235+0000","last_clean":"2024-09-21T10:25:15.350235+0000","last_became_active":"2024-09-21T10:23:25.544215+0000","last_became_peered":"2024-09-21T10:23:25.544215+0000","last_unstale":"2024-09-21T10:25:15.350235+0000","last_undegraded":"2024-09-21T10:25:15.350235+0000","last_fullsized":"2024-09-21T10:25:15.350235+0000","mapping_epoch":1164,"log_start":"1169'1400","ondisk_log_start":"1169'1400","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7183,"log_dups_size":0,"ondisk_log_size":7183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:47.760255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505479168,"num_objects":7713,"num_object_clones":0,"num_object_copies":23139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8583,"num_write_kb":521472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.a","version":"1182'8407","reported_seq":8433,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.588802+0000","last_change":"2024-09-21T10:23:24.926178+0000","last_active":"2024-09-21T10:25:13.588802+0000","last_peered":"2024-09-21T10:25:13.588802+0000","last_clean":"2024-09-21T10:25:13.588802+0000","last_became_active":"2024-09-21T10:23:24.925883+0000","last_became_peered":"2024-09-21T10:23:24.925883+0000","last_unstale":"2024-09-21T10:25:13.588802+0000","last_undegraded":"2024-09-21T10:25:13.588802+0000","last_fullsized":"2024-09-21T10:25:13.588802+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8407,"log_dups_size":0,"ondisk_log_size":8407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:23.995550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504168448,"num_objects":7693,"num_object_clones":0,"num_object_copies":23079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8407,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.9","version":"1182'8663","reported_seq":8689,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.529465+0000","last_change":"2024-09-21T10:23:24.924699+0000","last_active":"2024-09-21T10:25:13.529465+0000","last_peered":"2024-09-21T10:25:13.529465+0000","last_clean":"2024-09-21T10:25:13.529465+0000","last_became_active":"2024-09-21T10:23:24.924416+0000","last_became_peered":"2024-09-21T10:23:24.924416+0000","last_unstale":"2024-09-21T10:25:13.529465+0000","last_undegraded":"2024-09-21T10:25:13.529465+0000","last_fullsized":"2024-09-21T10:25:13.529465+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8663,"log_dups_size":0,"ondisk_log_size":8663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:37.348206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520945664,"num_objects":7949,"num_object_clones":0,"num_object_copies":23847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8663,"num_write_kb":531584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.8","version":"1182'8530","reported_seq":8577,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:15.345897+0000","last_change":"2024-09-21T10:23:25.544569+0000","last_active":"2024-09-21T10:25:15.345897+0000","last_peered":"2024-09-21T10:25:15.345897+0000","last_clean":"2024-09-21T10:25:15.345897+0000","last_became_active":"2024-09-21T10:23:25.544266+0000","last_became_peered":"2024-09-21T10:23:25.544266+0000","last_unstale":"2024-09-21T10:25:15.345897+0000","last_undegraded":"2024-09-21T10:25:15.345897+0000","last_fullsized":"2024-09-21T10:25:15.345897+0000","mapping_epoch":1164,"log_start":"1169'1300","ondisk_log_start":"1169'1300","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:16.355961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":497680384,"num_objects":7594,"num_object_clones":0,"num_object_copies":22782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8530,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.7","version":"1182'8557","reported_seq":8581,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.575660+0000","last_change":"2024-09-21T10:23:24.926188+0000","last_active":"2024-09-21T10:25:13.575660+0000","last_peered":"2024-09-21T10:25:13.575660+0000","last_clean":"2024-09-21T10:25:13.575660+0000","last_became_active":"2024-09-21T10:23:24.925897+0000","last_became_peered":"2024-09-21T10:23:24.925897+0000","last_unstale":"2024-09-21T10:25:13.575660+0000","last_undegraded":"2024-09-21T10:25:13.575660+0000","last_fullsized":"2024-09-21T10:25:13.575660+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8557,"log_dups_size":0,"ondisk_log_size":8557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:46.332054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515833856,"num_objects":7871,"num_object_clones":0,"num_object_copies":23613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8557,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.6","version":"1182'8545","reported_seq":8592,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.937232+0000","last_change":"2024-09-21T10:23:25.546048+0000","last_active":"2024-09-21T10:25:13.937232+0000","last_peered":"2024-09-21T10:25:13.937232+0000","last_clean":"2024-09-21T10:25:13.937232+0000","last_became_active":"2024-09-21T10:23:25.545764+0000","last_became_peered":"2024-09-21T10:23:25.545764+0000","last_unstale":"2024-09-21T10:25:13.937232+0000","last_undegraded":"2024-09-21T10:25:13.937232+0000","last_fullsized":"2024-09-21T10:25:13.937232+0000","mapping_epoch":1164,"log_start":"1169'600","ondisk_log_start":"1169'600","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7945,"log_dups_size":0,"ondisk_log_size":7945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:48:39.460646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512032768,"num_objects":7813,"num_object_clones":0,"num_object_copies":23439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8545,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.5","version":"1182'8136","reported_seq":8183,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:12.458408+0000","last_change":"2024-09-21T10:23:24.927838+0000","last_active":"2024-09-21T10:25:12.458408+0000","last_peered":"2024-09-21T10:25:12.458408+0000","last_clean":"2024-09-21T10:25:12.458408+0000","last_became_active":"2024-09-21T10:23:24.927541+0000","last_became_peered":"2024-09-21T10:23:24.927541+0000","last_unstale":"2024-09-21T10:25:12.458408+0000","last_undegraded":"2024-09-21T10:25:12.458408+0000","last_fullsized":"2024-09-21T10:25:12.458408+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8136,"log_dups_size":0,"ondisk_log_size":8136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:16.618768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495583232,"num_objects":7562,"num_object_clones":0,"num_object_copies":22686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8136,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.4","version":"1182'8623","reported_seq":8670,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:14.182957+0000","last_change":"2024-09-21T10:23:25.544626+0000","last_active":"2024-09-21T10:25:14.182957+0000","last_peered":"2024-09-21T10:25:14.182957+0000","last_clean":"2024-09-21T10:25:14.182957+0000","last_became_active":"2024-09-21T10:23:25.544315+0000","last_became_peered":"2024-09-21T10:23:25.544315+0000","last_unstale":"2024-09-21T10:25:14.182957+0000","last_undegraded":"2024-09-21T10:25:14.182957+0000","last_fullsized":"2024-09-21T10:25:14.182957+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8623,"log_dups_size":0,"ondisk_log_size":8623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:41.350175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513081344,"num_objects":7829,"num_object_clones":0,"num_object_copies":23487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8623,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.3","version":"1182'8505","reported_seq":8552,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.913046+0000","last_change":"2024-09-21T10:23:24.927687+0000","last_active":"2024-09-21T10:25:13.913046+0000","last_peered":"2024-09-21T10:25:13.913046+0000","last_clean":"2024-09-21T10:25:13.913046+0000","last_became_active":"2024-09-21T10:23:24.927161+0000","last_became_peered":"2024-09-21T10:23:24.927161+0000","last_unstale":"2024-09-21T10:25:13.913046+0000","last_undegraded":"2024-09-21T10:25:13.913046+0000","last_fullsized":"2024-09-21T10:25:13.913046+0000","mapping_epoch":1164,"log_start":"1169'600","ondisk_log_start":"1169'600","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7905,"log_dups_size":0,"ondisk_log_size":7905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:28.137305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506920960,"num_objects":7735,"num_object_clones":0,"num_object_copies":23205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8505,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.2","version":"1182'8412","reported_seq":8438,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.554306+0000","last_change":"2024-09-21T10:23:24.927278+0000","last_active":"2024-09-21T10:25:13.554306+0000","last_peered":"2024-09-21T10:25:13.554306+0000","last_clean":"2024-09-21T10:25:13.554306+0000","last_became_active":"2024-09-21T10:23:24.926970+0000","last_became_peered":"2024-09-21T10:23:24.926970+0000","last_unstale":"2024-09-21T10:25:13.554306+0000","last_undegraded":"2024-09-21T10:25:13.554306+0000","last_fullsized":"2024-09-21T10:25:13.554306+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8412,"log_dups_size":0,"ondisk_log_size":8412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:17.823705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508297216,"num_objects":7756,"num_object_clones":0,"num_object_copies":23268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8412,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.1","version":"1182'8694","reported_seq":8720,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.522238+0000","last_change":"2024-09-21T10:23:25.545025+0000","last_active":"2024-09-21T10:25:13.522238+0000","last_peered":"2024-09-21T10:25:13.522238+0000","last_clean":"2024-09-21T10:25:13.522238+0000","last_became_active":"2024-09-21T10:23:25.544657+0000","last_became_peered":"2024-09-21T10:23:25.544657+0000","last_unstale":"2024-09-21T10:25:13.522238+0000","last_undegraded":"2024-09-21T10:25:13.522238+0000","last_fullsized":"2024-09-21T10:25:13.522238+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8694,"log_dups_size":0,"ondisk_log_size":8694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:36.979666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524025856,"num_objects":7996,"num_object_clones":0,"num_object_copies":23988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8694,"num_write_kb":534080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"45.0","version":"1182'8412","reported_seq":8459,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:13.912876+0000","last_change":"2024-09-21T10:23:25.545529+0000","last_active":"2024-09-21T10:25:13.912876+0000","last_peered":"2024-09-21T10:25:13.912876+0000","last_clean":"2024-09-21T10:25:13.912876+0000","last_became_active":"2024-09-21T10:23:25.545200+0000","last_became_peered":"2024-09-21T10:23:25.545200+0000","last_unstale":"2024-09-21T10:25:13.912876+0000","last_undegraded":"2024-09-21T10:25:13.912876+0000","last_fullsized":"2024-09-21T10:25:13.912876+0000","mapping_epoch":1164,"log_start":"1169'500","ondisk_log_start":"1169'500","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:39.621176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508952576,"num_objects":7766,"num_object_clones":0,"num_object_copies":23298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8412,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":1799,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283430+0000","last_change":"2024-09-21T10:08:06.538833+0000","last_active":"2024-09-21T10:25:10.283430+0000","last_peered":"2024-09-21T10:25:10.283430+0000","last_clean":"2024-09-21T10:25:10.283430+0000","last_became_active":"2024-09-21T10:08:06.538526+0000","last_became_peered":"2024-09-21T10:08:06.538526+0000","last_unstale":"2024-09-21T10:25:10.283430+0000","last_undegraded":"2024-09-21T10:25:10.283430+0000","last_fullsized":"2024-09-21T10:25:10.283430+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:59.194474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.269848+0000","last_change":"2024-09-21T10:06:42.626454+0000","last_active":"2024-09-21T10:25:06.269848+0000","last_peered":"2024-09-21T10:25:06.269848+0000","last_clean":"2024-09-21T10:25:06.269848+0000","last_became_active":"2024-09-21T10:06:42.626116+0000","last_became_peered":"2024-09-21T10:06:42.626116+0000","last_unstale":"2024-09-21T10:25:06.269848+0000","last_undegraded":"2024-09-21T10:25:06.269848+0000","last_fullsized":"2024-09-21T10:25:06.269848+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:01.968111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1667,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.293667+0000","last_change":"2024-09-21T10:08:55.142962+0000","last_active":"2024-09-21T10:25:10.293667+0000","last_peered":"2024-09-21T10:25:10.293667+0000","last_clean":"2024-09-21T10:25:10.293667+0000","last_became_active":"2024-09-21T10:08:55.139681+0000","last_became_peered":"2024-09-21T10:08:55.139681+0000","last_unstale":"2024-09-21T10:25:10.293667+0000","last_undegraded":"2024-09-21T10:25:10.293667+0000","last_fullsized":"2024-09-21T10:25:10.293667+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:52.577104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1697,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281342+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:25:10.281342+0000","last_peered":"2024-09-21T10:25:10.281342+0000","last_clean":"2024-09-21T10:25:10.281342+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:25:10.281342+0000","last_undegraded":"2024-09-21T10:25:10.281342+0000","last_fullsized":"2024-09-21T10:25:10.281342+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":1124,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268663+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T10:25:06.268663+0000","last_peered":"2024-09-21T10:25:06.268663+0000","last_clean":"2024-09-21T10:25:06.268663+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T10:25:06.268663+0000","last_undegraded":"2024-09-21T10:25:06.268663+0000","last_fullsized":"2024-09-21T10:25:06.268663+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270164+0000","last_change":"2024-09-21T10:08:55.108381+0000","last_active":"2024-09-21T10:25:06.270164+0000","last_peered":"2024-09-21T10:25:06.270164+0000","last_clean":"2024-09-21T10:25:06.270164+0000","last_became_active":"2024-09-21T10:08:55.108074+0000","last_became_peered":"2024-09-21T10:08:55.108074+0000","last_unstale":"2024-09-21T10:25:06.270164+0000","last_undegraded":"2024-09-21T10:25:06.270164+0000","last_fullsized":"2024-09-21T10:25:06.270164+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.846660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":1141,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268801+0000","last_change":"2024-09-21T10:06:42.622644+0000","last_active":"2024-09-21T10:25:06.268801+0000","last_peered":"2024-09-21T10:25:06.268801+0000","last_clean":"2024-09-21T10:25:06.268801+0000","last_became_active":"2024-09-21T10:06:42.621612+0000","last_became_peered":"2024-09-21T10:06:42.621612+0000","last_unstale":"2024-09-21T10:25:06.268801+0000","last_undegraded":"2024-09-21T10:25:06.268801+0000","last_fullsized":"2024-09-21T10:25:06.268801+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:21.301285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270377+0000","last_change":"2024-09-21T10:08:55.109486+0000","last_active":"2024-09-21T10:25:06.270377+0000","last_peered":"2024-09-21T10:25:06.270377+0000","last_clean":"2024-09-21T10:25:06.270377+0000","last_became_active":"2024-09-21T10:08:55.109050+0000","last_became_peered":"2024-09-21T10:08:55.109050+0000","last_unstale":"2024-09-21T10:25:06.270377+0000","last_undegraded":"2024-09-21T10:25:06.270377+0000","last_fullsized":"2024-09-21T10:25:06.270377+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:13.242342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1795,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283222+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:25:10.283222+0000","last_peered":"2024-09-21T10:25:10.283222+0000","last_clean":"2024-09-21T10:25:10.283222+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:25:10.283222+0000","last_undegraded":"2024-09-21T10:25:10.283222+0000","last_fullsized":"2024-09-21T10:25:10.283222+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1695,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.297064+0000","last_change":"2024-09-21T10:08:55.117054+0000","last_active":"2024-09-21T10:25:10.297064+0000","last_peered":"2024-09-21T10:25:10.297064+0000","last_clean":"2024-09-21T10:25:10.297064+0000","last_became_active":"2024-09-21T10:08:55.116767+0000","last_became_peered":"2024-09-21T10:08:55.116767+0000","last_unstale":"2024-09-21T10:25:10.297064+0000","last_undegraded":"2024-09-21T10:25:10.297064+0000","last_fullsized":"2024-09-21T10:25:10.297064+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:13.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1158,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270290+0000","last_change":"2024-09-21T10:08:06.539436+0000","last_active":"2024-09-21T10:25:06.270290+0000","last_peered":"2024-09-21T10:25:06.270290+0000","last_clean":"2024-09-21T10:25:06.270290+0000","last_became_active":"2024-09-21T10:08:06.538856+0000","last_became_peered":"2024-09-21T10:08:06.538856+0000","last_unstale":"2024-09-21T10:25:06.270290+0000","last_undegraded":"2024-09-21T10:25:06.270290+0000","last_fullsized":"2024-09-21T10:25:06.270290+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:43.502357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1662,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.285973+0000","last_change":"2024-09-21T10:08:55.116807+0000","last_active":"2024-09-21T10:25:10.285973+0000","last_peered":"2024-09-21T10:25:10.285973+0000","last_clean":"2024-09-21T10:25:10.285973+0000","last_became_active":"2024-09-21T10:08:55.115948+0000","last_became_peered":"2024-09-21T10:08:55.115948+0000","last_unstale":"2024-09-21T10:25:10.285973+0000","last_undegraded":"2024-09-21T10:25:10.285973+0000","last_fullsized":"2024-09-21T10:25:10.285973+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:21.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1767,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288224+0000","last_change":"2024-09-21T10:08:55.109470+0000","last_active":"2024-09-21T10:25:10.288224+0000","last_peered":"2024-09-21T10:25:10.288224+0000","last_clean":"2024-09-21T10:25:10.288224+0000","last_became_active":"2024-09-21T10:08:55.106885+0000","last_became_peered":"2024-09-21T10:08:55.106885+0000","last_unstale":"2024-09-21T10:25:10.288224+0000","last_undegraded":"2024-09-21T10:25:10.288224+0000","last_fullsized":"2024-09-21T10:25:10.288224+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:51.866025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1793,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283386+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:25:10.283386+0000","last_peered":"2024-09-21T10:25:10.283386+0000","last_clean":"2024-09-21T10:25:10.283386+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:25:10.283386+0000","last_undegraded":"2024-09-21T10:25:10.283386+0000","last_fullsized":"2024-09-21T10:25:10.283386+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1698,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283631+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:25:10.283631+0000","last_peered":"2024-09-21T10:25:10.283631+0000","last_clean":"2024-09-21T10:25:10.283631+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:25:10.283631+0000","last_undegraded":"2024-09-21T10:25:10.283631+0000","last_fullsized":"2024-09-21T10:25:10.283631+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276358+0000","last_change":"2024-09-21T10:06:42.631648+0000","last_active":"2024-09-21T10:25:06.276358+0000","last_peered":"2024-09-21T10:25:06.276358+0000","last_clean":"2024-09-21T10:25:06.276358+0000","last_became_active":"2024-09-21T10:06:42.631301+0000","last_became_peered":"2024-09-21T10:06:42.631301+0000","last_unstale":"2024-09-21T10:25:06.276358+0000","last_undegraded":"2024-09-21T10:25:06.276358+0000","last_fullsized":"2024-09-21T10:25:06.276358+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:52.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1680,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283902+0000","last_change":"2024-09-21T10:08:55.134315+0000","last_active":"2024-09-21T10:25:10.283902+0000","last_peered":"2024-09-21T10:25:10.283902+0000","last_clean":"2024-09-21T10:25:10.283902+0000","last_became_active":"2024-09-21T10:08:55.132999+0000","last_became_peered":"2024-09-21T10:08:55.132999+0000","last_unstale":"2024-09-21T10:25:10.283902+0000","last_undegraded":"2024-09-21T10:25:10.283902+0000","last_fullsized":"2024-09-21T10:25:10.283902+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:24.870318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1790,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283528+0000","last_change":"2024-09-21T10:08:55.105723+0000","last_active":"2024-09-21T10:25:10.283528+0000","last_peered":"2024-09-21T10:25:10.283528+0000","last_clean":"2024-09-21T10:25:10.283528+0000","last_became_active":"2024-09-21T10:08:55.105283+0000","last_became_peered":"2024-09-21T10:08:55.105283+0000","last_unstale":"2024-09-21T10:25:10.283528+0000","last_undegraded":"2024-09-21T10:25:10.283528+0000","last_fullsized":"2024-09-21T10:25:10.283528+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:25.633107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1663,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281583+0000","last_change":"2024-09-21T10:08:55.131455+0000","last_active":"2024-09-21T10:25:10.281583+0000","last_peered":"2024-09-21T10:25:10.281583+0000","last_clean":"2024-09-21T10:25:10.281583+0000","last_became_active":"2024-09-21T10:08:55.129639+0000","last_became_peered":"2024-09-21T10:08:55.129639+0000","last_unstale":"2024-09-21T10:25:10.281583+0000","last_undegraded":"2024-09-21T10:25:10.281583+0000","last_fullsized":"2024-09-21T10:25:10.281583+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:50.934309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271761+0000","last_change":"2024-09-21T10:06:42.623974+0000","last_active":"2024-09-21T10:25:06.271761+0000","last_peered":"2024-09-21T10:25:06.271761+0000","last_clean":"2024-09-21T10:25:06.271761+0000","last_became_active":"2024-09-21T10:06:42.623598+0000","last_became_peered":"2024-09-21T10:06:42.623598+0000","last_unstale":"2024-09-21T10:25:06.271761+0000","last_undegraded":"2024-09-21T10:25:06.271761+0000","last_fullsized":"2024-09-21T10:25:06.271761+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.261148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1311,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.289053+0000","last_change":"2024-09-21T10:08:55.106181+0000","last_active":"2024-09-21T10:25:10.289053+0000","last_peered":"2024-09-21T10:25:10.289053+0000","last_clean":"2024-09-21T10:25:10.289053+0000","last_became_active":"2024-09-21T10:08:55.105882+0000","last_became_peered":"2024-09-21T10:08:55.105882+0000","last_unstale":"2024-09-21T10:25:10.289053+0000","last_undegraded":"2024-09-21T10:25:10.289053+0000","last_fullsized":"2024-09-21T10:25:10.289053+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:30.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1766,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288801+0000","last_change":"2024-09-21T10:08:06.538126+0000","last_active":"2024-09-21T10:25:10.288801+0000","last_peered":"2024-09-21T10:25:10.288801+0000","last_clean":"2024-09-21T10:25:10.288801+0000","last_became_active":"2024-09-21T10:08:06.537779+0000","last_became_peered":"2024-09-21T10:08:06.537779+0000","last_unstale":"2024-09-21T10:25:10.288801+0000","last_undegraded":"2024-09-21T10:25:10.288801+0000","last_fullsized":"2024-09-21T10:25:10.288801+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:34.731581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1145,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270602+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T10:25:06.270602+0000","last_peered":"2024-09-21T10:25:06.270602+0000","last_clean":"2024-09-21T10:25:06.270602+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T10:25:06.270602+0000","last_undegraded":"2024-09-21T10:25:06.270602+0000","last_fullsized":"2024-09-21T10:25:06.270602+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270444+0000","last_change":"2024-09-21T10:06:42.633523+0000","last_active":"2024-09-21T10:25:06.270444+0000","last_peered":"2024-09-21T10:25:06.270444+0000","last_clean":"2024-09-21T10:25:06.270444+0000","last_became_active":"2024-09-21T10:06:42.633019+0000","last_became_peered":"2024-09-21T10:06:42.633019+0000","last_unstale":"2024-09-21T10:25:06.270444+0000","last_undegraded":"2024-09-21T10:25:06.270444+0000","last_fullsized":"2024-09-21T10:25:06.270444+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:54.272637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1122,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270255+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T10:25:06.270255+0000","last_peered":"2024-09-21T10:25:06.270255+0000","last_clean":"2024-09-21T10:25:06.270255+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T10:25:06.270255+0000","last_undegraded":"2024-09-21T10:25:06.270255+0000","last_fullsized":"2024-09-21T10:25:06.270255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1153,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276426+0000","last_change":"2024-09-21T10:08:06.551874+0000","last_active":"2024-09-21T10:25:06.276426+0000","last_peered":"2024-09-21T10:25:06.276426+0000","last_clean":"2024-09-21T10:25:06.276426+0000","last_became_active":"2024-09-21T10:08:06.551572+0000","last_became_peered":"2024-09-21T10:08:06.551572+0000","last_unstale":"2024-09-21T10:25:06.276426+0000","last_undegraded":"2024-09-21T10:25:06.276426+0000","last_fullsized":"2024-09-21T10:25:06.276426+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:56.324479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1795,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283182+0000","last_change":"2024-09-21T10:06:42.626709+0000","last_active":"2024-09-21T10:25:10.283182+0000","last_peered":"2024-09-21T10:25:10.283182+0000","last_clean":"2024-09-21T10:25:10.283182+0000","last_became_active":"2024-09-21T10:06:42.626389+0000","last_became_peered":"2024-09-21T10:06:42.626389+0000","last_unstale":"2024-09-21T10:25:10.283182+0000","last_undegraded":"2024-09-21T10:25:10.283182+0000","last_fullsized":"2024-09-21T10:25:10.283182+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:39.634788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1682,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283719+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:25:10.283719+0000","last_peered":"2024-09-21T10:25:10.283719+0000","last_clean":"2024-09-21T10:25:10.283719+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:25:10.283719+0000","last_undegraded":"2024-09-21T10:25:10.283719+0000","last_fullsized":"2024-09-21T10:25:10.283719+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1765,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.291796+0000","last_change":"2024-09-21T10:06:42.630257+0000","last_active":"2024-09-21T10:25:10.291796+0000","last_peered":"2024-09-21T10:25:10.291796+0000","last_clean":"2024-09-21T10:25:10.291796+0000","last_became_active":"2024-09-21T10:06:42.628412+0000","last_became_peered":"2024-09-21T10:06:42.628412+0000","last_unstale":"2024-09-21T10:25:10.291796+0000","last_undegraded":"2024-09-21T10:25:10.291796+0000","last_fullsized":"2024-09-21T10:25:10.291796+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:30.166565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1673,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.282177+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:25:10.282177+0000","last_peered":"2024-09-21T10:25:10.282177+0000","last_clean":"2024-09-21T10:25:10.282177+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:25:10.282177+0000","last_undegraded":"2024-09-21T10:25:10.282177+0000","last_fullsized":"2024-09-21T10:25:10.282177+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1782,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283872+0000","last_change":"2024-09-21T10:06:09.039187+0000","last_active":"2024-09-21T10:25:10.283872+0000","last_peered":"2024-09-21T10:25:10.283872+0000","last_clean":"2024-09-21T10:25:10.283872+0000","last_became_active":"2024-09-21T10:06:09.038741+0000","last_became_peered":"2024-09-21T10:06:09.038741+0000","last_unstale":"2024-09-21T10:25:10.283872+0000","last_undegraded":"2024-09-21T10:25:10.283872+0000","last_fullsized":"2024-09-21T10:25:10.283872+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:40.920128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1152,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.273185+0000","last_change":"2024-09-21T10:06:09.028808+0000","last_active":"2024-09-21T10:25:06.273185+0000","last_peered":"2024-09-21T10:25:06.273185+0000","last_clean":"2024-09-21T10:25:06.273185+0000","last_became_active":"2024-09-21T10:06:09.028492+0000","last_became_peered":"2024-09-21T10:06:09.028492+0000","last_unstale":"2024-09-21T10:25:06.273185+0000","last_undegraded":"2024-09-21T10:25:06.273185+0000","last_fullsized":"2024-09-21T10:25:06.273185+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:41.134185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1767,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288671+0000","last_change":"2024-09-21T10:08:06.539915+0000","last_active":"2024-09-21T10:25:10.288671+0000","last_peered":"2024-09-21T10:25:10.288671+0000","last_clean":"2024-09-21T10:25:10.288671+0000","last_became_active":"2024-09-21T10:08:06.539121+0000","last_became_peered":"2024-09-21T10:08:06.539121+0000","last_unstale":"2024-09-21T10:25:10.288671+0000","last_undegraded":"2024-09-21T10:25:10.288671+0000","last_fullsized":"2024-09-21T10:25:10.288671+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:23.549392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1756,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288927+0000","last_change":"2024-09-21T10:06:42.625569+0000","last_active":"2024-09-21T10:25:10.288927+0000","last_peered":"2024-09-21T10:25:10.288927+0000","last_clean":"2024-09-21T10:25:10.288927+0000","last_became_active":"2024-09-21T10:06:42.625259+0000","last_became_peered":"2024-09-21T10:06:42.625259+0000","last_unstale":"2024-09-21T10:25:10.288927+0000","last_undegraded":"2024-09-21T10:25:10.288927+0000","last_fullsized":"2024-09-21T10:25:10.288927+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:45.833707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1781,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288822+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:25:10.288822+0000","last_peered":"2024-09-21T10:25:10.288822+0000","last_clean":"2024-09-21T10:25:10.288822+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:25:10.288822+0000","last_undegraded":"2024-09-21T10:25:10.288822+0000","last_fullsized":"2024-09-21T10:25:10.288822+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268673+0000","last_change":"2024-09-21T10:08:55.102678+0000","last_active":"2024-09-21T10:25:06.268673+0000","last_peered":"2024-09-21T10:25:06.268673+0000","last_clean":"2024-09-21T10:25:06.268673+0000","last_became_active":"2024-09-21T10:08:55.102022+0000","last_became_peered":"2024-09-21T10:08:55.102022+0000","last_unstale":"2024-09-21T10:25:06.268673+0000","last_undegraded":"2024-09-21T10:25:06.268673+0000","last_fullsized":"2024-09-21T10:25:06.268673+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:03.178484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1778,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288752+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:25:10.288752+0000","last_peered":"2024-09-21T10:25:10.288752+0000","last_clean":"2024-09-21T10:25:10.288752+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:25:10.288752+0000","last_undegraded":"2024-09-21T10:25:10.288752+0000","last_fullsized":"2024-09-21T10:25:10.288752+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1140,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270516+0000","last_change":"2024-09-21T10:06:09.043111+0000","last_active":"2024-09-21T10:25:06.270516+0000","last_peered":"2024-09-21T10:25:06.270516+0000","last_clean":"2024-09-21T10:25:06.270516+0000","last_became_active":"2024-09-21T10:06:09.042813+0000","last_became_peered":"2024-09-21T10:06:09.042813+0000","last_unstale":"2024-09-21T10:25:06.270516+0000","last_undegraded":"2024-09-21T10:25:06.270516+0000","last_fullsized":"2024-09-21T10:25:06.270516+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:50.398531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":1153,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268243+0000","last_change":"2024-09-21T10:06:42.622375+0000","last_active":"2024-09-21T10:25:06.268243+0000","last_peered":"2024-09-21T10:25:06.268243+0000","last_clean":"2024-09-21T10:25:06.268243+0000","last_became_active":"2024-09-21T10:06:42.622019+0000","last_became_peered":"2024-09-21T10:06:42.622019+0000","last_unstale":"2024-09-21T10:25:06.268243+0000","last_undegraded":"2024-09-21T10:25:06.268243+0000","last_fullsized":"2024-09-21T10:25:06.268243+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:22.917477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1769,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.289525+0000","last_change":"2024-09-21T10:06:42.627252+0000","last_active":"2024-09-21T10:25:10.289525+0000","last_peered":"2024-09-21T10:25:10.289525+0000","last_clean":"2024-09-21T10:25:10.289525+0000","last_became_active":"2024-09-21T10:06:42.626942+0000","last_became_peered":"2024-09-21T10:06:42.626942+0000","last_unstale":"2024-09-21T10:25:10.289525+0000","last_undegraded":"2024-09-21T10:25:10.289525+0000","last_fullsized":"2024-09-21T10:25:10.289525+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:21.844784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1768,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288254+0000","last_change":"2024-09-21T10:08:06.532136+0000","last_active":"2024-09-21T10:25:10.288254+0000","last_peered":"2024-09-21T10:25:10.288254+0000","last_clean":"2024-09-21T10:25:10.288254+0000","last_became_active":"2024-09-21T10:08:06.531195+0000","last_became_peered":"2024-09-21T10:08:06.531195+0000","last_unstale":"2024-09-21T10:25:10.288254+0000","last_undegraded":"2024-09-21T10:25:10.288254+0000","last_fullsized":"2024-09-21T10:25:10.288254+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:02.094677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1711,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.297118+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:25:10.297118+0000","last_peered":"2024-09-21T10:25:10.297118+0000","last_clean":"2024-09-21T10:25:10.297118+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:25:10.297118+0000","last_undegraded":"2024-09-21T10:25:10.297118+0000","last_fullsized":"2024-09-21T10:25:10.297118+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1803,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283463+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:25:10.283463+0000","last_peered":"2024-09-21T10:25:10.283463+0000","last_clean":"2024-09-21T10:25:10.283463+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:25:10.283463+0000","last_undegraded":"2024-09-21T10:25:10.283463+0000","last_fullsized":"2024-09-21T10:25:10.283463+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1804,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283814+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:25:10.283814+0000","last_peered":"2024-09-21T10:25:10.283814+0000","last_clean":"2024-09-21T10:25:10.283814+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:25:10.283814+0000","last_undegraded":"2024-09-21T10:25:10.283814+0000","last_fullsized":"2024-09-21T10:25:10.283814+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1169,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271665+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:25:06.271665+0000","last_peered":"2024-09-21T10:25:06.271665+0000","last_clean":"2024-09-21T10:25:06.271665+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:25:06.271665+0000","last_undegraded":"2024-09-21T10:25:06.271665+0000","last_fullsized":"2024-09-21T10:25:06.271665+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1122,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276479+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T10:25:06.276479+0000","last_peered":"2024-09-21T10:25:06.276479+0000","last_clean":"2024-09-21T10:25:06.276479+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T10:25:06.276479+0000","last_undegraded":"2024-09-21T10:25:06.276479+0000","last_fullsized":"2024-09-21T10:25:06.276479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1665,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.284642+0000","last_change":"2024-09-21T10:08:55.129563+0000","last_active":"2024-09-21T10:25:10.284642+0000","last_peered":"2024-09-21T10:25:10.284642+0000","last_clean":"2024-09-21T10:25:10.284642+0000","last_became_active":"2024-09-21T10:08:55.129268+0000","last_became_peered":"2024-09-21T10:08:55.129268+0000","last_unstale":"2024-09-21T10:25:10.284642+0000","last_undegraded":"2024-09-21T10:25:10.284642+0000","last_fullsized":"2024-09-21T10:25:10.284642+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:32.246505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":1152,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268561+0000","last_change":"2024-09-21T10:08:06.540841+0000","last_active":"2024-09-21T10:25:06.268561+0000","last_peered":"2024-09-21T10:25:06.268561+0000","last_clean":"2024-09-21T10:25:06.268561+0000","last_became_active":"2024-09-21T10:08:06.539359+0000","last_became_peered":"2024-09-21T10:08:06.539359+0000","last_unstale":"2024-09-21T10:25:06.268561+0000","last_undegraded":"2024-09-21T10:25:06.268561+0000","last_fullsized":"2024-09-21T10:25:06.268561+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:53.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1256,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271506+0000","last_change":"2024-09-21T10:06:09.040194+0000","last_active":"2024-09-21T10:25:06.271506+0000","last_peered":"2024-09-21T10:25:06.271506+0000","last_clean":"2024-09-21T10:25:06.271506+0000","last_became_active":"2024-09-21T10:06:09.039512+0000","last_became_peered":"2024-09-21T10:06:09.039512+0000","last_unstale":"2024-09-21T10:25:06.271506+0000","last_undegraded":"2024-09-21T10:25:06.271506+0000","last_fullsized":"2024-09-21T10:25:06.271506+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:37.194722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1767,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.289438+0000","last_change":"2024-09-21T10:08:55.112951+0000","last_active":"2024-09-21T10:25:10.289438+0000","last_peered":"2024-09-21T10:25:10.289438+0000","last_clean":"2024-09-21T10:25:10.289438+0000","last_became_active":"2024-09-21T10:08:55.112577+0000","last_became_peered":"2024-09-21T10:08:55.112577+0000","last_unstale":"2024-09-21T10:25:10.289438+0000","last_undegraded":"2024-09-21T10:25:10.289438+0000","last_fullsized":"2024-09-21T10:25:10.289438+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:06.470913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1682,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283798+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:25:10.283798+0000","last_peered":"2024-09-21T10:25:10.283798+0000","last_clean":"2024-09-21T10:25:10.283798+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:25:10.283798+0000","last_undegraded":"2024-09-21T10:25:10.283798+0000","last_fullsized":"2024-09-21T10:25:10.283798+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271094+0000","last_change":"2024-09-21T10:06:42.634333+0000","last_active":"2024-09-21T10:25:06.271094+0000","last_peered":"2024-09-21T10:25:06.271094+0000","last_clean":"2024-09-21T10:25:06.271094+0000","last_became_active":"2024-09-21T10:06:42.634037+0000","last_became_peered":"2024-09-21T10:06:42.634037+0000","last_unstale":"2024-09-21T10:25:06.271094+0000","last_undegraded":"2024-09-21T10:25:06.271094+0000","last_fullsized":"2024-09-21T10:25:06.271094+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:27.609590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1140,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270333+0000","last_change":"2024-09-21T10:08:06.537347+0000","last_active":"2024-09-21T10:25:06.270333+0000","last_peered":"2024-09-21T10:25:06.270333+0000","last_clean":"2024-09-21T10:25:06.270333+0000","last_became_active":"2024-09-21T10:08:06.537034+0000","last_became_peered":"2024-09-21T10:08:06.537034+0000","last_unstale":"2024-09-21T10:25:06.270333+0000","last_undegraded":"2024-09-21T10:25:06.270333+0000","last_fullsized":"2024-09-21T10:25:06.270333+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:26.325664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1149,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270773+0000","last_change":"2024-09-21T10:08:06.541436+0000","last_active":"2024-09-21T10:25:06.270773+0000","last_peered":"2024-09-21T10:25:06.270773+0000","last_clean":"2024-09-21T10:25:06.270773+0000","last_became_active":"2024-09-21T10:08:06.540616+0000","last_became_peered":"2024-09-21T10:08:06.540616+0000","last_unstale":"2024-09-21T10:25:06.270773+0000","last_undegraded":"2024-09-21T10:25:06.270773+0000","last_fullsized":"2024-09-21T10:25:06.270773+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:18.252764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1140,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.277921+0000","last_change":"2024-09-21T10:06:42.631470+0000","last_active":"2024-09-21T10:25:06.277921+0000","last_peered":"2024-09-21T10:25:06.277921+0000","last_clean":"2024-09-21T10:25:06.277921+0000","last_became_active":"2024-09-21T10:06:42.631081+0000","last_became_peered":"2024-09-21T10:06:42.631081+0000","last_unstale":"2024-09-21T10:25:06.277921+0000","last_undegraded":"2024-09-21T10:25:06.277921+0000","last_fullsized":"2024-09-21T10:25:06.277921+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:16.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270532+0000","last_change":"2024-09-21T10:08:55.113216+0000","last_active":"2024-09-21T10:25:06.270532+0000","last_peered":"2024-09-21T10:25:06.270532+0000","last_clean":"2024-09-21T10:25:06.270532+0000","last_became_active":"2024-09-21T10:08:55.111808+0000","last_became_peered":"2024-09-21T10:08:55.111808+0000","last_unstale":"2024-09-21T10:25:06.270532+0000","last_undegraded":"2024-09-21T10:25:06.270532+0000","last_fullsized":"2024-09-21T10:25:06.270532+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:24:21.526156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1310,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.288514+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:25:10.288514+0000","last_peered":"2024-09-21T10:25:10.288514+0000","last_clean":"2024-09-21T10:25:10.288514+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:25:10.288514+0000","last_undegraded":"2024-09-21T10:25:10.288514+0000","last_fullsized":"2024-09-21T10:25:10.288514+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1792,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283130+0000","last_change":"2024-09-21T10:06:42.626688+0000","last_active":"2024-09-21T10:25:10.283130+0000","last_peered":"2024-09-21T10:25:10.283130+0000","last_clean":"2024-09-21T10:25:10.283130+0000","last_became_active":"2024-09-21T10:06:42.626326+0000","last_became_peered":"2024-09-21T10:06:42.626326+0000","last_unstale":"2024-09-21T10:25:10.283130+0000","last_undegraded":"2024-09-21T10:25:10.283130+0000","last_fullsized":"2024-09-21T10:25:10.283130+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:00.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268877+0000","last_change":"2024-09-21T10:08:55.099610+0000","last_active":"2024-09-21T10:25:06.268877+0000","last_peered":"2024-09-21T10:25:06.268877+0000","last_clean":"2024-09-21T10:25:06.268877+0000","last_became_active":"2024-09-21T10:08:55.099313+0000","last_became_peered":"2024-09-21T10:08:55.099313+0000","last_unstale":"2024-09-21T10:25:06.268877+0000","last_undegraded":"2024-09-21T10:25:06.268877+0000","last_fullsized":"2024-09-21T10:25:06.268877+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:01.266798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.277737+0000","last_change":"2024-09-21T10:06:42.631675+0000","last_active":"2024-09-21T10:25:06.277737+0000","last_peered":"2024-09-21T10:25:06.277737+0000","last_clean":"2024-09-21T10:25:06.277737+0000","last_became_active":"2024-09-21T10:06:42.631098+0000","last_became_peered":"2024-09-21T10:06:42.631098+0000","last_unstale":"2024-09-21T10:25:06.277737+0000","last_undegraded":"2024-09-21T10:25:06.277737+0000","last_fullsized":"2024-09-21T10:25:06.277737+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:46.855597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1124,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.273109+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T10:25:06.273109+0000","last_peered":"2024-09-21T10:25:06.273109+0000","last_clean":"2024-09-21T10:25:06.273109+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T10:25:06.273109+0000","last_undegraded":"2024-09-21T10:25:06.273109+0000","last_fullsized":"2024-09-21T10:25:06.273109+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270127+0000","last_change":"2024-09-21T10:08:55.107939+0000","last_active":"2024-09-21T10:25:06.270127+0000","last_peered":"2024-09-21T10:25:06.270127+0000","last_clean":"2024-09-21T10:25:06.270127+0000","last_became_active":"2024-09-21T10:08:55.107640+0000","last_became_peered":"2024-09-21T10:08:55.107640+0000","last_unstale":"2024-09-21T10:25:06.270127+0000","last_undegraded":"2024-09-21T10:25:06.270127+0000","last_fullsized":"2024-09-21T10:25:06.270127+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:11.287664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1142,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.277518+0000","last_change":"2024-09-21T10:06:42.632153+0000","last_active":"2024-09-21T10:25:06.277518+0000","last_peered":"2024-09-21T10:25:06.277518+0000","last_clean":"2024-09-21T10:25:06.277518+0000","last_became_active":"2024-09-21T10:06:42.631728+0000","last_became_peered":"2024-09-21T10:06:42.631728+0000","last_unstale":"2024-09-21T10:25:06.277518+0000","last_undegraded":"2024-09-21T10:25:06.277518+0000","last_fullsized":"2024-09-21T10:25:06.277518+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:53.412364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1121,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271761+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T10:25:06.271761+0000","last_peered":"2024-09-21T10:25:06.271761+0000","last_clean":"2024-09-21T10:25:06.271761+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T10:25:06.271761+0000","last_undegraded":"2024-09-21T10:25:06.271761+0000","last_fullsized":"2024-09-21T10:25:06.271761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1141,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.273341+0000","last_change":"2024-09-21T10:06:09.030444+0000","last_active":"2024-09-21T10:25:06.273341+0000","last_peered":"2024-09-21T10:25:06.273341+0000","last_clean":"2024-09-21T10:25:06.273341+0000","last_became_active":"2024-09-21T10:06:09.030054+0000","last_became_peered":"2024-09-21T10:06:09.030054+0000","last_unstale":"2024-09-21T10:25:06.273341+0000","last_undegraded":"2024-09-21T10:25:06.273341+0000","last_fullsized":"2024-09-21T10:25:06.273341+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:27.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1148,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276034+0000","last_change":"2024-09-21T10:06:09.032476+0000","last_active":"2024-09-21T10:25:06.276034+0000","last_peered":"2024-09-21T10:25:06.276034+0000","last_clean":"2024-09-21T10:25:06.276034+0000","last_became_active":"2024-09-21T10:06:09.032121+0000","last_became_peered":"2024-09-21T10:06:09.032121+0000","last_unstale":"2024-09-21T10:25:06.276034+0000","last_undegraded":"2024-09-21T10:25:06.276034+0000","last_fullsized":"2024-09-21T10:25:06.276034+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:41.862129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1672,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281353+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:25:10.281353+0000","last_peered":"2024-09-21T10:25:10.281353+0000","last_clean":"2024-09-21T10:25:10.281353+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:25:10.281353+0000","last_undegraded":"2024-09-21T10:25:10.281353+0000","last_fullsized":"2024-09-21T10:25:10.281353+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1150,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.274920+0000","last_change":"2024-09-21T10:08:06.546911+0000","last_active":"2024-09-21T10:25:06.274920+0000","last_peered":"2024-09-21T10:25:06.274920+0000","last_clean":"2024-09-21T10:25:06.274920+0000","last_became_active":"2024-09-21T10:08:06.546417+0000","last_became_peered":"2024-09-21T10:08:06.546417+0000","last_unstale":"2024-09-21T10:25:06.274920+0000","last_undegraded":"2024-09-21T10:25:06.274920+0000","last_fullsized":"2024-09-21T10:25:06.274920+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:46.544701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1791,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283306+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:25:10.283306+0000","last_peered":"2024-09-21T10:25:10.283306+0000","last_clean":"2024-09-21T10:25:10.283306+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:25:10.283306+0000","last_undegraded":"2024-09-21T10:25:10.283306+0000","last_fullsized":"2024-09-21T10:25:10.283306+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1791,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283052+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:25:10.283052+0000","last_peered":"2024-09-21T10:25:10.283052+0000","last_clean":"2024-09-21T10:25:10.283052+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:25:10.283052+0000","last_undegraded":"2024-09-21T10:25:10.283052+0000","last_fullsized":"2024-09-21T10:25:10.283052+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1138,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268607+0000","last_change":"2024-09-21T10:08:55.102502+0000","last_active":"2024-09-21T10:25:06.268607+0000","last_peered":"2024-09-21T10:25:06.268607+0000","last_clean":"2024-09-21T10:25:06.268607+0000","last_became_active":"2024-09-21T10:08:55.101777+0000","last_became_peered":"2024-09-21T10:08:55.101777+0000","last_unstale":"2024-09-21T10:25:06.268607+0000","last_undegraded":"2024-09-21T10:25:06.268607+0000","last_fullsized":"2024-09-21T10:25:06.268607+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:51.293401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1172,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.274746+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:25:06.274746+0000","last_peered":"2024-09-21T10:25:06.274746+0000","last_clean":"2024-09-21T10:25:06.274746+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:25:06.274746+0000","last_undegraded":"2024-09-21T10:25:06.274746+0000","last_fullsized":"2024-09-21T10:25:06.274746+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1680,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281678+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:25:10.281678+0000","last_peered":"2024-09-21T10:25:10.281678+0000","last_clean":"2024-09-21T10:25:10.281678+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:25:10.281678+0000","last_undegraded":"2024-09-21T10:25:10.281678+0000","last_fullsized":"2024-09-21T10:25:10.281678+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":45,"num_pg":16,"stat_sum":{"num_bytes":8151040024,"num_objects":124376,"num_object_clones":0,"num_object_copies":373128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":124376,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135960,"num_write_kb":8330689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24455229440,"data_stored":24455217224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130960,"ondisk_log_size":130960,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":124,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397868,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3327176,"kb_used_data":3029824,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":90417976,"statfs":{"total":95995035648,"available":92588007424,"internally_reserved":0,"allocated":3102539776,"data_stored":3097986615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594097,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3331720,"kb_used_data":3030208,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90413432,"statfs":{"total":95995035648,"available":92583354368,"internally_reserved":0,"allocated":3102932992,"data_stored":3098379826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790326,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3846840,"kb_used_data":3537008,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89898312,"statfs":{"total":95995035648,"available":92055871488,"internally_reserved":0,"allocated":3621896192,"data_stored":3617354660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986554,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1842624,"kb_used_data":1532792,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91902528,"statfs":{"total":95995035648,"available":94108188672,"internally_reserved":0,"allocated":1569579008,"data_stored":1565033481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644486,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4095860,"kb_used_data":3959404,"kb_used_omap":67,"kb_used_meta":136124,"kb_avail":89649292,"statfs":{"total":95995035648,"available":91800875008,"internally_reserved":0,"allocated":4054429696,"data_stored":4049891735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":139391837},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527504,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3640372,"kb_used_data":3495980,"kb_used_omap":75,"kb_used_meta":144244,"kb_avail":90104780,"statfs":{"total":95995035648,"available":92267294720,"internally_reserved":0,"allocated":3579883520,"data_stored":3575346057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":147706010},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274657,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2588984,"kb_used_data":2471152,"kb_used_omap":51,"kb_used_meta":117708,"kb_avail":91156168,"statfs":{"total":95995035648,"available":93343916032,"internally_reserved":0,"allocated":2530459648,"data_stored":2525917628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":120533333},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353895,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3142716,"kb_used_data":3030196,"kb_used_omap":66,"kb_used_meta":112317,"kb_avail":90602436,"statfs":{"total":95995035648,"available":92776894464,"internally_reserved":0,"allocated":3102920704,"data_stored":3098375613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":115012932},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3070693376,"data_stored":3070689304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2515468288,"data_stored":2515468288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3548905472,"data_stored":3548905472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4039577600,"data_stored":4039573528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1551695872,"data_stored":1551695872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3586260992,"data_stored":3586260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3070234624,"data_stored":3070230552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3072393216,"data_stored":3072393216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:25:18.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:25:18.334 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:18.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:18.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:18.459 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:18.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:18.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:18.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:18.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:18.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:18.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:18.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:18.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:18.954 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:18.981 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:18.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:19.083 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:19.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:19.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:19.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:19.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:19.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:19.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:19.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:19.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:19.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:19.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:19.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:19.821 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:19.850 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:19.920 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:25:19.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:19.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:19.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:20.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:20.083 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:19 smithi119 ceph-mon[101432]: pgmap v1644: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 7.6 MiB/s wr, 908 op/s 2024-09-21T10:25:20.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/901345964' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:20.084 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:19 smithi119 ceph-mon[105288]: pgmap v1644: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 7.6 MiB/s wr, 908 op/s 2024-09-21T10:25:20.084 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/901345964' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:20.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:20.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:20.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:19 smithi145 ceph-mon[89359]: pgmap v1644: 89 pgs: 89 active+clean; 7.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 102 B/s rd, 7.6 MiB/s wr, 908 op/s 2024-09-21T10:25:20.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/901345964' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:20.238 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:20.238 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.248 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:20.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:20.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:20.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:20.520 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:25:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:25:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:20.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:20.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:20.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:20.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:20.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:20.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:20.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:20.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:20.995 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:20 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:20.995 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/901345964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:20.995 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:20 smithi119 ceph-mon[105288]: osdmap e1183: 8 total, 8 up, 8 in 2024-09-21T10:25:20.995 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:20 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:20.995 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/901345964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:20.995 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:20 smithi119 ceph-mon[101432]: osdmap e1183: 8 total, 8 up, 8 in 2024-09-21T10:25:21.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:21.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:21.160 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.186 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:21.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:21.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:20 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/901345964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:25:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:20 smithi145 ceph-mon[89359]: osdmap e1183: 8 total, 8 up, 8 in 2024-09-21T10:25:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:21.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:21.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:21.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:21.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:21.699 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:21.725 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:21.811 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:21.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:21.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:21.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:22.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:21 smithi119 ceph-mon[101432]: pgmap v1646: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 KiB/s wr, 910 op/s 2024-09-21T10:25:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:21 smithi119 ceph-mon[105288]: pgmap v1646: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 KiB/s wr, 910 op/s 2024-09-21T10:25:22.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.156 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:22.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:22.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:22.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:25:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:21 smithi145 ceph-mon[89359]: pgmap v1646: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 58 KiB/s wr, 910 op/s 2024-09-21T10:25:22.299 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:22.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:22.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:22.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:22.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:22.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:22.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:22.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:22.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:22.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:22.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:22.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:23.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:23.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:23.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:23.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:23.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:23.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:23.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:23.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:23.500 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:23.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:23.626 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:23.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:23.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.755 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:23.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:23.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:23.890 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:23.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:23.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:24.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:23 smithi119 ceph-mon[105288]: pgmap v1647: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-21T10:25:24.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:23 smithi119 ceph-mon[101432]: pgmap v1647: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-21T10:25:24.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:24.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:25:24.225 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:25:24.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:24.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:24.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:23 smithi145 ceph-mon[89359]: pgmap v1647: 89 pgs: 89 active+clean; 7.3 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 910 op/s 2024-09-21T10:25:24.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:24.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:24.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:24.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:24.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:24.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:24.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:24.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:25:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:25:24] "GET /metrics HTTP/1.1" 200 36767 "" "Prometheus/2.51.0" 2024-09-21T10:25:24.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:24.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:24.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:25:25.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:25.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:25.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:25.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:25.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:24 smithi145 ceph-mon[89359]: from='client.36069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:25:25.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:24 smithi119 ceph-mon[101432]: from='client.36069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:25:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:24 smithi119 ceph-mon[105288]: from='client.36069 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:25:25.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:25.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:25.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:25.494 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:25.540 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1647,"stamp":"2024-09-21T10:25:22.838560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7816288229,"num_objects":119285,"num_object_clones":0,"num_object_copies":357855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119285,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":141289,"num_write_kb":8332348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":590492,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134335,"ondisk_log_size":134335,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24834120,"kb_used_data":23104772,"kb_used_omap":268,"kb_used_meta":1728883,"kb_avail":725127096,"statfs":{"total":767960285184,"available":742530146304,"internally_reserved":0,"allocated":23659286528,"data_stored":23622915857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":274448,"internal_metadata":1770377200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-596901888,"num_objects":-9108,"num_object_clones":0,"num_object_copies":-27324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002656"},"pg_stats":[{"pgid":"45.f","version":"1183'8944","reported_seq":8993,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:22.508830+0000","last_change":"2024-09-21T10:23:24.928314+0000","last_active":"2024-09-21T10:25:22.508830+0000","last_peered":"2024-09-21T10:25:22.508830+0000","last_clean":"2024-09-21T10:25:22.508830+0000","last_became_active":"2024-09-21T10:23:24.928005+0000","last_became_peered":"2024-09-21T10:23:24.928005+0000","last_unstale":"2024-09-21T10:25:22.508830+0000","last_undegraded":"2024-09-21T10:25:22.508830+0000","last_fullsized":"2024-09-21T10:25:22.508830+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8944,"log_dups_size":0,"ondisk_log_size":8944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:15.220461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474480640,"num_objects":7240,"num_object_clones":0,"num_object_copies":21720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8944,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.e","version":"1182'8705","reported_seq":8731,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.519142+0000","last_change":"2024-09-21T10:23:25.544908+0000","last_active":"2024-09-21T10:25:18.519142+0000","last_peered":"2024-09-21T10:25:18.519142+0000","last_clean":"2024-09-21T10:25:18.519142+0000","last_became_active":"2024-09-21T10:23:25.544575+0000","last_became_peered":"2024-09-21T10:23:25.544575+0000","last_unstale":"2024-09-21T10:25:18.519142+0000","last_undegraded":"2024-09-21T10:25:18.519142+0000","last_fullsized":"2024-09-21T10:25:18.519142+0000","mapping_epoch":1164,"log_start":"1169'100","ondisk_log_start":"1169'100","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8605,"log_dups_size":0,"ondisk_log_size":8605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:15:16.735472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488177664,"num_objects":7449,"num_object_clones":0,"num_object_copies":22347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.d","version":"1182'8817","reported_seq":8864,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.947225+0000","last_change":"2024-09-21T10:23:24.927661+0000","last_active":"2024-09-21T10:25:18.947225+0000","last_peered":"2024-09-21T10:25:18.947225+0000","last_clean":"2024-09-21T10:25:18.947225+0000","last_became_active":"2024-09-21T10:23:24.927398+0000","last_became_peered":"2024-09-21T10:23:24.927398+0000","last_unstale":"2024-09-21T10:25:18.947225+0000","last_undegraded":"2024-09-21T10:25:18.947225+0000","last_fullsized":"2024-09-21T10:25:18.947225+0000","mapping_epoch":1164,"log_start":"1169'900","ondisk_log_start":"1169'900","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7917,"log_dups_size":0,"ondisk_log_size":7917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:19.561056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488767512,"num_objects":7459,"num_object_clones":0,"num_object_copies":22377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7459,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8817,"num_write_kb":520769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.c","version":"1182'8866","reported_seq":8892,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.534256+0000","last_change":"2024-09-21T10:23:24.924772+0000","last_active":"2024-09-21T10:25:18.534256+0000","last_peered":"2024-09-21T10:25:18.534256+0000","last_clean":"2024-09-21T10:25:18.534256+0000","last_became_active":"2024-09-21T10:23:24.924461+0000","last_became_peered":"2024-09-21T10:23:24.924461+0000","last_unstale":"2024-09-21T10:25:18.534256+0000","last_undegraded":"2024-09-21T10:25:18.534256+0000","last_fullsized":"2024-09-21T10:25:18.534256+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8866,"log_dups_size":0,"ondisk_log_size":8866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:25:04.341096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492306432,"num_objects":7512,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8866,"num_write_kb":524096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.b","version":"1183'8856","reported_seq":8905,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:20.349256+0000","last_change":"2024-09-21T10:23:25.544535+0000","last_active":"2024-09-21T10:25:20.349256+0000","last_peered":"2024-09-21T10:25:20.349256+0000","last_clean":"2024-09-21T10:25:20.349256+0000","last_became_active":"2024-09-21T10:23:25.544215+0000","last_became_peered":"2024-09-21T10:23:25.544215+0000","last_unstale":"2024-09-21T10:25:20.349256+0000","last_undegraded":"2024-09-21T10:25:20.349256+0000","last_fullsized":"2024-09-21T10:25:20.349256+0000","mapping_epoch":1164,"log_start":"1170'1700","ondisk_log_start":"1170'1700","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7156,"log_dups_size":0,"ondisk_log_size":7156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:47.760255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":487587840,"num_objects":7440,"num_object_clones":0,"num_object_copies":22320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8856,"num_write_kb":521472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.a","version":"1182'8713","reported_seq":8739,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.582076+0000","last_change":"2024-09-21T10:23:24.926178+0000","last_active":"2024-09-21T10:25:18.582076+0000","last_peered":"2024-09-21T10:25:18.582076+0000","last_clean":"2024-09-21T10:25:18.582076+0000","last_became_active":"2024-09-21T10:23:24.925883+0000","last_became_peered":"2024-09-21T10:23:24.925883+0000","last_unstale":"2024-09-21T10:25:18.582076+0000","last_undegraded":"2024-09-21T10:25:18.582076+0000","last_fullsized":"2024-09-21T10:25:18.582076+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8713,"log_dups_size":0,"ondisk_log_size":8713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:23.995550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484114432,"num_objects":7387,"num_object_clones":0,"num_object_copies":22161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8713,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.9","version":"1182'8928","reported_seq":8954,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.556622+0000","last_change":"2024-09-21T10:23:24.924699+0000","last_active":"2024-09-21T10:25:18.556622+0000","last_peered":"2024-09-21T10:25:18.556622+0000","last_clean":"2024-09-21T10:25:18.556622+0000","last_became_active":"2024-09-21T10:23:24.924416+0000","last_became_peered":"2024-09-21T10:23:24.924416+0000","last_unstale":"2024-09-21T10:25:18.556622+0000","last_undegraded":"2024-09-21T10:25:18.556622+0000","last_fullsized":"2024-09-21T10:25:18.556622+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8928,"log_dups_size":0,"ondisk_log_size":8928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:07:37.348206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503578624,"num_objects":7684,"num_object_clones":0,"num_object_copies":23052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":531584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.8","version":"1183'8835","reported_seq":8884,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:20.368178+0000","last_change":"2024-09-21T10:23:25.544569+0000","last_active":"2024-09-21T10:25:20.368178+0000","last_peered":"2024-09-21T10:25:20.368178+0000","last_clean":"2024-09-21T10:25:20.368178+0000","last_became_active":"2024-09-21T10:23:25.544266+0000","last_became_peered":"2024-09-21T10:23:25.544266+0000","last_unstale":"2024-09-21T10:25:20.368178+0000","last_undegraded":"2024-09-21T10:25:20.368178+0000","last_fullsized":"2024-09-21T10:25:20.368178+0000","mapping_epoch":1164,"log_start":"1170'1600","ondisk_log_start":"1170'1600","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7235,"log_dups_size":0,"ondisk_log_size":7235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:16.355961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477691904,"num_objects":7289,"num_object_clones":0,"num_object_copies":21867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8835,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.7","version":"1182'8812","reported_seq":8838,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.546235+0000","last_change":"2024-09-21T10:23:24.926188+0000","last_active":"2024-09-21T10:25:18.546235+0000","last_peered":"2024-09-21T10:25:18.546235+0000","last_clean":"2024-09-21T10:25:18.546235+0000","last_became_active":"2024-09-21T10:23:24.925897+0000","last_became_peered":"2024-09-21T10:23:24.925897+0000","last_unstale":"2024-09-21T10:25:18.546235+0000","last_undegraded":"2024-09-21T10:25:18.546235+0000","last_fullsized":"2024-09-21T10:25:18.546235+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8812,"log_dups_size":0,"ondisk_log_size":8812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:59:46.332054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":499122176,"num_objects":7616,"num_object_clones":0,"num_object_copies":22848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8812,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.6","version":"1182'8846","reported_seq":8890,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.942615+0000","last_change":"2024-09-21T10:23:25.546048+0000","last_active":"2024-09-21T10:25:18.942615+0000","last_peered":"2024-09-21T10:25:18.942615+0000","last_clean":"2024-09-21T10:25:18.942615+0000","last_became_active":"2024-09-21T10:23:25.545764+0000","last_became_peered":"2024-09-21T10:23:25.545764+0000","last_unstale":"2024-09-21T10:25:18.942615+0000","last_undegraded":"2024-09-21T10:25:18.942615+0000","last_fullsized":"2024-09-21T10:25:18.942615+0000","mapping_epoch":1164,"log_start":"1169'900","ondisk_log_start":"1169'900","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7946,"log_dups_size":0,"ondisk_log_size":7946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:48:39.460646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":492306432,"num_objects":7512,"num_object_clones":0,"num_object_copies":22536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.5","version":"1183'8689","reported_seq":8738,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:22.382172+0000","last_change":"2024-09-21T10:23:24.927838+0000","last_active":"2024-09-21T10:25:22.382172+0000","last_peered":"2024-09-21T10:25:22.382172+0000","last_clean":"2024-09-21T10:25:22.382172+0000","last_became_active":"2024-09-21T10:23:24.927541+0000","last_became_peered":"2024-09-21T10:23:24.927541+0000","last_unstale":"2024-09-21T10:25:22.382172+0000","last_undegraded":"2024-09-21T10:25:22.382172+0000","last_fullsized":"2024-09-21T10:25:22.382172+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8689,"log_dups_size":0,"ondisk_log_size":8689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:16.618768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459341824,"num_objects":7009,"num_object_clones":0,"num_object_copies":21027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8689,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.4","version":"1182'8888","reported_seq":8934,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.191457+0000","last_change":"2024-09-21T10:23:25.544626+0000","last_active":"2024-09-21T10:25:19.191457+0000","last_peered":"2024-09-21T10:25:19.191457+0000","last_clean":"2024-09-21T10:25:19.191457+0000","last_became_active":"2024-09-21T10:23:25.544315+0000","last_became_peered":"2024-09-21T10:23:25.544315+0000","last_unstale":"2024-09-21T10:25:19.191457+0000","last_undegraded":"2024-09-21T10:25:19.191457+0000","last_fullsized":"2024-09-21T10:25:19.191457+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8888,"log_dups_size":0,"ondisk_log_size":8888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:03:41.350175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495714304,"num_objects":7564,"num_object_clones":0,"num_object_copies":22692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8888,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.3","version":"1182'8787","reported_seq":8834,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.923906+0000","last_change":"2024-09-21T10:23:24.927687+0000","last_active":"2024-09-21T10:25:18.923906+0000","last_peered":"2024-09-21T10:25:18.923906+0000","last_clean":"2024-09-21T10:25:18.923906+0000","last_became_active":"2024-09-21T10:23:24.927161+0000","last_became_peered":"2024-09-21T10:23:24.927161+0000","last_unstale":"2024-09-21T10:25:18.923906+0000","last_undegraded":"2024-09-21T10:25:18.923906+0000","last_fullsized":"2024-09-21T10:25:18.923906+0000","mapping_epoch":1164,"log_start":"1169'800","ondisk_log_start":"1169'800","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7987,"log_dups_size":0,"ondisk_log_size":7987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:28.137305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":488439808,"num_objects":7453,"num_object_clones":0,"num_object_copies":22359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8787,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.2","version":"1182'8699","reported_seq":8725,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.581385+0000","last_change":"2024-09-21T10:23:24.927278+0000","last_active":"2024-09-21T10:25:18.581385+0000","last_peered":"2024-09-21T10:25:18.581385+0000","last_clean":"2024-09-21T10:25:18.581385+0000","last_became_active":"2024-09-21T10:23:24.926970+0000","last_became_peered":"2024-09-21T10:23:24.926970+0000","last_unstale":"2024-09-21T10:25:18.581385+0000","last_undegraded":"2024-09-21T10:25:18.581385+0000","last_fullsized":"2024-09-21T10:25:18.581385+0000","mapping_epoch":1164,"log_start":"1169'100","ondisk_log_start":"1169'100","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8599,"log_dups_size":0,"ondisk_log_size":8599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:15:17.823705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489488384,"num_objects":7469,"num_object_clones":0,"num_object_copies":22407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8699,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.1","version":"1182'8977","reported_seq":9003,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.511162+0000","last_change":"2024-09-21T10:23:25.545025+0000","last_active":"2024-09-21T10:25:18.511162+0000","last_peered":"2024-09-21T10:25:18.511162+0000","last_clean":"2024-09-21T10:25:18.511162+0000","last_became_active":"2024-09-21T10:23:25.544657+0000","last_became_peered":"2024-09-21T10:23:25.544657+0000","last_unstale":"2024-09-21T10:25:18.511162+0000","last_undegraded":"2024-09-21T10:25:18.511162+0000","last_fullsized":"2024-09-21T10:25:18.511162+0000","mapping_epoch":1164,"log_start":"0'0","ondisk_log_start":"0'0","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8977,"log_dups_size":0,"ondisk_log_size":8977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:36.979666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505479168,"num_objects":7713,"num_object_clones":0,"num_object_copies":23139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8977,"num_write_kb":534080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"45.0","version":"1182'8715","reported_seq":8762,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:18.930728+0000","last_change":"2024-09-21T10:23:25.545529+0000","last_active":"2024-09-21T10:25:18.930728+0000","last_peered":"2024-09-21T10:25:18.930728+0000","last_clean":"2024-09-21T10:25:18.930728+0000","last_became_active":"2024-09-21T10:23:25.545200+0000","last_became_peered":"2024-09-21T10:23:25.545200+0000","last_unstale":"2024-09-21T10:25:18.930728+0000","last_undegraded":"2024-09-21T10:25:18.930728+0000","last_fullsized":"2024-09-21T10:25:18.930728+0000","mapping_epoch":1164,"log_start":"1169'800","ondisk_log_start":"1169'800","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7915,"log_dups_size":0,"ondisk_log_size":7915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:47:39.621176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489095168,"num_objects":7463,"num_object_clones":0,"num_object_copies":22389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8715,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":1801,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933560+0000","last_change":"2024-09-21T10:08:06.538833+0000","last_active":"2024-09-21T10:25:19.933560+0000","last_peered":"2024-09-21T10:25:19.933560+0000","last_clean":"2024-09-21T10:25:19.933560+0000","last_became_active":"2024-09-21T10:08:06.538526+0000","last_became_peered":"2024-09-21T10:08:06.538526+0000","last_unstale":"2024-09-21T10:25:19.933560+0000","last_undegraded":"2024-09-21T10:25:19.933560+0000","last_fullsized":"2024-09-21T10:25:19.933560+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:03:59.194474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.269848+0000","last_change":"2024-09-21T10:06:42.626454+0000","last_active":"2024-09-21T10:25:06.269848+0000","last_peered":"2024-09-21T10:25:06.269848+0000","last_clean":"2024-09-21T10:25:06.269848+0000","last_became_active":"2024-09-21T10:06:42.626116+0000","last_became_peered":"2024-09-21T10:06:42.626116+0000","last_unstale":"2024-09-21T10:25:06.269848+0000","last_undegraded":"2024-09-21T10:25:06.269848+0000","last_fullsized":"2024-09-21T10:25:06.269848+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:01.968111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1667,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.293667+0000","last_change":"2024-09-21T10:08:55.142962+0000","last_active":"2024-09-21T10:25:10.293667+0000","last_peered":"2024-09-21T10:25:10.293667+0000","last_clean":"2024-09-21T10:25:10.293667+0000","last_became_active":"2024-09-21T10:08:55.139681+0000","last_became_peered":"2024-09-21T10:08:55.139681+0000","last_unstale":"2024-09-21T10:25:10.293667+0000","last_undegraded":"2024-09-21T10:25:10.293667+0000","last_fullsized":"2024-09-21T10:25:10.293667+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:00:52.577104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1697,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281342+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:25:10.281342+0000","last_peered":"2024-09-21T10:25:10.281342+0000","last_clean":"2024-09-21T10:25:10.281342+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:25:10.281342+0000","last_undegraded":"2024-09-21T10:25:10.281342+0000","last_fullsized":"2024-09-21T10:25:10.281342+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":1124,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268663+0000","last_change":"2024-09-21T09:57:00.008356+0000","last_active":"2024-09-21T10:25:06.268663+0000","last_peered":"2024-09-21T10:25:06.268663+0000","last_clean":"2024-09-21T10:25:06.268663+0000","last_became_active":"2024-09-21T09:57:00.008029+0000","last_became_peered":"2024-09-21T09:57:00.008029+0000","last_unstale":"2024-09-21T10:25:06.268663+0000","last_undegraded":"2024-09-21T10:25:06.268663+0000","last_fullsized":"2024-09-21T10:25:06.268663+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:14:28.811556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270164+0000","last_change":"2024-09-21T10:08:55.108381+0000","last_active":"2024-09-21T10:25:06.270164+0000","last_peered":"2024-09-21T10:25:06.270164+0000","last_clean":"2024-09-21T10:25:06.270164+0000","last_became_active":"2024-09-21T10:08:55.108074+0000","last_became_peered":"2024-09-21T10:08:55.108074+0000","last_unstale":"2024-09-21T10:25:06.270164+0000","last_undegraded":"2024-09-21T10:25:06.270164+0000","last_fullsized":"2024-09-21T10:25:06.270164+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:13:01.846660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":1141,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268801+0000","last_change":"2024-09-21T10:06:42.622644+0000","last_active":"2024-09-21T10:25:06.268801+0000","last_peered":"2024-09-21T10:25:06.268801+0000","last_clean":"2024-09-21T10:25:06.268801+0000","last_became_active":"2024-09-21T10:06:42.621612+0000","last_became_peered":"2024-09-21T10:06:42.621612+0000","last_unstale":"2024-09-21T10:25:06.268801+0000","last_undegraded":"2024-09-21T10:25:06.268801+0000","last_fullsized":"2024-09-21T10:25:06.268801+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:21.301285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270377+0000","last_change":"2024-09-21T10:08:55.109486+0000","last_active":"2024-09-21T10:25:06.270377+0000","last_peered":"2024-09-21T10:25:06.270377+0000","last_clean":"2024-09-21T10:25:06.270377+0000","last_became_active":"2024-09-21T10:08:55.109050+0000","last_became_peered":"2024-09-21T10:08:55.109050+0000","last_unstale":"2024-09-21T10:25:06.270377+0000","last_undegraded":"2024-09-21T10:25:06.270377+0000","last_fullsized":"2024-09-21T10:25:06.270377+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:13.242342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1797,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933072+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:25:19.933072+0000","last_peered":"2024-09-21T10:25:19.933072+0000","last_clean":"2024-09-21T10:25:19.933072+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:25:19.933072+0000","last_undegraded":"2024-09-21T10:25:19.933072+0000","last_fullsized":"2024-09-21T10:25:19.933072+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1695,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.297064+0000","last_change":"2024-09-21T10:08:55.117054+0000","last_active":"2024-09-21T10:25:10.297064+0000","last_peered":"2024-09-21T10:25:10.297064+0000","last_clean":"2024-09-21T10:25:10.297064+0000","last_became_active":"2024-09-21T10:08:55.116767+0000","last_became_peered":"2024-09-21T10:08:55.116767+0000","last_unstale":"2024-09-21T10:25:10.297064+0000","last_undegraded":"2024-09-21T10:25:10.297064+0000","last_fullsized":"2024-09-21T10:25:10.297064+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:13.520809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1158,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270290+0000","last_change":"2024-09-21T10:08:06.539436+0000","last_active":"2024-09-21T10:25:06.270290+0000","last_peered":"2024-09-21T10:25:06.270290+0000","last_clean":"2024-09-21T10:25:06.270290+0000","last_became_active":"2024-09-21T10:08:06.538856+0000","last_became_peered":"2024-09-21T10:08:06.538856+0000","last_unstale":"2024-09-21T10:25:06.270290+0000","last_undegraded":"2024-09-21T10:25:06.270290+0000","last_fullsized":"2024-09-21T10:25:06.270290+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:06:43.502357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1662,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.285973+0000","last_change":"2024-09-21T10:08:55.116807+0000","last_active":"2024-09-21T10:25:10.285973+0000","last_peered":"2024-09-21T10:25:10.285973+0000","last_clean":"2024-09-21T10:25:10.285973+0000","last_became_active":"2024-09-21T10:08:55.115948+0000","last_became_peered":"2024-09-21T10:08:55.115948+0000","last_unstale":"2024-09-21T10:25:10.285973+0000","last_undegraded":"2024-09-21T10:25:10.285973+0000","last_fullsized":"2024-09-21T10:25:10.285973+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:21.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1769,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930143+0000","last_change":"2024-09-21T10:08:55.109470+0000","last_active":"2024-09-21T10:25:19.930143+0000","last_peered":"2024-09-21T10:25:19.930143+0000","last_clean":"2024-09-21T10:25:19.930143+0000","last_became_active":"2024-09-21T10:08:55.106885+0000","last_became_peered":"2024-09-21T10:08:55.106885+0000","last_unstale":"2024-09-21T10:25:19.930143+0000","last_undegraded":"2024-09-21T10:25:19.930143+0000","last_fullsized":"2024-09-21T10:25:19.930143+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:51.866025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1795,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933684+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:25:19.933684+0000","last_peered":"2024-09-21T10:25:19.933684+0000","last_clean":"2024-09-21T10:25:19.933684+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:25:19.933684+0000","last_undegraded":"2024-09-21T10:25:19.933684+0000","last_fullsized":"2024-09-21T10:25:19.933684+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1698,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283631+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:25:10.283631+0000","last_peered":"2024-09-21T10:25:10.283631+0000","last_clean":"2024-09-21T10:25:10.283631+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:25:10.283631+0000","last_undegraded":"2024-09-21T10:25:10.283631+0000","last_fullsized":"2024-09-21T10:25:10.283631+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276358+0000","last_change":"2024-09-21T10:06:42.631648+0000","last_active":"2024-09-21T10:25:06.276358+0000","last_peered":"2024-09-21T10:25:06.276358+0000","last_clean":"2024-09-21T10:25:06.276358+0000","last_became_active":"2024-09-21T10:06:42.631301+0000","last_became_peered":"2024-09-21T10:06:42.631301+0000","last_unstale":"2024-09-21T10:25:06.276358+0000","last_undegraded":"2024-09-21T10:25:06.276358+0000","last_fullsized":"2024-09-21T10:25:06.276358+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:52.231304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1681,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283941+0000","last_change":"2024-09-21T10:08:55.134315+0000","last_active":"2024-09-21T10:25:10.283941+0000","last_peered":"2024-09-21T10:25:10.283941+0000","last_clean":"2024-09-21T10:25:10.283941+0000","last_became_active":"2024-09-21T10:08:55.132999+0000","last_became_peered":"2024-09-21T10:08:55.132999+0000","last_unstale":"2024-09-21T10:25:10.283941+0000","last_undegraded":"2024-09-21T10:25:10.283941+0000","last_fullsized":"2024-09-21T10:25:10.283941+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:24.870318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1792,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933647+0000","last_change":"2024-09-21T10:08:55.105723+0000","last_active":"2024-09-21T10:25:19.933647+0000","last_peered":"2024-09-21T10:25:19.933647+0000","last_clean":"2024-09-21T10:25:19.933647+0000","last_became_active":"2024-09-21T10:08:55.105283+0000","last_became_peered":"2024-09-21T10:08:55.105283+0000","last_unstale":"2024-09-21T10:25:19.933647+0000","last_undegraded":"2024-09-21T10:25:19.933647+0000","last_fullsized":"2024-09-21T10:25:19.933647+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:10:25.633107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1664,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281617+0000","last_change":"2024-09-21T10:08:55.131455+0000","last_active":"2024-09-21T10:25:10.281617+0000","last_peered":"2024-09-21T10:25:10.281617+0000","last_clean":"2024-09-21T10:25:10.281617+0000","last_became_active":"2024-09-21T10:08:55.129639+0000","last_became_peered":"2024-09-21T10:08:55.129639+0000","last_unstale":"2024-09-21T10:25:10.281617+0000","last_undegraded":"2024-09-21T10:25:10.281617+0000","last_fullsized":"2024-09-21T10:25:10.281617+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:50.934309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271761+0000","last_change":"2024-09-21T10:06:42.623974+0000","last_active":"2024-09-21T10:25:06.271761+0000","last_peered":"2024-09-21T10:25:06.271761+0000","last_clean":"2024-09-21T10:25:06.271761+0000","last_became_active":"2024-09-21T10:06:42.623598+0000","last_became_peered":"2024-09-21T10:06:42.623598+0000","last_unstale":"2024-09-21T10:25:06.271761+0000","last_undegraded":"2024-09-21T10:25:06.271761+0000","last_fullsized":"2024-09-21T10:25:06.271761+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:03.261148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1313,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.929918+0000","last_change":"2024-09-21T10:08:55.106181+0000","last_active":"2024-09-21T10:25:19.929918+0000","last_peered":"2024-09-21T10:25:19.929918+0000","last_clean":"2024-09-21T10:25:19.929918+0000","last_became_active":"2024-09-21T10:08:55.105882+0000","last_became_peered":"2024-09-21T10:08:55.105882+0000","last_unstale":"2024-09-21T10:25:19.929918+0000","last_undegraded":"2024-09-21T10:25:19.929918+0000","last_fullsized":"2024-09-21T10:25:19.929918+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:30.425874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1769,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.931009+0000","last_change":"2024-09-21T10:08:06.538126+0000","last_active":"2024-09-21T10:25:19.931009+0000","last_peered":"2024-09-21T10:25:19.931009+0000","last_clean":"2024-09-21T10:25:19.931009+0000","last_became_active":"2024-09-21T10:08:06.537779+0000","last_became_peered":"2024-09-21T10:08:06.537779+0000","last_unstale":"2024-09-21T10:25:19.931009+0000","last_undegraded":"2024-09-21T10:25:19.931009+0000","last_fullsized":"2024-09-21T10:25:19.931009+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:34.731581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1145,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270602+0000","last_change":"2024-09-21T09:57:00.007148+0000","last_active":"2024-09-21T10:25:06.270602+0000","last_peered":"2024-09-21T10:25:06.270602+0000","last_clean":"2024-09-21T10:25:06.270602+0000","last_became_active":"2024-09-21T09:57:00.005908+0000","last_became_peered":"2024-09-21T09:57:00.005908+0000","last_unstale":"2024-09-21T10:25:06.270602+0000","last_undegraded":"2024-09-21T10:25:06.270602+0000","last_fullsized":"2024-09-21T10:25:06.270602+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:14.605988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270444+0000","last_change":"2024-09-21T10:06:42.633523+0000","last_active":"2024-09-21T10:25:06.270444+0000","last_peered":"2024-09-21T10:25:06.270444+0000","last_clean":"2024-09-21T10:25:06.270444+0000","last_became_active":"2024-09-21T10:06:42.633019+0000","last_became_peered":"2024-09-21T10:06:42.633019+0000","last_unstale":"2024-09-21T10:25:06.270444+0000","last_undegraded":"2024-09-21T10:25:06.270444+0000","last_fullsized":"2024-09-21T10:25:06.270444+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:36:54.272637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1122,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270255+0000","last_change":"2024-09-21T09:57:00.093970+0000","last_active":"2024-09-21T10:25:06.270255+0000","last_peered":"2024-09-21T10:25:06.270255+0000","last_clean":"2024-09-21T10:25:06.270255+0000","last_became_active":"2024-09-21T09:57:00.093558+0000","last_became_peered":"2024-09-21T09:57:00.093558+0000","last_unstale":"2024-09-21T10:25:06.270255+0000","last_undegraded":"2024-09-21T10:25:06.270255+0000","last_fullsized":"2024-09-21T10:25:06.270255+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:36.246785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1153,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276426+0000","last_change":"2024-09-21T10:08:06.551874+0000","last_active":"2024-09-21T10:25:06.276426+0000","last_peered":"2024-09-21T10:25:06.276426+0000","last_clean":"2024-09-21T10:25:06.276426+0000","last_became_active":"2024-09-21T10:08:06.551572+0000","last_became_peered":"2024-09-21T10:08:06.551572+0000","last_unstale":"2024-09-21T10:25:06.276426+0000","last_undegraded":"2024-09-21T10:25:06.276426+0000","last_fullsized":"2024-09-21T10:25:06.276426+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:56.324479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1797,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933510+0000","last_change":"2024-09-21T10:06:42.626709+0000","last_active":"2024-09-21T10:25:19.933510+0000","last_peered":"2024-09-21T10:25:19.933510+0000","last_clean":"2024-09-21T10:25:19.933510+0000","last_became_active":"2024-09-21T10:06:42.626389+0000","last_became_peered":"2024-09-21T10:06:42.626389+0000","last_unstale":"2024-09-21T10:25:19.933510+0000","last_undegraded":"2024-09-21T10:25:19.933510+0000","last_fullsized":"2024-09-21T10:25:19.933510+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:39.634788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1682,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283719+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:25:10.283719+0000","last_peered":"2024-09-21T10:25:10.283719+0000","last_clean":"2024-09-21T10:25:10.283719+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:25:10.283719+0000","last_undegraded":"2024-09-21T10:25:10.283719+0000","last_fullsized":"2024-09-21T10:25:10.283719+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1767,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930534+0000","last_change":"2024-09-21T10:06:42.630257+0000","last_active":"2024-09-21T10:25:19.930534+0000","last_peered":"2024-09-21T10:25:19.930534+0000","last_clean":"2024-09-21T10:25:19.930534+0000","last_became_active":"2024-09-21T10:06:42.628412+0000","last_became_peered":"2024-09-21T10:06:42.628412+0000","last_unstale":"2024-09-21T10:25:19.930534+0000","last_undegraded":"2024-09-21T10:25:19.930534+0000","last_fullsized":"2024-09-21T10:25:19.930534+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:14:30.166565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1673,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.282177+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:25:10.282177+0000","last_peered":"2024-09-21T10:25:10.282177+0000","last_clean":"2024-09-21T10:25:10.282177+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:25:10.282177+0000","last_undegraded":"2024-09-21T10:25:10.282177+0000","last_fullsized":"2024-09-21T10:25:10.282177+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1784,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933255+0000","last_change":"2024-09-21T10:06:09.039187+0000","last_active":"2024-09-21T10:25:19.933255+0000","last_peered":"2024-09-21T10:25:19.933255+0000","last_clean":"2024-09-21T10:25:19.933255+0000","last_became_active":"2024-09-21T10:06:09.038741+0000","last_became_peered":"2024-09-21T10:06:09.038741+0000","last_unstale":"2024-09-21T10:25:19.933255+0000","last_undegraded":"2024-09-21T10:25:19.933255+0000","last_fullsized":"2024-09-21T10:25:19.933255+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:24:40.920128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1152,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.273185+0000","last_change":"2024-09-21T10:06:09.028808+0000","last_active":"2024-09-21T10:25:06.273185+0000","last_peered":"2024-09-21T10:25:06.273185+0000","last_clean":"2024-09-21T10:25:06.273185+0000","last_became_active":"2024-09-21T10:06:09.028492+0000","last_became_peered":"2024-09-21T10:06:09.028492+0000","last_unstale":"2024-09-21T10:25:06.273185+0000","last_undegraded":"2024-09-21T10:25:06.273185+0000","last_fullsized":"2024-09-21T10:25:06.273185+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:52:41.134185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1769,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.929944+0000","last_change":"2024-09-21T10:08:06.539915+0000","last_active":"2024-09-21T10:25:19.929944+0000","last_peered":"2024-09-21T10:25:19.929944+0000","last_clean":"2024-09-21T10:25:19.929944+0000","last_became_active":"2024-09-21T10:08:06.539121+0000","last_became_peered":"2024-09-21T10:08:06.539121+0000","last_unstale":"2024-09-21T10:25:19.929944+0000","last_undegraded":"2024-09-21T10:25:19.929944+0000","last_fullsized":"2024-09-21T10:25:19.929944+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:23.549392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1758,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.931093+0000","last_change":"2024-09-21T10:06:42.625569+0000","last_active":"2024-09-21T10:25:19.931093+0000","last_peered":"2024-09-21T10:25:19.931093+0000","last_clean":"2024-09-21T10:25:19.931093+0000","last_became_active":"2024-09-21T10:06:42.625259+0000","last_became_peered":"2024-09-21T10:06:42.625259+0000","last_unstale":"2024-09-21T10:25:19.931093+0000","last_undegraded":"2024-09-21T10:25:19.931093+0000","last_fullsized":"2024-09-21T10:25:19.931093+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:25:45.833707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1783,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930117+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:25:19.930117+0000","last_peered":"2024-09-21T10:25:19.930117+0000","last_clean":"2024-09-21T10:25:19.930117+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:25:19.930117+0000","last_undegraded":"2024-09-21T10:25:19.930117+0000","last_fullsized":"2024-09-21T10:25:19.930117+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268673+0000","last_change":"2024-09-21T10:08:55.102678+0000","last_active":"2024-09-21T10:25:06.268673+0000","last_peered":"2024-09-21T10:25:06.268673+0000","last_clean":"2024-09-21T10:25:06.268673+0000","last_became_active":"2024-09-21T10:08:55.102022+0000","last_became_peered":"2024-09-21T10:08:55.102022+0000","last_unstale":"2024-09-21T10:25:06.268673+0000","last_undegraded":"2024-09-21T10:25:06.268673+0000","last_fullsized":"2024-09-21T10:25:06.268673+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:03.178484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1780,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930036+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:25:19.930036+0000","last_peered":"2024-09-21T10:25:19.930036+0000","last_clean":"2024-09-21T10:25:19.930036+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:25:19.930036+0000","last_undegraded":"2024-09-21T10:25:19.930036+0000","last_fullsized":"2024-09-21T10:25:19.930036+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1140,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270516+0000","last_change":"2024-09-21T10:06:09.043111+0000","last_active":"2024-09-21T10:25:06.270516+0000","last_peered":"2024-09-21T10:25:06.270516+0000","last_clean":"2024-09-21T10:25:06.270516+0000","last_became_active":"2024-09-21T10:06:09.042813+0000","last_became_peered":"2024-09-21T10:06:09.042813+0000","last_unstale":"2024-09-21T10:25:06.270516+0000","last_undegraded":"2024-09-21T10:25:06.270516+0000","last_fullsized":"2024-09-21T10:25:06.270516+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:50.398531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":1153,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268243+0000","last_change":"2024-09-21T10:06:42.622375+0000","last_active":"2024-09-21T10:25:06.268243+0000","last_peered":"2024-09-21T10:25:06.268243+0000","last_clean":"2024-09-21T10:25:06.268243+0000","last_became_active":"2024-09-21T10:06:42.622019+0000","last_became_peered":"2024-09-21T10:06:42.622019+0000","last_unstale":"2024-09-21T10:25:06.268243+0000","last_undegraded":"2024-09-21T10:25:06.268243+0000","last_fullsized":"2024-09-21T10:25:06.268243+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:22.917477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073014599999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1771,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930787+0000","last_change":"2024-09-21T10:06:42.627252+0000","last_active":"2024-09-21T10:25:19.930787+0000","last_peered":"2024-09-21T10:25:19.930787+0000","last_clean":"2024-09-21T10:25:19.930787+0000","last_became_active":"2024-09-21T10:06:42.626942+0000","last_became_peered":"2024-09-21T10:06:42.626942+0000","last_unstale":"2024-09-21T10:25:19.930787+0000","last_undegraded":"2024-09-21T10:25:19.930787+0000","last_fullsized":"2024-09-21T10:25:19.930787+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:21.844784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1770,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.935361+0000","last_change":"2024-09-21T10:08:06.532136+0000","last_active":"2024-09-21T10:25:19.935361+0000","last_peered":"2024-09-21T10:25:19.935361+0000","last_clean":"2024-09-21T10:25:19.935361+0000","last_became_active":"2024-09-21T10:08:06.531195+0000","last_became_peered":"2024-09-21T10:08:06.531195+0000","last_unstale":"2024-09-21T10:25:19.935361+0000","last_undegraded":"2024-09-21T10:25:19.935361+0000","last_fullsized":"2024-09-21T10:25:19.935361+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:48:02.094677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1711,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.297118+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:25:10.297118+0000","last_peered":"2024-09-21T10:25:10.297118+0000","last_clean":"2024-09-21T10:25:10.297118+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:25:10.297118+0000","last_undegraded":"2024-09-21T10:25:10.297118+0000","last_fullsized":"2024-09-21T10:25:10.297118+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1805,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933759+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:25:19.933759+0000","last_peered":"2024-09-21T10:25:19.933759+0000","last_clean":"2024-09-21T10:25:19.933759+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:25:19.933759+0000","last_undegraded":"2024-09-21T10:25:19.933759+0000","last_fullsized":"2024-09-21T10:25:19.933759+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1806,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.937776+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:25:19.937776+0000","last_peered":"2024-09-21T10:25:19.937776+0000","last_clean":"2024-09-21T10:25:19.937776+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:25:19.937776+0000","last_undegraded":"2024-09-21T10:25:19.937776+0000","last_fullsized":"2024-09-21T10:25:19.937776+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1169,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271665+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:25:06.271665+0000","last_peered":"2024-09-21T10:25:06.271665+0000","last_clean":"2024-09-21T10:25:06.271665+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:25:06.271665+0000","last_undegraded":"2024-09-21T10:25:06.271665+0000","last_fullsized":"2024-09-21T10:25:06.271665+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1122,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276479+0000","last_change":"2024-09-21T09:57:01.043713+0000","last_active":"2024-09-21T10:25:06.276479+0000","last_peered":"2024-09-21T10:25:06.276479+0000","last_clean":"2024-09-21T10:25:06.276479+0000","last_became_active":"2024-09-21T09:57:01.043415+0000","last_became_peered":"2024-09-21T09:57:01.043415+0000","last_unstale":"2024-09-21T10:25:06.276479+0000","last_undegraded":"2024-09-21T10:25:06.276479+0000","last_fullsized":"2024-09-21T10:25:06.276479+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:33.915799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1665,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.284642+0000","last_change":"2024-09-21T10:08:55.129563+0000","last_active":"2024-09-21T10:25:10.284642+0000","last_peered":"2024-09-21T10:25:10.284642+0000","last_clean":"2024-09-21T10:25:10.284642+0000","last_became_active":"2024-09-21T10:08:55.129268+0000","last_became_peered":"2024-09-21T10:08:55.129268+0000","last_unstale":"2024-09-21T10:25:10.284642+0000","last_undegraded":"2024-09-21T10:25:10.284642+0000","last_fullsized":"2024-09-21T10:25:10.284642+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:42:32.246505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":1152,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268561+0000","last_change":"2024-09-21T10:08:06.540841+0000","last_active":"2024-09-21T10:25:06.268561+0000","last_peered":"2024-09-21T10:25:06.268561+0000","last_clean":"2024-09-21T10:25:06.268561+0000","last_became_active":"2024-09-21T10:08:06.539359+0000","last_became_peered":"2024-09-21T10:08:06.539359+0000","last_unstale":"2024-09-21T10:25:06.268561+0000","last_undegraded":"2024-09-21T10:25:06.268561+0000","last_fullsized":"2024-09-21T10:25:06.268561+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:51:53.459913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024984999999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1256,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271506+0000","last_change":"2024-09-21T10:06:09.040194+0000","last_active":"2024-09-21T10:25:06.271506+0000","last_peered":"2024-09-21T10:25:06.271506+0000","last_clean":"2024-09-21T10:25:06.271506+0000","last_became_active":"2024-09-21T10:06:09.039512+0000","last_became_peered":"2024-09-21T10:06:09.039512+0000","last_unstale":"2024-09-21T10:25:06.271506+0000","last_undegraded":"2024-09-21T10:25:06.271506+0000","last_fullsized":"2024-09-21T10:25:06.271506+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:40:37.194722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1769,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930713+0000","last_change":"2024-09-21T10:08:55.112951+0000","last_active":"2024-09-21T10:25:19.930713+0000","last_peered":"2024-09-21T10:25:19.930713+0000","last_clean":"2024-09-21T10:25:19.930713+0000","last_became_active":"2024-09-21T10:08:55.112577+0000","last_became_peered":"2024-09-21T10:08:55.112577+0000","last_unstale":"2024-09-21T10:25:19.930713+0000","last_undegraded":"2024-09-21T10:25:19.930713+0000","last_fullsized":"2024-09-21T10:25:19.930713+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:37:06.470913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1682,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.283798+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:25:10.283798+0000","last_peered":"2024-09-21T10:25:10.283798+0000","last_clean":"2024-09-21T10:25:10.283798+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:25:10.283798+0000","last_undegraded":"2024-09-21T10:25:10.283798+0000","last_fullsized":"2024-09-21T10:25:10.283798+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271094+0000","last_change":"2024-09-21T10:06:42.634333+0000","last_active":"2024-09-21T10:25:06.271094+0000","last_peered":"2024-09-21T10:25:06.271094+0000","last_clean":"2024-09-21T10:25:06.271094+0000","last_became_active":"2024-09-21T10:06:42.634037+0000","last_became_peered":"2024-09-21T10:06:42.634037+0000","last_unstale":"2024-09-21T10:25:06.271094+0000","last_undegraded":"2024-09-21T10:25:06.271094+0000","last_fullsized":"2024-09-21T10:25:06.271094+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:27.609590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1140,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270333+0000","last_change":"2024-09-21T10:08:06.537347+0000","last_active":"2024-09-21T10:25:06.270333+0000","last_peered":"2024-09-21T10:25:06.270333+0000","last_clean":"2024-09-21T10:25:06.270333+0000","last_became_active":"2024-09-21T10:08:06.537034+0000","last_became_peered":"2024-09-21T10:08:06.537034+0000","last_unstale":"2024-09-21T10:25:06.270333+0000","last_undegraded":"2024-09-21T10:25:06.270333+0000","last_fullsized":"2024-09-21T10:25:06.270333+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:26.325664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1149,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270773+0000","last_change":"2024-09-21T10:08:06.541436+0000","last_active":"2024-09-21T10:25:06.270773+0000","last_peered":"2024-09-21T10:25:06.270773+0000","last_clean":"2024-09-21T10:25:06.270773+0000","last_became_active":"2024-09-21T10:08:06.540616+0000","last_became_peered":"2024-09-21T10:08:06.540616+0000","last_unstale":"2024-09-21T10:25:06.270773+0000","last_undegraded":"2024-09-21T10:25:06.270773+0000","last_fullsized":"2024-09-21T10:25:06.270773+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:45:18.252764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1140,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.277921+0000","last_change":"2024-09-21T10:06:42.631470+0000","last_active":"2024-09-21T10:25:06.277921+0000","last_peered":"2024-09-21T10:25:06.277921+0000","last_clean":"2024-09-21T10:25:06.277921+0000","last_became_active":"2024-09-21T10:06:42.631081+0000","last_became_peered":"2024-09-21T10:06:42.631081+0000","last_unstale":"2024-09-21T10:25:06.277921+0000","last_undegraded":"2024-09-21T10:25:06.277921+0000","last_fullsized":"2024-09-21T10:25:06.277921+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:50:16.730529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270532+0000","last_change":"2024-09-21T10:08:55.113216+0000","last_active":"2024-09-21T10:25:06.270532+0000","last_peered":"2024-09-21T10:25:06.270532+0000","last_clean":"2024-09-21T10:25:06.270532+0000","last_became_active":"2024-09-21T10:08:55.111808+0000","last_became_peered":"2024-09-21T10:08:55.111808+0000","last_unstale":"2024-09-21T10:25:06.270532+0000","last_undegraded":"2024-09-21T10:25:06.270532+0000","last_fullsized":"2024-09-21T10:25:06.270532+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:24:21.526156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1312,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.930880+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:25:19.930880+0000","last_peered":"2024-09-21T10:25:19.930880+0000","last_clean":"2024-09-21T10:25:19.930880+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:25:19.930880+0000","last_undegraded":"2024-09-21T10:25:19.930880+0000","last_fullsized":"2024-09-21T10:25:19.930880+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1794,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.932832+0000","last_change":"2024-09-21T10:06:42.626688+0000","last_active":"2024-09-21T10:25:19.932832+0000","last_peered":"2024-09-21T10:25:19.932832+0000","last_clean":"2024-09-21T10:25:19.932832+0000","last_became_active":"2024-09-21T10:06:42.626326+0000","last_became_peered":"2024-09-21T10:06:42.626326+0000","last_unstale":"2024-09-21T10:25:19.932832+0000","last_undegraded":"2024-09-21T10:25:19.932832+0000","last_fullsized":"2024-09-21T10:25:19.932832+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:00.761023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268877+0000","last_change":"2024-09-21T10:08:55.099610+0000","last_active":"2024-09-21T10:25:06.268877+0000","last_peered":"2024-09-21T10:25:06.268877+0000","last_clean":"2024-09-21T10:25:06.268877+0000","last_became_active":"2024-09-21T10:08:55.099313+0000","last_became_peered":"2024-09-21T10:08:55.099313+0000","last_unstale":"2024-09-21T10:25:06.268877+0000","last_undegraded":"2024-09-21T10:25:06.268877+0000","last_fullsized":"2024-09-21T10:25:06.268877+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:42:01.266798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1139,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.277737+0000","last_change":"2024-09-21T10:06:42.631675+0000","last_active":"2024-09-21T10:25:06.277737+0000","last_peered":"2024-09-21T10:25:06.277737+0000","last_clean":"2024-09-21T10:25:06.277737+0000","last_became_active":"2024-09-21T10:06:42.631098+0000","last_became_peered":"2024-09-21T10:06:42.631098+0000","last_unstale":"2024-09-21T10:25:06.277737+0000","last_undegraded":"2024-09-21T10:25:06.277737+0000","last_fullsized":"2024-09-21T10:25:06.277737+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:46.855597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1124,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.273109+0000","last_change":"2024-09-21T09:57:00.001554+0000","last_active":"2024-09-21T10:25:06.273109+0000","last_peered":"2024-09-21T10:25:06.273109+0000","last_clean":"2024-09-21T10:25:06.273109+0000","last_became_active":"2024-09-21T09:57:00.001245+0000","last_became_peered":"2024-09-21T09:57:00.001245+0000","last_unstale":"2024-09-21T10:25:06.273109+0000","last_undegraded":"2024-09-21T10:25:06.273109+0000","last_fullsized":"2024-09-21T10:25:06.273109+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:47.022195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1156,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.270127+0000","last_change":"2024-09-21T10:08:55.107939+0000","last_active":"2024-09-21T10:25:06.270127+0000","last_peered":"2024-09-21T10:25:06.270127+0000","last_clean":"2024-09-21T10:25:06.270127+0000","last_became_active":"2024-09-21T10:08:55.107640+0000","last_became_peered":"2024-09-21T10:08:55.107640+0000","last_unstale":"2024-09-21T10:25:06.270127+0000","last_undegraded":"2024-09-21T10:25:06.270127+0000","last_fullsized":"2024-09-21T10:25:06.270127+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:11.287664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1142,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.277518+0000","last_change":"2024-09-21T10:06:42.632153+0000","last_active":"2024-09-21T10:25:06.277518+0000","last_peered":"2024-09-21T10:25:06.277518+0000","last_clean":"2024-09-21T10:25:06.277518+0000","last_became_active":"2024-09-21T10:06:42.631728+0000","last_became_peered":"2024-09-21T10:06:42.631728+0000","last_unstale":"2024-09-21T10:25:06.277518+0000","last_undegraded":"2024-09-21T10:25:06.277518+0000","last_fullsized":"2024-09-21T10:25:06.277518+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:53:53.412364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1121,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.271761+0000","last_change":"2024-09-21T09:57:01.008454+0000","last_active":"2024-09-21T10:25:06.271761+0000","last_peered":"2024-09-21T10:25:06.271761+0000","last_clean":"2024-09-21T10:25:06.271761+0000","last_became_active":"2024-09-21T09:57:01.008132+0000","last_became_peered":"2024-09-21T09:57:01.008132+0000","last_unstale":"2024-09-21T10:25:06.271761+0000","last_undegraded":"2024-09-21T10:25:06.271761+0000","last_fullsized":"2024-09-21T10:25:06.271761+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:18.509788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1141,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.273341+0000","last_change":"2024-09-21T10:06:09.030444+0000","last_active":"2024-09-21T10:25:06.273341+0000","last_peered":"2024-09-21T10:25:06.273341+0000","last_clean":"2024-09-21T10:25:06.273341+0000","last_became_active":"2024-09-21T10:06:09.030054+0000","last_became_peered":"2024-09-21T10:06:09.030054+0000","last_unstale":"2024-09-21T10:25:06.273341+0000","last_undegraded":"2024-09-21T10:25:06.273341+0000","last_fullsized":"2024-09-21T10:25:06.273341+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:27.913156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1148,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.276034+0000","last_change":"2024-09-21T10:06:09.032476+0000","last_active":"2024-09-21T10:25:06.276034+0000","last_peered":"2024-09-21T10:25:06.276034+0000","last_clean":"2024-09-21T10:25:06.276034+0000","last_became_active":"2024-09-21T10:06:09.032121+0000","last_became_peered":"2024-09-21T10:06:09.032121+0000","last_unstale":"2024-09-21T10:25:06.276034+0000","last_undegraded":"2024-09-21T10:25:06.276034+0000","last_fullsized":"2024-09-21T10:25:06.276034+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:41.862129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1672,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281353+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:25:10.281353+0000","last_peered":"2024-09-21T10:25:10.281353+0000","last_clean":"2024-09-21T10:25:10.281353+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:25:10.281353+0000","last_undegraded":"2024-09-21T10:25:10.281353+0000","last_fullsized":"2024-09-21T10:25:10.281353+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1150,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.274920+0000","last_change":"2024-09-21T10:08:06.546911+0000","last_active":"2024-09-21T10:25:06.274920+0000","last_peered":"2024-09-21T10:25:06.274920+0000","last_clean":"2024-09-21T10:25:06.274920+0000","last_became_active":"2024-09-21T10:08:06.546417+0000","last_became_peered":"2024-09-21T10:08:06.546417+0000","last_unstale":"2024-09-21T10:25:06.274920+0000","last_undegraded":"2024-09-21T10:25:06.274920+0000","last_fullsized":"2024-09-21T10:25:06.274920+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:39:46.544701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1793,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.933610+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:25:19.933610+0000","last_peered":"2024-09-21T10:25:19.933610+0000","last_clean":"2024-09-21T10:25:19.933610+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:25:19.933610+0000","last_undegraded":"2024-09-21T10:25:19.933610+0000","last_fullsized":"2024-09-21T10:25:19.933610+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1793,"reported_epoch":1183,"state":"active+clean","last_fresh":"2024-09-21T10:25:19.932736+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:25:19.932736+0000","last_peered":"2024-09-21T10:25:19.932736+0000","last_clean":"2024-09-21T10:25:19.932736+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:25:19.932736+0000","last_undegraded":"2024-09-21T10:25:19.932736+0000","last_fullsized":"2024-09-21T10:25:19.932736+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1138,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.268607+0000","last_change":"2024-09-21T10:08:55.102502+0000","last_active":"2024-09-21T10:25:06.268607+0000","last_peered":"2024-09-21T10:25:06.268607+0000","last_clean":"2024-09-21T10:25:06.268607+0000","last_became_active":"2024-09-21T10:08:55.101777+0000","last_became_peered":"2024-09-21T10:08:55.101777+0000","last_unstale":"2024-09-21T10:25:06.268607+0000","last_undegraded":"2024-09-21T10:25:06.268607+0000","last_fullsized":"2024-09-21T10:25:06.268607+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:51.293401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1172,"reported_epoch":1181,"state":"active+clean","last_fresh":"2024-09-21T10:25:06.274746+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:25:06.274746+0000","last_peered":"2024-09-21T10:25:06.274746+0000","last_clean":"2024-09-21T10:25:06.274746+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:25:06.274746+0000","last_undegraded":"2024-09-21T10:25:06.274746+0000","last_fullsized":"2024-09-21T10:25:06.274746+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1681,"reported_epoch":1182,"state":"active+clean","last_fresh":"2024-09-21T10:25:10.281717+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:25:10.281717+0000","last_peered":"2024-09-21T10:25:10.281717+0000","last_clean":"2024-09-21T10:25:10.281717+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:25:10.281717+0000","last_undegraded":"2024-09-21T10:25:10.281717+0000","last_fullsized":"2024-09-21T10:25:10.281717+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":45,"num_pg":16,"stat_sum":{"num_bytes":7815692312,"num_objects":119259,"num_object_clones":0,"num_object_copies":357777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119259,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":141077,"num_write_kb":8330689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23451676672,"data_stored":23451664456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134177,"ondisk_log_size":134177,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":124,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397869,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3225160,"kb_used_data":2919488,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90519992,"statfs":{"total":95995035648,"available":92692471808,"internally_reserved":0,"allocated":2989555712,"data_stored":2985002551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683594098,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3229320,"kb_used_data":2919488,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90515832,"statfs":{"total":95995035648,"available":92688211968,"internally_reserved":0,"allocated":2989555712,"data_stored":2985002546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790327,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3710712,"kb_used_data":3409136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90034440,"statfs":{"total":95995035648,"available":92195266560,"internally_reserved":0,"allocated":3490955264,"data_stored":3486413732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986555,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1776384,"kb_used_data":1479032,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91968768,"statfs":{"total":95995035648,"available":94176018432,"internally_reserved":0,"allocated":1514528768,"data_stored":1509983241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644487,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3959860,"kb_used_data":3815276,"kb_used_omap":67,"kb_used_meta":144444,"kb_avail":89785292,"statfs":{"total":95995035648,"available":91940139008,"internally_reserved":0,"allocated":3906842624,"data_stored":3902304663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":147911517},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527505,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3506676,"kb_used_data":3370668,"kb_used_omap":75,"kb_used_meta":135924,"kb_avail":90238476,"statfs":{"total":95995035648,"available":92404199424,"internally_reserved":0,"allocated":3451564032,"data_stored":3447026569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":139186330},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274658,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2505356,"kb_used_data":2379264,"kb_used_omap":51,"kb_used_meta":126028,"kb_avail":91239796,"statfs":{"total":95995035648,"available":93429551104,"internally_reserved":0,"allocated":2436366336,"data_stored":2431816941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":129053013},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353897,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2920652,"kb_used_data":2812420,"kb_used_omap":66,"kb_used_meta":108157,"kb_avail":90824500,"statfs":{"total":95995035648,"available":93004288000,"internally_reserved":0,"allocated":2879918080,"data_stored":2875365614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":110753092},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2847150080,"data_stored":2847146008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2423324672,"data_stored":2423324672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3421044736,"data_stored":3421044736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3889762304,"data_stored":3889758232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1497890816,"data_stored":1497890816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3453222912,"data_stored":3453222912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2958495744,"data_stored":2958491672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2960785408,"data_stored":2960785408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:25:25.544 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:25:25.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:25:25.544 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T10:25:25.544 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:25:25.544 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:25:25.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T10:25:25.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:25.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:25.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:25.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:25.790 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:25.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:25.932 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:25.934 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:25.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:26.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:26.115 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:26.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:25 smithi119 ceph-mon[101432]: pgmap v1648: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 911 op/s 2024-09-21T10:25:26.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:25 smithi119 ceph-mon[105288]: pgmap v1648: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 911 op/s 2024-09-21T10:25:26.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:26.258 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:25 smithi145 ceph-mon[89359]: pgmap v1648: 89 pgs: 89 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 911 op/s 2024-09-21T10:25:26.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:26.360 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:26.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:26.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:26.526 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:25:26.526 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:25:26.526 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:26.526 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:26.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:26.559 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:25:26.559 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:26.595 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:25:26.595 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:26.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:26.629 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:25:26.629 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:26.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:26.662 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:25:26.662 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:26.694 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:25:26.695 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:25:26.729 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:25:26.730 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:26.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:26.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:26.890 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:26.923 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:26.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:27.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:27.059 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3251176310' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3251176310' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:27.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:27.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:27.221 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3251176310' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:27.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:27.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:27.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:27.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:27.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:27.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:27.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:27.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.738 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:27.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:27.839 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:27.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:27.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:27.907 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:25:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:25:27.908 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:25:27.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:28.013 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:28.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.142 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:28.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:28.243 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:28.258 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:27 smithi145 ceph-mon[89359]: pgmap v1649: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 907 op/s 2024-09-21T10:25:28.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:27 smithi119 ceph-mon[105288]: pgmap v1649: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 907 op/s 2024-09-21T10:25:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:27 smithi119 ceph-mon[101432]: pgmap v1649: 89 pgs: 89 active+clean; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 907 op/s 2024-09-21T10:25:28.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:28.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:28.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:28.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:28.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:28.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:28.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:28.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:28.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:28.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:28.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:29.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:29.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:29.144 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:29.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.245 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:29.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:29.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.383 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:29.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:29.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:29.520 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:29.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:29.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:29.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:29.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:29.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:29.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:29.914 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:29.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:29.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:29.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:30.016 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:30.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:29 smithi119 ceph-mon[105288]: pgmap v1650: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:25:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:29 smithi119 ceph-mon[101432]: pgmap v1650: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:25:30.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:30.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:30.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:29 smithi145 ceph-mon[89359]: pgmap v1650: 89 pgs: 89 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 900 op/s 2024-09-21T10:25:30.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:30.349 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:30.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.486 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:30.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:25:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:25:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:30.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:30.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:30.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:30.857 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:30.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:30.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:31.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.035 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:31.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:31.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:31.235 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:25:31.235 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:25:31.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:64127 45.d 45:b9866091:::benchmark_data_smithi119_168706_object64126:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:07.610496+0000", 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.619148541000001, 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.099802874, 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.236 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 64127 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.610496+0000", 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.610496+0000", 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.610497+0000", 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9599999999999999e-07 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.610512+0000", 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4499e-05 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.610518+0000", 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8399999999999997e-06 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.610524+0000", 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4090000000000002e-06 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.614483+0000", 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003959583 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.614604+0000", 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012037299999999999 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.614735+0000", 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013137500000000001 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.620202+0000", 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005466551 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.620271+0000", 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9280000000000001e-05 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.622633+0000", 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023615559999999999 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.622677+0000", 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4161e-05 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710299+0000", 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.087622650999999996 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:64130 45.3 45:c0783a19:::benchmark_data_smithi119_168706_object64129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:07.612927+0000", 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.616717675000004, 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.097619865, 2024-09-21T10:25:31.242 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 64130 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.612927+0000", 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.612927+0000", 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.612929+0000", 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.539e-06 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.612952+0000", 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2764000000000001e-05 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.612962+0000", 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.045e-05 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.612969+0000", 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7800000000000003e-06 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.616480+0000", 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035112339999999998 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.616629+0000", 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000149288 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.616768+0000", 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000138863 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.619417+0000", 2024-09-21T10:25:31.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026487809999999998 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.622106+0000", 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026894060000000001 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710447+0000", 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.088340999000000003 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710505+0000", 2024-09-21T10:25:31.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7120999999999997e-05 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710547+0000", 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2639999999999998e-05 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:64124 45.5 e1172/1164 45:a612184b:::benchmark_data_smithi119_168706_object64123:head v 1172'3918, mlcod=1172'3915)", 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:07.615159+0000", 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.614486158000005, 2024-09-21T10:25:31.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.095430266999999999, 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.615159+0000", 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.615159+0000", 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.615160+0000", 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2399999999999996e-07 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.615187+0000", 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7041000000000001e-05 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.615197+0000", 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0543000000000001e-05 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.615204+0000", 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1789999999999996e-06 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617309+0000", 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002105872 2024-09-21T10:25:31.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617329+0000", 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9828999999999999e-05 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710506+0000", 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.093177168000000005 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710589+0000", 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2711000000000001e-05 2024-09-21T10:25:31.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:64133 45.d 45:bf160a1f:::benchmark_data_smithi119_168706_object64132:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:07.617331+0000", 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.612313732999993, 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.093376694999999996, 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.253 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 64133 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617331+0000", 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617331+0000", 2024-09-21T10:25:31.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617333+0000", 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.629e-06 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617373+0000", 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0225000000000002e-05 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617385+0000", 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1463e-05 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617392+0000", 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7239999999999992e-06 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.620297+0000", 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029041890000000002 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.256 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.620432+0000", 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013499000000000001 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.620556+0000", 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012463900000000001 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710499+0000", 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.089942470999999996 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710600+0000", 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101049 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710631+0000", 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1826000000000003e-05 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710677+0000", 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5219000000000002e-05 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710708+0000", 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1270999999999999e-05 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:64135 45.3 45:ce0e09a9:::benchmark_data_smithi119_168706_object64134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:07.617408+0000", 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.612236831000004, 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.093285621999999999, 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.259 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 64135 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617408+0000", 2024-09-21T10:25:31.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617408+0000", 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617409+0000", 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0299999999999999e-07 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617432+0000", 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3260000000000001e-05 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617439+0000", 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0589999999999997e-06 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617444+0000", 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2290000000000002e-06 2024-09-21T10:25:31.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.619644+0000", 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022000610000000001 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.619786+0000", 2024-09-21T10:25:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000141757 2024-09-21T10:25:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:25:31.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.619931+0000", 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014524699999999999 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.622139+0000", 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022072139999999999 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710389+0000", 2024-09-21T10:25:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.088250414999999999 2024-09-21T10:25:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710614+0000", 2024-09-21T10:25:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000224794 2024-09-21T10:25:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710660+0000", 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6573999999999998e-05 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710694+0000", 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3408999999999998e-05 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.35976.0:64122 45.d e1172/1164 ondisk, result = 0)", 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:07.617602+0000", 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.612043176, 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.092706912000000002, 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617602+0000", 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617602+0000", 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617603+0000", 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8299999999999995e-07 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617607+0000", 2024-09-21T10:25:31.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7419999999999997e-06 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617619+0000", 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1246e-05 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617625+0000", 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4690000000000002e-06 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617933+0000", 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00030753600000000002 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.617942+0000", 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8089999999999999e-06 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:07.710309+0000", 2024-09-21T10:25:31.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.092367226999999996 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:64791 45.0 45:07a39afa:::benchmark_data_smithi119_168706_object64790:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:08.128843+0000", 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.100801687000001, 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.087883055000000002, 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.275 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 64791 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128843+0000", 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128843+0000", 2024-09-21T10:25:31.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128845+0000", 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.517e-06 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128873+0000", 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8547999999999998e-05 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128887+0000", 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3738e-05 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128896+0000", 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5150000000000006e-06 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.128958+0000", 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1916999999999997e-05 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.129101+0000", 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014339800000000001 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.129235+0000", 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013397800000000001 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.211409+0000", 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.082174077999999998 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.216582+0000", 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0051733710000000004 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.216632+0000", 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0008e-05 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.216678+0000", 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5426999999999997e-05 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.216726+0000", 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8560000000000003e-05 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:64796 45.0 45:0cff6395:::benchmark_data_smithi119_168706_object64795:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:08.132786+0000", 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 83.096859034000005, 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.087532227000000004, 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.281 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 64796 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.132786+0000", 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.132786+0000", 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.132787+0000", 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0559999999999999e-06 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.132805+0000", 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8202e-05 2024-09-21T10:25:31.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.132817+0000", 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1584e-05 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.132824+0000", 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8770000000000004e-06 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.211445+0000", 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.078621257999999999 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.211604+0000", 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015855900000000001 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.211758+0000", 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015494099999999999 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.216620+0000", 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004861621 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.216733+0000", 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000113411 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.219250+0000", 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002516571 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.219538+0000", 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028806799999999999 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:08.220318+0000", 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00078007900000000004 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.287 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69505 45.6 45:637c4fd4:::benchmark_data_smithi119_168706_object69504:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.543562+0000", 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.686083354999994, 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.11146993199999999, 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69505 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.543562+0000", 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.543562+0000", 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.543566+0000", 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9949999999999999e-06 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.543595+0000", 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9385000000000001e-05 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.543612+0000", 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7047000000000001e-05 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.543618+0000", 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1369999999999999e-06 2024-09-21T10:25:31.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.544695+0000", 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001076839 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.544809+0000", 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011425999999999999 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.544942+0000", 2024-09-21T10:25:31.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013256800000000001 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.551368+0000", 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064265279999999999 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553253+0000", 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018847689999999999 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654948+0000", 2024-09-21T10:25:31.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10169484600000001 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654995+0000", 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6709000000000003e-05 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655032+0000", 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6848999999999999e-05 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.293 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69510 45.6 45:6bff2970:::benchmark_data_smithi119_168706_object69509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.545321+0000", 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.684324040999996, 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.109872449, 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.294 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69510 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.545321+0000", 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.545321+0000", 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.545322+0000", 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.198e-06 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.545343+0000", 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0497e-05 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.545354+0000", 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0874e-05 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.545360+0000", 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8199999999999999e-06 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547111+0000", 2024-09-21T10:25:31.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017503530000000001 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547237+0000", 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012630200000000001 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547373+0000", 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013586699999999999 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553288+0000", 2024-09-21T10:25:31.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059154109999999998 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553320+0000", 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1786999999999999e-05 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655121+0000", 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10180091600000001 2024-09-21T10:25:31.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655162+0000", 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1108e-05 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655193+0000", 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1316000000000003e-05 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:69508 45.e e1173/1164 45:7143865d:::benchmark_data_smithi119_168706_object69507:head v 1173'4299, mlcod=1173'4295)", 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.546036+0000", 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.683609481000005, 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.109078437, 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546036+0000", 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546036+0000", 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546036+0000", 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2200000000000004e-07 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546061+0000", 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4578000000000001e-05 2024-09-21T10:25:31.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546070+0000", 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0929999999999992e-06 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546076+0000", 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8050000000000003e-06 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546678+0000", 2024-09-21T10:25:31.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00060228699999999996 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.546702+0000", 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4318999999999999e-05 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655083+0000", 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10838072999999999 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655114+0000", 2024-09-21T10:25:31.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1003000000000001e-05 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.35976.0:69505 45.6 e1173/1164 ondisk, result = 0)", 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.547634+0000", 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.682010615999999, 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.107403314, 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547634+0000", 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547634+0000", 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547635+0000", 2024-09-21T10:25:31.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7899999999999998e-07 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547639+0000", 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3760000000000001e-06 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547649+0000", 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.679e-06 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.547656+0000", 2024-09-21T10:25:31.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6000000000000003e-06 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.551357+0000", 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037008369999999998 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.551363+0000", 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.28e-06 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655038+0000", 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10367486300000001 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:69511 45.e e1173/1164 45:7526af9a:::benchmark_data_smithi119_168706_object69510:head v 1173'4300, mlcod=1173'4295)", 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.548669+0000", 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.680976040999994, 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.106553192, 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.310 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.548669+0000", 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.548669+0000", 2024-09-21T10:25:31.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.548670+0000", 2024-09-21T10:25:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4600000000000004e-07 2024-09-21T10:25:31.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.548709+0000", 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9521000000000003e-05 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.548720+0000", 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0691e-05 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.548726+0000", 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6729999999999997e-06 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550931+0000", 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022053340000000002 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.314 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550948+0000", 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6648000000000001e-05 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655196+0000", 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.104247988 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655222+0000", 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6591e-05 2024-09-21T10:25:31.315 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69518 45.6 45:6f1da8f5:::benchmark_data_smithi119_168706_object69517:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.550852+0000", 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.678792924999996, 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10444060300000001, 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.316 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69518 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550852+0000", 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550852+0000", 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.317 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550853+0000", 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.268e-06 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550875+0000", 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1203999999999999e-05 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550886+0000", 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1636e-05 2024-09-21T10:25:31.318 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.550893+0000", 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1690000000000003e-06 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553354+0000", 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002461015 2024-09-21T10:25:31.319 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553479+0000", 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012456199999999999 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553610+0000", 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013145499999999999 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654649+0000", 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10103824 2024-09-21T10:25:31.320 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655076+0000", 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042695100000000001 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655228+0000", 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015247700000000001 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.321 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655264+0000", 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6226999999999997e-05 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655293+0000", 2024-09-21T10:25:31.322 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8399e-05 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69522 45.d 45:bb487116:::benchmark_data_smithi119_168706_object69521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.553638+0000", 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.676006897999997, 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.101157689, 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.323 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69522 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553638+0000", 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.324 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553638+0000", 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553640+0000", 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6109999999999999e-06 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.325 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553661+0000", 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0888999999999999e-05 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553673+0000", 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2268e-05 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553680+0000", 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3429999999999998e-06 2024-09-21T10:25:31.326 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553779+0000", 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9242999999999998e-05 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.553975+0000", 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019503399999999999 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.554093+0000", 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118249 2024-09-21T10:25:31.327 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557007+0000", 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002914514 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557583+0000", 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057544699999999998 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654700+0000", 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.097117131999999995 2024-09-21T10:25:31.328 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654753+0000", 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3078999999999998e-05 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654796+0000", 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2880000000000003e-05 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.329 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.35976.0:69518 45.6 e1173/1164 ondisk, result = 0)", 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.556307+0000", 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.673337767000007, 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.098348630000000006, 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.556307+0000", 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.330 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.556307+0000", 2024-09-21T10:25:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.333 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.556308+0000", 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6899999999999996e-07 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.556313+0000", 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5360000000000003e-06 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.556324+0000", 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1432e-05 2024-09-21T10:25:31.334 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.556331+0000", 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6200000000000001e-06 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654635+0000", 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.098304509999999998 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654643+0000", 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2649999999999996e-06 2024-09-21T10:25:31.335 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654656+0000", 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2398000000000001e-05 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69531 45.0 45:089c5d4b:::benchmark_data_smithi119_168706_object69530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.557884+0000", 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.671761450000005, 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.103360089, 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.336 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69531 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557884+0000", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557884+0000", 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.337 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557885+0000", 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4589999999999999e-06 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557905+0000", 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0276e-05 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557917+0000", 2024-09-21T10:25:31.338 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1287000000000001e-05 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.557923+0000", 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6370000000000002e-06 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654593+0000", 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.096669552000000006 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.339 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654729+0000", 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013637400000000001 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654869+0000", 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000139596 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.657826+0000", 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029577370000000002 2024-09-21T10:25:31.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.660676+0000", 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028497679999999999 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.661083+0000", 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040677699999999998 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.661179+0000", 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6465999999999994e-05 2024-09-21T10:25:31.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.661244+0000", 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4159999999999996e-05 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:69533 45.d 45:b0400815:::benchmark_data_smithi119_168706_object69532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.558658+0000", 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.670987018000005, 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.108117716, 2024-09-21T10:25:31.342 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:25:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:25:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:31.343 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69533 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558658+0000", 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558658+0000", 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558659+0000", 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.159e-06 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558678+0000", 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9227000000000001e-05 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654485+0000", 2024-09-21T10:25:31.345 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.095806192999999998 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654594+0000", 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109617 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.660922+0000", 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063274409999999996 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.346 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.661040+0000", 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118229 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.661460+0000", 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042008399999999998 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.665851+0000", 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043910429999999999 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.666191+0000", 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00034004700000000001 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.666688+0000", 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049660700000000004 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.666739+0000", 2024-09-21T10:25:31.348 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1310000000000002e-05 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.666776+0000", 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6758999999999998e-05 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.35976.0:69522 45.d e1173/1164 ondisk, result = 0)", 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.558898+0000", 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.670746980000004, 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.095902652000000005, 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.349 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558898+0000", 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558898+0000", 2024-09-21T10:25:31.350 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558899+0000", 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8599999999999997e-07 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.558902+0000", 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.641e-06 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654503+0000", 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.095600856999999997 2024-09-21T10:25:31.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654529+0000", 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.6069000000000001e-05 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654689+0000", 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015916199999999999 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654694+0000", 2024-09-21T10:25:31.352 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6899999999999997e-06 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654801+0000", 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010644700000000001 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:69540 45.a e1173/1164 45:58887e69:::benchmark_data_smithi119_168706_object69539:head v 1173'4335, mlcod=1173'4333)", 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:24:11.563189+0000", 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 79.666455798000001, 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.097051317999999998, 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:25:31.353 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.563189+0000", 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.563189+0000", 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.354 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.563190+0000", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.277e-06 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.563279+0000", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8109000000000006e-05 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654956+0000", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.091677416999999997 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:25:31.355 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.654964+0000", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5310000000000004e-06 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655597+0000", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00063311800000000001 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.655616+0000", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9417999999999999e-05 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.660191+0000", 2024-09-21T10:25:31.356 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045753119999999998 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:24:11.660241+0000", 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9135999999999998e-05 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:25:31.357 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:25:31.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:31.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.513 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:31.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:31.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:31.651 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:31.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:31.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:31.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:31.926 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:31.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:32.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:32.057 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:32.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:32.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.196 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:32.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:32.270 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:31 smithi119 ceph-mon[105288]: pgmap v1651: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-09-21T10:25:32.270 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:31 smithi119 ceph-mon[101432]: pgmap v1651: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-09-21T10:25:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:31 smithi145 ceph-mon[89359]: pgmap v1651: 89 pgs: 89 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 876 op/s 2024-09-21T10:25:32.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:32.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:32.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:32.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:32.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:32.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:32.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.741 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:32.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:32.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:32.868 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:32.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:32.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:32.971 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:25:33.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:33.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.158 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:33.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:33.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:25:33.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2148003784' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:33.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:33.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:33.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:33.278 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2148003784' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:33.278 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:33.278 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:33.278 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:33.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:25:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2148003784' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:32 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:33.318 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:25:33.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:33.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.425 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:33.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:33.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:33.560 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:33.602 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:25:33.602 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:25:33.603 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:25:33.603 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T10:25:33.603 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T10:25:33.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:33.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:33.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:33.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:33.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:33.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:33.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:33 smithi145 ceph-mon[89359]: pgmap v1652: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-09-21T10:25:33.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:33 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:33.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:33.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:33 smithi145 ceph-mon[89359]: osdmap e1184: 8 total, 8 up, 8 in 2024-09-21T10:25:33.991 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:25:33.991 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:33 smithi145 systemd[1]: Stopping Ceph osd.6 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:25:34.049 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:34.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:34.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:34.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:34.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[101432]: pgmap v1652: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-09-21T10:25:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:34.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:34.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[101432]: osdmap e1184: 8 total, 8 up, 8 in 2024-09-21T10:25:34.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:25:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[105288]: pgmap v1652: 89 pgs: 89 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 794 op/s 2024-09-21T10:25:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:34.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:34.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[105288]: osdmap e1184: 8 total, 8 up, 8 in 2024-09-21T10:25:34.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:25:34.384 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:34.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:34.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:34.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:34.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:34.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:34.746 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:34.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:34.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:25:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:25:34] "GET /metrics HTTP/1.1" 200 36769 "" "Prometheus/2.51.0" 2024-09-21T10:25:34.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:34.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:34.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:35.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:35.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:35.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:35.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:35.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:35.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:35.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:35.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:35.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[50814]: 2024-09-21T10:25:35.103+0000 7f90ef8e0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:25:35.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[50814]: 2024-09-21T10:25:35.103+0000 7f90ef8e0640 -1 osd.6 1184 *** Got signal Terminated *** 2024-09-21T10:25:35.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:35 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[50814]: 2024-09-21T10:25:35.103+0000 7f90ef8e0640 -1 osd.6 1184 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:25:35.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:35.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:35.719 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:35.750 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:35.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:35.884 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:35.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:35.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:36.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:36.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:36.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:36.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:36.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:36.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:35 smithi145 ceph-mon[89359]: pgmap v1654: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:25:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:36 smithi145 ceph-mon[89359]: osd.6 marked itself down and dead 2024-09-21T10:25:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:35 smithi119 ceph-mon[101432]: pgmap v1654: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:25:36.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:36 smithi119 ceph-mon[101432]: osd.6 marked itself down and dead 2024-09-21T10:25:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:36 smithi119 ceph-mon[105288]: pgmap v1654: 89 pgs: 89 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:25:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:36 smithi119 ceph-mon[105288]: osd.6 marked itself down and dead 2024-09-21T10:25:36.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:36.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:36.499 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:36.522 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:36.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:36.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:36.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:36.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:36.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:36.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:36.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:36.893 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:36.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:36.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:37.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:37.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:37.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:37.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:37.243 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:37 smithi119 ceph-mon[105288]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:25:37.243 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:37 smithi119 ceph-mon[105288]: osdmap e1185: 8 total, 7 up, 8 in 2024-09-21T10:25:37.243 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:37 smithi119 ceph-mon[101432]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:25:37.243 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:37 smithi119 ceph-mon[101432]: osdmap e1185: 8 total, 7 up, 8 in 2024-09-21T10:25:37.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:37.269 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:37.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:37.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:37 smithi145 ceph-mon[89359]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:25:37.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:37 smithi145 ceph-mon[89359]: osdmap e1185: 8 total, 7 up, 8 in 2024-09-21T10:25:37.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:37.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:37.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:37.649 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:37.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:37.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:37.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:37.920 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:37.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:38.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:38.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:38.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:38.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.178 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:38.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:38.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:38 smithi119 ceph-mon[105288]: pgmap v1656: 89 pgs: 10 stale+active+clean, 79 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 795 op/s 2024-09-21T10:25:38.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:38 smithi119 ceph-mon[105288]: osdmap e1186: 8 total, 7 up, 8 in 2024-09-21T10:25:38.195 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:38 smithi119 ceph-mon[101432]: pgmap v1656: 89 pgs: 10 stale+active+clean, 79 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 795 op/s 2024-09-21T10:25:38.195 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:38 smithi119 ceph-mon[101432]: osdmap e1186: 8 total, 7 up, 8 in 2024-09-21T10:25:38.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:38 smithi145 ceph-mon[89359]: pgmap v1656: 89 pgs: 10 stale+active+clean, 79 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 795 op/s 2024-09-21T10:25:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:38 smithi145 ceph-mon[89359]: osdmap e1186: 8 total, 7 up, 8 in 2024-09-21T10:25:38.294 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:37 smithi145 podman[113621]: 2024-09-21 10:25:37.822618904 +0000 UTC m=+3.410581353 container died 772774e9be4a8b3042f7c6c3e827f6dfda7138ab29c004415da4070759288b4f (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:25:38.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:38.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:38.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:38.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:38.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:38.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:38.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:25:38.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:38.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:38.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:38.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:38.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:25:38.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:25:38.895 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:38.895 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:38.905 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:25:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:38.936 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:25:38.936 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:38.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:38.968 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:25:38.968 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:38.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:38.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:38.994 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:25:38.994 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:25:39.025 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:25:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:39.083 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:39.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:39.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:39.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:39.261 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1995747464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:39.262 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1995747464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1995747464' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:39.294 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:39 smithi145 podman[113621]: 2024-09-21 10:25:39.030387423 +0000 UTC m=+4.618349859 container cleanup 772774e9be4a8b3042f7c6c3e827f6dfda7138ab29c004415da4070759288b4f (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:25:39.294 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:39 smithi145 bash[113621]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6 2024-09-21T10:25:39.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:39.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:39.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:39.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:39.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:39.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:39.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:39.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:39.794 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:39 smithi145 podman[113838]: 2024-09-21 10:25:39.524458725 +0000 UTC m=+3.134140212 container remove 772774e9be4a8b3042f7c6c3e827f6dfda7138ab29c004415da4070759288b4f (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:25:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.797 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:39.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:39.898 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:39.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:39.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:25:39.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:25:39.993 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:40.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:40.050 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:40.113 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:40 smithi145 ceph-mon[89359]: pgmap v1658: 89 pgs: 13 active+undersized, 3 stale+active+clean, 9 active+undersized+degraded, 64 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 19620/315177 objects degraded (6.225%) 2024-09-21T10:25:40.114 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:40 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 19620/315177 objects degraded (6.225%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:40.126 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:40.126 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:40.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:40.179 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:40.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:40.273 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:40 smithi119 ceph-mon[105288]: pgmap v1658: 89 pgs: 13 active+undersized, 3 stale+active+clean, 9 active+undersized+degraded, 64 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 19620/315177 objects degraded (6.225%) 2024-09-21T10:25:40.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:40 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 19620/315177 objects degraded (6.225%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:40.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:40 smithi119 ceph-mon[101432]: pgmap v1658: 89 pgs: 13 active+undersized, 3 stale+active+clean, 9 active+undersized+degraded, 64 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 878 op/s; 19620/315177 objects degraded (6.225%) 2024-09-21T10:25:40.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:40 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 19620/315177 objects degraded (6.225%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:40.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:40.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:40.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:40.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:40.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.485 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:40.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:25:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:25:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:40.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:40.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:40.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:40.739 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:40.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:40.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:40.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:41.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:41.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:41.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:41.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:41.170 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:41.272 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:41.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:25:41.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:41.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:41.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:41.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:41.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:41.667 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:41.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:41.780 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:41.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:41.937 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:41.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:41.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:25:42.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:42.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:42.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:42.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:42.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:42.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:42.242 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:42 smithi145 ceph-mon[89359]: pgmap v1659: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 38671/309051 objects degraded (12.513%) 2024-09-21T10:25:42.297 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:42 smithi119 ceph-mon[101432]: pgmap v1659: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 38671/309051 objects degraded (12.513%) 2024-09-21T10:25:42.297 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:42 smithi119 ceph-mon[105288]: pgmap v1659: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 928 op/s; 38671/309051 objects degraded (12.513%) 2024-09-21T10:25:42.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.370 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:42.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:42.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:42.498 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:42.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.643 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:42.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:42.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:42.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:42.822 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:42.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:42.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:42.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:43.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:43.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:43.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:43.060 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:25:43.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:43.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:43.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2493301820' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.259 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:43 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.293 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:43.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2493301820' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:43 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2493301820' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:43 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:25:43.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:43.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:43.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:43.559 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:43.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:43.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:43.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:43.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:43.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:43.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:44.044 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:43 smithi145 podman[114670]: 2024-09-21 10:25:43.665098245 +0000 UTC m=+0.359627101 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:25:44.044 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:43 smithi145 podman[114670]: 2024-09-21 10:25:43.944928628 +0000 UTC m=+0.639457471 container create d46eda1507991cdf9ffe669874946fed9df98eece74873029a71dcc0c13176b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:25:44.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:44.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:44.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:44.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:44.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[105288]: pgmap v1660: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 38671/307446 objects degraded (12.578%) 2024-09-21T10:25:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:25:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[105288]: osdmap e1187: 8 total, 7 up, 8 in 2024-09-21T10:25:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[101432]: pgmap v1660: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 38671/307446 objects degraded (12.578%) 2024-09-21T10:25:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:25:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-mon[101432]: osdmap e1187: 8 total, 7 up, 8 in 2024-09-21T10:25:44.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:44.441 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:44 smithi145 podman[114670]: 2024-09-21 10:25:44.198118924 +0000 UTC m=+0.892647786 container init d46eda1507991cdf9ffe669874946fed9df98eece74873029a71dcc0c13176b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-fbd1a6a) 2024-09-21T10:25:44.441 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:44 smithi145 podman[114670]: 2024-09-21 10:25:44.204276551 +0000 UTC m=+0.898805397 container start d46eda1507991cdf9ffe669874946fed9df98eece74873029a71dcc0c13176b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T10:25:44.441 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:44 smithi145 podman[114670]: 2024-09-21 10:25:44.297909374 +0000 UTC m=+0.992438226 container attach d46eda1507991cdf9ffe669874946fed9df98eece74873029a71dcc0c13176b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T10:25:44.441 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:44 smithi145 ceph-mon[89359]: pgmap v1660: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 568 op/s; 38671/307446 objects degraded (12.578%) 2024-09-21T10:25:44.441 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:44 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:25:44.441 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:44 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:25:44.442 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:44 smithi145 ceph-mon[89359]: osdmap e1187: 8 total, 7 up, 8 in 2024-09-21T10:25:44.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:44.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:44.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:44.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:44.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:44.794 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:44 smithi145 podman[114670]: 2024-09-21 10:25:44.568969258 +0000 UTC m=+1.263498109 container died d46eda1507991cdf9ffe669874946fed9df98eece74873029a71dcc0c13176b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:25:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:25:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:25:44] "GET /metrics HTTP/1.1" 200 36769 "" "Prometheus/2.51.0" 2024-09-21T10:25:44.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:44.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:44.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:45.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:45.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:45.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:45.198 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:45.272 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:45.272 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:45.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:45.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:45 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 38671/307446 objects degraded (12.578%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:45 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 38671/307446 objects degraded (12.578%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:45.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:45 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 38671/307446 objects degraded (12.578%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:45.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:45.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:45.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:45.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:45.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:45.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:45.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:46.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:46.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:46.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:25:46.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:46.142 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:25:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.199 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:46.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:46.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:46.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:46 smithi119 ceph-mon[105288]: pgmap v1662: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 37255/298191 objects degraded (12.494%) 2024-09-21T10:25:46.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:46 smithi119 ceph-mon[101432]: pgmap v1662: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 37255/298191 objects degraded (12.494%) 2024-09-21T10:25:46.429 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:25:46.429 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:46.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:46.449 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:25:46.449 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:46.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.484 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:25:46.484 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:46.515 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:25:46.515 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:46.542 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:25:46.543 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:46 smithi145 ceph-mon[89359]: pgmap v1662: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 887 op/s; 37255/298191 objects degraded (12.494%) 2024-09-21T10:25:46.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:46.573 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:25:46.573 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:46.578 INFO:teuthology.orchestra.run.smithi119.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:25:46.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T10:25:46.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:46.610 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:25:46.611 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:46.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:25:46.650 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T10:25:46.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:46.737 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:46.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:46.866 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:46.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:46.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:47.032 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:47.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:47.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:47.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:47.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:47.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1825370345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:47.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1825370345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1825370345' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:47.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:47.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:25:47.492 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:25:47.538 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:47.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:47.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:47.739 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:47.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:47.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:47.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:47.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:48.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:48.056 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:48.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:48.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:48.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:48.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:48.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:48.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:48 smithi119 ceph-mon[105288]: pgmap v1663: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 36987/295731 objects degraded (12.507%) 2024-09-21T10:25:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:48 smithi119 ceph-mon[101432]: pgmap v1663: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 36987/295731 objects degraded (12.507%) 2024-09-21T10:25:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:48.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:48.381 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:48 smithi145 ceph-mon[89359]: pgmap v1663: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 806 op/s; 36987/295731 objects degraded (12.507%) 2024-09-21T10:25:48.381 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:48.381 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:25:48.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:48.456 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:48.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:48.590 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:48.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:48.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:48.818 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:48.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:48.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:48.989 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:48.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:49.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:49.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:49.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:49.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:49.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:49.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:49.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:25:49.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:49.590 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:49.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.750 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:49.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:49.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:49.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:49.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:49.988 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:25:49.988 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45614.0:4011 43.6 e985/865 43:6665b828:::rbd_data.b22e8cd911aa.0000000000000015:head v 985'45, mlcod=981'41)", 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.436954+0000", 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 503.54618521499998, 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048492489, 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:49.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436954+0000", 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436954+0000", 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436954+0000", 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.440736+0000", 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037826019999999999 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.440792+0000", 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5483999999999999e-05 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.440811+0000", 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9639e-05 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.440926+0000", 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000114899 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.440964+0000", 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7920999999999998e-05 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.485392+0000", 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.044428350999999998 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.485446+0000", 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3857999999999998e-05 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:4022 43.1d 43:bbe61236:::rbd_data.b22e8cd911aa.000000000000002f:head [writefull 0~4194304] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e985)", 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.441155+0000", 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 503.54198381999998, 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048851745000000002, 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:49.993 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4022 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441155+0000", 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441155+0000", 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441153+0000", 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451533+0000", 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010379779 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451556+0000", 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2419e-05 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451566+0000", 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9520000000000006e-06 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451628+0000", 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2218999999999995e-05 2024-09-21T10:25:49.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451739+0000", 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011141600000000001 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451920+0000", 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018092999999999999 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:49.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.470629+0000", 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018709309 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.488939+0000", 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018309938000000001 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489907+0000", 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00096788099999999997 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489962+0000", 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4796000000000002e-05 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.490007+0000", 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5003999999999998e-05 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45614.0:4020 43.11 e985/865 43:89d9059d:::rbd_data.b22e8cd911aa.0000000000000054:head v 985'76, mlcod=984'72)", 2024-09-21T10:25:49.999 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.441190+0000", 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 503.541948574, 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047710292000000001, 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441190+0000", 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441190+0000", 2024-09-21T10:25:50.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441190+0000", 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441711+0000", 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052145499999999999 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441731+0000", 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9888e-05 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441742+0000", 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.039e-05 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441824+0000", 2024-09-21T10:25:50.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2670000000000006e-05 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.441859+0000", 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4805000000000003e-05 2024-09-21T10:25:50.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.488861+0000", 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047001712000000001 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.488901+0000", 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0042e-05 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.004 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35751.0:1561 44.1a 44:5ce24ee5:::rbd_data.8ba731f97b7c.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1090)", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:20:19.049983+0000", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 330.93315656499999, 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.057038775999999999, 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35751", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1503148914", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1561 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.049983+0000", 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.049983+0000", 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.049981+0000", 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061633+0000", 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011652186 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061653+0000", 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9428000000000001e-05 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061662+0000", 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2110000000000004e-06 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061750+0000", 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8603000000000005e-05 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061962+0000", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000211499 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.062227+0000", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026555799999999999 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.085245+0000", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023017124 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.086922+0000", 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001677481 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.106947+0000", 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020024579000000001 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.106993+0000", 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6421000000000002e-05 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.107021+0000", 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8204e-05 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35751.0:1562 44.c 44:345bf4d8:::rbd_data.8ba731f97b7c.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1090)", 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:20:19.061682+0000", 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 330.92145686999999, 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.058307354999999998, 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35751", 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1503148914", 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1562 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061682+0000", 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061682+0000", 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.061680+0000", 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.070042+0000", 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083617329999999997 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.070061+0000", 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9012999999999999e-05 2024-09-21T10:25:50.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.070068+0000", 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3760000000000002e-06 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.070125+0000", 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6684999999999997e-05 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.070281+0000", 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015633500000000001 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.070453+0000", 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017127599999999999 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.087124+0000", 2024-09-21T10:25:50.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016671743999999999 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.092973+0000", 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058480659999999999 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.119894+0000", 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.026921255000000002 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.119948+0000", 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4691000000000002e-05 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:19.119990+0000", 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1100000000000003e-05 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45632.0:1188 44.c 44:3340e42a:::rbd_data.b2407cc57e66.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1093)", 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:20:29.796788+0000", 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 320.18635131000002, 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.062928933000000006, 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45632", 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1188 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.796788+0000", 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.796788+0000", 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.796785+0000", 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:25:50.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.808877+0000", 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012091655 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.808955+0000", 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8063000000000001e-05 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.808997+0000", 2024-09-21T10:25:50.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1526999999999997e-05 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.809083+0000", 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6645999999999997e-05 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.809295+0000", 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021194900000000001 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.809501+0000", 2024-09-21T10:25:50.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020521600000000001 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.836830+0000", 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.027329057 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.840609+0000", 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037789310000000001 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.859656+0000", 2024-09-21T10:25:50.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019047028000000001 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.859693+0000", 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7475000000000003e-05 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:20:29.859717+0000", 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3688000000000001e-05 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45632.0:2771 44.17 44:ed8bf0d4:::rbd_data.b240933cb63.0000000000000016:head [writefull 0~4194304] snapc 71=[71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:31.233510+0000", 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 258.74962925, 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.063025697000000006, 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45632", 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2771 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.233510+0000", 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.233510+0000", 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.233507+0000", 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.024 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.258219+0000", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.024711107999999999 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.258238+0000", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9772e-05 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.258247+0000", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8270000000000004e-06 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.258297+0000", 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9477000000000001e-05 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.258396+0000", 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8938999999999996e-05 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.258563+0000", 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016730400000000001 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.273163+0000", 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014600422 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.279449+0000", 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062860989999999999 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.296465+0000", 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017015354 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.296505+0000", 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0595e-05 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.296536+0000", 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0130000000000001e-05 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45632.0:2930 44.0 44:07e55497:::rbd_data.b240933cb63.0000000000000036:head [writefull 0~4194304] snapc 74=[74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1131)", 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:34.196601+0000", 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 255.78653792700001, 2024-09-21T10:25:50.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.046202839000000002, 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45632", 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2930 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.196601+0000", 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.196601+0000", 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.196599+0000", 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.206726+0000", 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010126568000000001 2024-09-21T10:25:50.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.206748+0000", 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.16e-05 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.206755+0000", 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4889999999999998e-06 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.206805+0000", 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.007e-05 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.206927+0000", 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000121441 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.207114+0000", 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018721499999999999 2024-09-21T10:25:50.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.213251+0000", 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061371769999999997 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.241956+0000", 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.028705398 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.242731+0000", 2024-09-21T10:25:50.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000774244 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.242765+0000", 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4366999999999998e-05 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:34.242804+0000", 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8996999999999999e-05 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:6938 44.4 e1157/1014 44:210af00f:::rbd_data.b240bb5bb67f.0000000000000001:head v 1157'67, mlcod=1155'64)", 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:22:00.497620+0000", 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 229.485519001, 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.053634339000000003, 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.497620+0000", 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.497620+0000", 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.497619+0000", 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.522843+0000", 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.025223821 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.522875+0000", 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2045999999999998e-05 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.522884+0000", 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5129999999999996e-06 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.522932+0000", 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8025999999999997e-05 2024-09-21T10:25:50.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.522951+0000", 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9768000000000001e-05 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.551202+0000", 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.028251054000000001 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.551254+0000", 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1931000000000001e-05 2024-09-21T10:25:50.038 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45632.0:7360 44.0 44:06e2e6f0:::rbd_data.b240938dee57.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1160)", 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:22:03.551041+0000", 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 226.43209856499999, 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.055721828000000001, 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45632", 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:25:50.039 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 7360 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.551041+0000", 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.551041+0000", 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.551040+0000", 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.569806+0000", 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018766189999999999 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.569844+0000", 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8720000000000002e-05 2024-09-21T10:25:50.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.569858+0000", 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3722e-05 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.569955+0000", 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6575000000000003e-05 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.570173+0000", 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000218478 2024-09-21T10:25:50.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.570393+0000", 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00021927200000000001 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.596039+0000", 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.025646207000000001 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.602423+0000", 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0063842969999999997 2024-09-21T10:25:50.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.606593+0000", 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004169842 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.606681+0000", 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7954000000000005e-05 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:03.606762+0000", 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1503999999999998e-05 2024-09-21T10:25:50.045 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:13812 45.1 45:861ebb57:::benchmark_data_smithi119_168706_object13811:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1169)", 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:23:41.518321+0000", 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 128.46481834100001, 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048652014, 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 13812 2024-09-21T10:25:50.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518321+0000", 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518321+0000", 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518319+0000", 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518375+0000", 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6035000000000001e-05 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518387+0000", 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1544e-05 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518391+0000", 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5109999999999997e-06 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518433+0000", 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1938999999999999e-05 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518527+0000", 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3871999999999998e-05 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.518623+0000", 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5623000000000002e-05 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.519294+0000", 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067166899999999996 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.519478+0000", 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183869 2024-09-21T10:25:50.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.566920+0000", 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047441469 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.566951+0000", 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1448000000000001e-05 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:23:41.566973+0000", 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1631e-05 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:69547 45.1 45:8d361491:::benchmark_data_smithi119_168706_object69546:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:11.570509+0000", 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.412630307000001, 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.058754270999999997, 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69547 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570509+0000", 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570509+0000", 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570508+0000", 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570595+0000", 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7057000000000004e-05 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570607+0000", 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2238e-05 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570613+0000", 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3300000000000004e-06 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570646+0000", 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2601000000000002e-05 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570730+0000", 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.407e-05 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.570812+0000", 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1679000000000002e-05 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.572565+0000", 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017536940000000001 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.572683+0000", 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117466 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.629185+0000", 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.056502366999999998 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.629224+0000", 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8994e-05 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.629263+0000", 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8931999999999999e-05 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.057 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:103659 45.1 45:852843b1:::benchmark_data_smithi119_168706_object103658:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1176)", 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:40.087520+0000", 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 69.895619060000001, 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.046329267, 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.058 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 103659 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087520+0000", 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087520+0000", 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087519+0000", 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087611+0000", 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2357999999999999e-05 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087626+0000", 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4775e-05 2024-09-21T10:25:50.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087631+0000", 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4960000000000004e-06 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087664+0000", 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2651999999999998e-05 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087775+0000", 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011075599999999999 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.087906+0000", 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131586 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.091520+0000", 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036141229999999999 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.092889+0000", 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001368518 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.133769+0000", 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.040880113000000003 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.133806+0000", 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6665000000000002e-05 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:40.133849+0000", 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3565e-05 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.063 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:127266 45.1 45:84b29ef5:::benchmark_data_smithi119_168706_object127265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1180)", 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:04.134251+0000", 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 45.848888528000003, 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.058292297999999999, 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 127266 2024-09-21T10:25:50.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134251+0000", 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134251+0000", 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134249+0000", 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134302+0000", 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2947e-05 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134317+0000", 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5023e-05 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134322+0000", 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8390000000000003e-06 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134364+0000", 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1415999999999998e-05 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.134487+0000", 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012269900000000001 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.135629+0000", 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011429560000000001 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.148415+0000", 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012785502000000001 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.148581+0000", 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000166207 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192420+0000", 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.043838974000000003 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192474+0000", 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3817000000000003e-05 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192543+0000", 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8875999999999996e-05 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:127269 45.1 45:880f31ab:::benchmark_data_smithi119_168706_object127268:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1180)", 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:04.136411+0000", 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 45.846727825999999, 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.056263984000000003, 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 127269 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.070 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.136411+0000", 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.136411+0000", 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.136410+0000", 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.136482+0000", 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2162999999999993e-05 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.136499+0000", 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6945000000000001e-05 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.136505+0000", 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9109999999999998e-06 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139647+0000", 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031424619999999999 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139757+0000", 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110208 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139888+0000", 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000130443 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.148419+0000", 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085314459999999998 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.149679+0000", 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012593839999999999 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192592+0000", 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.042912973 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192643+0000", 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1320000000000003e-05 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192675+0000", 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2175999999999998e-05 2024-09-21T10:25:50.075 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:127272 45.1 45:86f27695:::benchmark_data_smithi119_168706_object127271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1180)", 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:04.139455+0000", 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 45.843683906999999, 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.053330952000000001, 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.076 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 127272 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139455+0000", 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139455+0000", 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139454+0000", 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139478+0000", 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4754000000000001e-05 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139495+0000", 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6348e-05 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.139501+0000", 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3509999999999998e-06 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.143776+0000", 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042750059999999996 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.143891+0000", 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115244 2024-09-21T10:25:50.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.144268+0000", 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000376893 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.148555+0000", 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042864310000000003 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.154433+0000", 2024-09-21T10:25:50.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058785629999999998 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192717+0000", 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.038284002999999997 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192756+0000", 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8841000000000003e-05 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:04.192786+0000", 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0049e-05 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:153847 45.1 45:8c73d9e3:::benchmark_data_smithi119_168706_object23677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1184)", 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:33.799837+0000", 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.183301595, 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048673510000000003, 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.083 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.083 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.083 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 153847 2024-09-21T10:25:50.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.083 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.799837+0000", 2024-09-21T10:25:50.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.799837+0000", 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.799835+0000", 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.799846+0000", 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0471e-05 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.799862+0000", 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6456999999999998e-05 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.799869+0000", 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7390000000000002e-06 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.816226+0000", 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016356834000000001 2024-09-21T10:25:50.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.816503+0000", 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027666799999999999 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.816833+0000", 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033067900000000001 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.825930+0000", 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0090964449999999999 2024-09-21T10:25:50.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.825956+0000", 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6469e-05 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.848246+0000", 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022290213 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.848388+0000", 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141941 2024-09-21T10:25:50.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:33.848511+0000", 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000122662 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155692 45.7 45:e02c39ca:::benchmark_data_smithi119_168706_object25522:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.022122+0000", 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.961017313999999, 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.044059455, 2024-09-21T10:25:50.089 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155692 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022122+0000", 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022122+0000", 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022121+0000", 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022126+0000", 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4940000000000003e-06 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022144+0000", 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7997999999999999e-05 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022151+0000", 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.024e-06 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022209+0000", 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7553999999999998e-05 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.022226+0000", 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6735000000000002e-05 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.048126+0000", 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0259006959999999 2024-09-21T10:25:50.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.050896+0000", 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002769915 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1", 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.050993+0000", 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7165999999999998e-05 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.063784+0000", 2024-09-21T10:25:50.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012790295 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066035+0000", 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002251734 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066086+0000", 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0736999999999999e-05 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066181+0000", 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5030000000000003e-05 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155704 45.7 45:e7a5874c:::benchmark_data_smithi119_168706_object25534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.048300+0000", 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 13.934839437000001, 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.018060607, 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.096 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155704 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048300+0000", 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048300+0000", 2024-09-21T10:25:50.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048298+0000", 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048304+0000", 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4010000000000002e-06 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048319+0000", 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4791e-05 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048325+0000", 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2920000000000001e-06 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048379+0000", 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3633e-05 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.048386+0000", 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3849999999999999e-06 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.054961+0000", 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.006575923 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.055078+0000", 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116222 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1", 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.055155+0000", 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7021999999999995e-05 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.063797+0000", 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086420219999999992 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066233+0000", 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024358919999999998 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066281+0000", 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7966999999999997e-05 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066360+0000", 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9682e-05 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.102 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:164487 45.7 45:ec96faa7:::benchmark_data_smithi119_168706_object34317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1187)", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:47.068604+0000", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.9145354289999998, 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048142328999999998, 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 164487 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.068604+0000", 2024-09-21T10:25:50.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.068604+0000", 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.068603+0000", 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:25:50.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.068606+0000", 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.509e-06 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.068623+0000", 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7552000000000001e-05 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.068627+0000", 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2710000000000001e-06 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:25:50.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.078997+0000", 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010370195 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.079102+0000", 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105037 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1", 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.079168+0000", 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5703000000000004e-05 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.084953+0000", 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057855529999999997 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.116608+0000", 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.031654564000000003 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.116657+0000", 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9493999999999997e-05 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:25:50.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:47.116746+0000", 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8750000000000002e-05 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:25:50.108 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:25:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:50.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:50.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:50.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:50.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:50.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:50.358 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:25:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:25:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:25:50.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:50 smithi145 ceph-mon[89359]: pgmap v1664: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 36133/289890 objects degraded (12.464%) 2024-09-21T10:25:50.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:50 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 36133/289890 objects degraded (12.464%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:50.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:50.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:50.359 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/961013428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[105288]: pgmap v1664: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 36133/289890 objects degraded (12.464%) 2024-09-21T10:25:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 36133/289890 objects degraded (12.464%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:50.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/961013428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:50.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[101432]: pgmap v1664: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.9 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 842 op/s; 36133/289890 objects degraded (12.464%) 2024-09-21T10:25:50.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 36133/289890 objects degraded (12.464%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:50.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:50.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:50.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/961013428' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:25:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.370 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:50.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:50.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:50.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.510 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:50.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:50.611 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:50.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:50.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:50.755 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:25:50.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:50.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:50.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:50.950 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:50.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:51.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:51.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:51.133 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:25:51.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:51.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:51.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:51.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:51.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/961013428' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[105288]: osdmap e1188: 8 total, 7 up, 8 in 2024-09-21T10:25:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/961013428' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:51 smithi119 ceph-mon[101432]: osdmap e1188: 8 total, 7 up, 8 in 2024-09-21T10:25:51.408 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:25:51.408 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:25:51.408 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:25:51.409 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:51 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:25:51.409 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:51 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/961013428' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:25:51.409 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:51 smithi145 ceph-mon[89359]: osdmap e1188: 8 total, 7 up, 8 in 2024-09-21T10:25:51.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:51.492 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.493 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:51.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:51.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:51.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:51.650 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [6] live_osds: [0, 1, 2, 3, 4, 5, 7] 2024-09-21T10:25:51.651 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:25:51.651 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-09-21T10:25:51.651 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-09-21T10:25:51.651 DEBUG:teuthology.orchestra.run.smithi145:osd.6> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T10:25:51.662 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl start ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T10:25:51.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:51.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:51.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:51.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:51.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:52.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:52.019 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:51 smithi145 systemd[1]: Starting Ceph osd.6 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:25:52.046 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:52.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:52.178 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:52.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:52.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:52 smithi145 ceph-mon[89359]: pgmap v1665: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 35349/282573 objects degraded (12.510%) 2024-09-21T10:25:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:52 smithi145 ceph-mon[89359]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:25:52.303 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:52.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:52 smithi119 ceph-mon[105288]: pgmap v1665: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 35349/282573 objects degraded (12.510%) 2024-09-21T10:25:52.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:52 smithi119 ceph-mon[105288]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:25:52.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:52 smithi119 ceph-mon[101432]: pgmap v1665: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 904 op/s; 35349/282573 objects degraded (12.510%) 2024-09-21T10:25:52.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:52 smithi119 ceph-mon[101432]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T10:25:52.404 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:52.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:52.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:52.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:52.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:52.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.700 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:52.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:52.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:52.837 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:52.893 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:25:52.894 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:25:52.925 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:25:52.925 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:25:52.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:52.963 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:25:52.963 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:25:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:52.973 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:52.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:53.002 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:25:53.003 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:25:53.034 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:25:53.034 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:25:53.064 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:25:53.064 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:25:53.065 INFO:teuthology.orchestra.run.smithi119.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:25:53.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:53.098 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:25:53.098 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:25:53.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:52 smithi145 podman[115856]: 2024-09-21 10:25:52.905674169 +0000 UTC m=+0.034951929 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:25:53.101 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 podman[115856]: 2024-09-21 10:25:53.041554869 +0000 UTC m=+0.170832604 container create b08d58b5e9c6f246b1394a008e168f3082d5a636c24f9d4f5fe666c4491fcaf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:25:53.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:53.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:53.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:53.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:53.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/22347132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:53.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/22347132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:53.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:53.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:53.438 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/22347132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:25:53.439 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 podman[115856]: 2024-09-21 10:25:53.328744875 +0000 UTC m=+0.458022623 container init b08d58b5e9c6f246b1394a008e168f3082d5a636c24f9d4f5fe666c4491fcaf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:25:53.439 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 podman[115856]: 2024-09-21 10:25:53.335077278 +0000 UTC m=+0.464355017 container start b08d58b5e9c6f246b1394a008e168f3082d5a636c24f9d4f5fe666c4491fcaf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T10:25:53.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:53.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:53.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:53.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:53.658 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:53.699 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 podman[115856]: 2024-09-21 10:25:53.4369581 +0000 UTC m=+0.566235839 container attach b08d58b5e9c6f246b1394a008e168f3082d5a636c24f9d4f5fe666c4491fcaf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:25:53.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:53.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:53.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:25:53.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:25:53.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:53.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:53.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:53.984 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate[115902]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-21T10:25:53.985 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:53 smithi145 bash[115856]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-21T10:25:53.994 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:53.994 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:54.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:54.061 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:54.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:54.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:54.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:25:54.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:54 smithi145 ceph-mon[89359]: pgmap v1667: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 35349/280932 objects degraded (12.583%) 2024-09-21T10:25:54.294 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:54 smithi145 podman[115856]: 2024-09-21 10:25:54.207185283 +0000 UTC m=+1.336463034 container died b08d58b5e9c6f246b1394a008e168f3082d5a636c24f9d4f5fe666c4491fcaf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T10:25:54.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:54.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:54.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:54 smithi119 ceph-mon[105288]: pgmap v1667: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 35349/280932 objects degraded (12.583%) 2024-09-21T10:25:54.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:54 smithi119 ceph-mon[101432]: pgmap v1667: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 925 op/s; 35349/280932 objects degraded (12.583%) 2024-09-21T10:25:54.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.467 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:54.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:54.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:54.590 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:54.678 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:54.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:54.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:54.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:54.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:25:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:25:54] "GET /metrics HTTP/1.1" 200 36773 "" "Prometheus/2.51.0" 2024-09-21T10:25:54.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:54.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:54.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:54.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:55.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:55.116 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:55.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:55.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:55.294 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:55 smithi145 podman[116095]: 2024-09-21 10:25:55.03177796 +0000 UTC m=+1.244791865 container remove b08d58b5e9c6f246b1394a008e168f3082d5a636c24f9d4f5fe666c4491fcaf9 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:25:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:55 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 35349/280932 objects degraded (12.583%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:55.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:55 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 35349/280932 objects degraded (12.583%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:55 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 35349/280932 objects degraded (12.583%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:25:55.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:55.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:55.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:55.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:25:55.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:55.651 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:55.753 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:55.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:55.794 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:55 smithi145 podman[116340]: 2024-09-21 10:25:55.607079569 +0000 UTC m=+0.025687716 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:25:55.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:25:55.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:55.937 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:55.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:56.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:56.074 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:56.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:56.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:56.222 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:56 smithi145 ceph-mon[89359]: pgmap v1668: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 33906/271572 objects degraded (12.485%) 2024-09-21T10:25:56.222 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:55 smithi145 podman[116340]: 2024-09-21 10:25:55.945536257 +0000 UTC m=+0.364144393 container create e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-21T10:25:56.223 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:56 smithi145 podman[116340]: 2024-09-21 10:25:56.182282996 +0000 UTC m=+0.600891133 container init e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T10:25:56.223 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:56 smithi145 podman[116340]: 2024-09-21 10:25:56.194271978 +0000 UTC m=+0.612880120 container start e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a) 2024-09-21T10:25:56.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:56.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:25:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.334 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:56.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:56 smithi119 ceph-mon[105288]: pgmap v1668: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 33906/271572 objects degraded (12.485%) 2024-09-21T10:25:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:56 smithi119 ceph-mon[101432]: pgmap v1668: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 909 op/s; 33906/271572 objects degraded (12.485%) 2024-09-21T10:25:56.397 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:56.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:56.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:56.534 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:56 smithi145 ceph-osd[116444]: -- 172.21.15.145:0/2404002586 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561a116c0d00 con 0x561a11692c00 2024-09-21T10:25:56.534 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:56 smithi145 bash[116340]: e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 2024-09-21T10:25:56.535 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:56 smithi145 systemd[1]: Started Ceph osd.6 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:25:56.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:56.603 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:56.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:56.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:56.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:56.794 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:25:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:25:56.741+0000 7fde25e8a440 -1 Falling back to public interface 2024-09-21T10:25:56.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:56.868 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:56.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:56.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:25:57.005 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:25:57.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:57.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:57.222 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:25:57.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:57.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:57.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:57.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:57.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:57.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:57.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:57.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:25:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:57.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:25:57.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:57.825 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:57.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:57.926 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:57.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:57.952 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:57.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:58.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.087 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.189 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.227 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:58 smithi119 ceph-mon[105288]: pgmap v1669: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 33641/269202 objects degraded (12.497%) 2024-09-21T10:25:58.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:58 smithi119 ceph-mon[101432]: pgmap v1669: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 33641/269202 objects degraded (12.497%) 2024-09-21T10:25:58.315 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:58 smithi145 ceph-mon[89359]: pgmap v1669: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 906 op/s; 33641/269202 objects degraded (12.497%) 2024-09-21T10:25:58.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.337 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:25:58.337 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:25:58.345 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.386 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:25:58.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:58.470 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:58.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:58.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:58.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:58.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:58.826 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:58.851 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:58.953 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:25:58.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:58.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:25:59.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:25:59.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:59.141 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:25:59.146 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:25:59.164 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1727479500' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:59.164 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:25:59 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:59.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1727479500' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:59.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:25:59 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:59.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:59.285 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:59.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1727479500' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:25:59 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:25:59.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:59.420 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:59.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:25:59.557 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:25:59.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:25:59.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:25:59.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:25:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:25:59.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:25:59.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:25:59.950 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:25:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:25:59.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:00.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:00.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:00.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:00.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:00.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[105288]: pgmap v1670: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 33327/265533 objects degraded (12.551%) 2024-09-21T10:26:00.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:00.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:00.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[105288]: osdmap e1189: 8 total, 7 up, 8 in 2024-09-21T10:26:00.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 33327/265533 objects degraded (12.551%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:00.345 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[101432]: pgmap v1670: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 33327/265533 objects degraded (12.551%) 2024-09-21T10:26:00.345 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:00.345 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:00.345 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[101432]: osdmap e1189: 8 total, 7 up, 8 in 2024-09-21T10:26:00.345 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 33327/265533 objects degraded (12.551%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:00.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:00.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:00.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:00.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:26:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:26:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:00 smithi145 ceph-mon[89359]: pgmap v1670: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 832 op/s; 33327/265533 objects degraded (12.551%) 2024-09-21T10:26:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:00 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:00 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:00 smithi145 ceph-mon[89359]: osdmap e1189: 8 total, 7 up, 8 in 2024-09-21T10:26:00.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 33327/265533 objects degraded (12.551%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:00.603 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:00.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:00.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:00.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:00.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:00.900 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:00.924 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:00.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:01.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:01.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:01.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:01.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:01.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:01.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:01.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:01.420 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:01.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:01.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:01.594 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:01.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:01.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:01.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:01.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:01.947 INFO:teuthology.orchestra.run.smithi145.stderr:no valid command found; 10 closest matches: 2024-09-21T10:26:01.947 INFO:teuthology.orchestra.run.smithi145.stderr:0 2024-09-21T10:26:01.947 INFO:teuthology.orchestra.run.smithi145.stderr:1 2024-09-21T10:26:01.947 INFO:teuthology.orchestra.run.smithi145.stderr:2 2024-09-21T10:26:01.947 INFO:teuthology.orchestra.run.smithi145.stderr:abort 2024-09-21T10:26:01.947 INFO:teuthology.orchestra.run.smithi145.stderr:assert 2024-09-21T10:26:01.948 INFO:teuthology.orchestra.run.smithi145.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:26:01.948 INFO:teuthology.orchestra.run.smithi145.stderr:bluefs files list 2024-09-21T10:26:01.948 INFO:teuthology.orchestra.run.smithi145.stderr:bluefs stats 2024-09-21T10:26:01.948 INFO:teuthology.orchestra.run.smithi145.stderr:bluestore allocator dump block 2024-09-21T10:26:01.948 INFO:teuthology.orchestra.run.smithi145.stderr:bluestore allocator fragmentation block 2024-09-21T10:26:01.948 INFO:teuthology.orchestra.run.smithi145.stderr:admin_socket: invalid command 2024-09-21T10:26:01.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:01.993 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:01.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:02.044 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:26:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:26:01.685+0000 7fde25e8a440 -1 osd.6 1184 log_to_monitors true 2024-09-21T10:26:02.095 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:02.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:02.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:02.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:02.370 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:02.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:26:02.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:02.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[105288]: pgmap v1672: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 31902/255267 objects degraded (12.498%) 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2339765722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[105288]: from='osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[105288]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[101432]: pgmap v1672: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 31902/255267 objects degraded (12.498%) 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2339765722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[101432]: from='osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T10:26:02.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:02 smithi119 ceph-mon[101432]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T10:26:02.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:02.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:02.532 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:26:02.532 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:26:02.532 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:26:02.532 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:26:02.532 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:26:02.532 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-09-21T10:26:02.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:02.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:26:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:26:02.167+0000 7fde1c433640 -1 osd.6 1184 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:26:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:02 smithi145 ceph-mon[89359]: pgmap v1672: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 924 op/s; 31902/255267 objects degraded (12.498%) 2024-09-21T10:26:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2339765722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:02 smithi145 ceph-mon[89359]: from='osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T10:26:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:02 smithi145 ceph-mon[89359]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-21T10:26:02.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:02.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:02.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:02.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:02.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:02.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:02.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:02.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:26:03.000 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-6, ['dump_ops_in_flight'] 2024-09-21T10:26:03.042 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:03.067 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.168 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:03.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:03 smithi145 ceph-mon[89359]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T10:26:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:03 smithi145 ceph-mon[89359]: from='osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:26:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:03 smithi145 ceph-mon[89359]: osdmap e1190: 8 total, 7 up, 8 in 2024-09-21T10:26:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:03 smithi145 ceph-mon[89359]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:26:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:03.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:03.332 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:03.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:03.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:03.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:03.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:03.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[101432]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T10:26:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[101432]: from='osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:26:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[101432]: osdmap e1190: 8 total, 7 up, 8 in 2024-09-21T10:26:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[101432]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:26:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[105288]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T10:26:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[105288]: from='osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:26:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[105288]: osdmap e1190: 8 total, 7 up, 8 in 2024-09-21T10:26:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[105288]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:26:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:03.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:03.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:03.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:03.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:03.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:03.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:03.952 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:03.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:03.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:3572 43.9 43:9723441a:::rbd_data.b22e8cd911aa.0000000000000007:head [writefull 0~4194304] snapc 6f=[6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e977)", 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:18.358331+0000", 2024-09-21T10:26:04.049 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 525.68698515200003, 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047855743999999999, 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3572 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.358331+0000", 2024-09-21T10:26:04.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.358331+0000", 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.358329+0000", 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.364546+0000", 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006216577 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.364561+0000", 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4611e-05 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.364568+0000", 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.323e-06 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.364617+0000", 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9115e-05 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.364695+0000", 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7669000000000005e-05 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.364868+0000", 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017331399999999999 2024-09-21T10:26:04.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.377462+0000", 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012594138 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.405868+0000", 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.028405308000000001 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.406117+0000", 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024911699999999998 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.406156+0000", 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9079000000000003e-05 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:18.406187+0000", 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1480999999999999e-05 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.055 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45614.0:3665 43.2 e979/865 43:46ff053a:::rbd_data.b22e8cd911aa.0000000000000041:head v 979'23, mlcod=973'20)", 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:20.375841+0000", 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 523.66947546300003, 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.046330604999999997, 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.375841+0000", 2024-09-21T10:26:04.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.375841+0000", 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.375841+0000", 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.401087+0000", 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.025245882000000001 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.401112+0000", 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5488000000000001e-05 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.401123+0000", 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.025e-05 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.401205+0000", 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2663000000000005e-05 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.401235+0000", 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9499e-05 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.422130+0000", 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020895145 2024-09-21T10:26:04.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.422172+0000", 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1919999999999998e-05 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:3679 43.11 43:8b3beb6b:::rbd_data.b22e8cd911aa.000000000000002e:head [write 234219~3960085] snapc 71=[71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e979)", 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:20.429484+0000", 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 523.61583270699998, 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.045993033000000003, 2024-09-21T10:26:04.060 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3679 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.429484+0000", 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.429484+0000", 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.429482+0000", 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.442179+0000", 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012696742 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.442201+0000", 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2334e-05 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.442210+0000", 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7059999999999993e-06 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.442277+0000", 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6353000000000007e-05 2024-09-21T10:26:04.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.442379+0000", 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102699 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.442572+0000", 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000192516 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.471109+0000", 2024-09-21T10:26:04.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.028536786000000001 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.475218+0000", 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041094169999999998 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.475401+0000", 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000183423 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.475438+0000", 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6665999999999998e-05 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:20.475477+0000", 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8937999999999999e-05 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:3763 43.11 43:89d9059d:::rbd_data.b22e8cd911aa.0000000000000054:head [write 531889~3662415] snapc 73=[73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e981)", 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:22.385837+0000", 2024-09-21T10:26:04.066 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 521.65947969499996, 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047561135999999997, 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 3763 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.385837+0000", 2024-09-21T10:26:04.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.385837+0000", 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.385833+0000", 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.391434+0000", 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005600636 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.391456+0000", 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2113999999999999e-05 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.391479+0000", 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3417e-05 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.391550+0000", 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1179000000000004e-05 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op must wait for map", 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.391556+0000", 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9030000000000001e-06 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.394140+0000", 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002583835 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.394258+0000", 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011749 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.394626+0000", 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036817099999999998 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.427870+0000", 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.033243811999999998 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.431257+0000", 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003387415 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.433276+0000", 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020191369999999998 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.433342+0000", 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5514999999999999e-05 2024-09-21T10:26:04.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:22.433398+0000", 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6459000000000003e-05 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:4011 43.6 43:6665b828:::rbd_data.b22e8cd911aa.0000000000000015:head [write 160560~3902398] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e985)", 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.422138+0000", 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 517.623178263, 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.063529064999999996, 2024-09-21T10:26:04.073 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4011 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.422138+0000", 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.422138+0000", 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.422136+0000", 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436177+0000", 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014041036999999999 2024-09-21T10:26:04.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436206+0000", 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8890999999999998e-05 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436218+0000", 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1542000000000001e-05 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436297+0000", 2024-09-21T10:26:04.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:04.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8829e-05 2024-09-21T10:26:04.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436423+0000", 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012667900000000001 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,4", 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.436665+0000", 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024123100000000001 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.450912+0000", 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014247306 2024-09-21T10:26:04.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.478720+0000", 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.027808451000000001 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.485595+0000", 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068751029999999996 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.485636+0000", 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0831999999999998e-05 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.485667+0000", 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1133999999999997e-05 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:4020 43.11 43:89d9059d:::rbd_data.b22e8cd911aa.0000000000000054:head [write 0~669543] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e985)", 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.438211+0000", 2024-09-21T10:26:04.081 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 517.60710558899996, 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.050880482999999997, 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4020 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438211+0000", 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438211+0000", 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438208+0000", 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999971 2024-09-21T10:26:04.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438907+0000", 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00069915699999999995 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438947+0000", 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9949e-05 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438963+0000", 2024-09-21T10:26:04.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5447e-05 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.439038+0000", 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5279999999999998e-05 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.439157+0000", 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118466 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.439350+0000", 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019324000000000001 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.479783+0000", 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.040432958999999997 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.486395+0000", 2024-09-21T10:26:04.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0066125089999999999 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489012+0000", 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026167 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489051+0000", 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8936000000000001e-05 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489092+0000", 2024-09-21T10:26:04.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0669e-05 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:4021 43.11 43:8b3beb6b:::rbd_data.b22e8cd911aa.000000000000002e:head [write 2054388~2139916] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e985)", 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.438992+0000", 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 517.60632468999995, 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.050217325, 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.088 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4021 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438992+0000", 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438992+0000", 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.089 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.438991+0000", 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442216+0000", 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032255780000000002 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442245+0000", 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8433999999999999e-05 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.090 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442253+0000", 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6950000000000006e-06 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442324+0000", 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1136999999999993e-05 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442509+0000", 2024-09-21T10:26:04.091 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018421600000000001 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442772+0000", 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026343100000000001 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.485945+0000", 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.043172532999999999 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.092 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.486422+0000", 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00047729400000000001 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489127+0000", 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027050170000000001 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489171+0000", 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3792999999999999e-05 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.093 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.489209+0000", 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8512e-05 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:4023 43.11 43:89fc5b03:::rbd_data.b22e8cd911aa.0000000000000030:head [write 0~1198801] snapc 77=[77,76,75,74,73,72,71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e985)", 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:26.442274+0000", 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 517.60304284100005, 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048784763000000002, 2024-09-21T10:26:04.094 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 4023 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442274+0000", 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.095 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442274+0000", 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.442273+0000", 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.096 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.444150+0000", 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018778550000000001 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.444171+0000", 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0154000000000001e-05 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.444179+0000", 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5539999999999998e-06 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.097 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.450920+0000", 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067409820000000004 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451001+0000", 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0501999999999996e-05 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.098 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.451230+0000", 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022928600000000001 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.488621+0000", 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.037391420000000002 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.490835+0000", 2024-09-21T10:26:04.099 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022135240000000001 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.490970+0000", 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013526799999999999 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.491021+0000", 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0550999999999998e-05 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.100 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:26.491059+0000", 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7917999999999998e-05 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45614.0:40974 43.9 43:9595c13c:::rbd_data.b22e9bd61ac7.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1008)", 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:17:50.417657+0000", 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 493.62765947899999, 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.046394764999999998, 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.101 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45614", 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/2931341001", 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 40974 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.417657+0000", 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.417657+0000", 2024-09-21T10:26:04.102 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.417655+0000", 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.435905+0000", 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018249718000000002 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.103 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.435945+0000", 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0796e-05 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.435962+0000", 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6835000000000001e-05 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.436038+0000", 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5271999999999994e-05 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.104 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.436225+0000", 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018796000000000001 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.436402+0000", 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000176985 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.457759+0000", 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021356935000000001 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.105 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.463906+0000", 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061464809999999996 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.463995+0000", 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8937999999999994e-05 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.106 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.464028+0000", 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3538999999999999e-05 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:17:50.464052+0000", 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3536e-05 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.107 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45632.0:1646 44.5 44:a4430663:::rbd_data.b240cbecc4a7.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1105)", 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:08.206868+0000", 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 295.838448637, 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.049585384000000003, 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45632", 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 1646 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.108 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.206868+0000", 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.206868+0000", 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.109 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.206866+0000", 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.220727+0000", 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013860921999999999 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.220783+0000", 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6344000000000001e-05 2024-09-21T10:26:04.110 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.220801+0000", 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7965000000000002e-05 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.220909+0000", 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000107916 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.111 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.221107+0000", 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019741299999999999 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.221314+0000", 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020740999999999999 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.237599+0000", 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016284837999999999 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.112 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.254031+0000", 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016432031 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.256370+0000", 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002339261 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.256411+0000", 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0649000000000003e-05 2024-09-21T10:26:04.113 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.256453+0000", 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2629999999999997e-05 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:1649 44.e e1105/1014 44:746c80ec:::rbd_data.b240cbecc4a7.0000000000000003:head v 1105'20, mlcod=1085'18)", 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:08.211467+0000", 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 295.83384951199997, 2024-09-21T10:26:04.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.046142649000000001, 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.211467+0000", 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.211467+0000", 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.211465+0000", 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.225086+0000", 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013621276999999999 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.225121+0000", 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4406999999999998e-05 2024-09-21T10:26:04.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.225137+0000", 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6538999999999999e-05 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.225230+0000", 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2298000000000003e-05 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.225270+0000", 2024-09-21T10:26:04.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0580000000000001e-05 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.257570+0000", 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.032299495999999997 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:08.257610+0000", 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0126999999999997e-05 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:2207 44.14 e1118/1014 44:2bfa428e:::rbd_data.b240933cb63.0000000000000006:head v 1118'44, mlcod=1106'42)", 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:21.110092+0000", 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 282.93522482899999, 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.045690640999999997, 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.110092+0000", 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.110092+0000", 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.110091+0000", 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.123756+0000", 2024-09-21T10:26:04.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013664703 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.123768+0000", 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2267e-05 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.123773+0000", 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1749999999999996e-06 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.123839+0000", 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5664e-05 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.123865+0000", 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5695e-05 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.155738+0000", 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.031873667000000001 2024-09-21T10:26:04.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:21.155782+0000", 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3949000000000002e-05 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.45632.0:2751 44.1e 44:7d92725e:::rbd_data.b240933cb63.0000000000000001:head [write 0~3839723] snapc 71=[71,70,6f,6e,6d,6c,6b,6a,69,68,67] ondisk+write+known_if_redirected+supports_pool_eio e1128)", 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:31.172716+0000", 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 272.87260107700001, 2024-09-21T10:26:04.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.052479198999999997, 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.45632", 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1506404560", 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 2751 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.172716+0000", 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.172716+0000", 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.172714+0000", 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.193697+0000", 2024-09-21T10:26:04.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020982358 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.193735+0000", 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8309000000000002e-05 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.193746+0000", 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0634e-05 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.193843+0000", 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7412999999999997e-05 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.193980+0000", 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000137389 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.194214+0000", 2024-09-21T10:26:04.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023389999999999999 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.216393+0000", 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022178521 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.223712+0000", 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073190649999999996 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.225109+0000", 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013972120000000001 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.225154+0000", 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4691999999999998e-05 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:31.225195+0000", 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1e-05 2024-09-21T10:26:04.129 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:3152 44.14 e1135/1014 44:2bfa428e:::rbd_data.b240933cb63.0000000000000006:head v 1135'107, mlcod=1134'103)", 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:38.228357+0000", 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 265.81695960000002, 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.058588199000000001, 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.228357+0000", 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.228357+0000", 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.228357+0000", 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241476+0000", 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013119004 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241487+0000", 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1501000000000001e-05 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241493+0000", 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5280000000000003e-06 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241555+0000", 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1921000000000006e-05 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241571+0000", 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6225000000000001e-05 2024-09-21T10:26:04.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.286902+0000", 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.045330840999999997 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.286945+0000", 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3568e-05 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:3158 44.13 e1135/1014 44:cc678c9c:::rbd_data.b240933cb63.000000000000004e:head v 1135'34, mlcod=1129'30)", 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:38.232071+0000", 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 265.81324603899998, 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.054785364000000003, 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.232071+0000", 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.232071+0000", 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.232070+0000", 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.248624+0000", 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016554019 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.248650+0000", 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5777000000000001e-05 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.248656+0000", 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2620000000000004e-06 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.248729+0000", 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3040000000000005e-05 2024-09-21T10:26:04.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.248753+0000", 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4070000000000002e-05 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.286803+0000", 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.038049207000000002 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.286856+0000", 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3291999999999997e-05 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:3161 44.11 e1135/1014 44:89734fb0:::rbd_data.b240933cb63.0000000000000056:head v 1135'30, mlcod=1133'26)", 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:21:38.241997+0000", 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 265.80331986599998, 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.045014581999999997, 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241997+0000", 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241997+0000", 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.241997+0000", 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967296 2024-09-21T10:26:04.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.250892+0000", 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0088954280000000004 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.250921+0000", 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9408e-05 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.141 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.250935+0000", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3303e-05 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.250998+0000", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3275000000000007e-05 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.251029+0000", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0562000000000003e-05 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.286957+0000", 2024-09-21T10:26:04.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.035928581000000001 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:21:38.287011+0000", 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4228000000000003e-05 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.45632.0:6937 44.e e1157/1014 44:727bac97:::rbd_data.b240bb5bb67f.0000000000000000:head v 1157'179, mlcod=1156'177)", 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:22:00.496915+0000", 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 243.54840113099999, 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.054286033999999997, 2024-09-21T10:26:04.143 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.496915+0000", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.496915+0000", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.496914+0000", 2024-09-21T10:26:04.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.515679+0000", 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018764632 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.515692+0000", 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3199e-05 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.515699+0000", 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2490000000000004e-06 2024-09-21T10:26:04.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.515749+0000", 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9743000000000001e-05 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.515776+0000", 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6372000000000001e-05 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.551083+0000", 2024-09-21T10:26:04.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.035307837000000002 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:22:00.551202+0000", 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011812 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:105637 45.6 e1177/1164 45:64209852:::benchmark_data_smithi119_168706_object105636:head v 1177'6705, mlcod=1177'6703)", 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:41.974527+0000", 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 82.070789955999999, 2024-09-21T10:26:04.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.052262724000000003, 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974527+0000", 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974527+0000", 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974526+0000", 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974556+0000", 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9532000000000001e-05 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974562+0000", 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4790000000000001e-06 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974566+0000", 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9509999999999999e-06 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974615+0000", 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8196999999999999e-05 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.974631+0000", 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6594000000000001e-05 2024-09-21T10:26:04.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.026756+0000", 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.052125284000000001 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.026789+0000", 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3006999999999998e-05 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:105644 45.e e1177/1164 45:7f63398d:::benchmark_data_smithi119_168706_object105643:head v 1177'6529, mlcod=1177'6525)", 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:41.976068+0000", 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 82.069248342999998, 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047686724999999999, 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976068+0000", 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976068+0000", 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976068+0000", 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976091+0000", 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3492000000000001e-05 2024-09-21T10:26:04.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017004+0000", 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.040913143999999999 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017012+0000", 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.712e-06 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017054+0000", 2024-09-21T10:26:04.154 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1980000000000001e-05 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017068+0000", 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4249e-05 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.023718+0000", 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006650174 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.023755+0000", 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6745000000000001e-05 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:105633 45.8 e1177/1164 45:111b003b:::benchmark_data_smithi119_168706_object105632:head v 1177'6555, mlcod=1177'6551)", 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:41.976127+0000", 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 82.069189159000004, 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.056221665999999997, 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:04.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976127+0000", 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976127+0000", 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976127+0000", 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:41.976181+0000", 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4215000000000002e-05 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.016999+0000", 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.040818053999999999 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017011+0000", 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1474000000000001e-05 2024-09-21T10:26:04.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017044+0000", 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3628999999999999e-05 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.017066+0000", 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1542000000000001e-05 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.032301+0000", 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015235316 2024-09-21T10:26:04.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:42.032349+0000", 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8050000000000002e-05 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:04.160 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:26:04.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:04.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:04.310 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[105288]: pgmap v1674: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 31902/253617 objects degraded (12.579%) 2024-09-21T10:26:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[105288]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:26:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[105288]: osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400] boot 2024-09-21T10:26:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[105288]: osdmap e1191: 8 total, 8 up, 8 in 2024-09-21T10:26:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T10:26:04.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[101432]: pgmap v1674: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 31902/253617 objects degraded (12.579%) 2024-09-21T10:26:04.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[101432]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:26:04.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[101432]: osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400] boot 2024-09-21T10:26:04.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[101432]: osdmap e1191: 8 total, 8 up, 8 in 2024-09-21T10:26:04.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T10:26:04.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:04.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:04.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:04 smithi145 ceph-mon[89359]: pgmap v1674: 89 pgs: 21 active+undersized, 12 active+undersized+degraded, 56 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 31902/253617 objects degraded (12.579%) 2024-09-21T10:26:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:04 smithi145 ceph-mon[89359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:26:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:04 smithi145 ceph-mon[89359]: osd.6 [v2:172.21.15.145:6816/2712241400,v1:172.21.15.145:6817/2712241400] boot 2024-09-21T10:26:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:04 smithi145 ceph-mon[89359]: osdmap e1191: 8 total, 8 up, 8 in 2024-09-21T10:26:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-21T10:26:04.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:04.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:04.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:04.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:04.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:04.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:26:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:26:04] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-21T10:26:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:26:04.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:26:04.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:04.914 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:04.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:26:04.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:04.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:04.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:05.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:05.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:05.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:05.171 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:05.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:05.200 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:05 smithi145 ceph-mon[89359]: osdmap e1192: 8 total, 8 up, 8 in 2024-09-21T10:26:05.200 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:05 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 31902/253617 objects degraded (12.579%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:05.200 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:05 smithi145 ceph-mon[89359]: osdmap e1193: 8 total, 8 up, 8 in 2024-09-21T10:26:05.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:05.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:05.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:05.448 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:05.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:05.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:05 smithi119 ceph-mon[105288]: osdmap e1192: 8 total, 8 up, 8 in 2024-09-21T10:26:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:05 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 31902/253617 objects degraded (12.579%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:05 smithi119 ceph-mon[105288]: osdmap e1193: 8 total, 8 up, 8 in 2024-09-21T10:26:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:05 smithi119 ceph-mon[101432]: osdmap e1192: 8 total, 8 up, 8 in 2024-09-21T10:26:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:05 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 31902/253617 objects degraded (12.579%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:05 smithi119 ceph-mon[101432]: osdmap e1193: 8 total, 8 up, 8 in 2024-09-21T10:26:05.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:05.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:05.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:05.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:05.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:06.000 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:06.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:06.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:06.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:06.279 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:06.348 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:06 smithi119 ceph-mon[101432]: pgmap v1677: 89 pgs: 2 remapped+peering, 20 peering, 8 active+undersized, 3 active+undersized+degraded, 56 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 10510/246726 objects degraded (4.260%) 2024-09-21T10:26:06.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:06 smithi119 ceph-mon[105288]: pgmap v1677: 89 pgs: 2 remapped+peering, 20 peering, 8 active+undersized, 3 active+undersized+degraded, 56 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 10510/246726 objects degraded (4.260%) 2024-09-21T10:26:06.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:06.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:06.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:06 smithi145 ceph-mon[89359]: pgmap v1677: 89 pgs: 2 remapped+peering, 20 peering, 8 active+undersized, 3 active+undersized+degraded, 56 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 10510/246726 objects degraded (4.260%) 2024-09-21T10:26:06.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:06.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:06.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:06.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:06.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:06.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:06.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:06.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:06.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:07.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:07.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:07.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:07.224 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:26:07.251 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:07.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:07.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:07.374 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4246215975' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:07.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:07.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:07.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4246215975' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4246215975' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:07.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.673 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:07.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:07.774 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:07.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:07.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:07.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:08.001 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:26:08.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:08.089 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:08.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:08.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:08.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:08.395 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:08.452 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.477 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:08 smithi145 ceph-mon[89359]: pgmap v1679: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 20 peering, 5 active+undersized, 1 active+undersized+degraded, 59 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 807 op/s; 2925/244467 objects degraded (1.196%); 0 B/s, 0 objects/s recovering 2024-09-21T10:26:08.477 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:08 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:08.478 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:08 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4246215975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:08.478 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:08 smithi145 ceph-mon[89359]: osdmap e1194: 8 total, 8 up, 8 in 2024-09-21T10:26:08.488 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:08.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[101432]: pgmap v1679: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 20 peering, 5 active+undersized, 1 active+undersized+degraded, 59 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 807 op/s; 2925/244467 objects degraded (1.196%); 0 B/s, 0 objects/s recovering 2024-09-21T10:26:08.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:08.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4246215975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:08.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[101432]: osdmap e1194: 8 total, 8 up, 8 in 2024-09-21T10:26:08.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[105288]: pgmap v1679: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 20 peering, 5 active+undersized, 1 active+undersized+degraded, 59 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 807 op/s; 2925/244467 objects degraded (1.196%); 0 B/s, 0 objects/s recovering 2024-09-21T10:26:08.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:08.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4246215975' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:08.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:08 smithi119 ceph-mon[105288]: osdmap e1194: 8 total, 8 up, 8 in 2024-09-21T10:26:08.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:08.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:08.738 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:08.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:181816 45.4 e1194/1164 45:293785ea:::benchmark_data_smithi119_168706_object51646:head v 1194'11448, mlcod=1194'11446)", 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:08.785635+0000", 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.023739485000000001, 2024-09-21T10:26:08.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023770975, 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785635+0000", 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785635+0000", 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785634+0000", 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785650+0000", 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6129000000000001e-05 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785667+0000", 2024-09-21T10:26:08.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7461000000000001e-05 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785675+0000", 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1670000000000002e-06 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785832+0000", 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000157701 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785884+0000", 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2237999999999997e-05 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:181817 45.c e1194/1192 45:3194a84e:::benchmark_data_smithi119_168706_object51647:head v 1194'11386, mlcod=1184'9817)", 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:08.785687+0000", 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.023687231999999999, 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023780701000000001, 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:08.819 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785687+0000", 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785687+0000", 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:08.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785687+0000", 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785691+0000", 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9199999999999997e-06 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785696+0000", 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.186e-06 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.785699+0000", 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9739999999999998e-06 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.791893+0000", 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061936550000000002 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.791947+0000", 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4324999999999999e-05 2024-09-21T10:26:08.822 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:181827 45.2 45:49b65119:::benchmark_data_smithi119_168706_object51657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1194)", 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:08.795486+0000", 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.01388874, 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014042114, 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:08.823 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 181827 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795486+0000", 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795486+0000", 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795484+0000", 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795493+0000", 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9570000000000008e-06 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795514+0000", 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0936000000000001e-05 2024-09-21T10:26:08.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795521+0000", 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6490000000000008e-06 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.795663+0000", 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000142096 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.796017+0000", 2024-09-21T10:26:08.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00035333800000000002 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.796473+0000", 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00045647100000000001 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.804115+0000", 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076424029999999999 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805572+0000", 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001456052 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:181822 45.c e1194/1192 45:374f50c9:::benchmark_data_smithi119_168706_object51652:head v 1194'11387, mlcod=1184'9817)", 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:08.799185+0000", 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.010189292000000001, 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010420899000000001, 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:08.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799185+0000", 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799185+0000", 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799184+0000", 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:08.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799193+0000", 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0669999999999998e-06 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799213+0000", 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0188000000000001e-05 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799220+0000", 2024-09-21T10:26:08.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8759999999999999e-06 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799418+0000", 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000198283 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799441+0000", 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2651e-05 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.35976.0:181824 45.4 e1194/1164 45:23198145:::benchmark_data_smithi119_168706_object51654:head v 1194'11449, mlcod=1194'11446)", 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:08.799259+0000", 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.010115585, 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010401205, 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799259+0000", 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799259+0000", 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799258+0000", 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799277+0000", 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8521000000000001e-05 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799283+0000", 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4609999999999996e-06 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.799287+0000", 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.839e-06 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.802048+0000", 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027611010000000002 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.802091+0000", 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2941000000000001e-05 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:181831 45.e 45:7e1d4fb6:::benchmark_data_smithi119_168706_object51661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1194)", 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:08.805479+0000", 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0038954800000000002, 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042373239999999998, 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:08.835 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 181831 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805479+0000", 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:08.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805479+0000", 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805477+0000", 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805486+0000", 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4689999999999997e-06 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805500+0000", 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4630999999999999e-05 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805507+0000", 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3269999999999997e-06 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805606+0000", 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9102000000000001e-05 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805726+0000", 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120494 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:08.805872+0000", 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145802 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 6 2024-09-21T10:26:08.839 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:26:08.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:08.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:08.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:09.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:09.068 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:09.170 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:09.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:09.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:09.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:09.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:09.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:09.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:09.584 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:09.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:09.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:26:09.726 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:09.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:09.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:09.925 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:09.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:10.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.059 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:10.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:10.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:10.207 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:10.304 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:26:10.304 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:10.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:10.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:10.342 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:26:10.343 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:10.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:26:10.377 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:26:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:10.409 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:26:10.409 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:10.441 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:26:10.442 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:10.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:10.474 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:10.480 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:26:10.480 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:10.522 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:26:10.522 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:10.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:10 smithi119 ceph-mon[105288]: pgmap v1681: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 9 peering, 76 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 2921/243012 objects degraded (1.202%); 0 B/s, 0 objects/s recovering 2024-09-21T10:26:10.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:10 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2921/243012 objects degraded (1.202%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:10.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:10 smithi119 ceph-mon[101432]: pgmap v1681: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 9 peering, 76 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 2921/243012 objects degraded (1.202%); 0 B/s, 0 objects/s recovering 2024-09-21T10:26:10.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:10 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2921/243012 objects degraded (1.202%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:26:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:26:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:10 smithi145 ceph-mon[89359]: pgmap v1681: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 remapped+peering, 9 peering, 76 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 650 op/s; 2921/243012 objects degraded (1.202%); 0 B/s, 0 objects/s recovering 2024-09-21T10:26:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:10 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2921/243012 objects degraded (1.202%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:10.550 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:10.567 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:26:10.567 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:10.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:10.634 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:10.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.772 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:10.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:10.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:10.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:10.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:11.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:11.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:11.156 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.196 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.298 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:11.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:11.391 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:11.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:11.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:11.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:11.527 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3275709791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:11.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3275709791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3275709791' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:11.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:11.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:11.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:11.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:11.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:11.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:11.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:11.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:12.020 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:12.121 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:12.156 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:12.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:12.297 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:12.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:12.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:12 smithi145 ceph-mon[89359]: pgmap v1682: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 7298/234528 objects degraded (3.112%); 0 B/s, 42 objects/s recovering 2024-09-21T10:26:12.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:12.584 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:12 smithi119 ceph-mon[105288]: pgmap v1682: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 7298/234528 objects degraded (3.112%); 0 B/s, 42 objects/s recovering 2024-09-21T10:26:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:12 smithi119 ceph-mon[101432]: pgmap v1682: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 7298/234528 objects degraded (3.112%); 0 B/s, 42 objects/s recovering 2024-09-21T10:26:12.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:12.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:12.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:12.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:12.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:12.869 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:12.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:12.969 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:13.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:13.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:13.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:13.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:13.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:13.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:13.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:13.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:13.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:13.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:13.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:13.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:13.742 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:13.820 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:13.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:13.883 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:13.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:13.985 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:14.027 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:14.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:14.172 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:14.273 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:26:14.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:14.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:14.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:14.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:14.511 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:14 smithi145 ceph-mon[89359]: pgmap v1683: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 7298/233148 objects degraded (3.130%); 0 B/s, 33 objects/s recovering 2024-09-21T10:26:14.511 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/277435388' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:14.511 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:14 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:14.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-mon[101432]: pgmap v1683: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 7298/233148 objects degraded (3.130%); 0 B/s, 33 objects/s recovering 2024-09-21T10:26:14.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/277435388' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:14.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:14.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-mon[105288]: pgmap v1683: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 565 op/s; 7298/233148 objects degraded (3.130%); 0 B/s, 33 objects/s recovering 2024-09-21T10:26:14.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/277435388' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:14.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:26:14.544 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:14.544 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:14.558 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.603 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:14.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:14.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:14.747 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:14.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:26:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:26:14] "GET /metrics HTTP/1.1" 200 36698 "" "Prometheus/2.51.0" 2024-09-21T10:26:14.883 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:14.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:14.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:15.124 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:15.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:15.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:15.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:15.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[101432]: osdmap e1195: 8 total, 8 up, 8 in 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 7298/233148 objects degraded (3.130%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:15.344 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[105288]: osdmap e1195: 8 total, 8 up, 8 in 2024-09-21T10:26:15.345 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:15 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 7298/233148 objects degraded (3.130%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:15.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:15.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:15.496 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:15 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:15.496 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:15 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:26:15.496 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:15 smithi145 ceph-mon[89359]: osdmap e1195: 8 total, 8 up, 8 in 2024-09-21T10:26:15.496 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:15 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 7298/233148 objects degraded (3.130%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:15.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:15.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:15.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:15.688 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:15.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:15.813 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:15.813 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:15.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:15.920 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:15.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:15.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:16.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:16.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:16.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:16.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:16.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:16.472 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.480 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:26:16.480 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:16.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.510 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:26:16.510 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:16.553 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:26:16.553 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:16.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:16 smithi119 ceph-mon[101432]: pgmap v1685: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 7298/225027 objects degraded (3.243%); 0 B/s, 85 objects/s recovering 2024-09-21T10:26:16.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:16 smithi119 ceph-mon[105288]: pgmap v1685: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 7298/225027 objects degraded (3.243%); 0 B/s, 85 objects/s recovering 2024-09-21T10:26:16.583 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:26:16.583 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:16.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:16.614 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:26:16.615 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:16.622 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:16 smithi145 ceph-mon[89359]: pgmap v1685: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 809 op/s; 7298/225027 objects degraded (3.243%); 0 B/s, 85 objects/s recovering 2024-09-21T10:26:16.645 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:26:16.646 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:16.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:16.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:26:16.683 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:26:16.683 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:16.715 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:26:16.715 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:16.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:16.808 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:16.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:16.938 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:16.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:17.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:17.121 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:17.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:17.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:17.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:17.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:17.450 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3842502645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:17.451 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3842502645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:17.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:17.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:17.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3842502645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:17.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:17.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:17.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:17.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:17.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:17.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:17.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:26:17.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:17.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:18.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:18.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:18.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:18.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:18.198 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:18.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:18.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:18.354 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:18.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:18.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:18.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:18.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:18 smithi119 ceph-mon[105288]: pgmap v1686: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 7298/221976 objects degraded (3.288%); 0 B/s, 72 objects/s recovering 2024-09-21T10:26:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:18 smithi119 ceph-mon[101432]: pgmap v1686: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 7298/221976 objects degraded (3.288%); 0 B/s, 72 objects/s recovering 2024-09-21T10:26:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:18.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.654 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:18.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:18.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:18 smithi145 ceph-mon[89359]: pgmap v1686: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 798 op/s; 7298/221976 objects degraded (3.288%); 0 B/s, 72 objects/s recovering 2024-09-21T10:26:18.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:18.710 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:18.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:18.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:18.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:18.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:18.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:18.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:19.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:19.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:19.199 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:19.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:19.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:19.329 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:19.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:19.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:19.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:19 smithi145 ceph-mon[89359]: pgmap v1687: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 7298/219684 objects degraded (3.322%); 0 B/s, 68 objects/s recovering 2024-09-21T10:26:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 7298/221976 objects degraded (3.288%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:19.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:19.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.602 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:19.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:19 smithi119 ceph-mon[105288]: pgmap v1687: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 7298/219684 objects degraded (3.322%); 0 B/s, 68 objects/s recovering 2024-09-21T10:26:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 7298/221976 objects degraded (3.288%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:19 smithi119 ceph-mon[101432]: pgmap v1687: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 7298/219684 objects degraded (3.322%); 0 B/s, 68 objects/s recovering 2024-09-21T10:26:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 7298/221976 objects degraded (3.288%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:19.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:19.741 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:59077 45.e 45:7f18b26d:::benchmark_data_smithi119_168706_object59076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:04.039261+0000", 2024-09-21T10:26:19.810 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 135.246245292, 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.076112791999999999, 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 59077 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.039261+0000", 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.039261+0000", 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.039260+0000", 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.039325+0000", 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5029000000000004e-05 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.039341+0000", 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6229e-05 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.039347+0000", 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5039999999999999e-06 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.040430+0000", 2024-09-21T10:26:19.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001083011 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.040544+0000", 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113356 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.040676+0000", 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131922 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.042737+0000", 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002061228 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.045771+0000", 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0030338829999999998 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.115316+0000", 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.069544800000000004 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.115349+0000", 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3420000000000002e-05 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:04.115373+0000", 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4414999999999999e-05 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:64159 45.e 45:73c85058:::benchmark_data_smithi119_168706_object64158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:26:19.816 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:07.631676+0000", 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 131.65382963499999, 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.088257927999999999, 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 64159 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631676+0000", 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631676+0000", 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631675+0000", 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631764+0000", 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9470999999999998e-05 2024-09-21T10:26:19.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631776+0000", 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2247e-05 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631788+0000", 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1753000000000001e-05 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631835+0000", 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7221000000000001e-05 2024-09-21T10:26:19.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.631960+0000", 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012467399999999999 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.632089+0000", 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012936499999999999 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.634264+0000", 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002175171 2024-09-21T10:26:19.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.634727+0000", 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00046298500000000001 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.719882+0000", 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.085154538000000002 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.719912+0000", 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0141000000000002e-05 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:07.719934+0000", 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2141999999999999e-05 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:64804 45.e 45:7759a48f:::benchmark_data_smithi119_168706_object64803:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:08.136979+0000", 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 131.14852716799999, 2024-09-21T10:26:19.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.082408843999999995, 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 64804 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.136979+0000", 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.136979+0000", 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.136978+0000", 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137044+0000", 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6229999999999994e-05 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137053+0000", 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8720000000000008e-06 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137057+0000", 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3290000000000004e-06 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137102+0000", 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4705e-05 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137214+0000", 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011183000000000001 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137294+0000", 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0175999999999999e-05 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.141606+0000", 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043117759999999998 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.141647+0000", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1326999999999999e-05 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.219307+0000", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.077660037000000001 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.219360+0000", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2904e-05 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.219388+0000", 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7453e-05 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:64805 45.e 45:76dd5091:::benchmark_data_smithi119_168706_object64804:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:08.137068+0000", 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 131.14843814299999, 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.086259765000000002, 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.828 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 64805 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137068+0000", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137068+0000", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137067+0000", 2024-09-21T10:26:19.829 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137119+0000", 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1396000000000001e-05 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137131+0000", 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1823e-05 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.137134+0000", 2024-09-21T10:26:19.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9909999999999998e-06 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.139063+0000", 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00192953 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.139176+0000", 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113323 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.139298+0000", 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012115700000000001 2024-09-21T10:26:19.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.141688+0000", 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023907099999999999 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.141737+0000", 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8826000000000003e-05 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.223163+0000", 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.081426078999999998 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.223221+0000", 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7540000000000001e-05 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.223328+0000", 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010688000000000001 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:64810 45.e 45:700be740:::benchmark_data_smithi119_168706_object64809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1172)", 2024-09-21T10:26:19.833 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:08.143653+0000", 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 131.14185344099999, 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.079817140999999994, 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 64810 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143653+0000", 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143653+0000", 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143651+0000", 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143697+0000", 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6657999999999999e-05 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143709+0000", 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.117e-05 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143713+0000", 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7779999999999998e-06 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143745+0000", 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1885999999999999e-05 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143837+0000", 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1643000000000003e-05 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.143921+0000", 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3658000000000004e-05 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.145481+0000", 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015607150000000001 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.145650+0000", 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016829900000000001 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.223374+0000", 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.077724689999999999 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.223433+0000", 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8789000000000002e-05 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:08.223470+0000", 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6650999999999999e-05 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:69508 45.e 45:7143865d:::benchmark_data_smithi119_168706_object69507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:11.544021+0000", 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 127.74148484299999, 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.111396535, 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.839 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69508 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.544021+0000", 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.544021+0000", 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.544020+0000", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.544081+0000", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0918999999999997e-05 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.544092+0000", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0563e-05 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.544097+0000", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4049999999999999e-06 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.545451+0000", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013536920000000001 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.545575+0000", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012392999999999999 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.545703+0000", 2024-09-21T10:26:19.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000127954 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.551490+0000", 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057870209999999998 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.553458+0000", 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0019683729999999998 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.655358+0000", 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10189968000000001 2024-09-21T10:26:19.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.655394+0000", 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5978e-05 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.655418+0000", 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3992999999999999e-05 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:69511 45.e 45:7526af9a:::benchmark_data_smithi119_168706_object69510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1173)", 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:24:11.547273+0000", 2024-09-21T10:26:19.844 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 127.738233392, 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.108244257, 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69511 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.547273+0000", 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.547273+0000", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.547271+0000", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.547368+0000", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6996000000000004e-05 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.547384+0000", 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5611999999999999e-05 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.547390+0000", 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2199999999999997e-06 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.548350+0000", 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00096032099999999998 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.548474+0000", 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012397299999999999 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.548639+0000", 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016505099999999999 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.551494+0000", 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002854489 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.553640+0000", 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021457469999999999 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.655449+0000", 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.101809257 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.655484+0000", 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4783999999999998e-05 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:24:11.655517+0000", 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3285999999999999e-05 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155685 45.9 45:92e63bd3:::benchmark_data_smithi119_168706_object25515:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.025775+0000", 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.259731111000001, 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024087424, 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.850 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155685 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025775+0000", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025775+0000", 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025773+0000", 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025785+0000", 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2551000000000001e-05 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025799+0000", 2024-09-21T10:26:19.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3098000000000001e-05 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025806+0000", 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3769999999999998e-06 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.026290+0000", 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048368199999999998 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.026298+0000", 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9239999999999996e-06 2024-09-21T10:26:19.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.047370+0000", 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0210716339999999 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.049862+0000", 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024921510000000002 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155697 45.9 45:9a5b4933:::benchmark_data_smithi119_168706_object25527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.035385+0000", 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.250120772000002, 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0486232090000001, 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155697 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035385+0000", 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035385+0000", 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035384+0000", 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035390+0000", 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2299999999999996e-06 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035401+0000", 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0463e-05 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035407+0000", 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3319999999999997e-06 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035443+0000", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6606999999999997e-05 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035454+0000", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0325e-05 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.049909+0000", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.014455369 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051425+0000", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001516143 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051989+0000", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056419700000000001 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066316+0000", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014326358000000001 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.083912+0000", 2024-09-21T10:26:19.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017595821000000001 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.083950+0000", 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8272999999999998e-05 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084008+0000", 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8479e-05 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155715 45.2 45:49e42279:::benchmark_data_smithi119_168706_object25545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.064069+0000", 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.221437189, 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.992269917, 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155715 2024-09-21T10:26:19.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.862 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064069+0000", 2024-09-21T10:26:19.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064069+0000", 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064068+0000", 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:19.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064071+0000", 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.179e-06 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064077+0000", 2024-09-21T10:26:19.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4770000000000002e-06 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064080+0000", 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.168e-06 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064113+0000", 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3312000000000002e-05 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064121+0000", 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2409999999999998e-06 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.046336+0000", 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98221509500000004 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.046474+0000", 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000138362 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.046606+0000", 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013158899999999999 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056161+0000", 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095556649999999996 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056215+0000", 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4036000000000002e-05 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056261+0000", 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5762e-05 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056339+0000", 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7568e-05 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.868 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155718 45.2 45:4826636e:::benchmark_data_smithi119_168706_object25548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.073004+0000", 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.212501566, 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99846614700000003, 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155718 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.869 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073004+0000", 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073004+0000", 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073003+0000", 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073009+0000", 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4750000000000001e-06 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073017+0000", 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6469999999999998e-06 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073022+0000", 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1070000000000004e-06 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073057+0000", 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5373999999999997e-05 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073063+0000", 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.071e-06 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.050905+0000", 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97784177800000005 2024-09-21T10:26:19.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051002+0000", 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6992999999999998e-05 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051108+0000", 2024-09-21T10:26:19.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105933 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056356+0000", 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052479650000000003 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.071365+0000", 2024-09-21T10:26:19.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015009688 2024-09-21T10:26:19.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.071401+0000", 2024-09-21T10:26:19.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5757999999999999e-05 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.071471+0000", 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9308999999999997e-05 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155719 45.9 45:98f146b7:::benchmark_data_smithi119_168706_object25549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.212456693999997, 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0110945950000001, 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155719 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.879 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:19.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073052+0000", 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.723e-06 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073057+0000", 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6509999999999998e-06 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073060+0000", 2024-09-21T10:26:19.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8669999999999998e-06 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073093+0000", 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2972000000000002e-05 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073100+0000", 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5869999999999998e-06 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.058920+0000", 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98582079600000005 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.059054+0000", 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013349999999999999 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.059198+0000", 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000144177 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066501+0000", 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073030980000000001 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084044+0000", 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017542881999999999 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084079+0000", 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4776e-05 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084144+0000", 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5009e-05 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155721 45.9 45:92074e92:::benchmark_data_smithi119_168706_object25551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.076715+0000", 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 43.208790981999996, 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0095854630000001, 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.886 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155721 2024-09-21T10:26:19.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.887 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.887 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.887 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076715+0000", 2024-09-21T10:26:19.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076715+0000", 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076714+0000", 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076722+0000", 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9370000000000004e-06 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076739+0000", 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7844999999999999e-05 2024-09-21T10:26:19.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076747+0000", 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5739999999999998e-06 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076851+0000", 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000104001 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076864+0000", 2024-09-21T10:26:19.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.292e-05 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.062819+0000", 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98595542599999997 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.062933+0000", 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113699 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:26:19.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.063037+0000", 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000104275 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066508+0000", 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003471081 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.086177+0000", 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019668622 2024-09-21T10:26:19.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.086223+0000", 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6283999999999999e-05 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.086300+0000", 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7169000000000006e-05 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178415 45.e 45:7ea6f60d:::benchmark_data_smithi119_168706_object48245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:02.578090+0000", 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.707415545, 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.121541864, 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178415 2024-09-21T10:26:19.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578090+0000", 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578090+0000", 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578089+0000", 2024-09-21T10:26:19.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578098+0000", 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8540000000000003e-06 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578113+0000", 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.535e-05 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578119+0000", 2024-09-21T10:26:19.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3570000000000003e-06 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578164+0000", 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4987999999999997e-05 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578267+0000", 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000102342 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578367+0000", 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010059400000000001 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594049+0000", 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015681555 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594057+0000", 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9419999999999995e-06 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.699486+0000", 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.105429652 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.699591+0000", 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000104627 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.699632+0000", 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1239000000000003e-05 2024-09-21T10:26:19.899 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178416 45.e 45:77c354bf:::benchmark_data_smithi119_168706_object48246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:02.578276+0000", 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.707230203999998, 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14788453100000001, 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.900 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178416 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578276+0000", 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578276+0000", 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578275+0000", 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578280+0000", 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5549999999999996e-06 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578289+0000", 2024-09-21T10:26:19.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2529999999999993e-06 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.578294+0000", 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8029999999999996e-06 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.582500+0000", 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042059460000000003 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.582608+0000", 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108111 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.582715+0000", 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010785099999999999 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594110+0000", 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011394089 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594152+0000", 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2789e-05 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726074+0000", 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.131921228 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726120+0000", 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6171000000000002e-05 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726160+0000", 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0550000000000003e-05 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.907 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178422 45.e 45:729e0548:::benchmark_data_smithi119_168706_object48252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:02.585906+0000", 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.699599692, 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14037707699999999, 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.908 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178422 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.585906+0000", 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.585906+0000", 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.585905+0000", 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.585912+0000", 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1960000000000002e-06 2024-09-21T10:26:19.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.585926+0000", 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3653e-05 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.585933+0000", 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0550000000000002e-06 2024-09-21T10:26:19.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.589349+0000", 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034166420000000001 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.589458+0000", 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108696 2024-09-21T10:26:19.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.589567+0000", 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010959900000000001 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594148+0000", 2024-09-21T10:26:19.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045806199999999997 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.597260+0000", 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003111567 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726208+0000", 2024-09-21T10:26:19.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.12894783300000001 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726249+0000", 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1548000000000003e-05 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726283+0000", 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4292000000000003e-05 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178429 45.e 45:764d950d:::benchmark_data_smithi119_168706_object48259:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:02.594078+0000", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.691428391999999, 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.13232155000000001, 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178429 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594078+0000", 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594078+0000", 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594077+0000", 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594082+0000", 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0100000000000003e-06 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594090+0000", 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5920000000000001e-06 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594098+0000", 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6720000000000004e-06 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594169+0000", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1135000000000002e-05 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594249+0000", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0061999999999993e-05 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.594321+0000", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1583999999999998e-05 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.597362+0000", 2024-09-21T10:26:19.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0030416660000000002 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.597486+0000", 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012330800000000001 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726327+0000", 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.12884127500000001 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726368+0000", 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0577000000000001e-05 2024-09-21T10:26:19.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726399+0000", 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1508999999999999e-05 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178434 45.e 45:78c9b596:::benchmark_data_smithi119_168706_object48264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:02.636018+0000", 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.649487791999999, 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.090502410000000005, 2024-09-21T10:26:19.922 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178434 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636018+0000", 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636018+0000", 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636017+0000", 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636022+0000", 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8810000000000004e-06 2024-09-21T10:26:19.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636032+0000", 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3989999999999993e-06 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636037+0000", 2024-09-21T10:26:19.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2190000000000003e-06 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636083+0000", 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6656999999999997e-05 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636182+0000", 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8720000000000003e-05 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.636273+0000", 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0649000000000006e-05 2024-09-21T10:26:19.928 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.641868+0000", 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055955390000000001 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.641964+0000", 2024-09-21T10:26:19.929 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5223999999999995e-05 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726450+0000", 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.084486277999999998 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.930 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726488+0000", 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8324000000000001e-05 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:02.726521+0000", 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2378000000000001e-05 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.931 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178894 45.2 45:48807ac9:::benchmark_data_smithi119_168706_object48724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:03.217696+0000", 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.067810229999999, 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0568011020000001, 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.932 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178894 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217696+0000", 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.933 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217696+0000", 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217694+0000", 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:26:19.934 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217699+0000", 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1830000000000002e-06 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217710+0000", 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0674e-05 2024-09-21T10:26:19.935 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217717+0000", 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5379999999999999e-06 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217768+0000", 2024-09-21T10:26:19.936 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.117e-05 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217778+0000", 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9250000000000007e-06 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.937 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181105+0000", 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96332745500000005 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181117+0000", 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1683e-05 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.265884+0000", 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0847669339999999 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.938 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.265974+0000", 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0136999999999994e-05 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.266061+0000", 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6847000000000004e-05 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.272892+0000", 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006831798 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.939 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.272900+0000", 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2080000000000003e-06 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274344+0000", 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001444702 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274384+0000", 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9468999999999998e-05 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.940 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274497+0000", 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011297499999999999 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178898 45.2 45:44d8b130:::benchmark_data_smithi119_168706_object48728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:03.220961+0000", 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 16.064545389999999, 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0539241750000001, 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:26:19.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178898 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220961+0000", 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220961+0000", 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220960+0000", 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220963+0000", 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2899999999999998e-06 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220969+0000", 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9170000000000003e-06 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220972+0000", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7489999999999999e-06 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.221004+0000", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2206000000000003e-05 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.221010+0000", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0809999999999999e-06 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181141+0000", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96013049699999997 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181148+0000", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.216e-06 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.269308+0000", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.088159447 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.269432+0000", 2024-09-21T10:26:19.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012429699999999999 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.269515+0000", 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3351999999999997e-05 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.272921+0000", 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034056339999999998 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274540+0000", 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016195370000000001 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274695+0000", 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015458799999999999 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274744+0000", 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8717999999999997e-05 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274885+0000", 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141113 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:26:19.948 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:26:19.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:19.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:19.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:20.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:20.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:20.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:20.269 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:26:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:26:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:20.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:20.382 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:20.416 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:20.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:20.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T10:26:20.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:20.536 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T10:26:20.536 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:26:20.537 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:26:20.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T10:26:20.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:20.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:20.657 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:20.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:20.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:20.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:26:20.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:20.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:20.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:20.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:20.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:21.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:21.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:21.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:21.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:21.242 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:21.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:21.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:21.380 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:21.482 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:21.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:21.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:21.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:21.691 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:21 smithi119 ceph-mon[101432]: pgmap v1688: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 7298/209193 objects degraded (3.489%); 0 B/s, 93 objects/s recovering 2024-09-21T10:26:21.692 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4064783109' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.692 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:21 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.692 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:21 smithi119 ceph-mon[105288]: pgmap v1688: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 7298/209193 objects degraded (3.489%); 0 B/s, 93 objects/s recovering 2024-09-21T10:26:21.692 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4064783109' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.692 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:21 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.692 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:26:21.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:21 smithi145 ceph-mon[89359]: pgmap v1688: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 844 op/s; 7298/209193 objects degraded (3.489%); 0 B/s, 93 objects/s recovering 2024-09-21T10:26:21.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4064783109' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:21 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:21.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:21.907 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:21.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:21.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:21.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:22.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:22.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:22.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:22.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:22.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:22.427 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:22.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:22.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:22.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:22.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:22.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:22.751 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:22 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:22.751 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:22 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:22.751 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:22 smithi119 ceph-mon[105288]: osdmap e1196: 8 total, 8 up, 8 in 2024-09-21T10:26:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:22 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:22 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:22.751 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:22 smithi119 ceph-mon[101432]: osdmap e1196: 8 total, 8 up, 8 in 2024-09-21T10:26:22.769 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:22 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:26:22.769 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:22 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:22.769 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:22 smithi145 ceph-mon[89359]: osdmap e1196: 8 total, 8 up, 8 in 2024-09-21T10:26:22.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:22.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:22.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:22.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:22.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:22.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:23.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:23.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:23.115 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:26:23.115 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:26:23.115 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:26:23.115 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:26:23.173 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:23.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:23.245 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:23.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:23.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:23.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:23.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:23.507 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:23.537 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:26:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:23.571 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:26:23.572 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:23.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:23.613 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:26:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.650 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:23.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:23.656 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:26:23.656 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:23.682 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:26:23.682 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:23.718 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:26:23.718 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:23.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:23 smithi119 ceph-mon[101432]: pgmap v1690: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 7298/207858 objects degraded (3.511%); 0 B/s, 62 objects/s recovering 2024-09-21T10:26:23.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:23 smithi119 ceph-mon[101432]: osdmap e1197: 8 total, 8 up, 8 in 2024-09-21T10:26:23.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2499995267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:23.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:23 smithi119 ceph-mon[105288]: pgmap v1690: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 7298/207858 objects degraded (3.511%); 0 B/s, 62 objects/s recovering 2024-09-21T10:26:23.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:23 smithi119 ceph-mon[105288]: osdmap e1197: 8 total, 8 up, 8 in 2024-09-21T10:26:23.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2499995267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:23.749 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:26:23.750 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:23.751 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:23.781 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:26:23.781 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:23.783 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:26:23.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:23.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:23.916 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:23.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:24.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:23 smithi145 ceph-mon[89359]: pgmap v1690: 89 pgs: 1 active+recovering+undersized+remapped, 5 active+recovery_wait+undersized+degraded+remapped, 83 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 7298/207858 objects degraded (3.511%); 0 B/s, 62 objects/s recovering 2024-09-21T10:26:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:23 smithi145 ceph-mon[89359]: osdmap e1197: 8 total, 8 up, 8 in 2024-09-21T10:26:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2499995267' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:24.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:24.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:24.161 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:24.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:24.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:24.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:24.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:24.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:24.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:26:24.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:24.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:24.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:24.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:24.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:26:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:26:24] "GET /metrics HTTP/1.1" 200 36859 "" "Prometheus/2.51.0" 2024-09-21T10:26:24.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:24 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 7298/207858 objects degraded (3.511%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:24.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:24 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 7298/207858 objects degraded (3.511%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:24.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:24.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:24.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:24.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:25.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:24 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 7298/207858 objects degraded (3.511%), 5 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:25.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:25.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:25.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:25.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:25.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:25.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:25.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.547 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:25.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:25.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:25.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.690 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:25.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:25.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:25 smithi145 ceph-mon[89359]: pgmap v1692: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 7298/201585 objects degraded (3.620%); 0 B/s, 95 objects/s recovering 2024-09-21T10:26:25.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:25 smithi119 ceph-mon[105288]: pgmap v1692: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 7298/201585 objects degraded (3.620%); 0 B/s, 95 objects/s recovering 2024-09-21T10:26:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:25 smithi119 ceph-mon[101432]: pgmap v1692: 89 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 849 op/s; 7298/201585 objects degraded (3.620%); 0 B/s, 95 objects/s recovering 2024-09-21T10:26:25.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:25.956 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:25.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:25.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:26.011 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:26.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:26.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:26.191 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:26.191 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:26.208 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:26.240 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:26.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.380 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:26.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:26.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:26.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:26.672 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:26.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:26.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:26.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:26.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:26.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:26.956 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:26.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:27.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:26:27.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:27.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:27.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:27.245 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:27.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:27.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:27.429 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:26:27.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:27.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:27.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:27.677 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:27.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:27.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:27.910 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:27 smithi145 ceph-mon[89359]: pgmap v1693: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 5899/199008 objects degraded (2.964%); 0 B/s, 158 objects/s recovering 2024-09-21T10:26:27.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:27.950 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:27.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:27.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:28.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:28.086 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:28.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:28.087 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:27 smithi119 ceph-mon[101432]: pgmap v1693: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 5899/199008 objects degraded (2.964%); 0 B/s, 158 objects/s recovering 2024-09-21T10:26:28.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:27 smithi119 ceph-mon[105288]: pgmap v1693: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 860 op/s; 5899/199008 objects degraded (2.964%); 0 B/s, 158 objects/s recovering 2024-09-21T10:26:28.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:28.189 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:26:28.189 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:26:28.189 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:26:28.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:28.328 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:28.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:28.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:28.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:28.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:28.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:28.630 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:28.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:28.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:28.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:28.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:28.903 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:28.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:29.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:29.035 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:29.137 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:29.175 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:29.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:29.277 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:29.314 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:29.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:29.436 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:29.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:29.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:29.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:29.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:29.744 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:29.745 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:29.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.837 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:29.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:29 smithi119 ceph-mon[101432]: pgmap v1694: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 522 op/s; 5899/196650 objects degraded (3.000%); 0 B/s, 93 objects/s recovering 2024-09-21T10:26:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5899/196650 objects degraded (3.000%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:29 smithi119 ceph-mon[105288]: pgmap v1694: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 522 op/s; 5899/196650 objects degraded (3.000%); 0 B/s, 93 objects/s recovering 2024-09-21T10:26:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5899/196650 objects degraded (3.000%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:29.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:29.986 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:29.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:29 smithi145 ceph-mon[89359]: pgmap v1694: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 522 op/s; 5899/196650 objects degraded (3.000%); 0 B/s, 93 objects/s recovering 2024-09-21T10:26:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5899/196650 objects degraded (3.000%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:30.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:30.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:30.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:30.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:30.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:30.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:30.447 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:26:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:26:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:30.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:30.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:26:30.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:30.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:30.655 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:30.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:30.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.787 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:30.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:30.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:30.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:30.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:31.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:31.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:31.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:31.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:31.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:31.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:31.316 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:31.417 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:31.438 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:31.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:31.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:31.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:31.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:31.709 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:31.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:31.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:31.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:31.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:31.973 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:31.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:31 smithi145 ceph-mon[89359]: pgmap v1695: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 5899/186846 objects degraded (3.157%); 0 B/s, 127 objects/s recovering 2024-09-21T10:26:32.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:32.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:31 smithi119 ceph-mon[101432]: pgmap v1695: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 5899/186846 objects degraded (3.157%); 0 B/s, 127 objects/s recovering 2024-09-21T10:26:32.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:31 smithi119 ceph-mon[105288]: pgmap v1695: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 833 op/s; 5899/186846 objects degraded (3.157%); 0 B/s, 127 objects/s recovering 2024-09-21T10:26:32.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:32.332 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:32.369 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:32.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:32.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:32.645 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:32.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:32.779 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:32.881 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:32.903 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:32.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:32.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:33.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:33.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:33.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:33.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:33.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:33.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:33.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:33.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:33.537 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:33.563 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:33.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:33.695 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:33.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:33.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:33.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:33.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:33.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:33 smithi145 ceph-mon[89359]: pgmap v1696: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 5899/186846 objects degraded (3.157%); 0 B/s, 114 objects/s recovering 2024-09-21T10:26:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:34.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:34.091 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:33 smithi119 ceph-mon[105288]: pgmap v1696: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 5899/186846 objects degraded (3.157%); 0 B/s, 114 objects/s recovering 2024-09-21T10:26:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:33 smithi119 ceph-mon[101432]: pgmap v1696: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 700 op/s; 5899/186846 objects degraded (3.157%); 0 B/s, 114 objects/s recovering 2024-09-21T10:26:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:34.193 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:34.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:34.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:34.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:34.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:34.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:34.526 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:34.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:34.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:34.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:34.692 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:34.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:34.767 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:26:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:26:34] "GET /metrics HTTP/1.1" 200 36860 "" "Prometheus/2.51.0" 2024-09-21T10:26:34.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:34.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:34.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:34 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5899/186846 objects degraded (3.157%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:35.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:34 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5899/186846 objects degraded (3.157%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:34 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5899/186846 objects degraded (3.157%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.126 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:35.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:35.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:35.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:35.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:35.397 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:35.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:35.532 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:35.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:35.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:35.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:35.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:35.924 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:35 smithi119 ceph-mon[101432]: pgmap v1697: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 5899/179088 objects degraded (3.294%); 0 B/s, 95 objects/s recovering 2024-09-21T10:26:35.925 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:35 smithi119 ceph-mon[105288]: pgmap v1697: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 5899/179088 objects degraded (3.294%); 0 B/s, 95 objects/s recovering 2024-09-21T10:26:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:35.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:35 smithi145 ceph-mon[89359]: pgmap v1697: 89 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 5899/179088 objects degraded (3.294%); 0 B/s, 95 objects/s recovering 2024-09-21T10:26:36.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:36.089 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:36.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:36.337 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:36.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:36.460 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:36.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:36.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:36.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:36.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:36.791 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:26:36.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:36 smithi119 ceph-mon[105288]: osdmap e1198: 8 total, 8 up, 8 in 2024-09-21T10:26:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1307823450' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:36 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:36 smithi119 ceph-mon[101432]: osdmap e1198: 8 total, 8 up, 8 in 2024-09-21T10:26:36.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1307823450' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:36.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:36 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:36.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:36.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:36.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:36.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:36 smithi145 ceph-mon[89359]: osdmap e1198: 8 total, 8 up, 8 in 2024-09-21T10:26:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1307823450' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:36 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:26:37.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:37.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:37.219 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:37.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:37.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:37.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:37.389 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:37.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:37.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:37.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:37.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:37.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:37.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:37.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:37.791 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:37.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:37.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:37.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:37.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:38.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:37 smithi145 ceph-mon[89359]: pgmap v1699: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 5899/176433 objects degraded (3.343%); 0 B/s, 89 objects/s recovering 2024-09-21T10:26:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:37 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:37 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:37 smithi145 ceph-mon[89359]: osdmap e1199: 8 total, 8 up, 8 in 2024-09-21T10:26:38.068 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[101432]: pgmap v1699: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 5899/176433 objects degraded (3.343%); 0 B/s, 89 objects/s recovering 2024-09-21T10:26:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:38.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:38.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[101432]: osdmap e1199: 8 total, 8 up, 8 in 2024-09-21T10:26:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[105288]: pgmap v1699: 89 pgs: 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 84 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 5899/176433 objects degraded (3.343%); 0 B/s, 89 objects/s recovering 2024-09-21T10:26:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:26:38.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:37 smithi119 ceph-mon[105288]: osdmap e1199: 8 total, 8 up, 8 in 2024-09-21T10:26:38.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:38.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:38.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:38.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:38.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.479 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:38.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:38.544 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:26:38.545 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:38.580 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:38.585 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:26:38.585 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:38.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:38.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:38.624 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:26:38.624 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:38.660 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:26:38.661 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:38.690 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:26:38.690 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:26:38.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:38.738 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:26:38.738 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:38.757 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:26:38.757 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:38.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:38.793 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:26:38.793 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:38.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:38.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:38.921 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1828063394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:38.921 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1828063394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:39.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:26:39.037 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:26:39.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:39.176 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:39.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1828063394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:39.307 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:39.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.442 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:39.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:39.471 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T10:26:39.471 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:39.471 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5, 7, 6] 2024-09-21T10:26:39.471 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T10:26:39.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4 2024-09-21T10:26:39.543 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:39.582 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:39.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.732 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:39.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:39.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:39 smithi145 systemd[1]: Stopping Ceph osd.4 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:26:39.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:39 smithi119 ceph-mon[105288]: pgmap v1701: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 510 op/s; 5899/174594 objects degraded (3.379%); 0 B/s, 62 objects/s recovering 2024-09-21T10:26:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5899/174594 objects degraded (3.379%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:39 smithi119 ceph-mon[101432]: pgmap v1701: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 510 op/s; 5899/174594 objects degraded (3.379%); 0 B/s, 62 objects/s recovering 2024-09-21T10:26:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5899/174594 objects degraded (3.379%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:39.869 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:39.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:39.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:40.010 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:40.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:40.153 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:39 smithi145 ceph-mon[89359]: pgmap v1701: 89 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 510 op/s; 5899/174594 objects degraded (3.379%); 0 B/s, 62 objects/s recovering 2024-09-21T10:26:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5899/174594 objects degraded (3.379%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:40.214 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[38603]: 2024-09-21T10:26:39.932+0000 7f50f1f72640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:26:40.214 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[38603]: 2024-09-21T10:26:39.932+0000 7f50f1f72640 -1 osd.4 1199 *** Got signal Terminated *** 2024-09-21T10:26:40.214 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:39 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[38603]: 2024-09-21T10:26:39.932+0000 7f50f1f72640 -1 osd.4 1199 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:26:40.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:40.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:40.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:40.428 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:40.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:26:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:26:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:40.569 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:40.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:40.705 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206215 45.0 e1199/1164 45:0d654129:::benchmark_data_smithi119_168706_object76045:head v 1199'12814, mlcod=1199'12811)", 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.758893+0000", 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.034247286000000002, 2024-09-21T10:26:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.034276009000000003, 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.758893+0000", 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.758893+0000", 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.758894+0000", 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6099999999999999e-07 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.758900+0000", 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2700000000000001e-06 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.758915+0000", 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4786999999999999e-05 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.758921+0000", 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4189999999999999e-06 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763717+0000", 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047963720000000001 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763822+0000", 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104947 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206216 45.0 e1199/1164 45:0053274c:::benchmark_data_smithi119_168706_object76046:head v 1199'12815, mlcod=1199'12811)", 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.763706+0000", 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.029434208999999999, 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.029516813999999999, 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763706+0000", 2024-09-21T10:26:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763706+0000", 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763707+0000", 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3820000000000001e-06 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763715+0000", 2024-09-21T10:26:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3440000000000001e-06 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763743+0000", 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7722e-05 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.763751+0000", 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8869999999999992e-06 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.774966+0000", 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011214905000000001 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.775009+0000", 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2691999999999997e-05 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206223 45.0 e1199/1164 45:08fa11fd:::benchmark_data_smithi119_168706_object76053:head v 1199'12816, mlcod=1199'12811)", 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.771292+0000", 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0218479, 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021977113, 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.771292+0000", 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.771292+0000", 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.771293+0000", 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6100000000000001e-07 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.771297+0000", 2024-09-21T10:26:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1899999999999997e-06 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.771312+0000", 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5702999999999999e-05 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.771318+0000", 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4310000000000002e-06 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.781428+0000", 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010110252 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.781473+0000", 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5111999999999998e-05 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206228 45.0 e1199/1164 45:0cbc7f72:::benchmark_data_smithi119_168706_object76058:head v 1199'12817, mlcod=1199'12812)", 2024-09-21T10:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.777233+0000", 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.015906884, 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016080766, 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.777233+0000", 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.777233+0000", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.777234+0000", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1800000000000005e-07 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.777239+0000", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3319999999999999e-06 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.777253+0000", 2024-09-21T10:26:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3672000000000001e-05 2024-09-21T10:26:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.777260+0000", 2024-09-21T10:26:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9190000000000002e-06 2024-09-21T10:26:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:26:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785997+0000", 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087371110000000005 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.786016+0000", 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9673e-05 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206229 45.1 e1199/1164 45:826bb1bd:::benchmark_data_smithi119_168706_object76059:head v 1199'13187, mlcod=1199'13185)", 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.779110+0000", 2024-09-21T10:26:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.01403018, 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01425011, 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779110+0000", 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779110+0000", 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779111+0000", 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.071e-06 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779117+0000", 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6170000000000003e-06 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779133+0000", 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5506e-05 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779140+0000", 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0149999999999998e-06 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779305+0000", 2024-09-21T10:26:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016498700000000001 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.779352+0000", 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7057e-05 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206235 45.1 e1199/1164)", 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.785867+0000", 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0072724169999999998, 2024-09-21T10:26:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007533606, 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785867+0000", 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785867+0000", 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785869+0000", 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3680000000000001e-06 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785875+0000", 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.635e-06 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785915+0000", 2024-09-21T10:26:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9793999999999999e-05 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.785923+0000", 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5630000000000003e-06 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:206239 45.5 45:ae2be516:::benchmark_data_smithi119_168706_object76069:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1199)", 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.788164+0000", 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0049758409999999999, 2024-09-21T10:26:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052774739999999999, 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 206239 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788164+0000", 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788164+0000", 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788165+0000", 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4219999999999999e-06 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788169+0000", 2024-09-21T10:26:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.72e-06 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788184+0000", 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4746000000000001e-05 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788192+0000", 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0779999999999996e-06 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788260+0000", 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8010999999999996e-05 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788506+0000", 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024583500000000002 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.788833+0000", 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000327394 2024-09-21T10:26:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:206236 45.0 e1199/1164)", 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:40.789601+0000", 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.003538935, 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038956659999999999, 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.789601+0000", 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.789601+0000", 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.789602+0000", 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2219999999999999e-06 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.789609+0000", 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2080000000000003e-06 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.789627+0000", 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7623000000000001e-05 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:40.789635+0000", 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9300000000000003e-06 2024-09-21T10:26:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:26:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:26:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:26:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 8 2024-09-21T10:26:40.827 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:26:40.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:40.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:40.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:40.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:41.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:41.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:41.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:40 smithi119 ceph-mon[105288]: osd.4 marked itself down and dead 2024-09-21T10:26:41.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:40 smithi119 ceph-mon[101432]: osd.4 marked itself down and dead 2024-09-21T10:26:41.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:40 smithi145 ceph-mon[89359]: osd.4 marked itself down and dead 2024-09-21T10:26:41.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:41.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:41.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:41.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.584 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:41.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:41.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:26:41.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:26:41.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:41.716 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:41 smithi145 podman[121266]: 2024-09-21 10:26:41.421615486 +0000 UTC m=+1.621100272 container died 18086eb220ec9cc96d2bdf9289cf74433f682e2bcb885ef6dfc8b002496bb3c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-21T10:26:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:41.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:41.844 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:41.926 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:41.946 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:41.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:41.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:42.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:41 smithi145 ceph-mon[89359]: pgmap v1702: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 4472/166875 objects degraded (2.680%); 0 B/s, 112 objects/s recovering 2024-09-21T10:26:42.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:41 smithi145 ceph-mon[89359]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:26:42.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:41 smithi145 ceph-mon[89359]: osdmap e1200: 8 total, 7 up, 8 in 2024-09-21T10:26:42.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:41 smithi145 podman[121266]: 2024-09-21 10:26:41.813451517 +0000 UTC m=+2.012936308 container cleanup 18086eb220ec9cc96d2bdf9289cf74433f682e2bcb885ef6dfc8b002496bb3c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-21T10:26:42.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:41 smithi145 bash[121266]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4 2024-09-21T10:26:42.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:41 smithi119 ceph-mon[105288]: pgmap v1702: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 4472/166875 objects degraded (2.680%); 0 B/s, 112 objects/s recovering 2024-09-21T10:26:42.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:41 smithi119 ceph-mon[105288]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:26:42.064 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:41 smithi119 ceph-mon[105288]: osdmap e1200: 8 total, 7 up, 8 in 2024-09-21T10:26:42.064 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:41 smithi119 ceph-mon[101432]: pgmap v1702: 89 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 85 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 831 op/s; 4472/166875 objects degraded (2.680%); 0 B/s, 112 objects/s recovering 2024-09-21T10:26:42.064 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:41 smithi119 ceph-mon[101432]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T10:26:42.064 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:41 smithi119 ceph-mon[101432]: osdmap e1200: 8 total, 7 up, 8 in 2024-09-21T10:26:42.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:42.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:26:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:42.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:42.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:42.237 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:42.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:42.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:42.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:42.486 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:42 smithi145 podman[121371]: 2024-09-21 10:26:42.132380071 +0000 UTC m=+0.712839158 container remove 18086eb220ec9cc96d2bdf9289cf74433f682e2bcb885ef6dfc8b002496bb3c8 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-21T10:26:42.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:42.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:42.707 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:42.716 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:42.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.729 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:42.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:42.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:42 smithi145 podman[121511]: 2024-09-21 10:26:42.387872767 +0000 UTC m=+0.016508401 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:26:42.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:42 smithi145 podman[121511]: 2024-09-21 10:26:42.525129996 +0000 UTC m=+0.153765672 container create a272cc663ae6c720e6cb0da7ae271eed0a34026dfa5cb00aff6198dd1ea72ef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T10:26:42.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:42.954 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:42.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:42.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:43.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:43.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:42 smithi119 ceph-mon[105288]: osdmap e1201: 8 total, 7 up, 8 in 2024-09-21T10:26:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:42 smithi119 ceph-mon[101432]: osdmap e1201: 8 total, 7 up, 8 in 2024-09-21T10:26:43.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:43.245 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:42 smithi145 ceph-mon[89359]: osdmap e1201: 8 total, 7 up, 8 in 2024-09-21T10:26:43.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:42 smithi145 podman[121511]: 2024-09-21 10:26:42.828518329 +0000 UTC m=+0.457153962 container init a272cc663ae6c720e6cb0da7ae271eed0a34026dfa5cb00aff6198dd1ea72ef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True) 2024-09-21T10:26:43.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:42 smithi145 podman[121511]: 2024-09-21 10:26:42.834104345 +0000 UTC m=+0.462739960 container start a272cc663ae6c720e6cb0da7ae271eed0a34026dfa5cb00aff6198dd1ea72ef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T10:26:43.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:42 smithi145 podman[121511]: 2024-09-21 10:26:42.928578548 +0000 UTC m=+0.557214169 container attach a272cc663ae6c720e6cb0da7ae271eed0a34026dfa5cb00aff6198dd1ea72ef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:26:43.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:26:43 smithi145 podman[121511]: 2024-09-21 10:26:43.041135943 +0000 UTC m=+0.669771569 container died a272cc663ae6c720e6cb0da7ae271eed0a34026dfa5cb00aff6198dd1ea72ef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T10:26:43.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:43.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:43.470 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:43.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:43.530 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:43.632 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:43.673 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:43.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:43.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.808 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:43.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:43 smithi119 ceph-mon[105288]: pgmap v1705: 89 pgs: 1 peering, 5 active+undersized, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 2950/166230 objects degraded (1.775%); 0 B/s, 66 objects/s recovering 2024-09-21T10:26:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:43 smithi119 ceph-mon[105288]: osdmap e1202: 8 total, 7 up, 8 in 2024-09-21T10:26:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:43 smithi119 ceph-mon[101432]: pgmap v1705: 89 pgs: 1 peering, 5 active+undersized, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 2950/166230 objects degraded (1.775%); 0 B/s, 66 objects/s recovering 2024-09-21T10:26:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:43 smithi119 ceph-mon[101432]: osdmap e1202: 8 total, 7 up, 8 in 2024-09-21T10:26:43.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T10:26:43.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:43.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:26:43.922 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T10:26:43.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:43.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:44.055 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:44.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:44.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:44.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:44.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:43 smithi145 ceph-mon[89359]: pgmap v1705: 89 pgs: 1 peering, 5 active+undersized, 11 stale+active+clean, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 69 active+clean; 3.4 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 566 op/s; 2950/166230 objects degraded (1.775%); 0 B/s, 66 objects/s recovering 2024-09-21T10:26:44.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:43 smithi145 ceph-mon[89359]: osdmap e1202: 8 total, 7 up, 8 in 2024-09-21T10:26:44.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:44.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:44.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:44.477 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:44.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:44.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:44.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:44.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:44.721 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:26:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:26:44] "GET /metrics HTTP/1.1" 200 36860 "" "Prometheus/2.51.0" 2024-09-21T10:26:44.741 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:44.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:44.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:44.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:44.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:45.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2950/166230 objects degraded (1.775%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:45.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2950/166230 objects degraded (1.775%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:45.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:45.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:45.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:45.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:45.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2950/166230 objects degraded (1.775%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:45.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:45.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:45.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:45.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:45.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:45.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:45.821 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:45.852 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:45.954 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:45.983 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:45.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:45.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:46.058 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:45 smithi119 ceph-mon[101432]: pgmap v1707: 89 pgs: 1 peering, 15 active+undersized, 1 active+recovering+undersized+remapped, 2 stale+active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 61 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 6400/159747 objects degraded (4.006%); 0 B/s, 213 objects/s recovering 2024-09-21T10:26:46.059 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:45 smithi119 ceph-mon[105288]: pgmap v1707: 89 pgs: 1 peering, 15 active+undersized, 1 active+recovering+undersized+remapped, 2 stale+active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 61 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 6400/159747 objects degraded (4.006%); 0 B/s, 213 objects/s recovering 2024-09-21T10:26:46.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:46.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:46.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:46.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:45 smithi145 ceph-mon[89359]: pgmap v1707: 89 pgs: 1 peering, 15 active+undersized, 1 active+recovering+undersized+remapped, 2 stale+active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 61 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 824 op/s; 6400/159747 objects degraded (4.006%); 0 B/s, 213 objects/s recovering 2024-09-21T10:26:46.317 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:26:46.318 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:26:46.318 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:26:46.318 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:26:46.318 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:26:46.335 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:46.356 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:46.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:46.479 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:46.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:46.603 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:46.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:46.726 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:46.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:46.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:46.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:46.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:46.926 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:46 smithi119 ceph-mon[105288]: osdmap e1203: 8 total, 7 up, 8 in 2024-09-21T10:26:46.926 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:46 smithi119 ceph-mon[101432]: osdmap e1203: 8 total, 7 up, 8 in 2024-09-21T10:26:46.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:46.979 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:46.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:47.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:47.122 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:47.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:47.250 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:47.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:26:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:46 smithi145 ceph-mon[89359]: osdmap e1203: 8 total, 7 up, 8 in 2024-09-21T10:26:47.352 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:47.380 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:47.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:47.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:47.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:47.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:47.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:47.865 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:26:47.865 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:26:47.866 INFO:teuthology.orchestra.run.smithi119.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-21T10:26:47.883 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:26:47.883 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:26:47.916 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:26:47.916 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:26:47.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:47.942 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:47.949 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:26:47.949 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:26:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:47.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:48.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:48.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[105288]: pgmap v1709: 89 pgs: 1 peering, 21 active+undersized, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 472 op/s; 9653/158667 objects degraded (6.084%); 0 B/s, 152 objects/s recovering 2024-09-21T10:26:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[105288]: osdmap e1204: 8 total, 7 up, 8 in 2024-09-21T10:26:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3419709190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[101432]: pgmap v1709: 89 pgs: 1 peering, 21 active+undersized, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 472 op/s; 9653/158667 objects degraded (6.084%); 0 B/s, 152 objects/s recovering 2024-09-21T10:26:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[101432]: osdmap e1204: 8 total, 7 up, 8 in 2024-09-21T10:26:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3419709190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:48.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:48.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:47 smithi145 ceph-mon[89359]: pgmap v1709: 89 pgs: 1 peering, 21 active+undersized, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 472 op/s; 9653/158667 objects degraded (6.084%); 0 B/s, 152 objects/s recovering 2024-09-21T10:26:48.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:47 smithi145 ceph-mon[89359]: osdmap e1204: 8 total, 7 up, 8 in 2024-09-21T10:26:48.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3419709190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:26:48.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:48.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:26:48.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:48.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:48.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:48.402 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:48.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:48.540 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:48.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:48.687 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:48.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:48.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:26:48.703 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:26:48.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:48.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:48.878 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:26:48.923 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 7, 6] 2024-09-21T10:26:48.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:26:48.923 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T10:26:48.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd out 5 2024-09-21T10:26:48.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:48.933 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/318587411' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:48.933 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:48 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:48.934 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/318587411' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:48.934 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:48 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:48.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:48.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:49.069 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:49.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:49.210 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/318587411' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.210 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:48 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:26:49.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.256 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:49.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:49.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:49.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:49.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:49.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:49.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:49.669 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:49.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:49.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:49.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:49.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:49.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:49.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:50.065 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:50.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[105288]: pgmap v1711: 89 pgs: 1 active+recovering+undersized+degraded, 21 active+undersized, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 12647/155685 objects degraded (8.123%); 0 B/s, 297 objects/s recovering 2024-09-21T10:26:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[105288]: osdmap e1205: 8 total, 7 up, 8 in 2024-09-21T10:26:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 12647/155685 objects degraded (8.123%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:50.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[101432]: pgmap v1711: 89 pgs: 1 active+recovering+undersized+degraded, 21 active+undersized, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 12647/155685 objects degraded (8.123%); 0 B/s, 297 objects/s recovering 2024-09-21T10:26:50.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:50.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:50.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[101432]: osdmap e1205: 8 total, 7 up, 8 in 2024-09-21T10:26:50.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:49 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 12647/155685 objects degraded (8.123%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:50.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:50.216 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:49 smithi145 ceph-mon[89359]: pgmap v1711: 89 pgs: 1 active+recovering+undersized+degraded, 21 active+undersized, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 55 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 585 op/s; 12647/155685 objects degraded (8.123%); 0 B/s, 297 objects/s recovering 2024-09-21T10:26:50.216 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:49 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:26:50.216 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:49 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:26:50.216 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:49 smithi145 ceph-mon[89359]: osdmap e1205: 8 total, 7 up, 8 in 2024-09-21T10:26:50.217 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:49 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 12647/155685 objects degraded (8.123%), 12 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:50.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:50.371 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:50.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:50.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:50.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:50.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:26:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:26:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:26:50.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:50.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:50.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:26:50.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:50.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:50.892 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:50.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:50.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:50.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:51.024 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:51.043 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:51.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:51.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:51.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:51.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:51.184 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:51.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:26:51.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:51.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:51.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:51.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.525 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:51.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:51.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:51.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:51.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:51.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:51.791 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:51.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:51.819 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:26:51.870 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:26:51.870 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:26:51.892 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:51.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:51.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:52.017 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:52.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:52.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:52.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:52.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:51 smithi145 ceph-mon[89359]: pgmap v1713: 89 pgs: 1 active+recovering+undersized+degraded, 21 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 56 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 11053/148491 objects degraded (7.444%); 0 B/s, 223 objects/s recovering 2024-09-21T10:26:52.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:52.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:51 smithi119 ceph-mon[101432]: pgmap v1713: 89 pgs: 1 active+recovering+undersized+degraded, 21 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 56 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 11053/148491 objects degraded (7.444%); 0 B/s, 223 objects/s recovering 2024-09-21T10:26:52.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:51 smithi119 ceph-mon[105288]: pgmap v1713: 89 pgs: 1 active+recovering+undersized+degraded, 21 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 9 active+undersized+degraded, 56 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 625 op/s; 11053/148491 objects degraded (7.444%); 0 B/s, 223 objects/s recovering 2024-09-21T10:26:52.328 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:52.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:52.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.606 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:52.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:52.707 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:52.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:52.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:52.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:52.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:52.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:53.023 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:53.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:53.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:26:53.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:53.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:26:53.227 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:53.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:53.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:53.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:53.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:53.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:53.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:53.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:53.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:53.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.798 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:53.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:53.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:53.946 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:53.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:54.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:54.072 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:54.084 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:53 smithi119 ceph-mon[105288]: pgmap v1714: 89 pgs: 21 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 56 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 10798/146901 objects degraded (7.351%); 0 B/s, 290 objects/s recovering 2024-09-21T10:26:54.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:53 smithi119 ceph-mon[101432]: pgmap v1714: 89 pgs: 21 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 56 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 10798/146901 objects degraded (7.351%); 0 B/s, 290 objects/s recovering 2024-09-21T10:26:54.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:54.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:53 smithi145 ceph-mon[89359]: pgmap v1714: 89 pgs: 21 active+undersized, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 10 active+undersized+degraded, 56 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 579 op/s; 10798/146901 objects degraded (7.351%); 0 B/s, 290 objects/s recovering 2024-09-21T10:26:54.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:54.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:54.506 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:54.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:54.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:54.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:54.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.779 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:54.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:54.818 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:26:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:26:54] "GET /metrics HTTP/1.1" 200 36864 "" "Prometheus/2.51.0" 2024-09-21T10:26:54.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:54.933 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:54.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:55.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:54 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10798/146901 objects degraded (7.351%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.065 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:55.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:54 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10798/146901 objects degraded (7.351%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:54 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10798/146901 objects degraded (7.351%), 11 pgs degraded (PG_DEGRADED) 2024-09-21T10:26:55.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:55.187 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:55.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:55.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:55.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:55.454 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:55.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:55.591 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:55.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:26:55.168+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1205 definitely_dead 0 2024-09-21T10:26:55.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:55.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:55.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:55.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:55.943 INFO:teuthology.orchestra.run.smithi119.stderr:marked out osd.5. 2024-09-21T10:26:55.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:55.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:55.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:56.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:56.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:56.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:56.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:56.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:55 smithi145 ceph-mon[89359]: pgmap v1715: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+undersized, 10 active+undersized+degraded, 56 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10537/137670 objects degraded (7.654%); 0 B/s, 277 objects/s recovering 2024-09-21T10:26:56.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3855483916' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:26:56.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:26:56.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:55 smithi145 ceph-mon[89359]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:26:56.344 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:56.365 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[105288]: pgmap v1715: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+undersized, 10 active+undersized+degraded, 56 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10537/137670 objects degraded (7.654%); 0 B/s, 277 objects/s recovering 2024-09-21T10:26:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3855483916' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:26:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:26:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[105288]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:26:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[101432]: pgmap v1715: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+undersized, 10 active+undersized+degraded, 56 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 874 op/s; 10537/137670 objects degraded (7.654%); 0 B/s, 277 objects/s recovering 2024-09-21T10:26:56.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3855483916' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:26:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:26:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:55 smithi119 ceph-mon[101432]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:26:56.466 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:56.488 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:56.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:56.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:56.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:56.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:56.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:56.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:56.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:26:56.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[101432]: osdmap e1206: 8 total, 7 up, 7 in 2024-09-21T10:26:56.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:56.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:56.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:56.979 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:56.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:57.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:57.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:57.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:57.225 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:26:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:56 smithi145 ceph-mon[89359]: osdmap e1206: 8 total, 7 up, 7 in 2024-09-21T10:26:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:57.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:57.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:26:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[105288]: osdmap e1206: 8 total, 7 up, 7 in 2024-09-21T10:26:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:26:57.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:57.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:57.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:57.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:57.592 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:57.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:57.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:57.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:57.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:57.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:57.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:57.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:58.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:58.093 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:58.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:58.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:57 smithi145 ceph-mon[89359]: pgmap v1717: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+undersized, 10 active+undersized+degraded, 56 active+clean; 2.8 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10297/136251 objects degraded (7.557%); 0 B/s, 164 objects/s recovering 2024-09-21T10:26:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:57 smithi145 ceph-mon[89359]: osdmap e1207: 8 total, 7 up, 7 in 2024-09-21T10:26:58.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:57 smithi119 ceph-mon[101432]: pgmap v1717: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+undersized, 10 active+undersized+degraded, 56 active+clean; 2.8 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10297/136251 objects degraded (7.557%); 0 B/s, 164 objects/s recovering 2024-09-21T10:26:58.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:57 smithi119 ceph-mon[101432]: osdmap e1207: 8 total, 7 up, 7 in 2024-09-21T10:26:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:57 smithi119 ceph-mon[105288]: pgmap v1717: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 21 active+undersized, 10 active+undersized+degraded, 56 active+clean; 2.8 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 809 op/s; 10297/136251 objects degraded (7.557%); 0 B/s, 164 objects/s recovering 2024-09-21T10:26:58.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:57 smithi119 ceph-mon[105288]: osdmap e1207: 8 total, 7 up, 7 in 2024-09-21T10:26:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:58.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:58.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:58.498 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:58.600 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:58.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:58.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:58.752 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:58.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:26:58.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:58.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:26:58.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:26:59.007 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:26:59.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:59.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:26:59.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:59.287 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.1 starting backfill to osd.6 from (0'0,0'0] MAX to 1205'13861 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.e starting backfill to osd.0 from (0'0,0'0] MAX to 1205'13407 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1205'13688 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: osdmap e1208: 8 total, 7 up, 7 in 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1205'13658 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.8 starting backfill to osd.7 from (0'0,0'0] MAX to 1205'13452 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.b starting backfill to osd.3 from (0'0,0'0] MAX to 1205'13512 2024-09-21T10:26:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:58 smithi145 ceph-mon[89359]: 45.0 starting backfill to osd.2 from (0'0,0'0] MAX to 1205'13434 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.1 starting backfill to osd.6 from (0'0,0'0] MAX to 1205'13861 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.e starting backfill to osd.0 from (0'0,0'0] MAX to 1205'13407 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1205'13688 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: osdmap e1208: 8 total, 7 up, 7 in 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1205'13658 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.8 starting backfill to osd.7 from (0'0,0'0] MAX to 1205'13452 2024-09-21T10:26:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.b starting backfill to osd.3 from (0'0,0'0] MAX to 1205'13512 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[101432]: 45.0 starting backfill to osd.2 from (0'0,0'0] MAX to 1205'13434 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.1 starting backfill to osd.6 from (0'0,0'0] MAX to 1205'13861 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.e starting backfill to osd.0 from (0'0,0'0] MAX to 1205'13407 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.4 starting backfill to osd.1 from (0'0,0'0] MAX to 1205'13688 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: osdmap e1208: 8 total, 7 up, 7 in 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.6 starting backfill to osd.6 from (0'0,0'0] MAX to 1205'13658 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.8 starting backfill to osd.7 from (0'0,0'0] MAX to 1205'13452 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.b starting backfill to osd.3 from (0'0,0'0] MAX to 1205'13512 2024-09-21T10:26:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:58 smithi119 ceph-mon[105288]: 45.0 starting backfill to osd.2 from (0'0,0'0] MAX to 1205'13434 2024-09-21T10:26:59.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:26:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.428 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:59.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:26:59.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:26:59.554 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:26:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:26:59.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:26:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.692 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:59.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:26:59.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:26:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:59.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:26:59.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:26:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:26:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:26:59.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:00.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:00.084 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:00.186 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:00.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:00.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:00.273 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:59 smithi145 ceph-mon[89359]: pgmap v1720: 89 pgs: 3 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 20 active+undersized, 8 active+undersized+degraded, 56 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 732 op/s; 7164/133704 objects degraded (5.358%); 94 B/s, 43 objects/s recovering 2024-09-21T10:27:00.273 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:59 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:27:00.273 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:26:59 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 7164/133704 objects degraded (5.358%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:00.274 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:27:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:27:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:00.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:59 smithi119 ceph-mon[101432]: pgmap v1720: 89 pgs: 3 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 20 active+undersized, 8 active+undersized+degraded, 56 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 732 op/s; 7164/133704 objects degraded (5.358%); 94 B/s, 43 objects/s recovering 2024-09-21T10:27:00.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:59 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:27:00.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:26:59 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 7164/133704 objects degraded (5.358%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:00.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:59 smithi119 ceph-mon[105288]: pgmap v1720: 89 pgs: 3 remapped+peering, 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 20 active+undersized, 8 active+undersized+degraded, 56 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 732 op/s; 7164/133704 objects degraded (5.358%); 94 B/s, 43 objects/s recovering 2024-09-21T10:27:00.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:59 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:27:00.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:26:59 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 7164/133704 objects degraded (5.358%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:00.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:00.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.373 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:00.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:00.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:00.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:00.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:00.647 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:00.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.782 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:00.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:00.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:00.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:00.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:01.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:01.063 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:01.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:01.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:01.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:01.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:01.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:01.465 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:01.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:01.597 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:01.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:01.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:01.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:01.883 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:01.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:01.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:02.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:02.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:02.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:02.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.145 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:02.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:02.246 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:02.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:02.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:01 smithi145 ceph-mon[89359]: pgmap v1721: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 3 remapped+peering, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 446 op/s; 5515/129633 objects degraded (4.254%); 15981/129633 objects misplaced (12.328%); 2.4 MiB/s, 159 objects/s recovering 2024-09-21T10:27:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:01 smithi119 ceph-mon[101432]: pgmap v1721: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 3 remapped+peering, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 446 op/s; 5515/129633 objects degraded (4.254%); 15981/129633 objects misplaced (12.328%); 2.4 MiB/s, 159 objects/s recovering 2024-09-21T10:27:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:01 smithi119 ceph-mon[105288]: pgmap v1721: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 5 active+clean+remapped, 3 remapped+peering, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.6 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 446 op/s; 5515/129633 objects degraded (4.254%); 15981/129633 objects misplaced (12.328%); 2.4 MiB/s, 159 objects/s recovering 2024-09-21T10:27:02.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:02.409 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:02.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:02.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:02.667 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:02.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:02.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:02.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:02.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:02.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:02.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:03.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:03.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:03.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:03.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:03.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:03.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:03.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:03.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:03.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:03.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:03.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:03.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:03.872 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:03.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:03.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:04.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:03 smithi145 ceph-mon[89359]: pgmap v1722: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 5314/128481 objects degraded (4.136%); 15982/128481 objects misplaced (12.439%); 2.1 MiB/s, 143 objects/s recovering 2024-09-21T10:27:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:03 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:27:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[105288]: pgmap v1722: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 5314/128481 objects degraded (4.136%); 15982/128481 objects misplaced (12.439%); 2.1 MiB/s, 143 objects/s recovering 2024-09-21T10:27:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:04.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:27:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[101432]: pgmap v1722: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.6 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 387 op/s; 5314/128481 objects degraded (4.136%); 15982/128481 objects misplaced (12.439%); 2.1 MiB/s, 143 objects/s recovering 2024-09-21T10:27:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:04.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:03 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-21T10:27:04.118 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:04.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:04.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:04.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:04.413 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:04.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.570 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:04.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:04.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:04.780 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:27:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:27:04] "GET /metrics HTTP/1.1" 200 36880 "" "Prometheus/2.51.0" 2024-09-21T10:27:04.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:04.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:04.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:04.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.002 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:05.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:04 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5314/128481 objects degraded (4.136%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:05.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:05.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:04 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5314/128481 objects degraded (4.136%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:05.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:04 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5314/128481 objects degraded (4.136%), 8 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:05.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:05.247 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:05.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:05.357 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:27:05.357 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:27:05.357 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:27:05.357 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:27:05.357 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-21T10:27:05.360 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:27:05.360 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:05.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:05.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:27:05.396 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:05.425 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:27:05.425 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:05.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:05.479 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:05.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:05.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:05.680 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:05.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:05.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:05.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:05.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:06.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:05 smithi145 ceph-mon[89359]: pgmap v1723: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.5 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 7946/120702 objects degraded (6.583%); 17140/120702 objects misplaced (14.200%); 6.6 MiB/s, 230 objects/s recovering 2024-09-21T10:27:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:05 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1579452732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:06.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:05 smithi119 ceph-mon[101432]: pgmap v1723: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.5 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 7946/120702 objects degraded (6.583%); 17140/120702 objects misplaced (14.200%); 6.6 MiB/s, 230 objects/s recovering 2024-09-21T10:27:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:05 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1579452732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:05 smithi119 ceph-mon[105288]: pgmap v1723: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.5 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 647 op/s; 7946/120702 objects degraded (6.583%); 17140/120702 objects misplaced (14.200%); 6.6 MiB/s, 230 objects/s recovering 2024-09-21T10:27:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:05 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1579452732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:06.137 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:06.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:06.272 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:06.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:06.398 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:06.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:06.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:06.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:06.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:06.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:06.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:06.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:06.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:06.946 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:06.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:06.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:07.048 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:07.059 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T10:27:07.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T10:27:07.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:07.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:07.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:07.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:07.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:07.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:07.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:07.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:07.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:07.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:07.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:07.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:07.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:07.956 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:07 smithi119 ceph-mon[105288]: pgmap v1724: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 578 op/s; 7432/119442 objects degraded (6.222%); 16720/119442 objects misplaced (13.998%); 7.6 MiB/s, 224 objects/s recovering 2024-09-21T10:27:07.957 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:07 smithi119 ceph-mon[101432]: pgmap v1724: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 578 op/s; 7432/119442 objects degraded (6.222%); 16720/119442 objects misplaced (13.998%); 7.6 MiB/s, 224 objects/s recovering 2024-09-21T10:27:07.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:07.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:07 smithi145 ceph-mon[89359]: pgmap v1724: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 578 op/s; 7432/119442 objects degraded (6.222%); 16720/119442 objects misplaced (13.998%); 7.6 MiB/s, 224 objects/s recovering 2024-09-21T10:27:08.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:08.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:08.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:08.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:08.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:08.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:08.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:08.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:08.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:08.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:08.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:08.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:08.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:08.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:09.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:27:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:09.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:09.272 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.337 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:09.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:09.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.490 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:09.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.536 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [4] live_osds: [0, 1, 2, 3, 5, 7, 6] 2024-09-21T10:27:09.536 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:09.536 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-21T10:27:09.537 INFO:tasks.cephadm.osd.4:Restarting osd.4 (starting--it wasn't running)... 2024-09-21T10:27:09.537 DEBUG:teuthology.orchestra.run.smithi145:osd.4> sudo journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T10:27:09.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl start ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4 2024-09-21T10:27:09.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:09.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:09.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:09.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:09.911 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:09 smithi145 ceph-mon[89359]: pgmap v1725: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 6979/116115 objects degraded (6.010%); 16499/116115 objects misplaced (14.209%); 6.9 MiB/s, 203 objects/s recovering 2024-09-21T10:27:09.911 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:09 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 6979/116115 objects degraded (6.010%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:09.911 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:09 smithi145 systemd[1]: Starting Ceph osd.4 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:27:09.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:09.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:10.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:10.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:10.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:09 smithi119 ceph-mon[105288]: pgmap v1725: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 6979/116115 objects degraded (6.010%); 16499/116115 objects misplaced (14.209%); 6.9 MiB/s, 203 objects/s recovering 2024-09-21T10:27:10.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:09 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 6979/116115 objects degraded (6.010%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:09 smithi119 ceph-mon[101432]: pgmap v1725: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.4 GiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 6979/116115 objects degraded (6.010%); 16499/116115 objects misplaced (14.209%); 6.9 MiB/s, 203 objects/s recovering 2024-09-21T10:27:10.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:09 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 6979/116115 objects degraded (6.010%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:10.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:10.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:10.215 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:09 smithi145 podman[123534]: 2024-09-21 10:27:09.813743446 +0000 UTC m=+0.027465437 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:27:10.215 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:09 smithi145 podman[123534]: 2024-09-21 10:27:09.955547364 +0000 UTC m=+0.169269341 container create 26f493f064328fbb67b8f4eb1d5b682c1aacc835c9cea22114cd870d8e7e4d6d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T10:27:10.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:10.334 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:10.435 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:10.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:27:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:10.544 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 podman[123534]: 2024-09-21 10:27:10.242348704 +0000 UTC m=+0.456070700 container init 26f493f064328fbb67b8f4eb1d5b682c1aacc835c9cea22114cd870d8e7e4d6d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:27:10.544 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 podman[123534]: 2024-09-21 10:27:10.248549078 +0000 UTC m=+0.462271059 container start 26f493f064328fbb67b8f4eb1d5b682c1aacc835c9cea22114cd870d8e7e4d6d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T10:27:10.544 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 podman[123534]: 2024-09-21 10:27:10.350344541 +0000 UTC m=+0.564066526 container attach 26f493f064328fbb67b8f4eb1d5b682c1aacc835c9cea22114cd870d8e7e4d6d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-21T10:27:10.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.601 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:10.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:10.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:10.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:10.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:10.871 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:10.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:10.972 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:10.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:11.007 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T10:27:11.007 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T10:27:11.007 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T10:27:11.008 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T10:27:11.009 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate[123563]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-21T10:27:11.009 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 bash[123534]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-21T10:27:11.009 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:10 smithi145 podman[123762]: 2024-09-21 10:27:10.67117701 +0000 UTC m=+0.029975635 container died 26f493f064328fbb67b8f4eb1d5b682c1aacc835c9cea22114cd870d8e7e4d6d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:27:11.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:11.147 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:11.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:11.288 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:11.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:11.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:11.518 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:11.534 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:27:11.534 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:225258 45.b 45:d56fa017:::benchmark_data_smithi119_168706_object95088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1208)", 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:10.330906+0000", 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.049641945999999999, 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.049665526000000002, 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:27:11.535 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 225258 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.330906+0000", 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.330906+0000", 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:11.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.330908+0000", 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.584e-06 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.330915+0000", 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4749999999999998e-06 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.330929+0000", 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4409999999999999e-05 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.330937+0000", 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5180000000000004e-06 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.331238+0000", 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000300298 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.331363+0000", 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012530700000000001 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.332508+0000", 2024-09-21T10:27:11.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001144515 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342309+0000", 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0098011330000000001 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.344145+0000", 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001836601 2024-09-21T10:27:11.540 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:225268 45.b 45:de0f4fae:::benchmark_data_smithi119_168706_object95098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1208)", 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:10.335347+0000", 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.045201369999999998, 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.045270115999999999, 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:27:11.541 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 225268 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.335347+0000", 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.335347+0000", 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.335349+0000", 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5990000000000001e-06 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.335360+0000", 2024-09-21T10:27:11.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1090000000000001e-05 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.335597+0000", 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00023739 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.335607+0000", 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0219e-05 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.337212+0000", 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001604762 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.337333+0000", 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012074300000000001 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:27:11.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.337458+0000", 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000125563 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342350+0000", 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048913120000000001 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.348533+0000", 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061829579999999997 2024-09-21T10:27:11.546 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:225272 45.b 45:d1ff5528:::benchmark_data_smithi119_168706_object95102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1208)", 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:10.342535+0000", 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.038013564, 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.038120168000000003, 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:27:11.547 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 225272 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342535+0000", 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:11.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342535+0000", 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342536+0000", 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5579999999999999e-06 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342541+0000", 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.933e-06 2024-09-21T10:27:11.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342553+0000", 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1314e-05 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.342560+0000", 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9720000000000003e-06 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.344171+0000", 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016112469999999999 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.344287+0000", 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116593 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.344415+0000", 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012796199999999999 2024-09-21T10:27:11.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.349833+0000", 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054171310000000004 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:10.350026+0000", 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019312800000000001 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.552 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:11.553 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:27:11.553 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 3 2024-09-21T10:27:11.553 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:27:11.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:11.616 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:11 smithi145 podman[123762]: 2024-09-21 10:27:11.322379705 +0000 UTC m=+0.681178310 container remove 26f493f064328fbb67b8f4eb1d5b682c1aacc835c9cea22114cd870d8e7e4d6d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-activate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:27:11.617 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:11 smithi145 podman[123836]: 2024-09-21 10:27:11.476725435 +0000 UTC m=+0.025393370 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:27:11.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.688 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:11.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:11.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:11.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:11.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:11 smithi145 ceph-mon[89359]: pgmap v1726: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 6456/108603 objects degraded (5.945%); 14600/108603 objects misplaced (13.443%); 11 MiB/s, 290 objects/s recovering 2024-09-21T10:27:11.937 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:11 smithi145 podman[123836]: 2024-09-21 10:27:11.61522577 +0000 UTC m=+0.163893695 container create 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a) 2024-09-21T10:27:11.977 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:11.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:11.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:12.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:27:12.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:11 smithi119 ceph-mon[105288]: pgmap v1726: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 6456/108603 objects degraded (5.945%); 14600/108603 objects misplaced (13.443%); 11 MiB/s, 290 objects/s recovering 2024-09-21T10:27:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:11 smithi119 ceph-mon[101432]: pgmap v1726: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.2 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 6456/108603 objects degraded (5.945%); 14600/108603 objects misplaced (13.443%); 11 MiB/s, 290 objects/s recovering 2024-09-21T10:27:12.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:12.225 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:12.268 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:12.294 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:11 smithi145 podman[123836]: 2024-09-21 10:27:11.93525875 +0000 UTC m=+0.483926680 container init 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:27:12.294 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:11 smithi145 podman[123836]: 2024-09-21 10:27:11.940944932 +0000 UTC m=+0.489612855 container start 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-21T10:27:12.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:11 smithi145 ceph-osd[123916]: -- 172.21.15.145:0/3252153011 <== mon.1 v2:172.21.15.145:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559a38d5ad00 con 0x559a38d2cc00 2024-09-21T10:27:12.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:12 smithi145 bash[123836]: 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 2024-09-21T10:27:12.295 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:12 smithi145 systemd[1]: Started Ceph osd.4 for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:27:12.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:12.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:27:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:12.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:12.428 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:12.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:12.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:12.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:12.704 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:12.750 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:27:12.488+0000 7fbe5d565440 -1 Falling back to public interface 2024-09-21T10:27:12.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:12.846 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:12.897 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:12.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:12.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:12.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:13.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:13.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:13.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:13.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:13.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:13.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:13.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:13.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:13.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:13.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:13.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:13.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:13.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:13 smithi119 ceph-mon[105288]: pgmap v1727: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 624 op/s; 6209/107145 objects degraded (5.795%); 14600/107145 objects misplaced (13.626%); 10 MiB/s, 211 objects/s recovering 2024-09-21T10:27:13.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:13 smithi119 ceph-mon[101432]: pgmap v1727: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 624 op/s; 6209/107145 objects degraded (5.795%); 14600/107145 objects misplaced (13.626%); 10 MiB/s, 211 objects/s recovering 2024-09-21T10:27:13.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:13.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:13.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:14.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:14.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:14.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:14.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:14.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:13 smithi145 ceph-mon[89359]: pgmap v1727: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.2 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 624 op/s; 6209/107145 objects degraded (5.795%); 14600/107145 objects misplaced (13.626%); 10 MiB/s, 211 objects/s recovering 2024-09-21T10:27:14.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:14.412 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:14.438 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:14.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.585 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:14.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:14.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:14.710 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:14.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:14.806 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:27:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:27:14] "GET /metrics HTTP/1.1" 200 36880 "" "Prometheus/2.51.0" 2024-09-21T10:27:14.806 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:14 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 6209/107145 objects degraded (5.795%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:14.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:14.861 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:14.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:14.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:14.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:14 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 6209/107145 objects degraded (5.795%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:15.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:14 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 6209/107145 objects degraded (5.795%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:15.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.136 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:15.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:15.274 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:15.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:15.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:15.517 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:15.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:15.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:15.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:15.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:15.825 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:27:15.835 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:15 smithi145 ceph-mon[89359]: pgmap v1728: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.0 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 5977/99042 objects degraded (6.035%); 13004/99042 objects misplaced (13.130%); 13 MiB/s, 286 objects/s recovering 2024-09-21T10:27:15.835 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/121078223' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:15.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:15.859 INFO:teuthology.orchestra.run.smithi145.stderr:no valid command found; 10 closest matches: 2024-09-21T10:27:15.859 INFO:teuthology.orchestra.run.smithi145.stderr:0 2024-09-21T10:27:15.859 INFO:teuthology.orchestra.run.smithi145.stderr:1 2024-09-21T10:27:15.859 INFO:teuthology.orchestra.run.smithi145.stderr:2 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:abort 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:assert 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:bluestore allocator dump block 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:bluestore allocator fragmentation block 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:bluestore allocator score block 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:config diff 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:config diff get 2024-09-21T10:27:15.860 INFO:teuthology.orchestra.run.smithi145.stderr:admin_socket: invalid command 2024-09-21T10:27:15.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:16.002 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:16.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:16.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:15 smithi119 ceph-mon[101432]: pgmap v1728: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.0 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 5977/99042 objects degraded (6.035%); 13004/99042 objects misplaced (13.130%); 13 MiB/s, 286 objects/s recovering 2024-09-21T10:27:16.108 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/121078223' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:16.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:15 smithi119 ceph-mon[105288]: pgmap v1728: 89 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.0 GiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 5977/99042 objects degraded (6.035%); 13004/99042 objects misplaced (13.130%); 13 MiB/s, 286 objects/s recovering 2024-09-21T10:27:16.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/121078223' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:27:16.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:16.237 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:16.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:16.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:16.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:16.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:16.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:16.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:16.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:27:16.768 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-21T10:27:16.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:16.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:16.911 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:16.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:16.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:17.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:17.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:27:16.819+0000 7fbe5d565440 -1 osd.4 1199 log_to_monitors true 2024-09-21T10:27:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:16 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/121078223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:16 smithi145 ceph-mon[89359]: osdmap e1209: 8 total, 7 up, 7 in 2024-09-21T10:27:17.076 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:16 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/121078223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:16 smithi119 ceph-mon[101432]: osdmap e1209: 8 total, 7 up, 7 in 2024-09-21T10:27:17.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:16 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:17.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/121078223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:27:17.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:16 smithi119 ceph-mon[105288]: osdmap e1209: 8 total, 7 up, 7 in 2024-09-21T10:27:17.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:17.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:17.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:17.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:17.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:17.497 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:17.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:17.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:17.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:17.780 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:17.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:17.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:17.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:18.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:18.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[105288]: pgmap v1730: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 5465/97563 objects degraded (5.602%); 12511/97563 objects misplaced (12.824%); 12 MiB/s, 257 objects/s recovering 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[105288]: from='osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[105288]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[101432]: pgmap v1730: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 5465/97563 objects degraded (5.602%); 12511/97563 objects misplaced (12.824%); 12 MiB/s, 257 objects/s recovering 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[101432]: from='osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[101432]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:18.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:18.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:18.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:18.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:18.267 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:18.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:18.294 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:27:17 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:27:17.854+0000 7fbe55b12640 -1 osd.4 1199 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T10:27:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:17 smithi145 ceph-mon[89359]: pgmap v1730: 89 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+degraded+remapped+backfill_wait, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 2.0 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 728 op/s; 5465/97563 objects degraded (5.602%); 12511/97563 objects misplaced (12.824%); 12 MiB/s, 257 objects/s recovering 2024-09-21T10:27:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:17 smithi145 ceph-mon[89359]: from='osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T10:27:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:17 smithi145 ceph-mon[89359]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-21T10:27:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:18.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:18.517 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:18.538 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:18.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:18.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:18.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:18.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:18.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:18.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:18.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:18.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:18.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[101432]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[101432]: from='osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[101432]: osdmap e1210: 8 total, 7 up, 7 in 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[101432]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[105288]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[105288]: from='osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[105288]: osdmap e1210: 8 total, 7 up, 7 in 2024-09-21T10:27:18.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:18 smithi119 ceph-mon[105288]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:27:19.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:19.040 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:19.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:19.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:19.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:18 smithi145 ceph-mon[89359]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T10:27:19.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:18 smithi145 ceph-mon[89359]: from='osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:27:19.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:18 smithi145 ceph-mon[89359]: osdmap e1210: 8 total, 7 up, 7 in 2024-09-21T10:27:19.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:18 smithi145 ceph-mon[89359]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]} : dispatch 2024-09-21T10:27:19.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:19.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:19.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:19.413 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:19.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:19.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:19.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:19.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:19.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:19.802 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:19.904 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:19.932 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:19.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:20.031 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: pgmap v1732: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 1.9 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 4997/94191 objects degraded (5.305%); 12278/94191 objects misplaced (13.035%); 7.4 MiB/s, 157 objects/s recovering 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597] boot 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: osdmap e1211: 8 total, 8 up, 7 in 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 4997/94191 objects degraded (5.305%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: osdmap e1212: 8 total, 8 up, 7 in 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[101432]: 45.b continuing backfill to osd.3 from (1181'8162,1210'14421] 45:ddaa8d25:::benchmark_data_smithi119_168706_object124159:head to 1210'14421 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: pgmap v1732: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 1.9 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 4997/94191 objects degraded (5.305%); 12278/94191 objects misplaced (13.035%); 7.4 MiB/s, 157 objects/s recovering 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597] boot 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: osdmap e1211: 8 total, 8 up, 7 in 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T10:27:20.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 4997/94191 objects degraded (5.305%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:20.033 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: osdmap e1212: 8 total, 8 up, 7 in 2024-09-21T10:27:20.033 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:19 smithi119 ceph-mon[105288]: 45.b continuing backfill to osd.3 from (1181'8162,1210'14421] 45:ddaa8d25:::benchmark_data_smithi119_168706_object124159:head to 1210'14421 2024-09-21T10:27:20.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: pgmap v1732: 89 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+clean+remapped, 4 active+remapped+backfill_wait, 18 active+undersized, 6 active+undersized+degraded, 49 active+clean; 1.9 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 600 op/s; 4997/94191 objects degraded (5.305%); 12278/94191 objects misplaced (13.035%); 7.4 MiB/s, 157 objects/s recovering 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: osd.4 [v2:172.21.15.145:6800/842981597,v1:172.21.15.145:6801/842981597] boot 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: osdmap e1211: 8 total, 8 up, 7 in 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 4997/94191 objects degraded (5.305%), 9 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: osdmap e1212: 8 total, 8 up, 7 in 2024-09-21T10:27:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:19 smithi145 ceph-mon[89359]: 45.b continuing backfill to osd.3 from (1181'8162,1210'14421] 45:ddaa8d25:::benchmark_data_smithi119_168706_object124159:head to 1210'14421 2024-09-21T10:27:20.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:20.070 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:27:20.070 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:20.099 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:27:20.099 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:20.125 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:27:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:20.139 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:27:20.140 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:27:20.140 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:27:20.140 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:27:20.140 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:27:20.149 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:27:20.149 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:20.160 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:20.172 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:27:20.172 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:20.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:20.196 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:27:20.196 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:20.219 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:27:20.219 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:20.243 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:27:20.243 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:20.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:20.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:20.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:20.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:27:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:27:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:20.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:20.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:20.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:20.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:20.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:27:20.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:20.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:20.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:20.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:21.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:21.086 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:20 smithi119 ceph-mon[105288]: 45.1 continuing backfill to osd.6 from (1177'6867,1210'14762] MIN to 1210'14762 2024-09-21T10:27:21.086 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3823894889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:21.086 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:20 smithi119 ceph-mon[105288]: osdmap e1213: 8 total, 8 up, 7 in 2024-09-21T10:27:21.086 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:20 smithi119 ceph-mon[101432]: 45.1 continuing backfill to osd.6 from (1177'6867,1210'14762] MIN to 1210'14762 2024-09-21T10:27:21.087 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3823894889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:21.087 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:20 smithi119 ceph-mon[101432]: osdmap e1213: 8 total, 8 up, 7 in 2024-09-21T10:27:21.091 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:21.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:21.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:21.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:20 smithi145 ceph-mon[89359]: 45.1 continuing backfill to osd.6 from (1177'6867,1210'14762] MIN to 1210'14762 2024-09-21T10:27:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3823894889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:20 smithi145 ceph-mon[89359]: osdmap e1213: 8 total, 8 up, 7 in 2024-09-21T10:27:21.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:21.346 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:21.448 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:21.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:27:21.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:21.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:21.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:21.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:21.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:27:21.782 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:21.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:21.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:21.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:21.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:22.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:22.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:22.049 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:22.132 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:27:22.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:22.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:22.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:22.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:22.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:22.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:22.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:22 smithi145 ceph-mon[89359]: pgmap v1736: 89 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 peering, 3 active+recovering+undersized+remapped, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 5 active+undersized+degraded, 55 active+clean; 1.8 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 3825/89040 objects degraded (4.296%); 8677/89040 objects misplaced (9.745%); 25 MiB/s, 478 objects/s recovering 2024-09-21T10:27:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:22 smithi145 ceph-mon[89359]: osdmap e1214: 8 total, 8 up, 7 in 2024-09-21T10:27:22.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:22.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:22 smithi119 ceph-mon[101432]: pgmap v1736: 89 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 peering, 3 active+recovering+undersized+remapped, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 5 active+undersized+degraded, 55 active+clean; 1.8 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 3825/89040 objects degraded (4.296%); 8677/89040 objects misplaced (9.745%); 25 MiB/s, 478 objects/s recovering 2024-09-21T10:27:22.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:22 smithi119 ceph-mon[101432]: osdmap e1214: 8 total, 8 up, 7 in 2024-09-21T10:27:22.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:22 smithi119 ceph-mon[105288]: pgmap v1736: 89 pgs: 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 peering, 3 active+recovering+undersized+remapped, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 14 active+undersized, 5 active+undersized+degraded, 55 active+clean; 1.8 GiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 710 op/s; 3825/89040 objects degraded (4.296%); 8677/89040 objects misplaced (9.745%); 25 MiB/s, 478 objects/s recovering 2024-09-21T10:27:22.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:22 smithi119 ceph-mon[105288]: osdmap e1214: 8 total, 8 up, 7 in 2024-09-21T10:27:22.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:22.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:22.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:22.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:22.849 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:22.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:22.976 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:22.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:23.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:23.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:23.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:23.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:23.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:23.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:23.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.542 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:23.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:23.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:23.694 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:23.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:23.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:23.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:23.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:24.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:24.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:24.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:24.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:24.251 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:24.259 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:24.259 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:24.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:24.404 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:24.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:24 smithi145 ceph-mon[89359]: pgmap v1738: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 peering, 1 active+recovering+undersized+remapped, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 12 active+undersized, 4 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 544 op/s; 3516/87651 objects degraded (4.011%); 8677/87651 objects misplaced (9.899%); 25 MiB/s, 637 objects/s recovering 2024-09-21T10:27:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:24 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3516/87651 objects degraded (4.011%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:24.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:24 smithi119 ceph-mon[101432]: pgmap v1738: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 peering, 1 active+recovering+undersized+remapped, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 12 active+undersized, 4 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 544 op/s; 3516/87651 objects degraded (4.011%); 8677/87651 objects misplaced (9.899%); 25 MiB/s, 637 objects/s recovering 2024-09-21T10:27:24.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:24 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3516/87651 objects degraded (4.011%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:24.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:24 smithi119 ceph-mon[105288]: pgmap v1738: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_wait, 2 peering, 1 active+recovering+undersized+remapped, 5 active+clean+remapped, 2 active+remapped+backfill_wait, 12 active+undersized, 4 active+undersized+degraded, 59 active+clean; 1.8 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 544 op/s; 3516/87651 objects degraded (4.011%); 8677/87651 objects misplaced (9.899%); 25 MiB/s, 637 objects/s recovering 2024-09-21T10:27:24.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:24 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3516/87651 objects degraded (4.011%), 6 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:24.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:24.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:27:24.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:24.669 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:24.772 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:24.804 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:27:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:27:24] "GET /metrics HTTP/1.1" 200 36875 "" "Prometheus/2.51.0" 2024-09-21T10:27:24.874 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:24.906 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:24.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:25.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:25.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.189 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:25.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:25.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:25.337 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:25.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:25.464 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:25.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:25.609 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:25.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:25.736 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:25.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:25.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:25.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:25.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:25.999 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[105288]: pgmap v1739: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 82 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 567 op/s; 3134/83118 objects degraded (3.771%); 6679/83118 objects misplaced (8.036%); 29 MiB/s, 627 objects/s recovering 2024-09-21T10:27:25.999 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[101432]: pgmap v1739: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 82 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 567 op/s; 3134/83118 objects degraded (3.771%); 6679/83118 objects misplaced (8.036%); 29 MiB/s, 627 objects/s recovering 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.000 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:26.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:26.001 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:25 smithi145 ceph-mon[89359]: pgmap v1739: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 82 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 567 op/s; 3134/83118 objects degraded (3.771%); 6679/83118 objects misplaced (8.036%); 29 MiB/s, 627 objects/s recovering 2024-09-21T10:27:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:27:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:27:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:26.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:26.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:26.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:26.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:26.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:26.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:26.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:26.601 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:26.601 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:26.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:26.725 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:26.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:26.865 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:26.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:26.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:26.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:26 smithi145 ceph-mon[89359]: pgmap v1740: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 82 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 3134/83118 objects degraded (3.771%); 6679/83118 objects misplaced (8.036%); 27 MiB/s, 587 objects/s recovering 2024-09-21T10:27:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:26 smithi145 ceph-mon[89359]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T10:27:27.100 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:26 smithi119 ceph-mon[105288]: pgmap v1740: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 82 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 3134/83118 objects degraded (3.771%); 6679/83118 objects misplaced (8.036%); 27 MiB/s, 587 objects/s recovering 2024-09-21T10:27:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:26 smithi119 ceph-mon[105288]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T10:27:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:26 smithi119 ceph-mon[101432]: pgmap v1740: 89 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 82 active+clean; 1.7 GiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 531 op/s; 3134/83118 objects degraded (3.771%); 6679/83118 objects misplaced (8.036%); 27 MiB/s, 587 objects/s recovering 2024-09-21T10:27:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:26 smithi119 ceph-mon[101432]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T10:27:27.133 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:27.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:27.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:27.276 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:27.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:27.407 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:27.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:27.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:27.545 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:27.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:27.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:27.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:27.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:27.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:27.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:27:27.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:28.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:28.048 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:28.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:28.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:28.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:28.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:28.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.424 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:28.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:28.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:28.564 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:28.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:28.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:28.699 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:28.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:28.828 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:28.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:28.964 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:28.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:28 smithi145 ceph-mon[89359]: pgmap v1741: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 82 active+clean; 1.6 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 494 op/s; 3134/79230 objects degraded (3.956%); 5633/79230 objects misplaced (7.110%); 30 MiB/s, 670 objects/s recovering 2024-09-21T10:27:29.065 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:29.089 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:29.102 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:28 smithi119 ceph-mon[105288]: pgmap v1741: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 82 active+clean; 1.6 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 494 op/s; 3134/79230 objects degraded (3.956%); 5633/79230 objects misplaced (7.110%); 30 MiB/s, 670 objects/s recovering 2024-09-21T10:27:29.102 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:28 smithi119 ceph-mon[101432]: pgmap v1741: 89 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 2 active+remapped+backfill_wait, 82 active+clean; 1.6 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 494 op/s; 3134/79230 objects degraded (3.956%); 5633/79230 objects misplaced (7.110%); 30 MiB/s, 670 objects/s recovering 2024-09-21T10:27:29.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:29.215 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:29.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:29.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:29.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:29.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:29.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:29.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:29.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:29.606 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:29.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:29.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:29.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:29.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:29.954 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:27:29.954 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:29.954 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T10:27:29.954 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:27:29.954 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:27:29.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-21T10:27:29.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:29.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:29.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:29 smithi145 ceph-mon[89359]: osdmap e1215: 8 total, 8 up, 7 in 2024-09-21T10:27:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3134/79230 objects degraded (3.956%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:29 smithi145 ceph-mon[89359]: osdmap e1216: 8 total, 8 up, 7 in 2024-09-21T10:27:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:29 smithi119 ceph-mon[105288]: osdmap e1215: 8 total, 8 up, 7 in 2024-09-21T10:27:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3134/79230 objects degraded (3.956%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:30.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:29 smithi119 ceph-mon[105288]: osdmap e1216: 8 total, 8 up, 7 in 2024-09-21T10:27:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:29 smithi119 ceph-mon[101432]: osdmap e1215: 8 total, 8 up, 7 in 2024-09-21T10:27:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3134/79230 objects degraded (3.956%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:27:30.081 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:29 smithi119 ceph-mon[101432]: osdmap e1216: 8 total, 8 up, 7 in 2024-09-21T10:27:30.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:30.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:30.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.238 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:30.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:30.339 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:30.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:30.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:30.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:30.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:27:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:27:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:30.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:30.635 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:30.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:30.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:30.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:30.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:30.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:30.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:31.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:31.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:31.129 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:31.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:31.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:31.253 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:31.326 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:27:31.355 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:31.376 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:31.478 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:31.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:31.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:31 smithi119 ceph-mon[101432]: pgmap v1744: 89 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 82 active+clean; 1.5 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 3134/72798 objects degraded (4.305%); 4585/72798 objects misplaced (6.298%); 27 MiB/s, 514 objects/s recovering 2024-09-21T10:27:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:31 smithi119 ceph-mon[101432]: osdmap e1217: 8 total, 8 up, 7 in 2024-09-21T10:27:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:31 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/68907343' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:31.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:31 smithi119 ceph-mon[105288]: pgmap v1744: 89 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 82 active+clean; 1.5 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 3134/72798 objects degraded (4.305%); 4585/72798 objects misplaced (6.298%); 27 MiB/s, 514 objects/s recovering 2024-09-21T10:27:31.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:31 smithi119 ceph-mon[105288]: osdmap e1217: 8 total, 8 up, 7 in 2024-09-21T10:27:31.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:31 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/68907343' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:31.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:31.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:31.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:31 smithi145 ceph-mon[89359]: pgmap v1744: 89 pgs: 1 peering, 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 82 active+clean; 1.5 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 721 op/s; 3134/72798 objects degraded (4.305%); 4585/72798 objects misplaced (6.298%); 27 MiB/s, 514 objects/s recovering 2024-09-21T10:27:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:31 smithi145 ceph-mon[89359]: osdmap e1217: 8 total, 8 up, 7 in 2024-09-21T10:27:31.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:31 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/68907343' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:27:31.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:31.943 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:31.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:32.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:32.073 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:32.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:32.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:32.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:32.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:32.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:32.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:32.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/68907343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[101432]: 45.b continuing backfill to osd.3 from (1182'8462,1216'14807] 45:dea9fb8f:::benchmark_data_smithi119_168706_object108070:head to 1216'14807 2024-09-21T10:27:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[101432]: osdmap e1218: 8 total, 8 up, 7 in 2024-09-21T10:27:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[101432]: pgmap v1747: 89 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 83 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 684 op/s; 3134/71013 objects degraded (4.413%); 4017/71013 objects misplaced (5.657%); 15 MiB/s, 441 objects/s recovering 2024-09-21T10:27:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:32.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/68907343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:32.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[105288]: 45.b continuing backfill to osd.3 from (1182'8462,1216'14807] 45:dea9fb8f:::benchmark_data_smithi119_168706_object108070:head to 1216'14807 2024-09-21T10:27:32.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[105288]: osdmap e1218: 8 total, 8 up, 7 in 2024-09-21T10:27:32.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:32 smithi119 ceph-mon[105288]: pgmap v1747: 89 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 83 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 684 op/s; 3134/71013 objects degraded (4.413%); 4017/71013 objects misplaced (5.657%); 15 MiB/s, 441 objects/s recovering 2024-09-21T10:27:32.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:32.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:32.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:32.793 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:32 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:27:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/68907343' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:27:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:32 smithi145 ceph-mon[89359]: 45.b continuing backfill to osd.3 from (1182'8462,1216'14807] 45:dea9fb8f:::benchmark_data_smithi119_168706_object108070:head to 1216'14807 2024-09-21T10:27:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:32 smithi145 ceph-mon[89359]: osdmap e1218: 8 total, 8 up, 7 in 2024-09-21T10:27:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:32 smithi145 ceph-mon[89359]: pgmap v1747: 89 pgs: 1 remapped+peering, 1 active+remapped+backfill_wait, 1 peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 83 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 684 op/s; 3134/71013 objects degraded (4.413%); 4017/71013 objects misplaced (5.657%); 15 MiB/s, 441 objects/s recovering 2024-09-21T10:27:32.845 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:32.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:32.870 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:27:32.871 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:32.901 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:27:32.901 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:32.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:32.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:32.937 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:27:32.937 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:32.975 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:27:32.975 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:33.004 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:27:33.004 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:33.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:33.035 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:27:33.035 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:33.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:33.064 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:27:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:33.093 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:27:33.093 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:33.142 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:33.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:33.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:33.307 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:33.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:33.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:33.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:33.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:33.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:33.719 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:33.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:33.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:33.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:33.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:33.942 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:33.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:33.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:33.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2375592531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:34.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:34.102 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2375592531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:27:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2375592531' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:34.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:34.229 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:34.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:34.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:34.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:34.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:34.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:34.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:27:34.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:34.600 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:34.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:34.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:34.799 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:27:34] "GET /metrics HTTP/1.1" 200 36875 "" "Prometheus/2.51.0" 2024-09-21T10:27:34.824 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:34.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:34.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:34.949 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:34.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:34.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:35.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:35.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.106 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:35.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-mon[101432]: pgmap v1748: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+remapped+backfill_wait, 1 peering, 1 active+remapped+backfilling, 83 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 569 op/s; 1563/67434 objects degraded (2.318%); 2441/67434 objects misplaced (3.620%); 13 MiB/s, 453 objects/s recovering 2024-09-21T10:27:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-mon[101432]: osdmap e1219: 8 total, 8 up, 7 in 2024-09-21T10:27:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1563/67434 objects degraded (2.318%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-mon[105288]: pgmap v1748: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+remapped+backfill_wait, 1 peering, 1 active+remapped+backfilling, 83 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 569 op/s; 1563/67434 objects degraded (2.318%); 2441/67434 objects misplaced (3.620%); 13 MiB/s, 453 objects/s recovering 2024-09-21T10:27:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-mon[105288]: osdmap e1219: 8 total, 8 up, 7 in 2024-09-21T10:27:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:34 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1563/67434 objects degraded (2.318%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:35.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:35.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:34 smithi145 ceph-mon[89359]: pgmap v1748: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+remapped+backfill_wait, 1 peering, 1 active+remapped+backfilling, 83 active+clean; 1.4 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 569 op/s; 1563/67434 objects degraded (2.318%); 2441/67434 objects misplaced (3.620%); 13 MiB/s, 453 objects/s recovering 2024-09-21T10:27:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:34 smithi145 ceph-mon[89359]: osdmap e1219: 8 total, 8 up, 7 in 2024-09-21T10:27:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:34 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1563/67434 objects degraded (2.318%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:35.337 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:35.381 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:35.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:35.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:35.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:35.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:35.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:35.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:35.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:35.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:35.933 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:35.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:36.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:36.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:36.086 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:35 smithi119 ceph-mon[105288]: osdmap e1220: 8 total, 8 up, 7 in 2024-09-21T10:27:36.087 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:35 smithi119 ceph-mon[101432]: osdmap e1220: 8 total, 8 up, 7 in 2024-09-21T10:27:36.173 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:36.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:36.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:35 smithi145 ceph-mon[89359]: osdmap e1220: 8 total, 8 up, 7 in 2024-09-21T10:27:36.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:36.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:36.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:36.456 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:36.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:36.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:36.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:36.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:36.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:36.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:36.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:36.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:36.970 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:36.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:37.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:37.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:36 smithi119 ceph-mon[101432]: pgmap v1751: 89 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 84 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 532 op/s; 1563/63762 objects degraded (2.451%); 1863/63762 objects misplaced (2.922%); 9.3 MiB/s, 375 objects/s recovering 2024-09-21T10:27:37.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:36 smithi119 ceph-mon[105288]: pgmap v1751: 89 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 84 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 532 op/s; 1563/63762 objects degraded (2.451%); 1863/63762 objects misplaced (2.922%); 9.3 MiB/s, 375 objects/s recovering 2024-09-21T10:27:37.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:37.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:37.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:37.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:36 smithi145 ceph-mon[89359]: pgmap v1751: 89 pgs: 2 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 remapped+peering, 84 active+clean; 1.3 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 532 op/s; 1563/63762 objects degraded (2.451%); 1863/63762 objects misplaced (2.922%); 9.3 MiB/s, 375 objects/s recovering 2024-09-21T10:27:37.348 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:37.375 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:37.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:37.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:37.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:37.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:37.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:37.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:37.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:37.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:37.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:38.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:38.080 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:38.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:38.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:38.328 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:38.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:38.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:38.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:38.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:38.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:38.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:38.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:38.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:38.925 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:38.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:39.027 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:39.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:39.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:38 smithi119 ceph-mon[101432]: pgmap v1752: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 85 active+clean; 1.2 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 580 op/s; 1563/60243 objects degraded (2.594%); 1863/60243 objects misplaced (3.092%); 4.5 MiB/s, 135 objects/s recovering 2024-09-21T10:27:39.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:38 smithi119 ceph-mon[105288]: pgmap v1752: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 85 active+clean; 1.2 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 580 op/s; 1563/60243 objects degraded (2.594%); 1863/60243 objects misplaced (3.092%); 4.5 MiB/s, 135 objects/s recovering 2024-09-21T10:27:39.157 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:39.181 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:39.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:38 smithi145 ceph-mon[89359]: pgmap v1752: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 1 active+recovering+undersized+remapped, 85 active+clean; 1.2 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 580 op/s; 1563/60243 objects degraded (2.594%); 1863/60243 objects misplaced (3.092%); 4.5 MiB/s, 135 objects/s recovering 2024-09-21T10:27:39.317 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:39.419 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:39.457 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:39.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:39.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:27:39.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:39.580 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:39.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:39.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:39.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:39.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:39.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:39.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:39.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:39.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:39.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:40.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:40.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:40.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1563/60243 objects degraded (2.594%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:40.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1563/60243 objects degraded (2.594%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:40.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1563/60243 objects degraded (2.594%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:40.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:40.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:40.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:40.460 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:40.491 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:27:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:27:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:40.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:40.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:40.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:40.750 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:40.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:40.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:40.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:40.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:40.980 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:41.003 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:41.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:41.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:41.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:40 smithi119 ceph-mon[105288]: pgmap v1753: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 1.1 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 1563/54168 objects degraded (2.885%); 1727/54168 objects misplaced (3.188%); 3.5 MiB/s, 170 objects/s recovering 2024-09-21T10:27:41.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:40 smithi119 ceph-mon[101432]: pgmap v1753: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 1.1 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 1563/54168 objects degraded (2.885%); 1727/54168 objects misplaced (3.188%); 3.5 MiB/s, 170 objects/s recovering 2024-09-21T10:27:41.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:41.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:41.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:40 smithi145 ceph-mon[89359]: pgmap v1753: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 1.1 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 701 op/s; 1563/54168 objects degraded (2.885%); 1727/54168 objects misplaced (3.188%); 3.5 MiB/s, 170 objects/s recovering 2024-09-21T10:27:41.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:41.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:41.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:41.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:41.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:41.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:41.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:41.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:41.794 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:41.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:41.895 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:41.923 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:41.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:42.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:42.052 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:42.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:42.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:42.191 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:42.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:42.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:42.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:42.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:42.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:42.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:42.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:42.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:42.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:42.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:42.993 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:42.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:43.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:42 smithi119 ceph-mon[101432]: pgmap v1754: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 1.1 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 1563/51921 objects degraded (3.010%); 1525/51921 objects misplaced (2.937%); 3.5 MiB/s, 121 objects/s recovering 2024-09-21T10:27:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:42 smithi119 ceph-mon[105288]: pgmap v1754: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 1.1 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 1563/51921 objects degraded (3.010%); 1525/51921 objects misplaced (2.937%); 3.5 MiB/s, 121 objects/s recovering 2024-09-21T10:27:43.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:43.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:43.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:43.275 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:42 smithi145 ceph-mon[89359]: pgmap v1754: 89 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 1.1 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 646 op/s; 1563/51921 objects degraded (3.010%); 1525/51921 objects misplaced (2.937%); 3.5 MiB/s, 121 objects/s recovering 2024-09-21T10:27:43.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:43.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:43.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:43.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:43.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:43.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:43.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:43.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:43.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:43.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:43.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:44.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:44.078 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:44.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:44.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:44.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:44.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:44.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:44.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:44.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:44.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:44.601 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:44.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:44.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.732 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:44.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:44.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:44.847 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:27:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:27:44] "GET /metrics HTTP/1.1" 200 36875 "" "Prometheus/2.51.0" 2024-09-21T10:27:44.864 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:44.965 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:44.993 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:44.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:45.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:44 smithi119 ceph-mon[105288]: pgmap v1755: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 976 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 1563/46887 objects degraded (3.334%); 1525/46887 objects misplaced (3.253%); 2.9 MiB/s, 144 objects/s recovering 2024-09-21T10:27:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1563/46887 objects degraded (3.334%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:44 smithi119 ceph-mon[101432]: pgmap v1755: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 976 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 1563/46887 objects degraded (3.334%); 1525/46887 objects misplaced (3.253%); 2.9 MiB/s, 144 objects/s recovering 2024-09-21T10:27:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1563/46887 objects degraded (3.334%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:45.125 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:45.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:45.251 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:45.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:44 smithi145 ceph-mon[89359]: pgmap v1755: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 976 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 669 op/s; 1563/46887 objects degraded (3.334%); 1525/46887 objects misplaced (3.253%); 2.9 MiB/s, 144 objects/s recovering 2024-09-21T10:27:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1563/46887 objects degraded (3.334%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:27:45.354 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:45.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:45.411 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:27:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:27:45.411 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:27:45.480 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:45.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:45.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:45.611 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:45.632 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:45.647 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:45.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:45.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:45.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:45.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:45.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:45.982 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:45 smithi119 ceph-mon[105288]: osdmap e1221: 8 total, 8 up, 7 in 2024-09-21T10:27:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:45 smithi119 ceph-mon[101432]: osdmap e1221: 8 total, 8 up, 7 in 2024-09-21T10:27:46.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:46.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:46.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:46.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:46.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:45 smithi145 ceph-mon[89359]: osdmap e1221: 8 total, 8 up, 7 in 2024-09-21T10:27:46.303 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:46.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:46.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:46.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:46.537 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:46.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:46.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:46.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:46.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:46.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:46.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:46.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:46.954 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:46.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:46.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:47.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:46 smithi119 ceph-mon[101432]: pgmap v1757: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 879 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 716 op/s; 1563/42258 objects degraded (3.699%); 1196/42258 objects misplaced (2.830%); 691 KiB/s, 106 objects/s recovering 2024-09-21T10:27:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:46 smithi119 ceph-mon[101432]: 45.1 continuing backfill to osd.6 from (1180'7967,1220'15890] 45:806b3ffb:::benchmark_data_smithi119_168706_object128743:head to 1220'15890 2024-09-21T10:27:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:46 smithi119 ceph-mon[101432]: osdmap e1222: 8 total, 8 up, 7 in 2024-09-21T10:27:47.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:46 smithi119 ceph-mon[105288]: pgmap v1757: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 879 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 716 op/s; 1563/42258 objects degraded (3.699%); 1196/42258 objects misplaced (2.830%); 691 KiB/s, 106 objects/s recovering 2024-09-21T10:27:47.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:46 smithi119 ceph-mon[105288]: 45.1 continuing backfill to osd.6 from (1180'7967,1220'15890] 45:806b3ffb:::benchmark_data_smithi119_168706_object128743:head to 1220'15890 2024-09-21T10:27:47.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:46 smithi119 ceph-mon[105288]: osdmap e1222: 8 total, 8 up, 7 in 2024-09-21T10:27:47.130 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:47.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:47.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:46 smithi145 ceph-mon[89359]: pgmap v1757: 89 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 879 MiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 716 op/s; 1563/42258 objects degraded (3.699%); 1196/42258 objects misplaced (2.830%); 691 KiB/s, 106 objects/s recovering 2024-09-21T10:27:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:46 smithi145 ceph-mon[89359]: 45.1 continuing backfill to osd.6 from (1180'7967,1220'15890] 45:806b3ffb:::benchmark_data_smithi119_168706_object128743:head to 1220'15890 2024-09-21T10:27:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:46 smithi145 ceph-mon[89359]: osdmap e1222: 8 total, 8 up, 7 in 2024-09-21T10:27:47.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:47.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:47.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:47.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:47.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:47.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:47.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:47.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:47.861 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:47.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:47.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:47.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:48.019 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:48.045 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:47 smithi119 ceph-mon[101432]: osdmap e1223: 8 total, 8 up, 7 in 2024-09-21T10:27:48.045 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:48.046 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:47 smithi119 ceph-mon[105288]: osdmap e1223: 8 total, 8 up, 7 in 2024-09-21T10:27:48.046 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:48.121 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:48.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:48.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:47 smithi145 ceph-mon[89359]: osdmap e1223: 8 total, 8 up, 7 in 2024-09-21T10:27:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:27:48.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:48.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:48.440 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:48.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:48.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:48.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:48.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:48.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:48.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:48.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:48.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:48.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:48.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:49.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:49.112 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:49.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:49.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:48 smithi145 ceph-mon[89359]: pgmap v1760: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 795 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 1196/38214 objects misplaced (3.130%); 0 B/s, 171 objects/s recovering 2024-09-21T10:27:49.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:48 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1563/42258 objects degraded (3.699%), 1 pg degraded) 2024-09-21T10:27:49.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:48 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:27:49.196 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:48 smithi145 ceph-mon[89359]: osdmap e1224: 8 total, 8 up, 7 in 2024-09-21T10:27:49.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:49.235 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:49.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:49.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[105288]: pgmap v1760: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 795 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 1196/38214 objects misplaced (3.130%); 0 B/s, 171 objects/s recovering 2024-09-21T10:27:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1563/42258 objects degraded (3.699%), 1 pg degraded) 2024-09-21T10:27:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:27:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[105288]: osdmap e1224: 8 total, 8 up, 7 in 2024-09-21T10:27:49.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[101432]: pgmap v1760: 89 pgs: 2 active+recovering+undersized+remapped, 2 active+remapped+backfill_wait, 85 active+clean; 795 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 761 op/s; 1196/38214 objects misplaced (3.130%); 0 B/s, 171 objects/s recovering 2024-09-21T10:27:49.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1563/42258 objects degraded (3.699%), 1 pg degraded) 2024-09-21T10:27:49.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:27:49.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:48 smithi119 ceph-mon[101432]: osdmap e1224: 8 total, 8 up, 7 in 2024-09-21T10:27:49.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:49.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:49.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:49.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:49.607 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:49.709 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:49.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:49.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:49.856 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:49.898 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:27:49.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:49.979 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:49 smithi119 ceph-mon[105288]: osdmap e1225: 8 total, 8 up, 7 in 2024-09-21T10:27:49.979 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1480616285' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:49.980 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:49 smithi119 ceph-mon[101432]: osdmap e1225: 8 total, 8 up, 7 in 2024-09-21T10:27:49.980 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1480616285' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:49.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:49.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:50.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.120 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:50.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:50.217 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:49 smithi145 ceph-mon[89359]: osdmap e1225: 8 total, 8 up, 7 in 2024-09-21T10:27:50.217 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1480616285' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:27:50.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:50.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:50.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:50.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:50.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:50.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:50.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:27:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:27:50.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:27:50.625 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:50.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:50.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:50.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:250853 45.9 e1226/1212 45:969bc2a4:::benchmark_data_smithi119_168706_object120683:head v 1226'16017, mlcod=1226'16014)", 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:50.684723+0000", 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0034072270000000001, 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034351749999999999, 2024-09-21T10:27:50.920 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684723+0000", 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684723+0000", 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.921 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684724+0000", 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4700000000000001e-07 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684730+0000", 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3999999999999997e-06 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:50.922 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684743+0000", 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.259e-05 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684750+0000", 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8070000000000002e-06 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684883+0000", 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013314800000000001 2024-09-21T10:27:50.923 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.684927+0000", 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4323000000000002e-05 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:250864 45.9 e1226/1212)", 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:50.685638+0000", 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0024922400000000002, 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025683360000000001, 2024-09-21T10:27:50.924 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.685638+0000", 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.685638+0000", 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.925 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.685639+0000", 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7100000000000002e-07 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.685642+0000", 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7459999999999998e-06 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.926 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.685655+0000", 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2543e-05 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.685662+0000", 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7750000000000004e-06 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:250851 45.4 45:2f2599ff:::benchmark_data_smithi119_168706_object120681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1226)", 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:50.674205+0000", 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.013925275000000001, 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014044253, 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 250851 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.928 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.674205+0000", 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.674205+0000", 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.674207+0000", 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5629999999999999e-06 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.674212+0000", 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:50.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.674225+0000", 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3013999999999999e-05 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.674233+0000", 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4850000000000003e-06 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.683563+0000", 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0093297229999999998 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.683673+0000", 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110188 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5,7", 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.683931+0000", 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025849299999999999 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:250850 45.0 e1226/1216 45:06e218fc:::benchmark_data_smithi119_168706_object120680:head v 1226'15571, mlcod=1226'15569)", 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:50.676045+0000", 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012085419999999999, 2024-09-21T10:27:50.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012262377, 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676045+0000", 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676045+0000", 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676046+0000", 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1500000000000003e-07 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676052+0000", 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1870000000000002e-06 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676068+0000", 2024-09-21T10:27:50.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6098000000000001e-05 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676075+0000", 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9240000000000001e-06 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676186+0000", 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110914 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:50.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676211+0000", 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5337000000000002e-05 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "MOSDPGPushReply(45.4 1226/1207 [PushReplyOp(45:23c0e218:::benchmark_data_smithi119_168706_object127729:head)])", 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:50.676317+0000", 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.011813324, 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012035311999999999, 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:50.936 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676317+0000", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676317+0000", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676317+0000", 2024-09-21T10:27:50.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5599999999999999e-07 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676321+0000", 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4230000000000001e-06 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676326+0000", 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8769999999999998e-06 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.676332+0000", 2024-09-21T10:27:50.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2129999999999999e-06 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:250858 45.2 e1226/1196 45:49076d7e:::benchmark_data_smithi119_168706_object120688:head v 1226'15615, mlcod=1226'15613)", 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:50.678742+0000", 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0093882440000000004, 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0096470400000000008, 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:27:50.939 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678742+0000", 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678742+0000", 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:27:50.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678743+0000", 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8299999999999995e-07 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678752+0000", 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7329999999999992e-06 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678769+0000", 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7453e-05 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678781+0000", 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2047e-05 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678881+0000", 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100146 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:50.678898+0000", 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6609999999999999e-05 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.942 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:27:50.943 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.943 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:27:50.943 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:27:50.943 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 6 2024-09-21T10:27:50.943 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:27:50.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:50.944 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:50.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:51.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:51.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[105288]: pgmap v1763: 89 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 86 active+clean; 688 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 690/33099 objects misplaced (2.085%); 19 MiB/s, 503 objects/s recovering 2024-09-21T10:27:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1480616285' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[105288]: osdmap e1226: 8 total, 8 up, 7 in 2024-09-21T10:27:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[101432]: pgmap v1763: 89 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 86 active+clean; 688 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 690/33099 objects misplaced (2.085%); 19 MiB/s, 503 objects/s recovering 2024-09-21T10:27:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1480616285' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:50 smithi119 ceph-mon[101432]: osdmap e1226: 8 total, 8 up, 7 in 2024-09-21T10:27:51.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:51.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:51.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:50 smithi145 ceph-mon[89359]: pgmap v1763: 89 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 86 active+clean; 688 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 763 op/s; 690/33099 objects misplaced (2.085%); 19 MiB/s, 503 objects/s recovering 2024-09-21T10:27:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:50 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:27:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1480616285' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:27:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:50 smithi145 ceph-mon[89359]: osdmap e1226: 8 total, 8 up, 7 in 2024-09-21T10:27:51.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:51.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:51.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:51.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:51.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:51.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:51.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:51.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:51.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:51.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:51.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:51.834 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:51.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:51.958 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:51.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:52.034 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:51 smithi119 ceph-mon[105288]: osdmap e1227: 8 total, 8 up, 7 in 2024-09-21T10:27:52.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:51 smithi119 ceph-mon[101432]: osdmap e1227: 8 total, 8 up, 7 in 2024-09-21T10:27:52.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:52.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:52.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:52.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:51 smithi145 ceph-mon[89359]: osdmap e1227: 8 total, 8 up, 7 in 2024-09-21T10:27:52.328 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:52.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:52.451 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:52.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:52.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:27:52.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:52.604 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:52.705 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:52.727 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:52.741 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:27:52.742 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:27:52.760 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:27:52.761 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:27:52.783 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:27:52.783 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:27:52.800 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:52.802 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:27:52.802 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:27:52.815 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:27:52.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:27:52.816 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:27:52.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:27:52.819 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:27:52.819 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:27:52.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:52.835 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:27:52.835 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:27:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.850 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:52.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:52.851 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:27:52.851 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:27:52.866 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:27:52.867 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:27:52.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:52.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:53.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:52 smithi119 ceph-mon[101432]: pgmap v1766: 89 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 86 active+clean; 658 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 554/31635 objects misplaced (1.751%); 19 MiB/s, 353 objects/s recovering 2024-09-21T10:27:53.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:52 smithi119 ceph-mon[101432]: osdmap e1228: 8 total, 8 up, 7 in 2024-09-21T10:27:53.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2396481607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:53.058 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:52 smithi119 ceph-mon[105288]: pgmap v1766: 89 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 86 active+clean; 658 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 554/31635 objects misplaced (1.751%); 19 MiB/s, 353 objects/s recovering 2024-09-21T10:27:53.058 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:52 smithi119 ceph-mon[105288]: osdmap e1228: 8 total, 8 up, 7 in 2024-09-21T10:27:53.058 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2396481607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:53.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:53.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:53.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:53.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:52 smithi145 ceph-mon[89359]: pgmap v1766: 89 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 86 active+clean; 658 MiB data, 3.7 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 548 op/s; 554/31635 objects misplaced (1.751%); 19 MiB/s, 353 objects/s recovering 2024-09-21T10:27:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:52 smithi145 ceph-mon[89359]: osdmap e1228: 8 total, 8 up, 7 in 2024-09-21T10:27:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2396481607' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:27:53.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:53.384 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:53.485 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:53.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:53.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:53.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:53.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:27:53.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:53.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:53.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:53.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:53.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:53.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:53.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:54.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:54.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:53 smithi119 ceph-mon[105288]: osdmap e1229: 8 total, 8 up, 7 in 2024-09-21T10:27:54.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:53 smithi119 ceph-mon[101432]: osdmap e1229: 8 total, 8 up, 7 in 2024-09-21T10:27:54.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:54.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:54.285 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:53 smithi145 ceph-mon[89359]: osdmap e1229: 8 total, 8 up, 7 in 2024-09-21T10:27:54.309 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:54.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:54.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:54.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.602 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:54.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:54.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:54.706 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:54.778 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:54.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:54.862 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:27:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:27:54] "GET /metrics HTTP/1.1" 200 36855 "" "Prometheus/2.51.0" 2024-09-21T10:27:54.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:54.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:54.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:55.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:55.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:55.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:54 smithi119 ceph-mon[105288]: pgmap v1769: 89 pgs: 1 active+remapped+backfill_wait, 88 active+clean; 593 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 382 op/s; 517/28503 objects misplaced (1.814%); 576 KiB/s, 249 objects/s recovering 2024-09-21T10:27:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:54 smithi119 ceph-mon[101432]: pgmap v1769: 89 pgs: 1 active+remapped+backfill_wait, 88 active+clean; 593 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 382 op/s; 517/28503 objects misplaced (1.814%); 576 KiB/s, 249 objects/s recovering 2024-09-21T10:27:55.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:55.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:55.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:55.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:54 smithi145 ceph-mon[89359]: pgmap v1769: 89 pgs: 1 active+remapped+backfill_wait, 88 active+clean; 593 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 0 B/s wr, 382 op/s; 517/28503 objects misplaced (1.814%); 576 KiB/s, 249 objects/s recovering 2024-09-21T10:27:55.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:55.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.460 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:55.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:55.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.604 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:55.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:55.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:55.743 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:55.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:55.870 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:55.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:55.972 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:56.002 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:56.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:56.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:56.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:27:56.229 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:56.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:56.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:27:56.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:27:56.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:56.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:56.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:56.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:56.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:56.819 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:56.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:56.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:56.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:56.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:57.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:57.104 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:57.206 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:57.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:57.283 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:56 smithi119 ceph-mon[105288]: pgmap v1770: 89 pgs: 89 active+clean; 525 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 436 op/s; 5.5 MiB/s, 259 objects/s recovering 2024-09-21T10:27:57.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:56 smithi119 ceph-mon[101432]: pgmap v1770: 89 pgs: 89 active+clean; 525 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 436 op/s; 5.5 MiB/s, 259 objects/s recovering 2024-09-21T10:27:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:56 smithi145 ceph-mon[89359]: pgmap v1770: 89 pgs: 89 active+clean; 525 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 436 op/s; 5.5 MiB/s, 259 objects/s recovering 2024-09-21T10:27:57.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:57.353 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:57.440 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:27:57.440 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:27:57.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:57.490 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:57.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:57.626 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:57.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:57.754 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:57.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:57.896 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:57.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:57.998 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:58.030 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:58.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:58.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:58.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:58.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:58.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:58.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:58.476 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:58.490 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:27:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:27:58.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:58.603 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:58.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:58.705 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:27:58.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:27:58.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:27:58.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:58.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:58.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:58.950 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:27:58.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:58.957 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:58 smithi119 ceph-mon[101432]: pgmap v1771: 89 pgs: 89 active+clean; 441 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 526 op/s; 4.7 MiB/s, 220 objects/s recovering 2024-09-21T10:27:58.977 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:58.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:59.078 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:27:59.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:27:59.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:27:59.244 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:59.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:27:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:58 smithi145 ceph-mon[89359]: pgmap v1771: 89 pgs: 89 active+clean; 441 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 526 op/s; 4.7 MiB/s, 220 objects/s recovering 2024-09-21T10:27:59.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:58 smithi119 ceph-mon[105288]: pgmap v1771: 89 pgs: 89 active+clean; 441 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 526 op/s; 4.7 MiB/s, 220 objects/s recovering 2024-09-21T10:27:59.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:27:59.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:27:59.533 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2024-09-21T10:27:57.518427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":462297061,"num_objects":7072,"num_object_clones":0,"num_object_copies":21216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7072,"num_whiteouts":0,"num_read":307,"num_read_kb":760,"num_write":253502,"num_write_kb":8332348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23688,"num_bytes_recovered":672859439,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121636,"ondisk_log_size":121636,"up":267,"acting":267,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":267,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":3320252,"kb_used_data":1570784,"kb_used_omap":729,"kb_used_meta":1748582,"kb_avail":652895812,"statfs":{"total":671965249536,"available":668565311488,"internally_reserved":0,"allocated":1608482816,"data_stored":1574342355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":747440,"internal_metadata":1790548048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,7,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-227606528,"num_objects":-3473,"num_object_clones":0,"num_object_copies":-10419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-554,"num_objects_unfound":0,"num_objects_dirty":-3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3473,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1457,"num_bytes_recovered":32309248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.594441"},"pg_stats":[{"pgid":"45.f","version":"1228'15599","reported_seq":15739,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.306100+0000","last_change":"2024-09-21T10:23:24.928314+0000","last_active":"2024-09-21T10:27:52.306100+0000","last_peered":"2024-09-21T10:27:52.306100+0000","last_clean":"2024-09-21T10:27:52.306100+0000","last_became_active":"2024-09-21T10:23:24.928005+0000","last_became_peered":"2024-09-21T10:23:24.928005+0000","last_unstale":"2024-09-21T10:27:52.306100+0000","last_undegraded":"2024-09-21T10:27:52.306100+0000","last_fullsized":"2024-09-21T10:27:52.306100+0000","mapping_epoch":1164,"log_start":"1180'7611","ondisk_log_start":"1180'7611","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7988,"log_dups_size":0,"ondisk_log_size":7988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:15.220461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15599,"num_write_kb":517888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.e","version":"1229'15636","reported_seq":20381,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:53.707197+0000","last_change":"2024-09-21T10:27:18.941008+0000","last_active":"2024-09-21T10:27:53.707197+0000","last_peered":"2024-09-21T10:27:53.707197+0000","last_clean":"2024-09-21T10:27:53.707197+0000","last_became_active":"2024-09-21T10:27:18.863836+0000","last_became_peered":"2024-09-21T10:27:18.863836+0000","last_unstale":"2024-09-21T10:27:53.707197+0000","last_undegraded":"2024-09-21T10:27:53.707197+0000","last_fullsized":"2024-09-21T10:27:53.707197+0000","mapping_epoch":1210,"log_start":"1179'7499","ondisk_log_start":"1179'7499","created":1164,"last_epoch_clean":1211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8137,"log_dups_size":0,"ondisk_log_size":8137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:09:38.442484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15636,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":152305664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.d","version":"1229'15821","reported_seq":18832,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:54.399336+0000","last_change":"2024-09-21T10:26:46.904802+0000","last_active":"2024-09-21T10:27:54.399336+0000","last_peered":"2024-09-21T10:27:54.399336+0000","last_clean":"2024-09-21T10:27:54.399336+0000","last_became_active":"2024-09-21T10:26:46.876329+0000","last_became_peered":"2024-09-21T10:26:46.876329+0000","last_unstale":"2024-09-21T10:27:54.399336+0000","last_undegraded":"2024-09-21T10:27:54.399336+0000","last_fullsized":"2024-09-21T10:27:54.399336+0000","mapping_epoch":1203,"log_start":"1182'8767","ondisk_log_start":"1182'8767","created":1164,"last_epoch_clean":1204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7054,"log_dups_size":0,"ondisk_log_size":7054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:41:45.679188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753368,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15821,"num_write_kb":520769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1428,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.c","version":"1229'16049","reported_seq":18868,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:57.237125+0000","last_change":"2024-09-21T10:27:21.432038+0000","last_active":"2024-09-21T10:27:57.237125+0000","last_peered":"2024-09-21T10:27:57.237125+0000","last_clean":"2024-09-21T10:27:57.237125+0000","last_became_active":"2024-09-21T10:27:21.331628+0000","last_became_peered":"2024-09-21T10:27:21.331628+0000","last_unstale":"2024-09-21T10:27:57.237125+0000","last_undegraded":"2024-09-21T10:27:57.237125+0000","last_fullsized":"2024-09-21T10:27:57.237125+0000","mapping_epoch":1213,"log_start":"1182'8532","ondisk_log_start":"1182'8532","created":1164,"last_epoch_clean":1214,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7517,"log_dups_size":0,"ondisk_log_size":7517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:04.740768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21561344,"num_objects":329,"num_object_clones":0,"num_object_copies":987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16049,"num_write_kb":524096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1390,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.b","version":"1229'15886","reported_seq":23244,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:55.631268+0000","last_change":"2024-09-21T10:27:34.839071+0000","last_active":"2024-09-21T10:27:55.631268+0000","last_peered":"2024-09-21T10:27:55.631268+0000","last_clean":"2024-09-21T10:27:55.631268+0000","last_became_active":"2024-09-21T10:27:34.796697+0000","last_became_peered":"2024-09-21T10:27:34.796697+0000","last_unstale":"2024-09-21T10:27:55.631268+0000","last_undegraded":"2024-09-21T10:27:55.631268+0000","last_fullsized":"2024-09-21T10:27:55.631268+0000","mapping_epoch":1219,"log_start":"1184'9562","ondisk_log_start":"1184'9562","created":1164,"last_epoch_clean":1220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":6324,"log_dups_size":0,"ondisk_log_size":6324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:20:17.325678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26869760,"num_objects":410,"num_object_clones":0,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15886,"num_write_kb":521472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3589,"num_bytes_recovered":135069696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.a","version":"1229'15803","reported_seq":18707,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:57.197522+0000","last_change":"2024-09-21T10:26:36.883340+0000","last_active":"2024-09-21T10:27:57.197522+0000","last_peered":"2024-09-21T10:27:57.197522+0000","last_clean":"2024-09-21T10:27:57.197522+0000","last_became_active":"2024-09-21T10:26:36.792565+0000","last_became_peered":"2024-09-21T10:26:36.792565+0000","last_unstale":"2024-09-21T10:27:57.197522+0000","last_undegraded":"2024-09-21T10:27:57.197522+0000","last_fullsized":"2024-09-21T10:27:57.197522+0000","mapping_epoch":1198,"log_start":"1182'8284","ondisk_log_start":"1182'8284","created":1164,"last_epoch_clean":1199,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:55.008029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19464192,"num_objects":297,"num_object_clones":0,"num_object_copies":891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15803,"num_write_kb":515200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1401,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.9","version":"1229'16319","reported_seq":19559,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:57.257163+0000","last_change":"2024-09-21T10:27:20.396915+0000","last_active":"2024-09-21T10:27:57.257163+0000","last_peered":"2024-09-21T10:27:57.257163+0000","last_clean":"2024-09-21T10:27:57.257163+0000","last_became_active":"2024-09-21T10:27:20.323626+0000","last_became_peered":"2024-09-21T10:27:20.323626+0000","last_unstale":"2024-09-21T10:27:57.257163+0000","last_undegraded":"2024-09-21T10:27:57.257163+0000","last_fullsized":"2024-09-21T10:27:57.257163+0000","mapping_epoch":1212,"log_start":"1182'8728","ondisk_log_start":"1182'8728","created":1164,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7591,"log_dups_size":0,"ondisk_log_size":7591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:25:39.031926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19202048,"num_objects":293,"num_object_clones":0,"num_object_copies":879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16319,"num_write_kb":531584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1561,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.8","version":"1229'15739","reported_seq":19206,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:55.626707+0000","last_change":"2024-09-21T10:27:29.368067+0000","last_active":"2024-09-21T10:27:55.626707+0000","last_peered":"2024-09-21T10:27:55.626707+0000","last_clean":"2024-09-21T10:27:55.626707+0000","last_became_active":"2024-09-21T10:27:29.327469+0000","last_became_peered":"2024-09-21T10:27:29.327469+0000","last_unstale":"2024-09-21T10:27:55.626707+0000","last_undegraded":"2024-09-21T10:27:55.626707+0000","last_fullsized":"2024-09-21T10:27:55.626707+0000","mapping_epoch":1215,"log_start":"1183'9402","ondisk_log_start":"1183'9402","created":1164,"last_epoch_clean":1216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":6337,"log_dups_size":0,"ondisk_log_size":6337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:33.812647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25231360,"num_objects":385,"num_object_clones":0,"num_object_copies":1155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15739,"num_write_kb":515968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1652,"num_bytes_recovered":108265472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"45.7","version":"1229'16088","reported_seq":22377,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:57.245694+0000","last_change":"2024-09-21T10:27:51.972970+0000","last_active":"2024-09-21T10:27:57.245694+0000","last_peered":"2024-09-21T10:27:57.245694+0000","last_clean":"2024-09-21T10:27:57.245694+0000","last_became_active":"2024-09-21T10:27:51.928507+0000","last_became_peered":"2024-09-21T10:27:51.928507+0000","last_unstale":"2024-09-21T10:27:57.245694+0000","last_undegraded":"2024-09-21T10:27:57.245694+0000","last_fullsized":"2024-09-21T10:27:57.245694+0000","mapping_epoch":1227,"log_start":"1182'8541","ondisk_log_start":"1182'8541","created":1164,"last_epoch_clean":1228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7547,"log_dups_size":0,"ondisk_log_size":7547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:49.804570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282240,"num_objects":340,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16088,"num_write_kb":525696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3085,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"45.6","version":"1229'15903","reported_seq":18010,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:54.410027+0000","last_change":"2024-09-21T10:27:47.919970+0000","last_active":"2024-09-21T10:27:54.410027+0000","last_peered":"2024-09-21T10:27:54.410027+0000","last_clean":"2024-09-21T10:27:54.410027+0000","last_became_active":"2024-09-21T10:27:47.885500+0000","last_became_peered":"2024-09-21T10:27:47.885500+0000","last_unstale":"2024-09-21T10:27:54.410027+0000","last_undegraded":"2024-09-21T10:27:54.410027+0000","last_fullsized":"2024-09-21T10:27:54.410027+0000","mapping_epoch":1223,"log_start":"1182'8882","ondisk_log_start":"1182'8882","created":1164,"last_epoch_clean":1224,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7021,"log_dups_size":0,"ondisk_log_size":7021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:56.424986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29818880,"num_objects":455,"num_object_clones":0,"num_object_copies":1365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15903,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":63635456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.5","version":"1228'15172","reported_seq":15313,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.329928+0000","last_change":"2024-09-21T10:23:24.927838+0000","last_active":"2024-09-21T10:27:52.329928+0000","last_peered":"2024-09-21T10:27:52.329928+0000","last_clean":"2024-09-21T10:27:52.329928+0000","last_became_active":"2024-09-21T10:23:24.927541+0000","last_became_peered":"2024-09-21T10:23:24.927541+0000","last_unstale":"2024-09-21T10:27:52.329928+0000","last_undegraded":"2024-09-21T10:27:52.329928+0000","last_fullsized":"2024-09-21T10:27:52.329928+0000","mapping_epoch":1164,"log_start":"1179'7230","ondisk_log_start":"1179'7230","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7942,"log_dups_size":0,"ondisk_log_size":7942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:16.618768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15172,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"45.4","version":"1229'15987","reported_seq":17707,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:54.684764+0000","last_change":"2024-09-21T10:27:52.970197+0000","last_active":"2024-09-21T10:27:54.684764+0000","last_peered":"2024-09-21T10:27:54.684764+0000","last_clean":"2024-09-21T10:27:54.684764+0000","last_became_active":"2024-09-21T10:27:52.928094+0000","last_became_peered":"2024-09-21T10:27:52.928094+0000","last_unstale":"2024-09-21T10:27:54.684764+0000","last_undegraded":"2024-09-21T10:27:54.684764+0000","last_fullsized":"2024-09-21T10:27:54.684764+0000","mapping_epoch":1228,"log_start":"1175'5900","ondisk_log_start":"1175'5900","created":1164,"last_epoch_clean":1229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:26:58.751067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30474240,"num_objects":465,"num_object_clones":0,"num_object_copies":1395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15987,"num_write_kb":526464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":779,"num_bytes_recovered":51052544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"45.3","version":"1229'15774","reported_seq":15917,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:54.392082+0000","last_change":"2024-09-21T10:23:24.927687+0000","last_active":"2024-09-21T10:27:54.392082+0000","last_peered":"2024-09-21T10:27:54.392082+0000","last_clean":"2024-09-21T10:27:54.392082+0000","last_became_active":"2024-09-21T10:23:24.927161+0000","last_became_peered":"2024-09-21T10:23:24.927161+0000","last_unstale":"2024-09-21T10:27:54.392082+0000","last_undegraded":"2024-09-21T10:27:54.392082+0000","last_fullsized":"2024-09-21T10:27:54.392082+0000","mapping_epoch":1164,"log_start":"1182'8707","ondisk_log_start":"1182'8707","created":1164,"last_epoch_clean":1165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:28.137305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15774,"num_write_kb":519680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"45.2","version":"1229'15684","reported_seq":18681,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:53.704536+0000","last_change":"2024-09-21T10:26:22.771152+0000","last_active":"2024-09-21T10:27:53.704536+0000","last_peered":"2024-09-21T10:27:53.704536+0000","last_clean":"2024-09-21T10:27:53.704536+0000","last_became_active":"2024-09-21T10:26:22.694511+0000","last_became_peered":"2024-09-21T10:26:22.694511+0000","last_unstale":"2024-09-21T10:27:53.704536+0000","last_undegraded":"2024-09-21T10:27:53.704536+0000","last_fullsized":"2024-09-21T10:27:53.704536+0000","mapping_epoch":1196,"log_start":"1179'7542","ondisk_log_start":"1179'7542","created":1164,"last_epoch_clean":1197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8142,"log_dups_size":0,"ondisk_log_size":8142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:28:29.995295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31719424,"num_objects":484,"num_object_clones":0,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15684,"num_write_kb":517376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1447,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"45.1","version":"1228'16143","reported_seq":20847,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.302159+0000","last_change":"2024-09-21T10:27:49.970236+0000","last_active":"2024-09-21T10:27:52.302159+0000","last_peered":"2024-09-21T10:27:52.302159+0000","last_clean":"2024-09-21T10:27:52.302159+0000","last_became_active":"2024-09-21T10:27:49.905192+0000","last_became_peered":"2024-09-21T10:27:49.905192+0000","last_unstale":"2024-09-21T10:27:52.302159+0000","last_undegraded":"2024-09-21T10:27:52.302159+0000","last_fullsized":"2024-09-21T10:27:52.302159+0000","mapping_epoch":1225,"log_start":"1180'7967","ondisk_log_start":"1180'7967","created":1164,"last_epoch_clean":1226,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":8176,"log_dups_size":0,"ondisk_log_size":8176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:27:34.040475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35848192,"num_objects":547,"num_object_clones":0,"num_object_copies":1641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16143,"num_write_kb":534080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2291,"num_bytes_recovered":47185920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"45.0","version":"1229'15687","reported_seq":19354,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:54.370148+0000","last_change":"2024-09-21T10:27:30.384557+0000","last_active":"2024-09-21T10:27:54.370148+0000","last_peered":"2024-09-21T10:27:54.370148+0000","last_clean":"2024-09-21T10:27:54.370148+0000","last_became_active":"2024-09-21T10:27:30.333897+0000","last_became_peered":"2024-09-21T10:27:30.333897+0000","last_unstale":"2024-09-21T10:27:54.370148+0000","last_undegraded":"2024-09-21T10:27:54.370148+0000","last_fullsized":"2024-09-21T10:27:54.370148+0000","mapping_epoch":1216,"log_start":"1182'8658","ondisk_log_start":"1182'8658","created":1164,"last_epoch_clean":1217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:23:23.908323+0000","last_clean_scrub_stamp":"2024-09-21T10:23:23.908323+0000","objects_scrubbed":0,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:34.281834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15687,"num_write_kb":517696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":114753536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":1905,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.980694+0000","last_change":"2024-09-21T10:26:57.030163+0000","last_active":"2024-09-21T10:27:51.980694+0000","last_peered":"2024-09-21T10:27:51.980694+0000","last_clean":"2024-09-21T10:27:51.980694+0000","last_became_active":"2024-09-21T10:26:56.956489+0000","last_became_peered":"2024-09-21T10:26:56.956489+0000","last_unstale":"2024-09-21T10:27:51.980694+0000","last_undegraded":"2024-09-21T10:27:51.980694+0000","last_fullsized":"2024-09-21T10:27:51.980694+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:29:04.563066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1196,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921239+0000","last_change":"2024-09-21T10:27:19.326034+0000","last_active":"2024-09-21T10:27:50.921239+0000","last_peered":"2024-09-21T10:27:50.921239+0000","last_clean":"2024-09-21T10:27:50.921239+0000","last_became_active":"2024-09-21T10:27:19.325763+0000","last_became_peered":"2024-09-21T10:27:19.325763+0000","last_unstale":"2024-09-21T10:27:50.921239+0000","last_undegraded":"2024-09-21T10:27:50.921239+0000","last_fullsized":"2024-09-21T10:27:50.921239+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:51.821446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1773,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.986622+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:27:52.986622+0000","last_peered":"2024-09-21T10:27:52.986622+0000","last_clean":"2024-09-21T10:27:52.986622+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:27:52.986622+0000","last_undegraded":"2024-09-21T10:27:52.986622+0000","last_fullsized":"2024-09-21T10:27:52.986622+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1792,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.985056+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:27:52.985056+0000","last_peered":"2024-09-21T10:27:52.985056+0000","last_clean":"2024-09-21T10:27:52.985056+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:27:52.985056+0000","last_undegraded":"2024-09-21T10:27:52.985056+0000","last_fullsized":"2024-09-21T10:27:52.985056+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":40,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.922032+0000","last_change":"2024-09-21T10:27:20.323228+0000","last_active":"2024-09-21T10:27:50.922032+0000","last_peered":"2024-09-21T10:27:50.922032+0000","last_clean":"2024-09-21T10:27:50.922032+0000","last_became_active":"2024-09-21T10:27:20.322808+0000","last_became_peered":"2024-09-21T10:27:20.322808+0000","last_unstale":"2024-09-21T10:27:50.922032+0000","last_undegraded":"2024-09-21T10:27:50.922032+0000","last_fullsized":"2024-09-21T10:27:50.922032+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:27.259692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1192,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921436+0000","last_change":"2024-09-21T10:27:19.333245+0000","last_active":"2024-09-21T10:27:50.921436+0000","last_peered":"2024-09-21T10:27:50.921436+0000","last_clean":"2024-09-21T10:27:50.921436+0000","last_became_active":"2024-09-21T10:27:19.332965+0000","last_became_peered":"2024-09-21T10:27:19.332965+0000","last_unstale":"2024-09-21T10:27:50.921436+0000","last_undegraded":"2024-09-21T10:27:50.921436+0000","last_fullsized":"2024-09-21T10:27:50.921436+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:04.828777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":40,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921732+0000","last_change":"2024-09-21T10:27:20.324804+0000","last_active":"2024-09-21T10:27:50.921732+0000","last_peered":"2024-09-21T10:27:50.921732+0000","last_clean":"2024-09-21T10:27:50.921732+0000","last_became_active":"2024-09-21T10:27:20.324499+0000","last_became_peered":"2024-09-21T10:27:20.324499+0000","last_unstale":"2024-09-21T10:27:50.921732+0000","last_undegraded":"2024-09-21T10:27:50.921732+0000","last_fullsized":"2024-09-21T10:27:50.921732+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:15.209140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1192,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921562+0000","last_change":"2024-09-21T10:27:19.370409+0000","last_active":"2024-09-21T10:27:50.921562+0000","last_peered":"2024-09-21T10:27:50.921562+0000","last_clean":"2024-09-21T10:27:50.921562+0000","last_became_active":"2024-09-21T10:27:19.370096+0000","last_became_peered":"2024-09-21T10:27:19.370096+0000","last_unstale":"2024-09-21T10:27:50.921562+0000","last_undegraded":"2024-09-21T10:27:50.921562+0000","last_fullsized":"2024-09-21T10:27:50.921562+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:03.124925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":1888,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.979666+0000","last_change":"2024-09-21T10:06:43.621989+0000","last_active":"2024-09-21T10:27:51.979666+0000","last_peered":"2024-09-21T10:27:51.979666+0000","last_clean":"2024-09-21T10:27:51.979666+0000","last_became_active":"2024-09-21T10:06:43.620817+0000","last_became_peered":"2024-09-21T10:06:43.620817+0000","last_unstale":"2024-09-21T10:27:51.979666+0000","last_undegraded":"2024-09-21T10:27:51.979666+0000","last_fullsized":"2024-09-21T10:27:51.979666+0000","mapping_epoch":543,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":544,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:30.745756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1797,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.925117+0000","last_change":"2024-09-21T10:26:56.958973+0000","last_active":"2024-09-21T10:27:52.925117+0000","last_peered":"2024-09-21T10:27:52.925117+0000","last_clean":"2024-09-21T10:27:52.925117+0000","last_became_active":"2024-09-21T10:26:56.958011+0000","last_became_peered":"2024-09-21T10:26:56.958011+0000","last_unstale":"2024-09-21T10:27:52.925117+0000","last_undegraded":"2024-09-21T10:27:52.925117+0000","last_fullsized":"2024-09-21T10:27:52.925117+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:48.664992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1167,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921455+0000","last_change":"2024-09-21T10:27:19.325859+0000","last_active":"2024-09-21T10:27:50.921455+0000","last_peered":"2024-09-21T10:27:50.921455+0000","last_clean":"2024-09-21T10:27:50.921455+0000","last_became_active":"2024-09-21T10:27:19.325540+0000","last_became_peered":"2024-09-21T10:27:19.325540+0000","last_unstale":"2024-09-21T10:27:50.921455+0000","last_undegraded":"2024-09-21T10:27:50.921455+0000","last_fullsized":"2024-09-21T10:27:50.921455+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:11.324592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1768,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.986984+0000","last_change":"2024-09-21T10:27:19.330394+0000","last_active":"2024-09-21T10:27:52.986984+0000","last_peered":"2024-09-21T10:27:52.986984+0000","last_clean":"2024-09-21T10:27:52.986984+0000","last_became_active":"2024-09-21T10:27:19.328017+0000","last_became_peered":"2024-09-21T10:27:19.328017+0000","last_unstale":"2024-09-21T10:27:52.986984+0000","last_undegraded":"2024-09-21T10:27:52.986984+0000","last_fullsized":"2024-09-21T10:27:52.986984+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:59:32.804082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1873,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.926752+0000","last_change":"2024-09-21T10:27:19.331712+0000","last_active":"2024-09-21T10:27:52.926752+0000","last_peered":"2024-09-21T10:27:52.926752+0000","last_clean":"2024-09-21T10:27:52.926752+0000","last_became_active":"2024-09-21T10:27:19.330959+0000","last_became_peered":"2024-09-21T10:27:19.330959+0000","last_unstale":"2024-09-21T10:27:52.926752+0000","last_undegraded":"2024-09-21T10:27:52.926752+0000","last_fullsized":"2024-09-21T10:27:52.926752+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:46.389484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1886,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.980211+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:27:51.980211+0000","last_peered":"2024-09-21T10:27:51.980211+0000","last_clean":"2024-09-21T10:27:51.980211+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:27:51.980211+0000","last_undegraded":"2024-09-21T10:27:51.980211+0000","last_fullsized":"2024-09-21T10:27:51.980211+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1793,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.983903+0000","last_change":"2024-09-21T10:08:55.115514+0000","last_active":"2024-09-21T10:27:52.983903+0000","last_peered":"2024-09-21T10:27:52.983903+0000","last_clean":"2024-09-21T10:27:52.983903+0000","last_became_active":"2024-09-21T10:08:55.114896+0000","last_became_peered":"2024-09-21T10:08:55.114896+0000","last_unstale":"2024-09-21T10:27:52.983903+0000","last_undegraded":"2024-09-21T10:27:52.983903+0000","last_fullsized":"2024-09-21T10:27:52.983903+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:45.967730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1177,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925990+0000","last_change":"2024-09-21T10:26:56.957568+0000","last_active":"2024-09-21T10:27:51.925990+0000","last_peered":"2024-09-21T10:27:51.925990+0000","last_clean":"2024-09-21T10:27:51.925990+0000","last_became_active":"2024-09-21T10:26:56.957191+0000","last_became_peered":"2024-09-21T10:26:56.957191+0000","last_unstale":"2024-09-21T10:27:51.925990+0000","last_undegraded":"2024-09-21T10:27:51.925990+0000","last_fullsized":"2024-09-21T10:27:51.925990+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:11:04.630815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1787,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.984146+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:27:52.984146+0000","last_peered":"2024-09-21T10:27:52.984146+0000","last_clean":"2024-09-21T10:27:52.984146+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:27:52.984146+0000","last_undegraded":"2024-09-21T10:27:52.984146+0000","last_fullsized":"2024-09-21T10:27:52.984146+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1891,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.980816+0000","last_change":"2024-09-21T10:26:56.956418+0000","last_active":"2024-09-21T10:27:51.980816+0000","last_peered":"2024-09-21T10:27:51.980816+0000","last_clean":"2024-09-21T10:27:51.980816+0000","last_became_active":"2024-09-21T10:26:56.956129+0000","last_became_peered":"2024-09-21T10:26:56.956129+0000","last_unstale":"2024-09-21T10:27:51.980816+0000","last_undegraded":"2024-09-21T10:27:51.980816+0000","last_fullsized":"2024-09-21T10:27:51.980816+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:27.517048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1770,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.985256+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:27:52.985256+0000","last_peered":"2024-09-21T10:27:52.985256+0000","last_clean":"2024-09-21T10:27:52.985256+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:27:52.985256+0000","last_undegraded":"2024-09-21T10:27:52.985256+0000","last_fullsized":"2024-09-21T10:27:52.985256+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":529,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925195+0000","last_change":"2024-09-21T10:26:56.955932+0000","last_active":"2024-09-21T10:27:51.925195+0000","last_peered":"2024-09-21T10:27:51.925195+0000","last_clean":"2024-09-21T10:27:51.925195+0000","last_became_active":"2024-09-21T10:26:56.955446+0000","last_became_peered":"2024-09-21T10:26:56.955446+0000","last_unstale":"2024-09-21T10:27:51.925195+0000","last_undegraded":"2024-09-21T10:27:51.925195+0000","last_fullsized":"2024-09-21T10:27:51.925195+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:27:15.329971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1417,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927252+0000","last_change":"2024-09-21T10:27:19.327772+0000","last_active":"2024-09-21T10:27:52.927252+0000","last_peered":"2024-09-21T10:27:52.927252+0000","last_clean":"2024-09-21T10:27:52.927252+0000","last_became_active":"2024-09-21T10:27:19.327467+0000","last_became_peered":"2024-09-21T10:27:19.327467+0000","last_unstale":"2024-09-21T10:27:52.927252+0000","last_undegraded":"2024-09-21T10:27:52.927252+0000","last_fullsized":"2024-09-21T10:27:52.927252+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:28:38.639278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1873,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.926998+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:27:52.926998+0000","last_peered":"2024-09-21T10:27:52.926998+0000","last_clean":"2024-09-21T10:27:52.926998+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:27:52.926998+0000","last_undegraded":"2024-09-21T10:27:52.926998+0000","last_fullsized":"2024-09-21T10:27:52.926998+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1202,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921640+0000","last_change":"2024-09-21T10:27:19.330694+0000","last_active":"2024-09-21T10:27:50.921640+0000","last_peered":"2024-09-21T10:27:50.921640+0000","last_clean":"2024-09-21T10:27:50.921640+0000","last_became_active":"2024-09-21T10:27:19.329455+0000","last_became_peered":"2024-09-21T10:27:19.329455+0000","last_unstale":"2024-09-21T10:27:50.921640+0000","last_undegraded":"2024-09-21T10:27:50.921640+0000","last_fullsized":"2024-09-21T10:27:50.921640+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:27.883673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1196,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921639+0000","last_change":"2024-09-21T10:27:19.371314+0000","last_active":"2024-09-21T10:27:50.921639+0000","last_peered":"2024-09-21T10:27:50.921639+0000","last_clean":"2024-09-21T10:27:50.921639+0000","last_became_active":"2024-09-21T10:27:19.370747+0000","last_became_peered":"2024-09-21T10:27:19.370747+0000","last_unstale":"2024-09-21T10:27:50.921639+0000","last_undegraded":"2024-09-21T10:27:50.921639+0000","last_fullsized":"2024-09-21T10:27:50.921639+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:00.522037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1177,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921661+0000","last_change":"2024-09-21T10:27:20.428677+0000","last_active":"2024-09-21T10:27:50.921661+0000","last_peered":"2024-09-21T10:27:50.921661+0000","last_clean":"2024-09-21T10:27:50.921661+0000","last_became_active":"2024-09-21T10:27:20.428376+0000","last_became_peered":"2024-09-21T10:27:20.428376+0000","last_unstale":"2024-09-21T10:27:50.921661+0000","last_undegraded":"2024-09-21T10:27:50.921661+0000","last_fullsized":"2024-09-21T10:27:50.921661+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:31.411176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1183,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925983+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:27:51.925983+0000","last_peered":"2024-09-21T10:27:51.925983+0000","last_clean":"2024-09-21T10:27:51.925983+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:27:51.925983+0000","last_undegraded":"2024-09-21T10:27:51.925983+0000","last_fullsized":"2024-09-21T10:27:51.925983+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":1900,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.980025+0000","last_change":"2024-09-21T10:27:19.329990+0000","last_active":"2024-09-21T10:27:51.980025+0000","last_peered":"2024-09-21T10:27:51.980025+0000","last_clean":"2024-09-21T10:27:51.980025+0000","last_became_active":"2024-09-21T10:27:19.329672+0000","last_became_peered":"2024-09-21T10:27:19.329672+0000","last_unstale":"2024-09-21T10:27:51.980025+0000","last_undegraded":"2024-09-21T10:27:51.980025+0000","last_fullsized":"2024-09-21T10:27:51.980025+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:55:39.827132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1777,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.983993+0000","last_change":"2024-09-21T10:08:55.105463+0000","last_active":"2024-09-21T10:27:52.983993+0000","last_peered":"2024-09-21T10:27:52.983993+0000","last_clean":"2024-09-21T10:27:52.983993+0000","last_became_active":"2024-09-21T10:08:55.103195+0000","last_became_peered":"2024-09-21T10:08:55.103195+0000","last_unstale":"2024-09-21T10:27:52.983993+0000","last_undegraded":"2024-09-21T10:27:52.983993+0000","last_fullsized":"2024-09-21T10:27:52.983993+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:42:54.856734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1882,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927553+0000","last_change":"2024-09-21T10:27:19.326897+0000","last_active":"2024-09-21T10:27:52.927553+0000","last_peered":"2024-09-21T10:27:52.927553+0000","last_clean":"2024-09-21T10:27:52.927553+0000","last_became_active":"2024-09-21T10:27:19.324476+0000","last_became_peered":"2024-09-21T10:27:19.324476+0000","last_unstale":"2024-09-21T10:27:52.927553+0000","last_undegraded":"2024-09-21T10:27:52.927553+0000","last_fullsized":"2024-09-21T10:27:52.927553+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:57.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1768,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.986593+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:27:52.986593+0000","last_peered":"2024-09-21T10:27:52.986593+0000","last_clean":"2024-09-21T10:27:52.986593+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:27:52.986593+0000","last_undegraded":"2024-09-21T10:27:52.986593+0000","last_fullsized":"2024-09-21T10:27:52.986593+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1882,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.979870+0000","last_change":"2024-09-21T10:26:57.026095+0000","last_active":"2024-09-21T10:27:51.979870+0000","last_peered":"2024-09-21T10:27:51.979870+0000","last_clean":"2024-09-21T10:27:51.979870+0000","last_became_active":"2024-09-21T10:26:56.958513+0000","last_became_peered":"2024-09-21T10:26:56.958513+0000","last_unstale":"2024-09-21T10:27:51.979870+0000","last_undegraded":"2024-09-21T10:27:51.979870+0000","last_fullsized":"2024-09-21T10:27:51.979870+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:23.065911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1216,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925073+0000","last_change":"2024-09-21T10:27:19.331827+0000","last_active":"2024-09-21T10:27:51.925073+0000","last_peered":"2024-09-21T10:27:51.925073+0000","last_clean":"2024-09-21T10:27:51.925073+0000","last_became_active":"2024-09-21T10:27:19.331523+0000","last_became_peered":"2024-09-21T10:27:19.331523+0000","last_unstale":"2024-09-21T10:27:51.925073+0000","last_undegraded":"2024-09-21T10:27:51.925073+0000","last_fullsized":"2024-09-21T10:27:51.925073+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:54.525605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1870,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927119+0000","last_change":"2024-09-21T10:26:57.025262+0000","last_active":"2024-09-21T10:27:52.927119+0000","last_peered":"2024-09-21T10:27:52.927119+0000","last_clean":"2024-09-21T10:27:52.927119+0000","last_became_active":"2024-09-21T10:26:57.024903+0000","last_became_peered":"2024-09-21T10:26:57.024903+0000","last_unstale":"2024-09-21T10:27:52.927119+0000","last_undegraded":"2024-09-21T10:27:52.927119+0000","last_fullsized":"2024-09-21T10:27:52.927119+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:40.642148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1868,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927134+0000","last_change":"2024-09-21T10:27:20.427309+0000","last_active":"2024-09-21T10:27:52.927134+0000","last_peered":"2024-09-21T10:27:52.927134+0000","last_clean":"2024-09-21T10:27:52.927134+0000","last_became_active":"2024-09-21T10:27:20.426992+0000","last_became_peered":"2024-09-21T10:27:20.426992+0000","last_unstale":"2024-09-21T10:27:52.927134+0000","last_undegraded":"2024-09-21T10:27:52.927134+0000","last_fullsized":"2024-09-21T10:27:52.927134+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:12.169478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1877,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927258+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:27:52.927258+0000","last_peered":"2024-09-21T10:27:52.927258+0000","last_clean":"2024-09-21T10:27:52.927258+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:27:52.927258+0000","last_undegraded":"2024-09-21T10:27:52.927258+0000","last_fullsized":"2024-09-21T10:27:52.927258+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":679,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.983845+0000","last_change":"2024-09-21T10:26:56.958589+0000","last_active":"2024-09-21T10:27:52.983845+0000","last_peered":"2024-09-21T10:27:52.983845+0000","last_clean":"2024-09-21T10:27:52.983845+0000","last_became_active":"2024-09-21T10:26:56.957824+0000","last_became_peered":"2024-09-21T10:26:56.957824+0000","last_unstale":"2024-09-21T10:27:52.983845+0000","last_undegraded":"2024-09-21T10:27:52.983845+0000","last_fullsized":"2024-09-21T10:27:52.983845+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:14:41.032119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1874,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927215+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:27:52.927215+0000","last_peered":"2024-09-21T10:27:52.927215+0000","last_clean":"2024-09-21T10:27:52.927215+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:27:52.927215+0000","last_undegraded":"2024-09-21T10:27:52.927215+0000","last_fullsized":"2024-09-21T10:27:52.927215+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1197,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921493+0000","last_change":"2024-09-21T10:27:19.333723+0000","last_active":"2024-09-21T10:27:50.921493+0000","last_peered":"2024-09-21T10:27:50.921493+0000","last_clean":"2024-09-21T10:27:50.921493+0000","last_became_active":"2024-09-21T10:27:19.333492+0000","last_became_peered":"2024-09-21T10:27:19.333492+0000","last_unstale":"2024-09-21T10:27:50.921493+0000","last_undegraded":"2024-09-21T10:27:50.921493+0000","last_fullsized":"2024-09-21T10:27:50.921493+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:03.860673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":75,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927036+0000","last_change":"2024-09-21T10:26:57.033704+0000","last_active":"2024-09-21T10:27:52.927036+0000","last_peered":"2024-09-21T10:27:52.927036+0000","last_clean":"2024-09-21T10:27:52.927036+0000","last_became_active":"2024-09-21T10:26:56.958387+0000","last_became_peered":"2024-09-21T10:26:56.958387+0000","last_unstale":"2024-09-21T10:27:52.927036+0000","last_undegraded":"2024-09-21T10:27:52.927036+0000","last_fullsized":"2024-09-21T10:27:52.927036+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:36.081861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1871,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927497+0000","last_change":"2024-09-21T10:26:57.025473+0000","last_active":"2024-09-21T10:27:52.927497+0000","last_peered":"2024-09-21T10:27:52.927497+0000","last_clean":"2024-09-21T10:27:52.927497+0000","last_became_active":"2024-09-21T10:26:56.958119+0000","last_became_peered":"2024-09-21T10:26:56.958119+0000","last_unstale":"2024-09-21T10:27:52.927497+0000","last_undegraded":"2024-09-21T10:27:52.927497+0000","last_fullsized":"2024-09-21T10:27:52.927497+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:12.873422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1874,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.926715+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:27:52.926715+0000","last_peered":"2024-09-21T10:27:52.926715+0000","last_clean":"2024-09-21T10:27:52.926715+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:27:52.926715+0000","last_undegraded":"2024-09-21T10:27:52.926715+0000","last_fullsized":"2024-09-21T10:27:52.926715+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1806,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.924833+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:27:52.924833+0000","last_peered":"2024-09-21T10:27:52.924833+0000","last_clean":"2024-09-21T10:27:52.924833+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:27:52.924833+0000","last_undegraded":"2024-09-21T10:27:52.924833+0000","last_fullsized":"2024-09-21T10:27:52.924833+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1897,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.980302+0000","last_change":"2024-09-21T10:08:06.543182+0000","last_active":"2024-09-21T10:27:51.980302+0000","last_peered":"2024-09-21T10:27:51.980302+0000","last_clean":"2024-09-21T10:27:51.980302+0000","last_became_active":"2024-09-21T10:08:06.542697+0000","last_became_peered":"2024-09-21T10:08:06.542697+0000","last_unstale":"2024-09-21T10:27:51.980302+0000","last_undegraded":"2024-09-21T10:27:51.980302+0000","last_fullsized":"2024-09-21T10:27:51.980302+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:42.967900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1897,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.979672+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:27:51.979672+0000","last_peered":"2024-09-21T10:27:51.979672+0000","last_clean":"2024-09-21T10:27:51.979672+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:27:51.979672+0000","last_undegraded":"2024-09-21T10:27:51.979672+0000","last_fullsized":"2024-09-21T10:27:51.979672+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1216,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924765+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:27:51.924765+0000","last_peered":"2024-09-21T10:27:51.924765+0000","last_clean":"2024-09-21T10:27:51.924765+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:27:51.924765+0000","last_undegraded":"2024-09-21T10:27:51.924765+0000","last_fullsized":"2024-09-21T10:27:51.924765+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1179,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925129+0000","last_change":"2024-09-21T10:27:19.327425+0000","last_active":"2024-09-21T10:27:51.925129+0000","last_peered":"2024-09-21T10:27:51.925129+0000","last_clean":"2024-09-21T10:27:51.925129+0000","last_became_active":"2024-09-21T10:27:19.326641+0000","last_became_peered":"2024-09-21T10:27:19.326641+0000","last_unstale":"2024-09-21T10:27:51.925129+0000","last_undegraded":"2024-09-21T10:27:51.925129+0000","last_fullsized":"2024-09-21T10:27:51.925129+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:31:37.389673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1784,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.985265+0000","last_change":"2024-09-21T10:27:20.323085+0000","last_active":"2024-09-21T10:27:52.985265+0000","last_peered":"2024-09-21T10:27:52.985265+0000","last_clean":"2024-09-21T10:27:52.985265+0000","last_became_active":"2024-09-21T10:27:20.322679+0000","last_became_peered":"2024-09-21T10:27:20.322679+0000","last_unstale":"2024-09-21T10:27:52.985265+0000","last_undegraded":"2024-09-21T10:27:52.985265+0000","last_fullsized":"2024-09-21T10:27:52.985265+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:10:30.641216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":71,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.925429+0000","last_change":"2024-09-21T10:26:57.033494+0000","last_active":"2024-09-21T10:27:52.925429+0000","last_peered":"2024-09-21T10:27:52.925429+0000","last_clean":"2024-09-21T10:27:52.925429+0000","last_became_active":"2024-09-21T10:26:56.957867+0000","last_became_peered":"2024-09-21T10:26:56.957867+0000","last_unstale":"2024-09-21T10:27:52.925429+0000","last_undegraded":"2024-09-21T10:27:52.925429+0000","last_fullsized":"2024-09-21T10:27:52.925429+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:32.453055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1320,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924450+0000","last_change":"2024-09-21T10:26:04.289253+0000","last_active":"2024-09-21T10:27:51.924450+0000","last_peered":"2024-09-21T10:27:51.924450+0000","last_clean":"2024-09-21T10:27:51.924450+0000","last_became_active":"2024-09-21T10:26:04.288861+0000","last_became_peered":"2024-09-21T10:26:04.288861+0000","last_unstale":"2024-09-21T10:27:51.924450+0000","last_undegraded":"2024-09-21T10:27:51.924450+0000","last_fullsized":"2024-09-21T10:27:51.924450+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:38.402224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1873,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927423+0000","last_change":"2024-09-21T10:27:19.327581+0000","last_active":"2024-09-21T10:27:52.927423+0000","last_peered":"2024-09-21T10:27:52.927423+0000","last_clean":"2024-09-21T10:27:52.927423+0000","last_became_active":"2024-09-21T10:27:19.327252+0000","last_became_peered":"2024-09-21T10:27:19.327252+0000","last_unstale":"2024-09-21T10:27:52.927423+0000","last_undegraded":"2024-09-21T10:27:52.927423+0000","last_fullsized":"2024-09-21T10:27:52.927423+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:46.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1777,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.984068+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:27:52.984068+0000","last_peered":"2024-09-21T10:27:52.984068+0000","last_clean":"2024-09-21T10:27:52.984068+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:27:52.984068+0000","last_undegraded":"2024-09-21T10:27:52.984068+0000","last_fullsized":"2024-09-21T10:27:52.984068+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1196,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921888+0000","last_change":"2024-09-21T10:27:19.329566+0000","last_active":"2024-09-21T10:27:50.921888+0000","last_peered":"2024-09-21T10:27:50.921888+0000","last_clean":"2024-09-21T10:27:50.921888+0000","last_became_active":"2024-09-21T10:27:19.329307+0000","last_became_peered":"2024-09-21T10:27:19.329307+0000","last_unstale":"2024-09-21T10:27:50.921888+0000","last_undegraded":"2024-09-21T10:27:50.921888+0000","last_fullsized":"2024-09-21T10:27:50.921888+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:59.305906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1149,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921735+0000","last_change":"2024-09-21T10:27:19.326055+0000","last_active":"2024-09-21T10:27:50.921735+0000","last_peered":"2024-09-21T10:27:50.921735+0000","last_clean":"2024-09-21T10:27:50.921735+0000","last_became_active":"2024-09-21T10:27:19.324778+0000","last_became_peered":"2024-09-21T10:27:19.324778+0000","last_unstale":"2024-09-21T10:27:50.921735+0000","last_undegraded":"2024-09-21T10:27:50.921735+0000","last_fullsized":"2024-09-21T10:27:50.921735+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:12.572967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1156,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921684+0000","last_change":"2024-09-21T10:27:20.323254+0000","last_active":"2024-09-21T10:27:50.921684+0000","last_peered":"2024-09-21T10:27:50.921684+0000","last_clean":"2024-09-21T10:27:50.921684+0000","last_became_active":"2024-09-21T10:27:20.322751+0000","last_became_peered":"2024-09-21T10:27:20.322751+0000","last_unstale":"2024-09-21T10:27:50.921684+0000","last_undegraded":"2024-09-21T10:27:50.921684+0000","last_fullsized":"2024-09-21T10:27:50.921684+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.909953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1185,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925234+0000","last_change":"2024-09-21T10:27:19.326938+0000","last_active":"2024-09-21T10:27:51.925234+0000","last_peered":"2024-09-21T10:27:51.925234+0000","last_clean":"2024-09-21T10:27:51.925234+0000","last_became_active":"2024-09-21T10:27:19.326640+0000","last_became_peered":"2024-09-21T10:27:19.326640+0000","last_unstale":"2024-09-21T10:27:51.925234+0000","last_undegraded":"2024-09-21T10:27:51.925234+0000","last_fullsized":"2024-09-21T10:27:51.925234+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:24.413432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1196,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921553+0000","last_change":"2024-09-21T10:27:19.332150+0000","last_active":"2024-09-21T10:27:50.921553+0000","last_peered":"2024-09-21T10:27:50.921553+0000","last_clean":"2024-09-21T10:27:50.921553+0000","last_became_active":"2024-09-21T10:27:19.331901+0000","last_became_peered":"2024-09-21T10:27:19.331901+0000","last_unstale":"2024-09-21T10:27:50.921553+0000","last_undegraded":"2024-09-21T10:27:50.921553+0000","last_fullsized":"2024-09-21T10:27:50.921553+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:07.758296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1405,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.927329+0000","last_change":"2024-09-21T10:08:06.530040+0000","last_active":"2024-09-21T10:27:52.927329+0000","last_peered":"2024-09-21T10:27:52.927329+0000","last_clean":"2024-09-21T10:27:52.927329+0000","last_became_active":"2024-09-21T10:08:06.529292+0000","last_became_peered":"2024-09-21T10:08:06.529292+0000","last_unstale":"2024-09-21T10:27:52.927329+0000","last_undegraded":"2024-09-21T10:27:52.927329+0000","last_fullsized":"2024-09-21T10:27:52.927329+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:15.702634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1896,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.979270+0000","last_change":"2024-09-21T10:27:19.370824+0000","last_active":"2024-09-21T10:27:51.979270+0000","last_peered":"2024-09-21T10:27:51.979270+0000","last_clean":"2024-09-21T10:27:51.979270+0000","last_became_active":"2024-09-21T10:27:19.370477+0000","last_became_peered":"2024-09-21T10:27:19.370477+0000","last_unstale":"2024-09-21T10:27:51.979270+0000","last_undegraded":"2024-09-21T10:27:51.979270+0000","last_fullsized":"2024-09-21T10:27:51.979270+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:17.020888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1214,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.984163+0000","last_change":"2024-09-21T10:26:56.956218+0000","last_active":"2024-09-21T10:27:52.984163+0000","last_peered":"2024-09-21T10:27:52.984163+0000","last_clean":"2024-09-21T10:27:52.984163+0000","last_became_active":"2024-09-21T10:26:56.955902+0000","last_became_peered":"2024-09-21T10:26:56.955902+0000","last_unstale":"2024-09-21T10:27:52.984163+0000","last_undegraded":"2024-09-21T10:27:52.984163+0000","last_fullsized":"2024-09-21T10:27:52.984163+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:08.908926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1167,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925093+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:27:51.925093+0000","last_peered":"2024-09-21T10:27:51.925093+0000","last_clean":"2024-09-21T10:27:51.925093+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:27:51.925093+0000","last_undegraded":"2024-09-21T10:27:51.925093+0000","last_fullsized":"2024-09-21T10:27:51.925093+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1202,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924999+0000","last_change":"2024-09-21T10:26:57.016488+0000","last_active":"2024-09-21T10:27:51.924999+0000","last_peered":"2024-09-21T10:27:51.924999+0000","last_clean":"2024-09-21T10:27:51.924999+0000","last_became_active":"2024-09-21T10:26:56.955144+0000","last_became_peered":"2024-09-21T10:26:56.955144+0000","last_unstale":"2024-09-21T10:27:51.924999+0000","last_undegraded":"2024-09-21T10:27:51.924999+0000","last_fullsized":"2024-09-21T10:27:51.924999+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:29.887586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1192,"reported_epoch":1227,"state":"active+clean","last_fresh":"2024-09-21T10:27:50.921801+0000","last_change":"2024-09-21T10:27:19.327246+0000","last_active":"2024-09-21T10:27:50.921801+0000","last_peered":"2024-09-21T10:27:50.921801+0000","last_clean":"2024-09-21T10:27:50.921801+0000","last_became_active":"2024-09-21T10:27:19.326343+0000","last_became_peered":"2024-09-21T10:27:19.326343+0000","last_unstale":"2024-09-21T10:27:50.921801+0000","last_undegraded":"2024-09-21T10:27:50.921801+0000","last_fullsized":"2024-09-21T10:27:50.921801+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:37.654823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1206,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925473+0000","last_change":"2024-09-21T10:27:19.324762+0000","last_active":"2024-09-21T10:27:51.925473+0000","last_peered":"2024-09-21T10:27:51.925473+0000","last_clean":"2024-09-21T10:27:51.925473+0000","last_became_active":"2024-09-21T10:27:19.324341+0000","last_became_peered":"2024-09-21T10:27:19.324341+0000","last_unstale":"2024-09-21T10:27:51.925473+0000","last_undegraded":"2024-09-21T10:27:51.925473+0000","last_fullsized":"2024-09-21T10:27:51.925473+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:42.722645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1188,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924626+0000","last_change":"2024-09-21T10:27:20.322710+0000","last_active":"2024-09-21T10:27:51.924626+0000","last_peered":"2024-09-21T10:27:51.924626+0000","last_clean":"2024-09-21T10:27:51.924626+0000","last_became_active":"2024-09-21T10:27:20.322390+0000","last_became_peered":"2024-09-21T10:27:20.322390+0000","last_unstale":"2024-09-21T10:27:51.924626+0000","last_undegraded":"2024-09-21T10:27:51.924626+0000","last_fullsized":"2024-09-21T10:27:51.924626+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:56.725786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1205,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.925036+0000","last_change":"2024-09-21T10:27:19.331984+0000","last_active":"2024-09-21T10:27:51.925036+0000","last_peered":"2024-09-21T10:27:51.925036+0000","last_clean":"2024-09-21T10:27:51.925036+0000","last_became_active":"2024-09-21T10:27:19.331649+0000","last_became_peered":"2024-09-21T10:27:19.331649+0000","last_unstale":"2024-09-21T10:27:51.925036+0000","last_undegraded":"2024-09-21T10:27:51.925036+0000","last_fullsized":"2024-09-21T10:27:51.925036+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:33:27.695066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1204,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924927+0000","last_change":"2024-09-21T10:26:57.031546+0000","last_active":"2024-09-21T10:27:51.924927+0000","last_peered":"2024-09-21T10:27:51.924927+0000","last_clean":"2024-09-21T10:27:51.924927+0000","last_became_active":"2024-09-21T10:26:56.953044+0000","last_became_peered":"2024-09-21T10:26:56.953044+0000","last_unstale":"2024-09-21T10:27:51.924927+0000","last_undegraded":"2024-09-21T10:27:51.924927+0000","last_fullsized":"2024-09-21T10:27:51.924927+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:33:36.912858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1767,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.986384+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:27:52.986384+0000","last_peered":"2024-09-21T10:27:52.986384+0000","last_clean":"2024-09-21T10:27:52.986384+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:27:52.986384+0000","last_undegraded":"2024-09-21T10:27:52.986384+0000","last_fullsized":"2024-09-21T10:27:52.986384+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1214,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924600+0000","last_change":"2024-09-21T10:26:04.187372+0000","last_active":"2024-09-21T10:27:51.924600+0000","last_peered":"2024-09-21T10:27:51.924600+0000","last_clean":"2024-09-21T10:27:51.924600+0000","last_became_active":"2024-09-21T10:26:04.186026+0000","last_became_peered":"2024-09-21T10:26:04.186026+0000","last_unstale":"2024-09-21T10:27:51.924600+0000","last_undegraded":"2024-09-21T10:27:51.924600+0000","last_fullsized":"2024-09-21T10:27:51.924600+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:55.130410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1885,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.980140+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:27:51.980140+0000","last_peered":"2024-09-21T10:27:51.980140+0000","last_clean":"2024-09-21T10:27:51.980140+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:27:51.980140+0000","last_undegraded":"2024-09-21T10:27:51.980140+0000","last_fullsized":"2024-09-21T10:27:51.980140+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1884,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.979386+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:27:51.979386+0000","last_peered":"2024-09-21T10:27:51.979386+0000","last_clean":"2024-09-21T10:27:51.979386+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:27:51.979386+0000","last_undegraded":"2024-09-21T10:27:51.979386+0000","last_fullsized":"2024-09-21T10:27:51.979386+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":639,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924863+0000","last_change":"2024-09-21T10:26:56.955563+0000","last_active":"2024-09-21T10:27:51.924863+0000","last_peered":"2024-09-21T10:27:51.924863+0000","last_clean":"2024-09-21T10:27:51.924863+0000","last_became_active":"2024-09-21T10:26:56.955148+0000","last_became_peered":"2024-09-21T10:26:56.955148+0000","last_unstale":"2024-09-21T10:27:51.924863+0000","last_undegraded":"2024-09-21T10:27:51.924863+0000","last_fullsized":"2024-09-21T10:27:51.924863+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:59.655293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1219,"reported_epoch":1228,"state":"active+clean","last_fresh":"2024-09-21T10:27:51.924690+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:27:51.924690+0000","last_peered":"2024-09-21T10:27:51.924690+0000","last_clean":"2024-09-21T10:27:51.924690+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:27:51.924690+0000","last_undegraded":"2024-09-21T10:27:51.924690+0000","last_fullsized":"2024-09-21T10:27:51.924690+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1776,"reported_epoch":1229,"state":"active+clean","last_fresh":"2024-09-21T10:27:52.985653+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:27:52.985653+0000","last_peered":"2024-09-21T10:27:52.985653+0000","last_clean":"2024-09-21T10:27:52.985653+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:27:52.985653+0000","last_undegraded":"2024-09-21T10:27:52.985653+0000","last_fullsized":"2024-09-21T10:27:52.985653+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":45,"num_pg":16,"stat_sum":{"num_bytes":461701144,"num_objects":7046,"num_object_clones":0,"num_object_copies":21138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7046,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253290,"num_write_kb":8330689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23670,"num_bytes_recovered":672268288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1733177344,"data_stored":1733165128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121478,"ondisk_log_size":121478,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":306,"num_read_kb":759,"num_write":149,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":764,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397900,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":564736,"kb_used_data":254896,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93180416,"statfs":{"total":95995035648,"available":95416745984,"internally_reserved":0,"allocated":261013504,"data_stored":256126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049560,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":533756,"kb_used_data":189364,"kb_used_omap":242,"kb_used_meta":344077,"kb_avail":93211396,"statfs":{"total":95995035648,"available":95448469504,"internally_reserved":0,"allocated":193908736,"data_stored":189018229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":352335071},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":158913790358,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395464,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":440024,"kb_used_data":124808,"kb_used_omap":224,"kb_used_meta":314975,"kb_avail":93305128,"statfs":{"total":95995035648,"available":95544451072,"internally_reserved":0,"allocated":127803392,"data_stored":122938871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":322535305},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644518,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":529644,"kb_used_data":294880,"kb_used_omap":67,"kb_used_meta":234684,"kb_avail":93215508,"statfs":{"total":95995035648,"available":95452680192,"internally_reserved":0,"allocated":301957120,"data_stored":297081318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":240317277},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527536,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":465452,"kb_used_data":262812,"kb_used_omap":75,"kb_used_meta":202548,"kb_avail":93279700,"statfs":{"total":95995035648,"available":95518412800,"internally_reserved":0,"allocated":269119488,"data_stored":264248226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":207409306},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274689,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":360232,"kb_used_data":184160,"kb_used_omap":51,"kb_used_meta":176012,"kb_avail":93384920,"statfs":{"total":95995035648,"available":95626158080,"internally_reserved":0,"allocated":188579840,"data_stored":183704005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":180236629},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353927,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":426408,"kb_used_data":259864,"kb_used_omap":66,"kb_used_meta":166461,"kb_avail":93318744,"statfs":{"total":95995035648,"available":95558393856,"internally_reserved":0,"allocated":266100736,"data_stored":261225178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":170456388},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":252055552,"data_stored":252051480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":157745152,"data_stored":157745152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":240517120,"data_stored":240517120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":275845120,"data_stored":275841048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":110100480,"data_stored":110100480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":302514176,"data_stored":302514176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":162402304,"data_stored":162398232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":45,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":231997440,"data_stored":231997440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:27:59.537 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:27:59.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T10:27:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:59.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:27:59.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:27:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.702 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:59.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:27:59.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:27:59.828 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:27:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:27:59.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:27:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:27:59.971 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:27:59.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:00.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:00.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:00.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:00.249 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:00.255 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:27:59 smithi145 ceph-mon[89359]: from='client.46085 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:28:00.256 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:28:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:28:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:00.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:27:59 smithi119 ceph-mon[105288]: from='client.46085 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:28:00.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:27:59 smithi119 ceph-mon[101432]: from='client.46085 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:28:00.352 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:00.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:00.442 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:00.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:00.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:00.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:00.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:00.675 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:00.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:00.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:00.902 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:00.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:00.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:01.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:01.076 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:01.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:01.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:01.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:00 smithi145 ceph-mon[89359]: pgmap v1772: 89 pgs: 89 active+clean; 290 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 3.9 MiB/s, 182 objects/s recovering 2024-09-21T10:28:01.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:01.342 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:01.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:00 smithi119 ceph-mon[105288]: pgmap v1772: 89 pgs: 89 active+clean; 290 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 3.9 MiB/s, 182 objects/s recovering 2024-09-21T10:28:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:00 smithi119 ceph-mon[101432]: pgmap v1772: 89 pgs: 89 active+clean; 290 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 735 op/s; 3.9 MiB/s, 182 objects/s recovering 2024-09-21T10:28:01.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:01.470 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:01.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:01.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:01.699 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:01.728 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:01.829 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:01.852 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:01.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:01.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:01.997 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:28:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:02.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:02.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:02.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:02.284 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:02.289 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:02.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1723256843' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1723256843' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1723256843' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:02.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:02.428 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:02.531 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:02.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:02.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:02.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:02.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:02.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:02.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:02.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:02.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:03.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:03.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:03.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:03.238 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[105288]: pgmap v1773: 89 pgs: 89 active+clean; 240 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 3.2 MiB/s, 151 objects/s recovering 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1723256843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[105288]: osdmap e1230: 8 total, 8 up, 7 in 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[101432]: pgmap v1773: 89 pgs: 89 active+clean; 240 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 3.2 MiB/s, 151 objects/s recovering 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1723256843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[101432]: osdmap e1230: 8 total, 8 up, 7 in 2024-09-21T10:28:03.239 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:03.240 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:02 smithi145 ceph-mon[89359]: pgmap v1773: 89 pgs: 89 active+clean; 240 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 696 op/s; 3.2 MiB/s, 151 objects/s recovering 2024-09-21T10:28:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:02 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1723256843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:02 smithi145 ceph-mon[89359]: osdmap e1230: 8 total, 8 up, 7 in 2024-09-21T10:28:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:03.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:03.353 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Removed 130167 objects 2024-09-21T10:28:03.353 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Clean up completed and total clean up time :175.805 2024-09-21T10:28:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.370 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:03.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:03.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:03.493 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:03.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:03.615 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:03.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:03.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:03.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:03.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:03.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:03.986 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:03.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:03.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:04.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:04.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:04.231 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:04.333 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:04.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:04.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:04.477 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:04.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1701315539' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:04.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1701315539' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1701315539' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:04.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:04.602 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:04.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:04.725 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:04.827 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:04.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.849 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:28:04] "GET /metrics HTTP/1.1" 200 36860 "" "Prometheus/2.51.0" 2024-09-21T10:28:04.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:04.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:05.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:05.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:05.073 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:05.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:05.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:05.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:05.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:05.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:05.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:05 smithi119 ceph-mon[105288]: pgmap v1775: 89 pgs: 89 active+clean; 127 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:28:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:05 smithi119 ceph-mon[101432]: pgmap v1775: 89 pgs: 89 active+clean; 127 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:28:05.448 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:05.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:05 smithi145 ceph-mon[89359]: pgmap v1775: 89 pgs: 89 active+clean; 127 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 745 op/s; 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:28:05.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:05.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:05.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:05.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:05.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:05.818 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:05.840 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:05.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:05.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:05.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:06.065 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:06.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:06.189 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:06.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:06.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:06.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:06.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:06.454 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:06.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:06.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:06.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:06.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:06.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:06.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:06.950 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:06.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:06.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:28:07.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:07.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:07.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:07.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:07.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:07.331 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:07 smithi119 ceph-mon[101432]: pgmap v1776: 89 pgs: 89 active+clean; 37 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T10:28:07.331 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:07 smithi119 ceph-mon[105288]: pgmap v1776: 89 pgs: 89 active+clean; 37 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T10:28:07.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:07.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:07 smithi145 ceph-mon[89359]: pgmap v1776: 89 pgs: 89 active+clean; 37 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 781 op/s 2024-09-21T10:28:07.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:07.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:07.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:07.703 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:07.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:07.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:07.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:07.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:07.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:08.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:28:08.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:08.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155681 45.c 45:39a24039:::benchmark_data_smithi119_168706_object25511:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:28:08.125 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.021899+0000", 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 152.09002651899999, 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.032164187, 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155681 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021899+0000", 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021899+0000", 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021901+0000", 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6559999999999999e-06 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021906+0000", 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8999999999999997e-06 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021918+0000", 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2607e-05 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021926+0000", 2024-09-21T10:28:08.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021976+0000", 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9929999999999998e-05 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.021982+0000", 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.251e-06 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.025827+0000", 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038450310000000001 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.025837+0000", 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7419999999999992e-06 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.051542+0000", 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0257053309999999 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.054063+0000", 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025211320000000001 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155699 45.c 45:3be5d6e0:::benchmark_data_smithi119_168706_object25529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.036739+0000", 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 152.07518696899999, 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.034692978, 2024-09-21T10:28:08.131 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155699 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.036739+0000", 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.036739+0000", 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.036740+0000", 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6559999999999999e-06 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.036745+0000", 2024-09-21T10:28:08.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1039999999999998e-06 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.036760+0000", 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4309e-05 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.036767+0000", 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.357e-06 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.037024+0000", 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025657499999999999 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.037044+0000", 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0431000000000001e-05 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.054445+0000", 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.017401314 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.055363+0000", 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00091751699999999996 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 7", 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.055474+0000", 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110543 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.064623+0000", 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0091491360000000004 2024-09-21T10:28:08.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.071146+0000", 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0065230920000000003 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.071243+0000", 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6992999999999998e-05 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.071432+0000", 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000188951 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178869 45.c 45:387aa2e4:::benchmark_data_smithi119_168706_object48699:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1192)", 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:04.174898+0000", 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 123.937027523, 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.144516316, 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.138 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178869 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174898+0000", 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174898+0000", 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174900+0000", 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4699999999999999e-06 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174904+0000", 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3499999999999999e-06 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174915+0000", 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1351e-05 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174935+0000", 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9712e-05 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174984+0000", 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8504999999999997e-05 2024-09-21T10:28:08.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174995+0000", 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1141999999999999e-05 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.290206+0000", 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1152118070000001 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for missing object", 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.290320+0000", 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011344399999999999 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.319314+0000", 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.028994418000000001 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.319414+0000", 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100117 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178895 45.c 45:3f295f1e:::benchmark_data_smithi119_168706_object48725:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1192)", 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:04.174951+0000", 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 123.936974202, 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.137615995, 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178895 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174951+0000", 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174951+0000", 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174952+0000", 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5799999999999999e-07 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174955+0000", 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7240000000000001e-06 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174962+0000", 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3989999999999998e-06 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174968+0000", 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.122e-06 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175021+0000", 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2339999999999997e-05 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175028+0000", 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7279999999999995e-06 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.290540+0000", 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.115511516 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.290653+0000", 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112642 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.290787+0000", 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013467500000000001 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.305156+0000", 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014369101 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.306037+0000", 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00088072400000000005 2024-09-21T10:28:08.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.312297+0000", 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0062595039999999999 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.312360+0000", 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3202999999999998e-05 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.312567+0000", 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020775899999999999 2024-09-21T10:28:08.150 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178906 45.c 45:3824badb:::benchmark_data_smithi119_168706_object48736:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1192)", 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:04.174977+0000", 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 123.93694911, 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.144574266, 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.151 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178906 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174977+0000", 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174977+0000", 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174977+0000", 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0600000000000001e-07 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174979+0000", 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1150000000000001e-06 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174986+0000", 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4269999999999996e-06 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174991+0000", 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1220000000000003e-06 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175054+0000", 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3380000000000006e-05 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175082+0000", 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8138000000000001e-05 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.295389+0000", 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1203068840000001 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.295478+0000", 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9136999999999997e-05 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.295555+0000", 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6688999999999997e-05 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.305165+0000", 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0096103769999999998 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.312611+0000", 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074458930000000003 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.319439+0000", 2024-09-21T10:28:08.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068280119999999996 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.319472+0000", 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2480000000000001e-05 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.319551+0000", 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9206e-05 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178908 45.c 45:39ed3bd0:::benchmark_data_smithi119_168706_object48738:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1192)", 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:04.174997+0000", 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 123.936928521, 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1507344559999999, 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178908 2024-09-21T10:28:08.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174997+0000", 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174997+0000", 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174998+0000", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.77e-07 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.174999+0000", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8810000000000001e-06 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175006+0000", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3069999999999999e-06 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175011+0000", 2024-09-21T10:28:08.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9300000000000002e-06 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175117+0000", 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106733 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175125+0000", 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5179999999999998e-06 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.300116+0000", 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1249913540000001 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.300247+0000", 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013091200000000001 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.300380+0000", 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013256699999999999 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.305209+0000", 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048292830000000002 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.312702+0000", 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0074926080000000004 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.325482+0000", 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012780039999999999 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.325538+0000", 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6187000000000003e-05 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.325732+0000", 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000193759 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178909 45.9 45:93c58d8b:::benchmark_data_smithi119_168706_object48739:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1192)", 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:04.175017+0000", 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 123.936909083, 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.154438012, 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178909 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175017+0000", 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175017+0000", 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175017+0000", 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9000000000000002e-07 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175019+0000", 2024-09-21T10:28:08.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9889999999999999e-06 2024-09-21T10:28:08.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175025+0000", 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.268e-06 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175030+0000", 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7559999999999999e-06 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175250+0000", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00022041399999999999 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.175274+0000", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3750000000000001e-05 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.295098+0000", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1198237129999999 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.298367+0000", 2024-09-21T10:28:08.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003269026 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.298493+0000", 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012618500000000001 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.311756+0000", 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013263059000000001 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.316260+0000", 2024-09-21T10:28:08.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045042290000000002 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.329174+0000", 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012913367 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.329240+0000", 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6268999999999999e-05 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.329455+0000", 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021459699999999999 2024-09-21T10:28:08.171 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:243863 45.4 45:2d786605:::benchmark_data_smithi119_168706_object113693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1220)", 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.516633+0000", 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.595292601000001, 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15719904900000001, 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.172 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 243863 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.516633+0000", 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.516633+0000", 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.516634+0000", 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0999999999999997e-07 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.516638+0000", 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1160000000000001e-06 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.516649+0000", 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.094e-05 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.516656+0000", 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7150000000000001e-06 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.524488+0000", 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078320219999999992 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.524587+0000", 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9350000000000003e-05 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5,7", 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.524671+0000", 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3526e-05 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530069+0000", 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0053984300000000001 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.533518+0000", 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003448764 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.533551+0000", 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3009999999999997e-05 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673648+0000", 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.140097324 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673703+0000", 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4404999999999998e-05 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673832+0000", 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012953699999999999 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243867 45.2 e1220/1196 45:49bb5890:::benchmark_data_smithi119_168706_object113697:head v 1220'15205, mlcod=1220'15203)", 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.526041+0000", 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.585885172000001, 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15638418300000001, 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.178 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526041+0000", 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526041+0000", 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526041+0000", 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0200000000000001e-07 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526050+0000", 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1390000000000001e-06 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526061+0000", 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0202e-05 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526067+0000", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7649999999999996e-06 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526228+0000", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000160707 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526246+0000", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8332999999999999e-05 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.682347+0000", 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15610020799999999 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.682425+0000", 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8027000000000004e-05 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.182 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243862 45.9 e1220/1212 45:93dd9c61:::benchmark_data_smithi119_168706_object113692:head v 1220'15553, mlcod=1220'15549)", 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.526096+0000", 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.585829796999999, 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.161246114, 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526096+0000", 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526096+0000", 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526096+0000", 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8200000000000001e-07 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526100+0000", 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2509999999999999e-06 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526106+0000", 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4489999999999996e-06 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.526110+0000", 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1710000000000004e-06 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.527893+0000", 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017826809999999999 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.527913+0000", 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0091000000000001e-05 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.687271+0000", 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15935812899999999 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.687342+0000", 2024-09-21T10:28:08.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0959999999999998e-05 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243868 45.9 e1220/1212 45:9408b879:::benchmark_data_smithi119_168706_object113698:head v 1220'15554, mlcod=1220'15551)", 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.529888+0000", 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.582037393, 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15750466799999999, 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.187 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529888+0000", 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529888+0000", 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529889+0000", 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7899999999999997e-07 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529895+0000", 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4060000000000004e-06 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529906+0000", 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1586000000000001e-05 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529913+0000", 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.759e-06 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.533086+0000", 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003172949 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.533102+0000", 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5931e-05 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.687344+0000", 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15424255000000001 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.687393+0000", 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8607999999999999e-05 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243870 45.2 e1220/1196 45:4e4a7846:::benchmark_data_smithi119_168706_object113700:head v 1220'15206, mlcod=1220'15203)", 2024-09-21T10:28:08.191 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.529973+0000", 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.58195302, 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15251404099999999, 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529973+0000", 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529973+0000", 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529973+0000", 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9400000000000001e-07 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529976+0000", 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0939999999999999e-06 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529982+0000", 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2020000000000001e-06 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529987+0000", 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2449999999999998e-06 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532437+0000", 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024491669999999999 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532477+0000", 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0500000000000002e-05 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.682434+0000", 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14995717 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.682487+0000", 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2369e-05 2024-09-21T10:28:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243864 45.3 e1220/1164 45:c0b6e3b7:::benchmark_data_smithi119_168706_object113694:head v 1220'15205, mlcod=1220'15202)", 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.529986+0000", 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.581939201000001, 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.143956942, 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529986+0000", 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529986+0000", 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529987+0000", 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6499999999999999e-07 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.529994+0000", 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7449999999999998e-06 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530005+0000", 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0959e-05 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530011+0000", 2024-09-21T10:28:08.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0739999999999998e-06 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.531538+0000", 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001527287 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.531557+0000", 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8499e-05 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673869+0000", 2024-09-21T10:28:08.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14231259900000001 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673943+0000", 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4214000000000005e-05 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243865 45.3 e1220/1164 45:cb854d19:::benchmark_data_smithi119_168706_object113695:head v 1220'15206, mlcod=1220'15202)", 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.530030+0000", 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.581895974999998, 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16567726499999999, 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.200 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530030+0000", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530030+0000", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530030+0000", 2024-09-21T10:28:08.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.48e-07 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530034+0000", 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2350000000000001e-06 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530040+0000", 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6099999999999997e-06 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530044+0000", 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6059999999999996e-06 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673973+0000", 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14392822699999999 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.674382+0000", 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000409584 2024-09-21T10:28:08.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.695659+0000", 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021277161999999999 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.695707+0000", 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7593000000000003e-05 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.204 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243869 45.5 e1220/1164 45:a1000f67:::benchmark_data_smithi119_168706_object113699:head v 1220'14701, mlcod=1220'14698)", 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.530065+0000", 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.581860348999999, 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14384502800000001, 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530065+0000", 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530065+0000", 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530066+0000", 2024-09-21T10:28:08.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7700000000000001e-07 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530069+0000", 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.439e-06 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530075+0000", 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3770000000000001e-06 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530080+0000", 2024-09-21T10:28:08.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.143e-06 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530169+0000", 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9617000000000006e-05 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.530187+0000", 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8236e-05 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673688+0000", 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.143500351 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673910+0000", 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00022258799999999999 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243871 45.9 e1220/1212 45:970afa64:::benchmark_data_smithi119_168706_object113701:head v 1220'15555, mlcod=1220'15551)", 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.532374+0000", 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.579551662, 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15506377499999999, 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532374+0000", 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532374+0000", 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532374+0000", 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.46e-07 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532379+0000", 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9690000000000002e-06 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532387+0000", 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0059999999999992e-06 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.532393+0000", 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1150000000000002e-06 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673592+0000", 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14119988 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673613+0000", 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0749000000000001e-05 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.687395+0000", 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013782054 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.687438+0000", 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2555999999999997e-05 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.213 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:243877 45.2 e1220/1196 45:42cc6825:::benchmark_data_smithi119_168706_object113707:head v 1220'15207, mlcod=1220'15203)", 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:39.535070+0000", 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.576856082999999, 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15084439399999999, 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.535070+0000", 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.535070+0000", 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.535070+0000", 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6899999999999996e-07 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.535076+0000", 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3190000000000002e-06 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673453+0000", 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.13837691799999999 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673468+0000", 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4909000000000001e-05 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673589+0000", 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012175099999999999 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.673940+0000", 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00035108399999999999 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.685823+0000", 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011883046 2024-09-21T10:28:08.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:39.685914+0000", 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0598000000000002e-05 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:244355 45.4 45:2a9a18b5:::benchmark_data_smithi119_168706_object114185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1220)", 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:40.245005+0000", 2024-09-21T10:28:08.218 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 27.866920314000001, 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.156904984, 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 244355 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.245005+0000", 2024-09-21T10:28:08.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.245005+0000", 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.245007+0000", 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.358e-06 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.245011+0000", 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0040000000000001e-06 2024-09-21T10:28:08.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.245021+0000", 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.027e-05 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.245027+0000", 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3409999999999999e-06 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.251883+0000", 2024-09-21T10:28:08.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068560180000000002 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.252012+0000", 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012887400000000001 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5,7", 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.252141+0000", 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129071 2024-09-21T10:28:08.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256383+0000", 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042419190000000002 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256421+0000", 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7492e-05 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.257349+0000", 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00092789800000000003 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.258499+0000", 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011502699999999999 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.258530+0000", 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0775000000000003e-05 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.401910+0000", 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.143380694 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:244365 45.2 e1220/1196 45:4c92c37b:::benchmark_data_smithi119_168706_object114195:head v 1220'15237, mlcod=1220'15235)", 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:40.256893+0000", 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 27.855032366, 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14560142200000001, 2024-09-21T10:28:08.225 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256893+0000", 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256893+0000", 2024-09-21T10:28:08.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256894+0000", 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5300000000000004e-07 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256899+0000", 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7700000000000001e-06 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256907+0000", 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9349999999999994e-06 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256911+0000", 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2089999999999999e-06 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256951+0000", 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0451999999999998e-05 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.256964+0000", 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.269e-05 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.402448+0000", 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14548443 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.402495+0000", 2024-09-21T10:28:08.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6383e-05 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.35976.0:244366 45.3 e1220/1164 45:cbe11682:::benchmark_data_smithi119_168706_object114196:head v 1220'15238, mlcod=1220'15236)", 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:40.258803+0000", 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 27.853123007000001, 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15491401799999999, 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:08.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.258803+0000", 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.258803+0000", 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.258803+0000", 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1200000000000003e-07 2024-09-21T10:28:08.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.258808+0000", 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7310000000000001e-06 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.401265+0000", 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.14245685899999999 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.401276+0000", 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1398e-05 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.401335+0000", 2024-09-21T10:28:08.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8825e-05 2024-09-21T10:28:08.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:08.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.401357+0000", 2024-09-21T10:28:08.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1690000000000001e-05 2024-09-21T10:28:08.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.413686+0000", 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012329582 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:08.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:40.413717+0000", 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0321e-05 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:08.236 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:28:08.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:08.238 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 1 pg_num to 11 2024-09-21T10:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:08.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:08 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3104786748' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:28:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:08 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3104786748' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:28:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:08.482 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:08 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3104786748' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-21T10:28:08.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:08.632 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:08.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:08.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:08.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:08.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:08.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:09.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:09.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:09.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:09.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:09.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:09.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:09.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:09.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:09.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:09 smithi145 ceph-mon[89359]: pgmap v1777: 89 pgs: 89 active+clean; 5.3 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 696 op/s 2024-09-21T10:28:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3104786748' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:28:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:09 smithi145 ceph-mon[89359]: osdmap e1231: 8 total, 8 up, 7 in 2024-09-21T10:28:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:09 smithi119 ceph-mon[101432]: pgmap v1777: 89 pgs: 89 active+clean; 5.3 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 696 op/s 2024-09-21T10:28:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3104786748' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:28:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:09 smithi119 ceph-mon[101432]: osdmap e1231: 8 total, 8 up, 7 in 2024-09-21T10:28:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:09 smithi119 ceph-mon[105288]: pgmap v1777: 89 pgs: 89 active+clean; 5.3 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 696 op/s 2024-09-21T10:28:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3104786748' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T10:28:09.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:09 smithi119 ceph-mon[105288]: osdmap e1231: 8 total, 8 up, 7 in 2024-09-21T10:28:09.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:09.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:09.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:09.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:09.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:09.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:09.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:09.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:10.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:10.124 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:10.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:10.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:10.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:10.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:10.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:10.408 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:10.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:10.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:10.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:28:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:28:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:28:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:28:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-21T10:28:10.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.665 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:10.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:10.764 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-21T10:28:10.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-21T10:28:10.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:10.789 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:10.799 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:10.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:10.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:10.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:11.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:11.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:11.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:11.162 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:11.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:11.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:11.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:11.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:11.408 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:11.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:11.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:11 smithi145 ceph-mon[89359]: pgmap v1779: 89 pgs: 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 479 op/s 2024-09-21T10:28:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:28:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:11 smithi145 ceph-mon[89359]: osdmap e1232: 8 total, 8 up, 7 in 2024-09-21T10:28:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:11 smithi119 ceph-mon[101432]: pgmap v1779: 89 pgs: 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 479 op/s 2024-09-21T10:28:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:28:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:11 smithi119 ceph-mon[101432]: osdmap e1232: 8 total, 8 up, 7 in 2024-09-21T10:28:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:11 smithi119 ceph-mon[105288]: pgmap v1779: 89 pgs: 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 479 op/s 2024-09-21T10:28:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T10:28:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:11 smithi119 ceph-mon[105288]: osdmap e1232: 8 total, 8 up, 7 in 2024-09-21T10:28:11.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:11.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:11.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:11.790 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:11.881 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:28:11.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:11.897 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:11.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:11.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:11.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:12.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:12.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:12.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:12.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:12.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:12.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:12.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:12.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:12.405 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:12 smithi119 ceph-mon[101432]: osdmap e1233: 8 total, 8 up, 7 in 2024-09-21T10:28:12.406 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:12 smithi119 ceph-mon[105288]: osdmap e1233: 8 total, 8 up, 7 in 2024-09-21T10:28:12.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:12.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:12.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:12 smithi145 ceph-mon[89359]: osdmap e1233: 8 total, 8 up, 7 in 2024-09-21T10:28:12.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:12.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:12.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:12.790 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:12.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:12.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:12.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:12.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:13.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:13.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:13.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:13.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:13.242 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:13 smithi119 ceph-mon[101432]: pgmap v1782: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 95 op/s 2024-09-21T10:28:13.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:13.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:13.395 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:13.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:13.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:13 smithi145 ceph-mon[89359]: pgmap v1782: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 95 op/s 2024-09-21T10:28:13.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:13.560 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:13 smithi119 ceph-mon[105288]: pgmap v1782: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 95 op/s 2024-09-21T10:28:13.646 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:13.668 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:13.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:13.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:13.797 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:13.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:13.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:13.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:14.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:14.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:14.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:14.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:14.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:14.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:14.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:14.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:14.418 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:14.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:14.543 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:28:14] "GET /metrics HTTP/1.1" 200 36860 "" "Prometheus/2.51.0" 2024-09-21T10:28:14.644 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:14.669 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:14.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:14.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:14.917 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:14.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:14.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:15.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:15.040 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:15.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:15.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:15.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:15.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:15.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:15.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:15.540 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:15 smithi145 ceph-mon[89359]: pgmap v1783: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 12 op/s 2024-09-21T10:28:15.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:15 smithi119 ceph-mon[105288]: pgmap v1783: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 12 op/s 2024-09-21T10:28:15.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:15 smithi119 ceph-mon[101432]: pgmap v1783: 99 pgs: 10 unknown, 89 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 12 op/s 2024-09-21T10:28:15.558 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:28:15.558 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:15.563 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:28:15.564 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:15.564 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:15.580 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:28:15.580 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:15.599 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:28:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:15.616 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:28:15.616 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:15.632 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:28:15.632 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:15.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:15.647 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:28:15.647 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:15.662 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:28:15.663 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:15.667 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:15.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:15.765 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:28:15.765 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:28:15.765 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:28:15.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T10:28:15.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:15.792 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:15.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:15.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:15.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:16.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:16.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:16.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:16.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:16.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:16.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:16.367 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:16.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:16.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:16.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:28:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/505379216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:16.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:28:16.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/505379216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:16.573 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-21T10:28:16.573 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/505379216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:16.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:16.615 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:16.716 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:16.739 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:16.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:16.864 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:16.965 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:16.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:16.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:16.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:17.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:17.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:17.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:17.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:17.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:17.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:17.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:17.460 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:17.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:17.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:17 smithi145 ceph-mon[89359]: pgmap v1784: 99 pgs: 99 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:28:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:17 smithi145 ceph-mon[89359]: osdmap e1234: 8 total, 8 up, 7 in 2024-09-21T10:28:17.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:17.595 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:17 smithi119 ceph-mon[105288]: pgmap v1784: 99 pgs: 99 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:17.595 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:28:17.595 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:17 smithi119 ceph-mon[105288]: osdmap e1234: 8 total, 8 up, 7 in 2024-09-21T10:28:17.596 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:17 smithi119 ceph-mon[101432]: pgmap v1784: 99 pgs: 99 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:17.596 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T10:28:17.596 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:17 smithi119 ceph-mon[101432]: osdmap e1234: 8 total, 8 up, 7 in 2024-09-21T10:28:17.610 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:17.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:17.733 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:17.814 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:17.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.856 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:17.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:17.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:17.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:17.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:18.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:18.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:18.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:18.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:18.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:18.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:18.353 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:18.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:18.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:18.479 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:18 smithi145 ceph-mon[89359]: osdmap e1235: 8 total, 8 up, 7 in 2024-09-21T10:28:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:18.563 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:18 smithi119 ceph-mon[105288]: osdmap e1235: 8 total, 8 up, 7 in 2024-09-21T10:28:18.563 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:18.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:18 smithi119 ceph-mon[101432]: osdmap e1235: 8 total, 8 up, 7 in 2024-09-21T10:28:18.563 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:18.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.603 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:18.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:18.704 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:18.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:18.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:18.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:18.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:18.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:18.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:18.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:19.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:19.108 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:19.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.232 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:19.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:19.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:28:19.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:19.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:19 smithi145 ceph-mon[89359]: pgmap v1787: 99 pgs: 2 activating, 97 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:19.554 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:19 smithi119 ceph-mon[101432]: pgmap v1787: 99 pgs: 2 activating, 97 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:19.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:19 smithi119 ceph-mon[105288]: pgmap v1787: 99 pgs: 2 activating, 97 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:19.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:19.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:19.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:19.729 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:19.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:19.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:19.851 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:28:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:28:19.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:19.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:19.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:20.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:20.097 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:20.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:20.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:20.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:20.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:20.272 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_0' removed 2024-09-21T10:28:20.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:20.368 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:20.470 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:20.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/878720233' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:28:20.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1841973632' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:28:20.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/878720233' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:28:20.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1841973632' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:28:20.491 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:20.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/878720233' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:28:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1841973632' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:28:20.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:28:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:28:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:20.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:20.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:20.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:20.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:20.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:20.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:20.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:20.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:20.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:20.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:20.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:21.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:21.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:21.138 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:28:21.138 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:28:21.138 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-21T10:28:21.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_1 16 2024-09-21T10:28:21.208 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:21.229 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:21.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:28:21.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:21.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[105288]: pgmap v1788: 99 pgs: 2 activating, 97 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-09-21T10:28:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/878720233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:28:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1841973632' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:28:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[105288]: osdmap e1236: 8 total, 8 up, 7 in 2024-09-21T10:28:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[101432]: pgmap v1788: 99 pgs: 2 activating, 97 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-09-21T10:28:21.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:21.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/878720233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:28:21.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1841973632' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:28:21.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:21 smithi119 ceph-mon[101432]: osdmap e1236: 8 total, 8 up, 7 in 2024-09-21T10:28:21.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:21.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:21 smithi145 ceph-mon[89359]: pgmap v1788: 99 pgs: 2 activating, 97 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-09-21T10:28:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:21 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/878720233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:28:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1841973632' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:28:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:21 smithi145 ceph-mon[89359]: osdmap e1236: 8 total, 8 up, 7 in 2024-09-21T10:28:21.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:21.599 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:21.693 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:28:21.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:21.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:21.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:21.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:21.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:21.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:21.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:21.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:22.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:22.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:22.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:22.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:22.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:22.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:22.337 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.3 scrub starts 2024-09-21T10:28:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.3 scrub ok 2024-09-21T10:28:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.4 scrub starts 2024-09-21T10:28:22.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.4 scrub ok 2024-09-21T10:28:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:28:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.0 scrub ok 2024-09-21T10:28:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.a scrub starts 2024-09-21T10:28:22.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[101432]: 1.a scrub ok 2024-09-21T10:28:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.3 scrub starts 2024-09-21T10:28:22.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.3 scrub ok 2024-09-21T10:28:22.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.4 scrub starts 2024-09-21T10:28:22.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.4 scrub ok 2024-09-21T10:28:22.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:28:22.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.0 scrub ok 2024-09-21T10:28:22.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.a scrub starts 2024-09-21T10:28:22.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:22 smithi119 ceph-mon[105288]: 1.a scrub ok 2024-09-21T10:28:22.403 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:22.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:22.460 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.3 scrub starts 2024-09-21T10:28:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.3 scrub ok 2024-09-21T10:28:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.4 scrub starts 2024-09-21T10:28:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.4 scrub ok 2024-09-21T10:28:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:28:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.0 scrub ok 2024-09-21T10:28:22.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.a scrub starts 2024-09-21T10:28:22.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:22 smithi145 ceph-mon[89359]: 1.a scrub ok 2024-09-21T10:28:22.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:22.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:22.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:22.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:22.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:22.828 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:22.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:22.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:22.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:23.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.074 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:23.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:23.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:23.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:23.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:23.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:23.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:23.497 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:23.497 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: pgmap v1790: 83 pgs: 2 activating, 81 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: 1.6 scrub starts 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: 1.6 scrub ok 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: 1.9 scrub starts 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: 1.9 scrub ok 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: 1.5 scrub starts 2024-09-21T10:28:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: 1.5 scrub ok 2024-09-21T10:28:23.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:23.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:23.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:23.585 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: pgmap v1790: 83 pgs: 2 activating, 81 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:23.585 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: 1.6 scrub starts 2024-09-21T10:28:23.585 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: 1.6 scrub ok 2024-09-21T10:28:23.585 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: 1.9 scrub starts 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: 1.9 scrub ok 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: 1.5 scrub starts 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: 1.5 scrub ok 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: pgmap v1790: 83 pgs: 2 activating, 81 active+clean; 582 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: 1.6 scrub starts 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: 1.6 scrub ok 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: 1.9 scrub starts 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: 1.9 scrub ok 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: 1.5 scrub starts 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: 1.5 scrub ok 2024-09-21T10:28:23.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:23.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:23.689 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:23.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:23.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:23.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:23.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:23.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:24.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:24.060 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:24.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:24.183 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:24.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:24.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:24.306 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:24.407 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:24.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:24.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:24.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:24.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:24.676 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:24.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:24.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:28:24.762 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:28:24] "GET /metrics HTTP/1.1" 200 36842 "" "Prometheus/2.51.0" 2024-09-21T10:28:24.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:24.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:24.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:24.923 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:24.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:25.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:25.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:25.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:25.170 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:25.271 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:25.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:25.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:25.416 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:25.518 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.540 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:25.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:25 smithi145 ceph-mon[89359]: pgmap v1791: 83 pgs: 83 active+clean; 477 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:25 smithi145 ceph-mon[89359]: 1.8 scrub starts 2024-09-21T10:28:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:25 smithi145 ceph-mon[89359]: 1.8 scrub ok 2024-09-21T10:28:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:25 smithi119 ceph-mon[105288]: pgmap v1791: 83 pgs: 83 active+clean; 477 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:25 smithi119 ceph-mon[105288]: 1.8 scrub starts 2024-09-21T10:28:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:25 smithi119 ceph-mon[105288]: 1.8 scrub ok 2024-09-21T10:28:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:25 smithi119 ceph-mon[101432]: pgmap v1791: 83 pgs: 83 active+clean; 477 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:25.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:25 smithi119 ceph-mon[101432]: 1.8 scrub starts 2024-09-21T10:28:25.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:25 smithi119 ceph-mon[101432]: 1.8 scrub ok 2024-09-21T10:28:25.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:25.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:25.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:25.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:25.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:25.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:25.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:25.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:26.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:26.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:26.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:26.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:26.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:26.279 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:26.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:26.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:26.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:26.524 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:28:26.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:26.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:26.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:26.766 INFO:teuthology.orchestra.run.smithi119.stdout:moved 0 / 249 (0%) 2024-09-21T10:28:26.766 INFO:teuthology.orchestra.run.smithi119.stdout:avg 35.5714 2024-09-21T10:28:26.766 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 5.28764 -> 5.28764 (expected baseline 5.52176) 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 25 -> 25 pgs (0.702811 -> 0.702811 * mean) 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 43 -> 43 pgs (1.20884 -> 1.20884 * mean) 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0029 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0291 2024-09-21T10:28:26.767 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:28:26.771 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:26.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:26.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:26.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:26.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:27.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:27.118 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:27.140 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:27.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:27.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:27.274 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:28:27.274 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:28:27.274 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:27.274 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:27.287 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:28:27.288 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:27.304 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:28:27.304 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:27.319 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:28:27.319 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:27.334 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:28:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:27.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:28:27.350 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:27.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:27.365 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:28:27.365 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:27.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:27.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:27.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:27 smithi145 ceph-mon[89359]: pgmap v1792: 83 pgs: 83 active+clean; 792 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:27 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:28:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:27 smithi145 ceph-mon[89359]: 1.2 scrub ok 2024-09-21T10:28:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/75800921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:27.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[105288]: pgmap v1792: 83 pgs: 83 active+clean; 792 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:27.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:28:27.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[105288]: 1.2 scrub ok 2024-09-21T10:28:27.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/75800921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:27.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[101432]: pgmap v1792: 83 pgs: 83 active+clean; 792 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:27.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:28:27.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[101432]: 1.2 scrub ok 2024-09-21T10:28:27.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/75800921' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:27.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:27.636 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:27.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:27.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:27.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:27.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:27.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:27.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:27.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:28.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:28.107 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:28.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:28.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:28.250 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:28.306 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_1' created 2024-09-21T10:28:28.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:28.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:28.495 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:28.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:28 smithi119 ceph-mon[105288]: from='client.36213 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:28 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2870546825' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:28:28.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:28 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:28:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:28 smithi119 ceph-mon[101432]: from='client.36213 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:28 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2870546825' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:28:28.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:28 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:28:28.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:28.677 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:28 smithi145 ceph-mon[89359]: from='client.36213 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:28.677 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:28 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2870546825' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:28:28.677 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:28 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-21T10:28:28.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:28.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:28.856 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:28:28.856 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:28:28.856 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:28:28.856 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:28:28.856 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:28:28.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:28.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:28.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:28.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:29.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:29.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:29.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:29.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:29.333 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:29.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:29.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:29.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-21T10:28:29.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:29.477 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:29.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:29.599 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:29 smithi145 ceph-mon[89359]: pgmap v1793: 83 pgs: 83 active+clean; 739 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:29.599 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:29 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:28:29.599 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:29 smithi145 ceph-mon[89359]: osdmap e1237: 8 total, 8 up, 7 in 2024-09-21T10:28:29.600 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:29.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:29 smithi119 ceph-mon[105288]: pgmap v1793: 83 pgs: 83 active+clean; 739 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:29 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:28:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:29 smithi119 ceph-mon[105288]: osdmap e1237: 8 total, 8 up, 7 in 2024-09-21T10:28:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:29 smithi119 ceph-mon[101432]: pgmap v1793: 83 pgs: 83 active+clean; 739 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:29 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-21T10:28:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:29 smithi119 ceph-mon[101432]: osdmap e1237: 8 total, 8 up, 7 in 2024-09-21T10:28:29.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:29.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:29.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:29.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:29.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:29.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:29.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:29.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:29.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:30.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:30.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:30.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:30.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:30.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:30.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:30.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:30.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:30.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:28:30.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:30.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:30.493 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:28:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:28:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:30.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:30 smithi145 ceph-mon[89359]: osdmap e1238: 8 total, 8 up, 7 in 2024-09-21T10:28:30.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:30 smithi145 ceph-mon[89359]: 1.7 scrub starts 2024-09-21T10:28:30.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:30 smithi145 ceph-mon[89359]: 1.7 scrub ok 2024-09-21T10:28:30.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:30.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:30.562 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:30.585 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[105288]: osdmap e1238: 8 total, 8 up, 7 in 2024-09-21T10:28:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[105288]: 1.7 scrub starts 2024-09-21T10:28:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[105288]: 1.7 scrub ok 2024-09-21T10:28:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[101432]: osdmap e1238: 8 total, 8 up, 7 in 2024-09-21T10:28:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[101432]: 1.7 scrub starts 2024-09-21T10:28:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[101432]: 1.7 scrub ok 2024-09-21T10:28:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:30.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:30.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:30.709 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:30.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:30.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:30.831 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:30.933 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:30.954 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:30.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:31.055 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:31.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:31.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:31.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:31.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:31.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:31.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:31.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:31.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:31.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:31.597 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:31 smithi119 ceph-mon[101432]: pgmap v1796: 99 pgs: 2 creating+peering, 13 unknown, 84 active+clean; 687 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:31.597 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:31 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:28:31.598 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:31 smithi119 ceph-mon[101432]: osdmap e1239: 8 total, 8 up, 7 in 2024-09-21T10:28:31.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:31 smithi119 ceph-mon[105288]: pgmap v1796: 99 pgs: 2 creating+peering, 13 unknown, 84 active+clean; 687 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:31.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:31 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:28:31.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:31 smithi119 ceph-mon[105288]: osdmap e1239: 8 total, 8 up, 7 in 2024-09-21T10:28:31.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:31.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:31 smithi145 ceph-mon[89359]: pgmap v1796: 99 pgs: 2 creating+peering, 13 unknown, 84 active+clean; 687 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:31 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:28:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:31 smithi145 ceph-mon[89359]: osdmap e1239: 8 total, 8 up, 7 in 2024-09-21T10:28:31.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:31.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:31.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:31.942 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:31.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:32.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:32.065 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:32.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:32.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:32.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:32.308 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:28:32.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-21T10:28:32.313 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:32.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:32.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:32.559 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:32.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:32.682 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:32.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:32.806 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:32.907 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:32.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:32.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:33.018 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:33.030 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.057 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:33.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:33.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:33.179 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:33.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:33.280 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:33.303 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:33.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:28:33.404 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:33.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.427 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:33.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:33.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:33.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:33.560 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[105288]: pgmap v1798: 99 pgs: 2 creating+peering, 8 unknown, 89 active+clean; 687 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:33.560 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:33.561 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:33.561 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:33.561 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[101432]: pgmap v1798: 99 pgs: 2 creating+peering, 8 unknown, 89 active+clean; 687 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:33.561 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:33.561 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:33.561 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:33.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:33.673 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:33.775 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:33 smithi145 ceph-mon[89359]: pgmap v1798: 99 pgs: 2 creating+peering, 8 unknown, 89 active+clean; 687 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:33.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:33.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:33.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:33.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:34.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:34.036 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:34.037 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:34.045 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:34.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:34.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:34.271 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:34.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:34.326 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:28:34.380 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2269401670' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:34.381 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:34 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:34.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2269401670' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:34.381 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:34 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:34.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:34.419 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:34.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:34.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:34.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:34.648 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:28:34] "GET /metrics HTTP/1.1" 200 36782 "" "Prometheus/2.51.0" 2024-09-21T10:28:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:34.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:34.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2269401670' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:34.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:34 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:28:34.799 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:34.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:34.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:34.925 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:34.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:35.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:35.051 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:35.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:35.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:35.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:35.297 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:35.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:35.413 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[101432]: pgmap v1799: 99 pgs: 2 creating+peering, 97 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:35.413 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:28:35.413 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:35.413 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[101432]: osdmap e1240: 8 total, 8 up, 7 in 2024-09-21T10:28:35.413 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[105288]: pgmap v1799: 99 pgs: 2 creating+peering, 97 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:35.413 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:28:35.414 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:35.414 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:35 smithi119 ceph-mon[105288]: osdmap e1240: 8 total, 8 up, 7 in 2024-09-21T10:28:35.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:35.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:35.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:35.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:35.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:35.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:35 smithi145 ceph-mon[89359]: pgmap v1799: 99 pgs: 2 creating+peering, 97 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:35 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:28:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:35 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:28:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:35 smithi145 ceph-mon[89359]: osdmap e1240: 8 total, 8 up, 7 in 2024-09-21T10:28:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.806 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:35.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:35.907 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:35.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:35.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:36.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:36.057 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:36.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:36.184 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:36.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:36.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:36.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:36.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:36.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:36.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:36.598 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T10:28:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-21T10:28:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T10:28:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-21T10:28:36.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:36.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]} : dispatch 2024-09-21T10:28:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]} : dispatch 2024-09-21T10:28:36.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:36.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:36.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:36.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:36.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:37.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:37.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:37.203 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:37.228 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:37.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:37.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:37.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:37.476 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:37.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:37.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[101432]: pgmap v1801: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T10:28:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T10:28:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[101432]: osdmap e1241: 8 total, 8 up, 7 in 2024-09-21T10:28:37.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[105288]: pgmap v1801: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:37.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T10:28:37.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T10:28:37.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:37 smithi119 ceph-mon[105288]: osdmap e1241: 8 total, 8 up, 7 in 2024-09-21T10:28:37.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:37.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:37 smithi145 ceph-mon[89359]: pgmap v1801: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:37 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-21T10:28:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:37 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-21T10:28:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:37 smithi145 ceph-mon[89359]: osdmap e1241: 8 total, 8 up, 7 in 2024-09-21T10:28:37.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:37.856 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:37.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:37.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:37.979 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:37.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:38.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:38.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:38.208 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:38.233 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:38.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:38.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:38.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:38.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:38.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:38.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:38.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:38.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:38 smithi119 ceph-mon[105288]: osdmap e1242: 8 total, 8 up, 7 in 2024-09-21T10:28:38.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:38 smithi119 ceph-mon[101432]: osdmap e1242: 8 total, 8 up, 7 in 2024-09-21T10:28:38.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:38.734 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:38.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:38.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:38 smithi145 ceph-mon[89359]: osdmap e1242: 8 total, 8 up, 7 in 2024-09-21T10:28:38.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:38.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:38.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:38.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:38.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:39.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:39.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:39.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:39.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:39.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.230 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:39.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:39.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:39.348 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-21T10:28:39.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:39.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:39.476 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:39.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:39.599 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:39.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:39 smithi119 ceph-mon[101432]: pgmap v1804: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3980895313' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:28:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:39 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:28:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:39 smithi119 ceph-mon[105288]: pgmap v1804: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3980895313' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:28:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:39 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:28:39.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:39.727 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:39.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:39 smithi145 ceph-mon[89359]: pgmap v1804: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3980895313' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:28:39.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:39 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:28:39.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:39.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:39.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:39.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:39.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:40.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:28:40.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:40.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:40.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:40.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:40.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:40.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:40.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:40.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:40.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:40 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:28:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:40 smithi145 ceph-mon[89359]: osdmap e1243: 8 total, 8 up, 7 in 2024-09-21T10:28:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:40 smithi145 ceph-mon[89359]: pgmap v1806: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:40.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:28:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:28:40.214Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:40.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:40.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:40 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:28:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:40 smithi119 ceph-mon[105288]: osdmap e1243: 8 total, 8 up, 7 in 2024-09-21T10:28:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:40 smithi119 ceph-mon[105288]: pgmap v1806: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:40 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:28:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:40 smithi119 ceph-mon[101432]: osdmap e1243: 8 total, 8 up, 7 in 2024-09-21T10:28:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:40 smithi119 ceph-mon[101432]: pgmap v1806: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:40.699 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:40.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155683 45.d 45:babfc7b3:::benchmark_data_smithi119_168706_object25513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1184)", 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.002573+0000", 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 184.74124068200001, 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0413941360000001, 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:28:40.758 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155683 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002573+0000", 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002573+0000", 2024-09-21T10:28:40.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002575+0000", 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5850000000000001e-06 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002580+0000", 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3859999999999998e-06 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002591+0000", 2024-09-21T10:28:40.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1117e-05 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002598+0000", 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9500000000000004e-06 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002701+0000", 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102567 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002827+0000", 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012616999999999999 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002972+0000", 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014517299999999999 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.007550+0000", 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045776310000000004 2024-09-21T10:28:40.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.008112+0000", 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00056214199999999996 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054647+0000", 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.046535188999999998 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.763 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054656+0000", 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1560000000000001e-06 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.043912+0000", 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98925529000000001 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.043967+0000", 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5779999999999998e-05 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.764 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155690 45.d 45:b8328bf4:::benchmark_data_smithi119_168706_object25520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1184)", 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 184.73007443200001, 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0306722479999999, 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155690 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.765 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4799999999999998e-07 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.766 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013743+0000", 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0630000000000002e-06 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013751+0000", 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4240000000000004e-06 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013756+0000", 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.767 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013891+0000", 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013522999999999999 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.014000+0000", 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109286 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.014107+0000", 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106871 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.768 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.031183+0000", 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017076131000000001 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054681+0000", 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023497977999999999 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054687+0000", 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9440000000000002e-06 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.769 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044372+0000", 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98968437799999998 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044412+0000", 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0054e-05 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155711 45.d 45:bba85108:::benchmark_data_smithi119_168706_object25541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:28:40.770 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 184.68447245999999, 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98960547399999998, 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155711 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.771 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9400000000000004e-07 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.772 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059346+0000", 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.608e-06 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059356+0000", 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7459999999999995e-06 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059363+0000", 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.754e-06 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.773 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059439+0000", 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6777e-05 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059448+0000", 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9230000000000008e-06 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044470+0000", 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98502151400000004 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.774 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044559+0000", 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9623999999999994e-05 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0", 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044648+0000", 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8647999999999996e-05 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.047982+0000", 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003334301 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.048818+0000", 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00083602399999999999 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.048868+0000", 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0108999999999997e-05 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.048947+0000", 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8552000000000003e-05 2024-09-21T10:28:40.776 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178862 45.d 45:b26a92ab:::benchmark_data_smithi119_168706_object48692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.165994+0000", 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 157.57781953200001, 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.116675195, 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.777 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178862 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.165994+0000", 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.165994+0000", 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.165996+0000", 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5510000000000001e-06 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.166002+0000", 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.166015+0000", 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2659e-05 2024-09-21T10:28:40.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.166023+0000", 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0339999999999997e-06 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.182261+0000", 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016238890999999998 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.182380+0000", 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011845099999999999 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0", 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.182494+0000", 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114459 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190731+0000", 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082361599999999993 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190743+0000", 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2019e-05 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180045+0000", 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98930244700000003 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180057+0000", 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2182e-05 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.267705+0000", 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0876482219999999 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for missing object", 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.267857+0000", 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015175400000000001 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282522+0000", 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014665025 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282670+0000", 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014749000000000001 2024-09-21T10:28:40.783 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178872 45.d 45:b9c4f220:::benchmark_data_smithi119_168706_object48702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.178058+0000", 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 157.56575630099999, 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.104982814, 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.784 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178872 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178058+0000", 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178058+0000", 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178059+0000", 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2079999999999999e-06 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178063+0000", 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5719999999999996e-06 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178074+0000", 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.045e-05 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178081+0000", 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7490000000000001e-06 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190779+0000", 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012698397 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190792+0000", 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3397000000000001e-05 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180088+0000", 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.989295857 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180096+0000", 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0439999999999996e-06 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.268140+0000", 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.088043756 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.268261+0000", 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121038 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.268400+0000", 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013852999999999999 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.281948+0000", 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013548098999999999 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282452+0000", 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050397899999999999 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282742+0000", 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029014699999999999 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282915+0000", 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017263700000000001 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283040+0000", 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012595400000000001 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178880 45.d 45:bd281a79:::benchmark_data_smithi119_168706_object48710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.195342+0000", 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 157.548471761, 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0879248580000001, 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.791 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178880 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195342+0000", 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195342+0000", 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195344+0000", 2024-09-21T10:28:40.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5260000000000001e-06 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195348+0000", 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.76e-06 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195364+0000", 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6892e-05 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195372+0000", 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195874+0000", 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050212599999999998 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195884+0000", 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9860000000000006e-06 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180139+0000", 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98425486299999998 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180147+0000", 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2349999999999999e-06 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.272917+0000", 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.092769825 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.273062+0000", 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014444600000000001 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.273222+0000", 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015990500000000001 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.281992+0000", 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087703120000000006 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282490+0000", 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049801700000000001 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283051+0000", 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00056076699999999999 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283136+0000", 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5342999999999999e-05 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283267+0000", 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013099499999999999 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178912 45.d 45:b9c12902:::benchmark_data_smithi119_168706_object48742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.235809+0000", 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 157.50800513999999, 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0489738179999999, 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.799 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178912 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235809+0000", 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235809+0000", 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235810+0000", 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9900000000000009e-07 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235813+0000", 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4740000000000001e-06 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235823+0000", 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9599999999999995e-06 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235828+0000", 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.203e-06 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235890+0000", 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1362e-05 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235901+0000", 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0788000000000001e-05 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180176+0000", 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.94427550999999998 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180184+0000", 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2130000000000008e-06 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.277238+0000", 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0970531889999999 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.277354+0000", 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116568 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.277518+0000", 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016381499999999999 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.804 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283276+0000", 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057580340000000004 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284465+0000", 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011892070000000001 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284546+0000", 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0364000000000005e-05 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.805 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284630+0000", 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4439999999999998e-05 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284783+0000", 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015269200000000001 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209128 45.d 45:b002702d:::benchmark_data_smithi119_168706_object78958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.806 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.86668192, 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.028074398, 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209128 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:28:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9999999999999998e-07 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.808 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877135+0000", 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4449999999999999e-06 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877141+0000", 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3230000000000003e-06 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877146+0000", 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.578e-06 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.809 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877233+0000", 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6698000000000002e-05 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877244+0000", 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1333e-05 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.901205+0000", 2024-09-21T10:28:40.810 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.023961245 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.901302+0000", 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6879999999999994e-05 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.812 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.901404+0000", 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010234399999999999 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905038+0000", 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036335450000000002 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905099+0000", 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1057999999999997e-05 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.813 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905113+0000", 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4389999999999999e-05 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905167+0000", 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3831e-05 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905206+0000", 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9328e-05 2024-09-21T10:28:40.814 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209152 45.d 45:b5404417:::benchmark_data_smithi119_168706_object78982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.904264+0000", 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.83954990700001, 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.008213383, 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.815 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209152 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904264+0000", 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904264+0000", 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.816 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904265+0000", 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5199999999999995e-07 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904269+0000", 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9509999999999999e-06 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904279+0000", 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9150000000000008e-06 2024-09-21T10:28:40.817 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904286+0000", 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4449999999999999e-06 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904368+0000", 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1626999999999996e-05 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904378+0000", 2024-09-21T10:28:40.818 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7759999999999992e-06 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905241+0000", 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.000863007 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905313+0000", 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2460999999999996e-05 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.819 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905383+0000", 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0304999999999999e-05 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912371+0000", 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069873690000000002 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912405+0000", 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4709999999999998e-05 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.820 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912414+0000", 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1920000000000005e-06 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912444+0000", 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9844000000000001e-05 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912477+0000", 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3929e-05 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.821 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209157 45.d 45:b1a7ba1c:::benchmark_data_smithi119_168706_object78987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.913010+0000", 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.83080423, 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99960250799999995, 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209157 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.822 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913010+0000", 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913010+0000", 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913011+0000", 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.384e-06 2024-09-21T10:28:40.823 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913015+0000", 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.245e-06 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913024+0000", 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.653e-06 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913030+0000", 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4700000000000001e-06 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.824 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913113+0000", 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3172999999999998e-05 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913122+0000", 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3659999999999998e-06 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.908526+0000", 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99540369799999995 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.825 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.908645+0000", 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118922 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.908772+0000", 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012708399999999999 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912517+0000", 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037448450000000001 2024-09-21T10:28:40.826 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912556+0000", 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9308999999999999e-05 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912563+0000", 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.17e-06 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.827 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912587+0000", 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4009e-05 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912612+0000", 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5179999999999999e-05 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209163 45.d 45:b689418d:::benchmark_data_smithi119_168706_object78993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.919351+0000", 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.82446278899999, 2024-09-21T10:28:40.828 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0055039210000001, 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209163 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919351+0000", 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.829 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919351+0000", 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919352+0000", 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0720000000000001e-06 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919357+0000", 2024-09-21T10:28:40.830 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5859999999999998e-06 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919366+0000", 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1919999999999994e-06 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919373+0000", 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7510000000000002e-06 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.831 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919434+0000", 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1178000000000003e-05 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919443+0000", 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5790000000000004e-06 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912643+0000", 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99320071899999995 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.832 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912761+0000", 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000117687 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912902+0000", 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014121700000000001 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924550+0000", 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011648304999999999 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.833 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924605+0000", 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4215000000000002e-05 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924752+0000", 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014749899999999999 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924808+0000", 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5566999999999998e-05 2024-09-21T10:28:40.834 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924855+0000", 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7354000000000001e-05 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209174 45.d 45:b582510b:::benchmark_data_smithi119_168706_object79004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.926123+0000", 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.81769139799999, 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99885655200000001, 2024-09-21T10:28:40.835 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209174 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926123+0000", 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.836 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926123+0000", 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926124+0000", 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6300000000000001e-06 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926130+0000", 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3489999999999999e-06 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.837 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926147+0000", 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6952999999999998e-05 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926155+0000", 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8380000000000008e-06 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926222+0000", 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6861999999999998e-05 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.838 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926233+0000", 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1249e-05 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.916796+0000", 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99056217499999999 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.916925+0000", 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129627 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.839 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.917050+0000", 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012504399999999999 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924704+0000", 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076538480000000004 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924718+0000", 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3835000000000001e-05 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.840 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924902+0000", 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018354499999999999 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924945+0000", 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3322999999999998e-05 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924979+0000", 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4273999999999997e-05 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.841 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209223 45.d 45:bad164a9:::benchmark_data_smithi119_168706_object79053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.988555+0000", 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.755258962, 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.94998562099999995, 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209223 2024-09-21T10:28:40.842 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988555+0000", 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988555+0000", 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.843 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988556+0000", 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1960000000000001e-06 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988562+0000", 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6160000000000001e-06 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988571+0000", 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5259999999999999e-06 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.844 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988578+0000", 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.657e-06 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988641+0000", 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2915000000000003e-05 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988651+0000", 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0458999999999999e-05 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.845 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.920862+0000", 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93221024399999997 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.920977+0000", 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011578699999999999 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.921096+0000", 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011887099999999999 2024-09-21T10:28:40.846 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924868+0000", 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037720840000000002 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925022+0000", 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015386999999999999 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938445+0000", 2024-09-21T10:28:40.847 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013423091999999999 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938500+0000", 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4416999999999998e-05 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938541+0000", 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0886999999999997e-05 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.848 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209244 45.d 45:b1a7562f:::benchmark_data_smithi119_168706_object79074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:46.008462+0000", 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.73535204, 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93854806899999998, 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209244 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.849 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008462+0000", 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008462+0000", 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.850 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008463+0000", 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.31e-06 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008468+0000", 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5800000000000002e-06 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008477+0000", 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0839999999999998e-06 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.851 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008483+0000", 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5300000000000002e-06 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008554+0000", 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0437000000000003e-05 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008564+0000", 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0229e-05 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.852 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925049+0000", 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.916484462 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925168+0000", 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011900000000000001 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925306+0000", 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013886599999999999 2024-09-21T10:28:40.853 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.937933+0000", 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012626818 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938352+0000", 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0004191 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.946903+0000", 2024-09-21T10:28:40.854 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085509980000000006 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.946966+0000", 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2939000000000002e-05 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.947010+0000", 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3716e-05 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.855 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209252 45.d 45:bc201d98:::benchmark_data_smithi119_168706_object79082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:46.017131+0000", 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 114.726682569, 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93874006399999999, 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209252 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.856 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017131+0000", 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017131+0000", 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017133+0000", 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7999999999999999e-06 2024-09-21T10:28:40.857 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017139+0000", 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6450000000000002e-06 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017156+0000", 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6719999999999999e-05 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017164+0000", 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1210000000000007e-06 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.858 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017298+0000", 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013479 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017333+0000", 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4885000000000002e-05 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938885+0000", 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.92155175899999997 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.859 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938993+0000", 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107836 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.939110+0000", 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116883 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.947020+0000", 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079101529999999996 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.860 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.947200+0000", 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000180058 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.955756+0000", 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085562299999999997 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.955807+0000", 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0748000000000003e-05 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.861 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.955871+0000", 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4436000000000004e-05 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248855 45.6 45:68fea44d:::benchmark_data_smithi119_168706_object118685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1222)", 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.870752+0000", 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 53.873062244000003, 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0490298819999999, 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.862 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248855 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870752+0000", 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.863 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870752+0000", 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870753+0000", 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5340000000000001e-06 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870758+0000", 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0849999999999996e-06 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.864 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870771+0000", 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2322e-05 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870778+0000", 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0809999999999996e-06 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870833+0000", 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5318000000000001e-05 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.865 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870964+0000", 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013073800000000001 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.871080+0000", 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000115803 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.879130+0000", 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080503699999999994 2024-09-21T10:28:40.866 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.882842+0000", 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003711733 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.886807+0000", 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039652749999999999 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.886816+0000", 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5169999999999999e-06 2024-09-21T10:28:40.867 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.919735+0000", 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0329197619999999 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.919782+0000", 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6344000000000002e-05 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.868 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248869 45.6 45:63b23490:::benchmark_data_smithi119_168706_object118699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.885958+0000", 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 53.857855974000003, 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0414795889999999, 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248869 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885958+0000", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885958+0000", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885960+0000", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.579e-06 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885964+0000", 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4120000000000003e-06 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885975+0000", 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1449e-05 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885984+0000", 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0579999999999998e-06 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.901089+0000", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015105578999999999 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.901098+0000", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9879999999999993e-06 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.920124+0000", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0190259349999999 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.920202+0000", 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7834000000000001e-05 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.920288+0000", 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6397999999999995e-05 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927179+0000", 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068907489999999998 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927310+0000", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013100300000000001 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927355+0000", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5219000000000002e-05 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927398+0000", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2404000000000003e-05 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927438+0000", 2024-09-21T10:28:40.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9981999999999998e-05 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248896 45.6 45:6c795fd8:::benchmark_data_smithi119_168706_object118726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 53.828713391000001, 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.012468983, 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.875 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248896 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:28:40.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7000000000000004e-07 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915105+0000", 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7390000000000001e-06 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915113+0000", 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1969999999999996e-06 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915120+0000", 2024-09-21T10:28:40.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4729999999999997e-06 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915251+0000", 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013153199999999999 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915258+0000", 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8260000000000004e-06 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.923776+0000", 2024-09-21T10:28:40.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0085175230000001 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.923864+0000", 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8670000000000003e-05 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.923941+0000", 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7124000000000002e-05 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927348+0000", 2024-09-21T10:28:40.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003406203 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927444+0000", 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5995000000000005e-05 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927485+0000", 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1341000000000003e-05 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927534+0000", 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9212999999999998e-05 2024-09-21T10:28:40.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927570+0000", 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5376999999999997e-05 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248913 45.6 45:673d7454:::benchmark_data_smithi119_168706_object118743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.933139+0000", 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 53.810675164999999, 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0031919119999999, 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.881 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248913 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933139+0000", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933139+0000", 2024-09-21T10:28:40.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933140+0000", 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3149999999999999e-06 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933145+0000", 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9099999999999996e-06 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933161+0000", 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5722999999999998e-05 2024-09-21T10:28:40.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933169+0000", 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7430000000000002e-06 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933225+0000", 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6651999999999999e-05 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933234+0000", 2024-09-21T10:28:40.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9889999999999998e-06 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927630+0000", 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99439544599999996 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927735+0000", 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105407 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.928257+0000", 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052212700000000005 2024-09-21T10:28:40.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936069+0000", 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078119030000000002 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936108+0000", 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9035000000000001e-05 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936229+0000", 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012085400000000001 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936282+0000", 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3511999999999999e-05 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936331+0000", 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8296e-05 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248931 45.6 45:633ee5d8:::benchmark_data_smithi119_168706_object118761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.949798+0000", 2024-09-21T10:28:40.887 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 53.794016270999997, 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99733362000000003, 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248931 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949798+0000", 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.888 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949798+0000", 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949799+0000", 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.55e-06 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949805+0000", 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3700000000000003e-06 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.889 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949817+0000", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2244e-05 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949824+0000", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4070000000000004e-06 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949984+0000", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015923399999999999 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:28:40.890 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949992+0000", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.932278+0000", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98228570400000004 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.932400+0000", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012246699999999999 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.932535+0000", 2024-09-21T10:28:40.891 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013443400000000001 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936215+0000", 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036803069999999998 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.940108+0000", 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003892923 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.947038+0000", 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069303389999999998 2024-09-21T10:28:40.892 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.947086+0000", 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8074000000000001e-05 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.947131+0000", 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4904000000000002e-05 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:28:40.893 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:28:40.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:40.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:40.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:41.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:41.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:41.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:41.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:41.168 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:41.207 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-21T10:28:41.211 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:28:41.211 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:28:41.256 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:hints = 1 2024-09-21T10:28:41.256 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:28:41.256 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object prefix: benchmark_data_smithi119_206743 2024-09-21T10:28:41.256 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:28:41.256 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:28:41.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:41.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:41.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:41.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:41.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:41.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:41 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:28:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:41 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:28:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:41 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:28:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:41 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:28:41.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:41.704 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:41 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:28:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:41 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:28:41.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:41.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:41.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:41.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:41.983 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:41.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:42.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:42.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:42.224 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:42.253 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:42.256 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 1 16 3559 3543 221.426 221.438 0.00207557 0.00450143 2024-09-21T10:28:42.355 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:42.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:42.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:42.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:42.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:42.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:42.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:42.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:42.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:42 smithi145 ceph-mon[89359]: pgmap v1807: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:42.863 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:42 smithi119 ceph-mon[101432]: pgmap v1807: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:42.864 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:42 smithi119 ceph-mon[105288]: pgmap v1807: 99 pgs: 99 active+clean; 634 KiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-21T10:28:42.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:42.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:43.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:43.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:43.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:43.211 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:43.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:43.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 2 16 6845 6829 213.391 205.375 0.00354167 0.00467528 2024-09-21T10:28:43.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:43.350 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:43.452 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:43.481 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:43.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:43.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:43.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:43.760 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:43.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:43.897 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:43.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:43.998 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:44.043 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:44.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:44.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:28:44.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:44.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:44.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 3 16 9878 9862 205.443 189.562 0.00935778 0.00485945 2024-09-21T10:28:44.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:44.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:44.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:44.419 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:44.443 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:28:44.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:44.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:44.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:44.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:44.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:44.702 INFO:teuthology.orchestra.run.smithi119.stdout:moved 19 / 297 (6.39731%) 2024-09-21T10:28:44.702 INFO:teuthology.orchestra.run.smithi119.stdout:avg 42.4286 2024-09-21T10:28:44.702 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 4.30472 -> 4.30472 (expected baseline 6.03053) 2024-09-21T10:28:44.702 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 35 -> 41 pgs (0.824916 -> 0.96633 * mean) 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 48 -> 48 pgs (1.13131 -> 1.13131 * mean) 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:oload 110 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 3 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0033 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0037 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4 weight 1.0000 -> 0.7658 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 weight 1.0000 -> 0.8418 2024-09-21T10:28:44.703 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 1.0000 -> 0.8083 2024-09-21T10:28:44.704 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:28:44.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:44.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:44.833 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:28:44] "GET /metrics HTTP/1.1" 200 36782 "" "Prometheus/2.51.0" 2024-09-21T10:28:44.833 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:28:44.696+0000 7fc0be6c7640 -1 mgr.server reply reply (3) No such process no change 2024-09-21T10:28:44.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:44.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:44.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:45.007 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:45.048 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:45.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:45.179 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:45.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 4 16 12466 12450 194.516 161.75 0.011411 0.00513293 2024-09-21T10:28:45.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:45.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:45.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:45.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:45.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:45.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:45 smithi119 ceph-mon[105288]: pgmap v1808: 99 pgs: 99 active+clean; 143 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s wr, 316 op/s 2024-09-21T10:28:45.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:45 smithi119 ceph-mon[101432]: pgmap v1808: 99 pgs: 99 active+clean; 143 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s wr, 316 op/s 2024-09-21T10:28:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:45 smithi145 ceph-mon[89359]: pgmap v1808: 99 pgs: 99 active+clean; 143 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s wr, 316 op/s 2024-09-21T10:28:45.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:45.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:45.717 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:45.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:45.847 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:45.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:45.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:45.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:45.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:46.043 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:28:46.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 1.0 2024-09-21T10:28:46.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.133 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:46.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:46.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:46.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 5 16 15152 15136 189.185 167.875 0.00982778 0.00527818 2024-09-21T10:28:46.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:46.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:46.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:46.449 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:46 smithi145 ceph-mon[89359]: from='client.36249 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:46.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:46.564 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:46 smithi119 ceph-mon[105288]: from='client.36249 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:46.565 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:46 smithi119 ceph-mon[101432]: from='client.36249 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:46.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:46.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:46.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:46.768 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:28:46.769 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:28:46.769 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:28:46.769 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:28:46.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:46.856 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:28:46.856 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:28:46.856 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:28:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:46.865 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:46.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:46.871 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:46.883 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:28:46.884 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:46.911 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:28:46.912 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:46.937 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:46.942 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:28:46.942 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:46.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:46.973 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:28:46.973 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:47.007 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:28:47.007 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:47.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:47.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:47.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:47.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 6 16 17101 17085 177.955 121.812 0.008187 0.00561281 2024-09-21T10:28:47.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:47.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: pgmap v1809: 99 pgs: 99 active+clean; 386 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 47 MiB/s wr, 750 op/s 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/132731603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:47.405 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:47.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:28:47.444 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: pgmap v1809: 99 pgs: 99 active+clean; 386 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 47 MiB/s wr, 750 op/s 2024-09-21T10:28:47.444 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.444 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.444 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/132731603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:47.444 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:47.444 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: pgmap v1809: 99 pgs: 99 active+clean; 386 MiB data, 2.8 GiB used, 623 GiB / 626 GiB avail; 47 MiB/s wr, 750 op/s 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/132731603' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:28:47.445 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:47.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:47.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:47.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:47.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:47.803 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:28:47.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:47.848 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:47.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:47.971 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:47.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:48.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:48.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:48.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:48.228 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:48.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 7 16 19441 19425 173.424 146.25 0.00245221 0.00575429 2024-09-21T10:28:48.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:48.366 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:48.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:48.496 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:48.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:48.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:48.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:48.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:48.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:48.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:48.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:48.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:49.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:49.037 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:48 smithi145 ceph-mon[89359]: pgmap v1810: 99 pgs: 99 active+clean; 699 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:28:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:48 smithi119 ceph-mon[101432]: pgmap v1810: 99 pgs: 99 active+clean; 699 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:28:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:48 smithi119 ceph-mon[105288]: pgmap v1810: 99 pgs: 99 active+clean; 699 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:28:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:28:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:28:49.139 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:49.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:49.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 8 15 21586 21571 168.51 134.125 0.00472837 0.00592795 2024-09-21T10:28:49.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:49.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:49.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:49.443 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:49.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:49.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:49.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:49.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:49.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:49.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:49.944 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:28:49.944 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:28:49.961 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:49.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:49.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:49.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:50.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:50.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:50.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:50.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:50.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 9 16 23932 23916 166.07 146.562 0.00523557 0.00601571 2024-09-21T10:28:50.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:50.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:50.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:50.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:28:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:28:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:28:50.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:50.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:50.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:50.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:50.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:50.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:50.873 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:50 smithi145 ceph-mon[89359]: pgmap v1811: 99 pgs: 99 active+clean; 1.0 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-21T10:28:50.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:50.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:50.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:50 smithi119 ceph-mon[105288]: pgmap v1811: 99 pgs: 99 active+clean; 1.0 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-21T10:28:50.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:50 smithi119 ceph-mon[101432]: pgmap v1811: 99 pgs: 99 active+clean; 1.0 GiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 102 MiB/s wr, 1.63k op/s 2024-09-21T10:28:51.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:51.065 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:51.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:51.193 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:51.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 10 16 25862 25846 161.524 120.625 0.005109 0.00618449 2024-09-21T10:28:51.295 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:51.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:51.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:51.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:51.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:51.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:51.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:28:51.593 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:51.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:51.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:51.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:51.837 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:51.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:51.966 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:51.995 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:51.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:52.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:52.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.126 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:52.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:52.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:52.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:52.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 11 16 27911 27895 158.48 128.062 0.0054868 0.00630427 2024-09-21T10:28:52.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:52.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:28:52.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:52.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:52.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:52.522 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:52.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:52.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:52.716 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:28:52.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:52.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:52.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:52.924 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:52.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:53.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:53.033 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:52 smithi119 ceph-mon[105288]: pgmap v1812: 99 pgs: 99 active+clean; 1.2 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:28:53.033 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:52 smithi119 ceph-mon[101432]: pgmap v1812: 99 pgs: 99 active+clean; 1.2 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:28:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:52 smithi145 ceph-mon[89359]: pgmap v1812: 99 pgs: 99 active+clean; 1.2 GiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-09-21T10:28:53.052 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:53.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:53.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:53.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 12 16 29763 29747 154.919 115.75 0.0027374 0.00644949 2024-09-21T10:28:53.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:53.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:53.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:53.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:53.596 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:53.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:53.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:53.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:53.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:53.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:53.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:53.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:53.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:54.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:54.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:54.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:54.252 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:54.257 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 13 16 31621 31605 151.934 116.125 0.00649224 0.00657637 2024-09-21T10:28:54.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:54.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:54.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:54.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:54.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:28:54.558 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:28:54.616 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:54.644 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:54.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:54.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:54.795 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:28:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:28:54] "GET /metrics HTTP/1.1" 200 36843 "" "Prometheus/2.51.0" 2024-09-21T10:28:54.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:54 smithi119 ceph-mon[105288]: pgmap v1813: 99 pgs: 99 active+clean; 1.6 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-21T10:28:54.796 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:54 smithi119 ceph-mon[101432]: pgmap v1813: 99 pgs: 99 active+clean; 1.6 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-21T10:28:54.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:54.916 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:54.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:55.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:54 smithi145 ceph-mon[89359]: pgmap v1813: 99 pgs: 99 active+clean; 1.6 GiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 134 MiB/s wr, 2.14k op/s 2024-09-21T10:28:55.057 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:55.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.205 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:55.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:55.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 14 16 33290 33274 148.532 104.312 0.0116176 0.00672562 2024-09-21T10:28:55.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:55.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:55.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:55.471 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:55.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:55.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:55.624 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:55.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:55.727 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:moved 14 / 297 (4.7138%) 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:avg 42.4286 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 4.30472 -> 5.31459 (expected baseline 6.03053) 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 35 -> 38 pgs (0.824916 -> 0.895623 * mean) 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 48 -> 50 pgs (1.13131 -> 1.17845 * mean) 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:oload 110 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 1 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:28:55.807 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0101 2024-09-21T10:28:55.808 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0111 2024-09-21T10:28:55.808 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 1.0000 -> 0.7610 2024-09-21T10:28:55.808 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 weight 1.0000 -> 0.8746 2024-09-21T10:28:55.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:55.852 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:55.930 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'49871','6':'57321'}"} : dispatch 2024-09-21T10:28:55.931 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'49871','6':'57321'}"} : dispatch 2024-09-21T10:28:55.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:55.975 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:55.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'49871','6':'57321'}"} : dispatch 2024-09-21T10:28:56.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:56.096 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:56.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:56.221 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:56.224 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:28:56.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 15 16 34218 34202 142.495 58 0.00196396 0.00681491 2024-09-21T10:28:56.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:56.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:56.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:56.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:56.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:56.510 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:28:56.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.590 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:56.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:56.663 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:28:56.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 0.05 2024-09-21T10:28:56.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.712 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:56.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:56.810 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:28:56.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:56.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[101432]: from='client.55435 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[101432]: pgmap v1814: 99 pgs: 99 active+clean; 1.8 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'49871','6':'57321'}"}]': finished 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[101432]: osdmap e1244: 8 total, 8 up, 7 in 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3317902110' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[105288]: from='client.55435 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[105288]: pgmap v1814: 99 pgs: 99 active+clean; 1.8 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'49871','6':'57321'}"}]': finished 2024-09-21T10:28:56.916 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[105288]: osdmap e1244: 8 total, 8 up, 7 in 2024-09-21T10:28:56.917 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3317902110' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:28:56.917 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:28:56.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:56.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:56.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:56.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:56 smithi145 ceph-mon[89359]: from='client.55435 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:28:56.976 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:56 smithi145 ceph-mon[89359]: pgmap v1814: 99 pgs: 99 active+clean; 1.8 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 140 MiB/s wr, 2.24k op/s 2024-09-21T10:28:56.976 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'49871','6':'57321'}"}]': finished 2024-09-21T10:28:56.976 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:56 smithi145 ceph-mon[89359]: osdmap e1244: 8 total, 8 up, 7 in 2024-09-21T10:28:56.976 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3317902110' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:28:56.976 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:28:57.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:57.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:57.186 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:57.208 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:57.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 16 16 34218 34202 133.59 0 - 0.00681491 2024-09-21T10:28:57.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:57.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:57.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:57.377 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:28:57.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:57.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:57.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:57.556 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:28:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.578 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:57.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:28:57.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:57.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:57.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:57.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:57.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:57.956 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:57.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:58.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:57 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:58.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:57 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:28:58.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:57 smithi145 ceph-mon[89359]: osdmap e1245: 8 total, 8 up, 7 in 2024-09-21T10:28:58.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:57 smithi145 ceph-mon[89359]: osdmap e1246: 8 total, 8 up, 7 in 2024-09-21T10:28:58.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:28:58.082 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:28:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:28:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[105288]: osdmap e1245: 8 total, 8 up, 7 in 2024-09-21T10:28:58.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[105288]: osdmap e1246: 8 total, 8 up, 7 in 2024-09-21T10:28:58.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:28:58.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:28:58.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[101432]: osdmap e1245: 8 total, 8 up, 7 in 2024-09-21T10:28:58.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:57 smithi119 ceph-mon[101432]: osdmap e1246: 8 total, 8 up, 7 in 2024-09-21T10:28:58.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:58.205 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:58.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 17 16 34474 34458 126.672 8 0.00238834 0.00745886 2024-09-21T10:28:58.269 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:28:58.269 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:28:58.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:58.329 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:58.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:58.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:58.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:58.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:28:58.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:28:58.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:28:58.697 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:28:58.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:58.907 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:58 smithi119 ceph-mon[105288]: pgmap v1817: 99 pgs: 1 remapped+peering, 98 active+clean; 1.9 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:58.907 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:58 smithi119 ceph-mon[101432]: pgmap v1817: 99 pgs: 1 remapped+peering, 98 active+clean; 1.9 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:58.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:28:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:58.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:28:58.981 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:58 smithi145 ceph-mon[89359]: pgmap v1817: 99 pgs: 1 remapped+peering, 98 active+clean; 1.9 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:28:59.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:28:59.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:59.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:59.132 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:28:59.132 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:28:59.140 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:28:59.140 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:28:59.157 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:28:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:28:59.171 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:59.173 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:28:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:28:59.190 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:28:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:28:59.193 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:59.205 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:28:59.206 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:28:59.221 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:28:59.221 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:28:59.236 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:28:59.236 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:28:59.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 18 16 34474 34458 119.635 0 - 0.00745886 2024-09-21T10:28:59.287 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:28:59.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:28:59.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:28:59.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:28:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:28:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:28:59.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:59.563 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:59.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:59.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:59.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:59.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:28:59.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:28:59.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:28:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:28:59.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:28:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2663851358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:00.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:00.085 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:28:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2663851358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:28:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2663851358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:00.188 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:00.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:00.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 19 16 35219 35203 115.789 23.2812 0.00743293 0.00863001 2024-09-21T10:29:00.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:00.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:29:00.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:00.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:00.494 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:00.508 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:29:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:29:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:00.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:00.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:00.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:00.778 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:00.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:00.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:00.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:01.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:01.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:00 smithi119 ceph-mon[101432]: pgmap v1819: 99 pgs: 1 remapped+peering, 98 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:29:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:00 smithi119 ceph-mon[105288]: pgmap v1819: 99 pgs: 1 remapped+peering, 98 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:29:01.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:01.171 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:00 smithi145 ceph-mon[89359]: pgmap v1819: 99 pgs: 1 remapped+peering, 98 active+clean; 2.0 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 76 MiB/s wr, 1.22k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:29:01.188 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:01.205 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:29:01.205 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:29:01.206 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:29:01.206 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:29:01.206 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:29:01.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:29:01.257138+0000 min lat: 0.00134105 max lat: 4.01826 avg lat: 0.00869774 2024-09-21T10:29:01.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:29:01.258 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 20 16 36787 36771 114.899 98 0.00581414 0.00869774 2024-09-21T10:29:01.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.330 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:01.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:01.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:01.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:01.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:01.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:01.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:01.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:01.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:01.866 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:01.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:01.907 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:29:01.925 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:01.968 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:02.026 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:01 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 8441/103956 objects degraded (8.120%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:01 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 8441/103956 objects degraded (8.120%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:02.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:02.164 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:02.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:02.259 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 21 16 38383 38367 114.174 99.75 0.0213162 0.00875185 2024-09-21T10:29:02.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:01 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 8441/103956 objects degraded (8.120%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.314 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:02.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:02.325 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:02.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:02.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:02.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:02.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:02.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:02.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:02.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:02.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:02.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:02.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:02.997 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:02 smithi145 ceph-mon[89359]: pgmap v1820: 99 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 95 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s; 8441/103956 objects degraded (8.120%); 11 KiB/s, 0 objects/s recovering 2024-09-21T10:29:02.997 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:02.997 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:03.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:03.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:02 smithi119 ceph-mon[101432]: pgmap v1820: 99 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 95 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s; 8441/103956 objects degraded (8.120%); 11 KiB/s, 0 objects/s recovering 2024-09-21T10:29:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:02 smithi119 ceph-mon[105288]: pgmap v1820: 99 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 remapped+peering, 95 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s; 8441/103956 objects degraded (8.120%); 11 KiB/s, 0 objects/s recovering 2024-09-21T10:29:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:03.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:03.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:03.259 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 22 16 39875 39859 113.223 93.25 0.0106976 0.00882485 2024-09-21T10:29:03.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:03.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:29:03.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:03.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:03.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.603 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:03.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:03.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:03.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:03.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:03.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:03.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:03.921 INFO:teuthology.orchestra.run.smithi119.stdout:moved 6 / 297 (2.0202%) 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:avg 42.4286 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 5.31459 -> 7.38448 (expected baseline 6.03053) 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 33 -> 28 pgs (0.777778 -> 0.659933 * mean) 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 50 -> 52 pgs (1.17845 -> 1.22559 * mean) 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:oload 110 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0132 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0145 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.7610 -> 0.7110 2024-09-21T10:29:03.922 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 weight 1.0000 -> 0.9500 2024-09-21T10:29:03.963 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:03 smithi119 ceph-mon[105288]: from='client.36273 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:29:03.963 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'46595','2':'62260'}"} : dispatch 2024-09-21T10:29:03.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:03 smithi119 ceph-mon[101432]: from='client.36273 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:29:03.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'46595','2':'62260'}"} : dispatch 2024-09-21T10:29:04.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:04.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:04.070 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:03 smithi145 ceph-mon[89359]: from='client.36273 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:29:04.070 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'46595','2':'62260'}"} : dispatch 2024-09-21T10:29:04.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:04.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:04.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:04.259 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 23 16 41028 41012 111.433 72.0625 0.00284462 0.00886937 2024-09-21T10:29:04.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:04.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:04.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:04.457 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:29:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:29:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:29:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:29:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.457 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155685 45.9 45:92e63bd3:::benchmark_data_smithi119_168706_object25515:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.025775+0000", 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.420755402, 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.024087424, 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155685 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.458 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025775+0000", 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025775+0000", 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025773+0000", 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:04.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025785+0000", 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2551000000000001e-05 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025799+0000", 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3098000000000001e-05 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.025806+0000", 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3769999999999998e-06 2024-09-21T10:29:04.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.026290+0000", 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00048368199999999998 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.026298+0000", 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9239999999999996e-06 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.047370+0000", 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0210716339999999 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.049862+0000", 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024921510000000002 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155697 45.9 45:9a5b4933:::benchmark_data_smithi119_168706_object25527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.035385+0000", 2024-09-21T10:29:04.462 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.41114506299999, 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0486232090000001, 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155697 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035385+0000", 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035385+0000", 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035384+0000", 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035390+0000", 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2299999999999996e-06 2024-09-21T10:29:04.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035401+0000", 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0463e-05 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035407+0000", 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3319999999999997e-06 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035443+0000", 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6606999999999997e-05 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.035454+0000", 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0325e-05 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.049909+0000", 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.014455369 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051425+0000", 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001516143 2024-09-21T10:29:04.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051989+0000", 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00056419700000000001 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066316+0000", 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014326358000000001 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.083912+0000", 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017595821000000001 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.083950+0000", 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8272999999999998e-05 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084008+0000", 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8479e-05 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.468 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155715 45.2 45:49e42279:::benchmark_data_smithi119_168706_object25545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.064069+0000", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.38246147999999, 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.992269917, 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155715 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064069+0000", 2024-09-21T10:29:04.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064069+0000", 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064068+0000", 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064071+0000", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.179e-06 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064077+0000", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4770000000000002e-06 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064080+0000", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.168e-06 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064113+0000", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3312000000000002e-05 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.064121+0000", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2409999999999998e-06 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.046336+0000", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98221509500000004 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.046474+0000", 2024-09-21T10:29:04.472 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000138362 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.046606+0000", 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013158899999999999 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056161+0000", 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095556649999999996 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056215+0000", 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4036000000000002e-05 2024-09-21T10:29:04.473 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056261+0000", 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5762e-05 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056339+0000", 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7568e-05 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.474 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155718 45.2 45:4826636e:::benchmark_data_smithi119_168706_object25548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.073004+0000", 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.373525857, 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99846614700000003, 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155718 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.475 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073004+0000", 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073004+0000", 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073003+0000", 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.476 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073009+0000", 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4750000000000001e-06 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073017+0000", 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6469999999999998e-06 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073022+0000", 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1070000000000004e-06 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.477 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073057+0000", 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5373999999999997e-05 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073063+0000", 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.071e-06 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.050905+0000", 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97784177800000005 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.478 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051002+0000", 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6992999999999998e-05 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.051108+0000", 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105933 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.056356+0000", 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052479650000000003 2024-09-21T10:29:04.479 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.071365+0000", 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015009688 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.071401+0000", 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5757999999999999e-05 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.071471+0000", 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9308999999999997e-05 2024-09-21T10:29:04.480 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155719 45.9 45:98f146b7:::benchmark_data_smithi119_168706_object25549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.37348098499999, 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0110945950000001, 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.481 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155719 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.482 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073049+0000", 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073052+0000", 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.723e-06 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073057+0000", 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6509999999999998e-06 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.483 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073060+0000", 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8669999999999998e-06 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073093+0000", 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2972000000000002e-05 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.073100+0000", 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5869999999999998e-06 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.484 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.058920+0000", 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98582079600000005 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.059054+0000", 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013349999999999999 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.059198+0000", 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000144177 2024-09-21T10:29:04.485 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066501+0000", 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073030980000000001 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084044+0000", 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017542881999999999 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084079+0000", 2024-09-21T10:29:04.486 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4776e-05 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.084144+0000", 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5009e-05 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:155721 45.9 45:92074e92:::benchmark_data_smithi119_168706_object25551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:25:36.076715+0000", 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.369815273, 2024-09-21T10:29:04.487 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0095854630000001, 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 155721 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076715+0000", 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.488 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076715+0000", 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076714+0000", 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076722+0000", 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9370000000000004e-06 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.489 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076739+0000", 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7844999999999999e-05 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076747+0000", 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5739999999999998e-06 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076851+0000", 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000104001 2024-09-21T10:29:04.490 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:36.076864+0000", 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.292e-05 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.062819+0000", 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98595542599999997 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.062933+0000", 2024-09-21T10:29:04.491 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113699 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2", 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.063037+0000", 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000104275 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.066508+0000", 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003471081 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.492 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.086177+0000", 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019668622 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.086223+0000", 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6283999999999999e-05 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:25:37.086300+0000", 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7169000000000006e-05 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.493 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178894 45.2 45:48807ac9:::benchmark_data_smithi119_168706_object48724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:03.217696+0000", 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 181.22883452100001, 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0568011020000001, 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178894 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217696+0000", 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217696+0000", 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217694+0000", 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.495 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217699+0000", 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1830000000000002e-06 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217710+0000", 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0674e-05 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217717+0000", 2024-09-21T10:29:04.496 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5379999999999999e-06 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217768+0000", 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.117e-05 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.497 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.498 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.217778+0000", 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9250000000000007e-06 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181105+0000", 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96332745500000005 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.499 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181117+0000", 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1683e-05 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.265884+0000", 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0847669339999999 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.265974+0000", 2024-09-21T10:29:04.500 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0136999999999994e-05 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.266061+0000", 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6847000000000004e-05 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.272892+0000", 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006831798 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.272900+0000", 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2080000000000003e-06 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274344+0000", 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001444702 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274384+0000", 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9468999999999998e-05 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274497+0000", 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011297499999999999 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:178898 45.2 45:44d8b130:::benchmark_data_smithi119_168706_object48728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:03.220961+0000", 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 181.225569681, 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0539241750000001, 2024-09-21T10:29:04.503 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 178898 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220961+0000", 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220961+0000", 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220960+0000", 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220963+0000", 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2899999999999998e-06 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220969+0000", 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9170000000000003e-06 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.220972+0000", 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7489999999999999e-06 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.221004+0000", 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2206000000000003e-05 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:03.221010+0000", 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0809999999999999e-06 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181141+0000", 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96013049699999997 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:04.181148+0000", 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.216e-06 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.269308+0000", 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.088159447 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.269432+0000", 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012429699999999999 2024-09-21T10:29:04.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.269515+0000", 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3351999999999997e-05 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.272921+0000", 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034056339999999998 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274540+0000", 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016195370000000001 2024-09-21T10:29:04.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274695+0000", 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015458799999999999 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274744+0000", 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8717999999999997e-05 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:05.274885+0000", 2024-09-21T10:29:04.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141113 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:192584 45.2 45:4f46063b:::benchmark_data_smithi119_168706_object62414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1196)", 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:21.707498+0000", 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 162.73903193300001, 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0639593540000001, 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.511 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 192584 2024-09-21T10:29:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707498+0000", 2024-09-21T10:29:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707498+0000", 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707497+0000", 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707503+0000", 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6049999999999997e-06 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707512+0000", 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7819999999999999e-06 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707515+0000", 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7440000000000001e-06 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707556+0000", 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0355999999999998e-05 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.707565+0000", 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4530000000000008e-06 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.717826+0000", 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0102602890000001 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.717964+0000", 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000138422 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.718114+0000", 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000149667 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.728455+0000", 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010341105999999999 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771270+0000", 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.042815619999999999 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771384+0000", 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011415400000000001 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771431+0000", 2024-09-21T10:29:04.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6461999999999997e-05 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771458+0000", 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6741e-05 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:192599 45.2 45:4070d208:::benchmark_data_smithi119_168706_object62429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1196)", 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:21.729667+0000", 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 162.716863295, 2024-09-21T10:29:04.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0419121920000001, 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 192599 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729667+0000", 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729667+0000", 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729665+0000", 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729673+0000", 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9400000000000002e-06 2024-09-21T10:29:04.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729692+0000", 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8632999999999999e-05 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729698+0000", 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4749999999999998e-06 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729753+0000", 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5096000000000002e-05 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.729765+0000", 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1252e-05 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.721275+0000", 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99151005800000003 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.721403+0000", 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012799099999999999 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.721534+0000", 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013128400000000001 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.734574+0000", 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0130396 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771309+0000", 2024-09-21T10:29:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.036735378999999999 2024-09-21T10:29:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771509+0000", 2024-09-21T10:29:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000200554 2024-09-21T10:29:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771543+0000", 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3847000000000003e-05 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771579+0000", 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5837000000000003e-05 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:192634 45.2 45:4f5577e2:::benchmark_data_smithi119_168706_object62464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1196)", 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:21.775852+0000", 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 162.670678151, 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99587057899999998, 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.528 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 192634 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775852+0000", 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775852+0000", 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775850+0000", 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775858+0000", 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4800000000000004e-06 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775874+0000", 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5979999999999999e-05 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775881+0000", 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7190000000000001e-06 2024-09-21T10:29:04.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775942+0000", 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0958999999999997e-05 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:21.775954+0000", 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1695999999999999e-05 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.725138+0000", 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.94918383500000003 2024-09-21T10:29:04.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.725204+0000", 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6544999999999994e-05 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.725274+0000", 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9586999999999996e-05 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.740426+0000", 2024-09-21T10:29:04.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015151929 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771632+0000", 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.031206484 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771665+0000", 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2626999999999998e-05 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771692+0000", 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6679e-05 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:22.771723+0000", 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0959e-05 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230678 45.e 45:79bfb7a2:::benchmark_data_smithi119_168706_object100508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.808009+0000", 2024-09-21T10:29:04.536 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.638521341, 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.097576466, 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230678 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808009+0000", 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808009+0000", 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808008+0000", 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808013+0000", 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0470000000000001e-06 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808022+0000", 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0080000000000008e-06 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808026+0000", 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9910000000000004e-06 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808055+0000", 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8611000000000001e-05 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808176+0000", 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000121403 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808311+0000", 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013477699999999999 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.812893+0000", 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045818250000000003 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.813982+0000", 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010895270000000001 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.817794+0000", 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038114450000000001 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840263+0000", 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022469335999999999 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840275+0000", 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1874e-05 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905517+0000", 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.065242453 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905585+0000", 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8132000000000004e-05 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230690 45.e 45:74649e72:::benchmark_data_smithi119_168706_object100520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.824956+0000", 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.62157424199999, 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.080745302, 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:29:04.543 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230690 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824956+0000", 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824956+0000", 2024-09-21T10:29:04.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824955+0000", 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824960+0000", 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8300000000000003e-06 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824969+0000", 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1929999999999999e-06 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824974+0000", 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.78e-06 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.825032+0000", 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8323999999999999e-05 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.825177+0000", 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014489799999999999 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.825262+0000", 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5150999999999998e-05 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840324+0000", 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0150616 2024-09-21T10:29:04.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840331+0000", 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4789999999999999e-06 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905636+0000", 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.065304714 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905701+0000", 2024-09-21T10:29:04.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5591000000000003e-05 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230693 45.e 45:7aa067b6:::benchmark_data_smithi119_168706_object100523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.828211+0000", 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.618319017, 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.077542488, 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.549 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230693 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828211+0000", 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828211+0000", 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828209+0000", 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828217+0000", 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8979999999999996e-06 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828237+0000", 2024-09-21T10:29:04.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0010999999999999e-05 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828245+0000", 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2429999999999999e-06 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828977+0000", 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073276999999999999 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.829121+0000", 2024-09-21T10:29:04.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014400599999999999 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.829260+0000", 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013883499999999999 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840365+0000", 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011104862 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840373+0000", 2024-09-21T10:29:04.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5220000000000001e-06 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905717+0000", 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0653447140000001 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905754+0000", 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6402999999999997e-05 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.554 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230694 45.e 45:79bbaa33:::benchmark_data_smithi119_168706_object100524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.828798+0000", 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.617732639, 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.077047415, 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230694 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828798+0000", 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828798+0000", 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828797+0000", 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828802+0000", 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4580000000000001e-06 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828825+0000", 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2708999999999999e-05 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828832+0000", 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.374e-06 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.834438+0000", 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056056700000000001 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.834557+0000", 2024-09-21T10:29:04.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000119207 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.834634+0000", 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7002000000000006e-05 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840428+0000", 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057940259999999999 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840436+0000", 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6480000000000003e-06 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905784+0000", 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0653486219999999 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905845+0000", 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0616999999999999e-05 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.560 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230699 45.e 45:73169608:::benchmark_data_smithi119_168706_object100529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.840651+0000", 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.605879413, 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1005979770000001, 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230699 2024-09-21T10:29:04.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840651+0000", 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840651+0000", 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840649+0000", 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840655+0000", 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2260000000000002e-06 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840666+0000", 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0587e-05 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840672+0000", 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840762+0000", 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0346999999999994e-05 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840772+0000", 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108999999999999e-05 2024-09-21T10:29:04.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905891+0000", 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.065118842 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.906014+0000", 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012271100000000001 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.906152+0000", 2024-09-21T10:29:04.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000138419 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.922422+0000", 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016269520999999999 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941090+0000", 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018668609999999999 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941186+0000", 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5977999999999995e-05 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941225+0000", 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8651e-05 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941249+0000", 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4031000000000001e-05 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.567 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230700 45.e 45:76633324:::benchmark_data_smithi119_168706_object100530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.842302+0000", 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.604228565, 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0991607729999999, 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230700 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.568 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842302+0000", 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842302+0000", 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842301+0000", 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842306+0000", 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9490000000000004e-06 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842316+0000", 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0369e-05 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842320+0000", 2024-09-21T10:29:04.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3359999999999997e-06 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842353+0000", 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2666999999999997e-05 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842358+0000", 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2329999999999997e-06 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.910012+0000", 2024-09-21T10:29:04.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0676535069999999 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.910143+0000", 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013090700000000001 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.910276+0000", 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000133215 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.922427+0000", 2024-09-21T10:29:04.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012150988 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941113+0000", 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018686472999999999 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941409+0000", 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029531300000000001 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941444+0000", 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5355000000000003e-05 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941462+0000", 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8413000000000001e-05 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.575 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230756 45.e 45:7f9a0f5a:::benchmark_data_smithi119_168706_object100586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.919711+0000", 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.526819084, 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0218782360000001, 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230756 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919711+0000", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919711+0000", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919710+0000", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919717+0000", 2024-09-21T10:29:04.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7500000000000003e-06 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919731+0000", 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.399e-05 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919737+0000", 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1330000000000004e-06 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919778+0000", 2024-09-21T10:29:04.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0710000000000002e-05 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919785+0000", 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.562e-06 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.913413+0000", 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99362865300000003 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.913493+0000", 2024-09-21T10:29:04.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9919999999999994e-05 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.913567+0000", 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3306999999999995e-05 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.928030+0000", 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014463877 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941141+0000", 2024-09-21T10:29:04.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013110157000000001 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941504+0000", 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036338499999999998 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941551+0000", 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6860999999999997e-05 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941589+0000", 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8544999999999998e-05 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230758 45.e 45:7f2353b2:::benchmark_data_smithi119_168706_object100588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.920715+0000", 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.525815114, 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.021031107, 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.582 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230758 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920715+0000", 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920715+0000", 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920714+0000", 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920719+0000", 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1290000000000004e-06 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920732+0000", 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2548000000000001e-05 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920737+0000", 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3800000000000002e-06 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920779+0000", 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2024999999999998e-05 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920784+0000", 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7940000000000002e-06 2024-09-21T10:29:04.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.916661+0000", 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99587639999999999 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.916752+0000", 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1729000000000002e-05 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:29:04.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.916849+0000", 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6251999999999999e-05 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.928037+0000", 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011188849000000001 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941283+0000", 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013245883999999999 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941697+0000", 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041416700000000001 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941726+0000", 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8523000000000001e-05 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941746+0000", 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0319e-05 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230770 45.e 45:7a02883a:::benchmark_data_smithi119_168706_object100600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.938360+0000", 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 106.508169955, 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0035411750000001, 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230770 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:04.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938360+0000", 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938360+0000", 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938359+0000", 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:04.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938365+0000", 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8340000000000004e-06 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938379+0000", 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4242e-05 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938386+0000", 2024-09-21T10:29:04.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2319999999999998e-06 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938451+0000", 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5400000000000004e-05 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938459+0000", 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9850000000000006e-06 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.919620+0000", 2024-09-21T10:29:04.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98116114600000004 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.919697+0000", 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6692000000000004e-05 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.919783+0000", 2024-09-21T10:29:04.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5957000000000004e-05 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.933063+0000", 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013280001 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941307+0000", 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082444389999999992 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941856+0000", 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054824699999999997 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941881+0000", 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5786e-05 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941901+0000", 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0157999999999999e-05 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:04.596 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:04.596 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:29:04.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:04.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:04.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:04.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:04.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:04.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:29:04] "GET /metrics HTTP/1.1" 200 36885 "" "Prometheus/2.51.0" 2024-09-21T10:29:04.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:04.969 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:04.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:04.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:05.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:05.114 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:05.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[101432]: pgmap v1821: 99 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 775 op/s; 8441/109191 objects degraded (7.730%); 1586/109191 objects misplaced (1.453%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'46595','2':'62260'}"}]': finished 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[101432]: osdmap e1247: 8 total, 8 up, 7 in 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[101432]: osdmap e1248: 8 total, 8 up, 7 in 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[105288]: pgmap v1821: 99 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 775 op/s; 8441/109191 objects degraded (7.730%); 1586/109191 objects misplaced (1.453%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'46595','2':'62260'}"}]': finished 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[105288]: osdmap e1247: 8 total, 8 up, 7 in 2024-09-21T10:29:05.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:04 smithi119 ceph-mon[105288]: osdmap e1248: 8 total, 8 up, 7 in 2024-09-21T10:29:05.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:05.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:05.259 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 24 16 41028 41012 106.79 0 - 0.00886937 2024-09-21T10:29:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:04 smithi145 ceph-mon[89359]: pgmap v1821: 99 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 2.2 GiB data, 8.6 GiB used, 617 GiB / 626 GiB avail; 48 MiB/s wr, 775 op/s; 8441/109191 objects degraded (7.730%); 1586/109191 objects misplaced (1.453%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:29:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'46595','2':'62260'}"}]': finished 2024-09-21T10:29:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:04 smithi145 ceph-mon[89359]: osdmap e1247: 8 total, 8 up, 7 in 2024-09-21T10:29:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:04 smithi145 ceph-mon[89359]: osdmap e1248: 8 total, 8 up, 7 in 2024-09-21T10:29:05.339 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:05.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:05.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:05.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:05.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:29:05.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:05.627 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:05.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:05.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:05.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:05.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:05.888 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:05.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:05.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:06.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:06.030 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:06.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:06.162 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:06.259 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 25 16 42133 42117 105.281 34.5312 0.0341631 0.00949033 2024-09-21T10:29:06.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:06.265 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:29:06.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-21T10:29:06.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:06.360 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:29:06.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.433 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:06.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:06.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:06 smithi145 ceph-mon[89359]: osdmap e1249: 8 total, 8 up, 7 in 2024-09-21T10:29:06.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/122825883' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:06.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.580 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:06.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:06 smithi119 ceph-mon[101432]: osdmap e1249: 8 total, 8 up, 7 in 2024-09-21T10:29:06.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/122825883' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:06 smithi119 ceph-mon[105288]: osdmap e1249: 8 total, 8 up, 7 in 2024-09-21T10:29:06.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/122825883' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:06.681 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:06.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:06.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:06.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:06.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:06.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:06.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:07.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:07.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:07.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:07.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:07.316 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 26 16 43290 43274 104.006 72.3125 0.012193 0.00960909 2024-09-21T10:29:07.367 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:07.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:07.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:07.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[105288]: pgmap v1825: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 8438/117624 objects degraded (7.174%); 1586/117624 objects misplaced (1.348%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/122825883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[105288]: osdmap e1250: 8 total, 8 up, 7 in 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[101432]: pgmap v1825: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 8438/117624 objects degraded (7.174%); 1586/117624 objects misplaced (1.348%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/122825883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:07.538 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:07 smithi119 ceph-mon[101432]: osdmap e1250: 8 total, 8 up, 7 in 2024-09-21T10:29:07.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:07.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:07.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:07.764 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:07 smithi145 ceph-mon[89359]: pgmap v1825: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 8438/117624 objects degraded (7.174%); 1586/117624 objects misplaced (1.348%); 6.4 MiB/s, 103 objects/s recovering 2024-09-21T10:29:07.764 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:07 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:07.764 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/122825883' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:07.765 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:07 smithi145 ceph-mon[89359]: osdmap e1250: 8 total, 8 up, 7 in 2024-09-21T10:29:07.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:07.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:07.926 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:07.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:08.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:08.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:08.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:08.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:08.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:08.261 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 27 16 44694 44678 103.404 87.75 0.00985842 0.00966395 2024-09-21T10:29:08.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:08.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:08.383 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:29:08.383 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:29:08.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36246.0:44843 46.6 46:6660f207:::benchmark_data_smithi119_206743_object44842:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1250)", 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:29:08.374385+0000", 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0065260730000000003, 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065527850000000002, 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36246", 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 44843 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374385+0000", 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374385+0000", 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374382+0000", 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374430+0000", 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7982000000000002e-05 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374446+0000", 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5515999999999998e-05 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374453+0000", 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2389999999999996e-06 2024-09-21T10:29:08.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374508+0000", 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5359999999999999e-05 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374730+0000", 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00022142 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374870+0000", 2024-09-21T10:29:08.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014057800000000001 2024-09-21T10:29:08.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:08.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.377665+0000", 2024-09-21T10:29:08.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002794876 2024-09-21T10:29:08.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.377943+0000", 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027719600000000001 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36246.0:44845 46.f 46:f5110f13:::benchmark_data_smithi119_206743_object44844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1250)", 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:29:08.374548+0000", 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0063625180000000002, 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0064443670000000003, 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:08.390 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36246", 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 44845 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374548+0000", 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:08.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374548+0000", 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374547+0000", 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374656+0000", 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108415 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374674+0000", 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8512000000000002e-05 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374680+0000", 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1510000000000001e-06 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374727+0000", 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6524999999999999e-05 2024-09-21T10:29:08.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.374866+0000", 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013908200000000001 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375025+0000", 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000159495 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.376651+0000", 2024-09-21T10:29:08.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016258150000000001 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.380334+0000", 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036824090000000002 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36246.0:44848 46.6 46:6c29f3f1:::benchmark_data_smithi119_206743_object44847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1250)", 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:29:08.375082+0000", 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0058288380000000002, 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005957286, 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:08.395 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36246", 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 44848 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375082+0000", 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:08.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375082+0000", 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375081+0000", 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375140+0000", 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9553999999999999e-05 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375151+0000", 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0691e-05 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.375163+0000", 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.226e-05 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.376270+0000", 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011070279999999999 2024-09-21T10:29:08.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.376387+0000", 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011723299999999999 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.376539+0000", 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015180599999999999 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:29:08.377829+0000", 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012892660000000001 2024-09-21T10:29:08.399 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.400 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:08.400 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.400 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:08.400 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:29:08.400 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 3 2024-09-21T10:29:08.400 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:29:08.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:08.442 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:08.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:08.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:08.540 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:29:08.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:08.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:08.576 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:29:08.576 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:08.605 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:08.612 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:29:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:08.644 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:29:08.645 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:08.677 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:29:08.677 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:08.707 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:08.709 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:29:08.709 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:08.736 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:08.741 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:29:08.741 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:08.771 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:29:08.771 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:08.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:08.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:08.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:08.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:08.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:09.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:09.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:09.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:29:09.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:09.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:09.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:09.248 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:09.261 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 28 16 45989 45973 102.601 80.9375 0.00625329 0.00974021 2024-09-21T10:29:09.278 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:09.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:09.421 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:09.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:09.466 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:09.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[101432]: pgmap v1827: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 5537/123555 objects degraded (4.481%); 3886/123555 objects misplaced (3.145%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:29:09.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:09.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5537/123555 objects degraded (4.481%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:09.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1876947951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:09.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[105288]: pgmap v1827: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 5537/123555 objects degraded (4.481%); 3886/123555 objects misplaced (3.145%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:29:09.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:09.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5537/123555 objects degraded (4.481%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:09.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1876947951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:09.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:09 smithi145 ceph-mon[89359]: pgmap v1827: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.5 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 5537/123555 objects degraded (4.481%); 3886/123555 objects misplaced (3.145%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:29:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:09 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5537/123555 objects degraded (4.481%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1876947951' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:09.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:09.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:09.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:09.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:09.837 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:09.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:09.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:09.973 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:09.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:10.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:10.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:10.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:29:10.205 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:10.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:10.261 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 29 16 47414 47398 102.134 89.0625 0.00906757 0.00978475 2024-09-21T10:29:10.332 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:10.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:10.461 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:10.488 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:29:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:29:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:10.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:10.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:10.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:10.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:10.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:10.896 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:10.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:10.997 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:11.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:11.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:11.164 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:11.261 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 30 16 48690 48674 101.388 79.75 0.00612366 0.00985616 2024-09-21T10:29:11.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:11.302 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:11.326 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:11 smithi145 ceph-mon[89359]: pgmap v1828: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s; 5537/127935 objects degraded (4.328%); 3886/127935 objects misplaced (3.037%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:29:11.365 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:11.365 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:11.403 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:11.423 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:11.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:11.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:29:11.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:11.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:11.590 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:11 smithi119 ceph-mon[101432]: pgmap v1828: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s; 5537/127935 objects degraded (4.328%); 3886/127935 objects misplaced (3.037%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:29:11.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:11 smithi119 ceph-mon[105288]: pgmap v1828: 99 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 61 MiB/s wr, 973 op/s; 5537/127935 objects degraded (4.328%); 3886/127935 objects misplaced (3.037%); 5.0 MiB/s, 80 objects/s recovering 2024-09-21T10:29:11.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.682 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:11.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:11.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:11.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:11.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:11.952 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:11.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:11.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:12.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:12.088 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:12.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:29:12.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:12.225 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:12.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:12.261 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 31 16 49797 49781 100.349 69.1875 0.00734369 0.00995862 2024-09-21T10:29:12.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:12.394 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:12.423 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:12.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:12.545 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:12.646 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:12.689 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.690 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:12.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:12.791 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:12.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:12.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:12.942 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:12.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:13.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:13.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:13.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:13.217 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:13.261 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 32 16 50911 50895 99.3887 69.625 0.0228503 0.0100542 2024-09-21T10:29:13.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:13.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:13 smithi119 ceph-mon[105288]: pgmap v1829: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 8083/137067 objects degraded (5.897%); 3516/137067 objects misplaced (2.565%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:29:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:13 smithi119 ceph-mon[101432]: pgmap v1829: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 8083/137067 objects degraded (5.897%); 3516/137067 objects misplaced (2.565%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:29:13.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:13.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:13 smithi145 ceph-mon[89359]: pgmap v1829: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.8 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 8083/137067 objects degraded (5.897%); 3516/137067 objects misplaced (2.565%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:29:13.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:13.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:13.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:13.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:13.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:13.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:13.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:13 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:29:13.625+0000 7fc0be6c7640 -1 mgr.server reply reply (3) No such process no change 2024-09-21T10:29:13.885 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:13.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:13.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:14.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:14.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:14.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:14.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:14.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:14.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 33 16 52059 52043 98.5502 71.75 0.00546965 0.0101405 2024-09-21T10:29:14.286 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:14.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:14.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:14.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:14.559 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:29:14] "GET /metrics HTTP/1.1" 200 36885 "" "Prometheus/2.51.0" 2024-09-21T10:29:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:14 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/144033 objects degraded (5.612%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:14 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/144033 objects degraded (5.612%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:14.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:14.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:14.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:14 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/144033 objects degraded (5.612%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:14.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:14.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:14.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:14.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:15.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:15.080 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:15.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:15.208 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:15.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 34 16 53183 53167 97.7177 70.25 0.00976413 0.0102282 2024-09-21T10:29:15.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:15.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:15.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:15.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:15.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:15.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:15 smithi119 ceph-mon[105288]: pgmap v1830: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s; 8083/144033 objects degraded (5.612%); 3516/144033 objects misplaced (2.441%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:15 smithi119 ceph-mon[105288]: from='client.55462 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:29:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:15 smithi119 ceph-mon[101432]: pgmap v1830: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s; 8083/144033 objects degraded (5.612%); 3516/144033 objects misplaced (2.441%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:15 smithi119 ceph-mon[101432]: from='client.55462 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:29:15.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:15.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:15 smithi145 ceph-mon[89359]: pgmap v1830: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 67 MiB/s wr, 1.07k op/s; 8083/144033 objects degraded (5.612%); 3516/144033 objects misplaced (2.441%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:15 smithi145 ceph-mon[89359]: from='client.55462 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:29:15.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:15.846 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:15.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:15.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:15.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:16.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:16.107 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:16.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:16.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:16.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 35 16 54182 54166 96.7096 62.4375 0.00634685 0.0103336 2024-09-21T10:29:16.352 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:16.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:16.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:16.507 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:16.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:16.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:16.644 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:16.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:16.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:16.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:16.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:16.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:16.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:17.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:17.040 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:17.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:17.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:17.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 36 16 55092 55076 95.603 56.875 0.00296384 0.0104183 2024-09-21T10:29:17.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:17.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.299 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:17.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:17.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:17.445 INFO:teuthology.orchestra.run.smithi119.stdout:moved 9 / 297 (3.0303%) 2024-09-21T10:29:17.445 INFO:teuthology.orchestra.run.smithi119.stdout:avg 42.4286 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 7.38448 -> 9.43939 (expected baseline 6.03053) 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 28 -> 23 pgs (0.659933 -> 0.542088 * mean) 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 52 -> 52 pgs (1.22559 -> 1.22559 * mean) 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:oload 110 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 1 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0172 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0189 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 weight 0.8746 -> 0.9099 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.7110 -> 0.6078 2024-09-21T10:29:17.446 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 weight 0.9500 -> 0.8410 2024-09-21T10:29:17.447 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:29:17.447 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:17 smithi119 ceph-mon[105288]: pgmap v1831: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 72 MiB/s wr, 1.14k op/s; 8083/151971 objects degraded (5.319%); 3516/151971 objects misplaced (2.314%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:29:17.447 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:17 smithi119 ceph-mon[101432]: pgmap v1831: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 72 MiB/s wr, 1.14k op/s; 8083/151971 objects degraded (5.319%); 3516/151971 objects misplaced (2.314%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:29:17.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:17 smithi145 ceph-mon[89359]: pgmap v1831: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.1 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 72 MiB/s wr, 1.14k op/s; 8083/151971 objects degraded (5.319%); 3516/151971 objects misplaced (2.314%); 4.0 MiB/s, 63 objects/s recovering 2024-09-21T10:29:17.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:17.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.722 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:17.730 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:17.730 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:17.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.845 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:17.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:17.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:17.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:17.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:18.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:18.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:18.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:18.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:18.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:18.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:18.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 37 16 56002 55986 94.5561 56.875 0.0025848 0.0105442 2024-09-21T10:29:18.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:18.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:18.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:29:18.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:18.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:18.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:18.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:18.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:18.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:18.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:18.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:18.795 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:18.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:18.915 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:18.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:19.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:19.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:19.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:19.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:19.170 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:19.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 38 16 57102 57086 93.8769 68.75 0.0112839 0.0106471 2024-09-21T10:29:19.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:19.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:19.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:19.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:19.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:19.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:19 smithi145 ceph-mon[89359]: pgmap v1832: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/158379 objects degraded (5.104%); 3168/158379 objects misplaced (2.000%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:29:19.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:19.608 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:19 smithi119 ceph-mon[101432]: pgmap v1832: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/158379 objects degraded (5.104%); 3168/158379 objects misplaced (2.000%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:29:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:19 smithi119 ceph-mon[105288]: pgmap v1832: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/158379 objects degraded (5.104%); 3168/158379 objects misplaced (2.000%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:29:19.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:19.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:19.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:19.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:19.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:19.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:19.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:19.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:19.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:20.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:20.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:20.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:20.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:20.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:20.264 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 39 16 58266 58250 93.3351 72.75 0.026524 0.0107081 2024-09-21T10:29:20.280 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:29:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:29:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:20.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:20.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:20.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:20.437 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:20 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/158379 objects degraded (5.104%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:20.438 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:20 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/158379 objects degraded (5.104%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:20.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:20.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:20 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/158379 objects degraded (5.104%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:20.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:20.643 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:20.709 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:20.709 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:20.745 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:20.788 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:20.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:20.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:20.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:21.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:21.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:21.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:21.179 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:29:21.180 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:21.194 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:21.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:21.216 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:29:21.216 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:21.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:29:21.261442+0000 min lat: 0.00134105 max lat: 4.01826 avg lat: 0.0107626 2024-09-21T10:29:21.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:29:21.262 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 40 16 59459 59443 92.8656 74.5625 0.0157044 0.0107626 2024-09-21T10:29:21.264 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:29:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:21.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:21.299 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:29:21.299 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:21.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:21.330 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:29:21.330 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:21.363 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:29:21.364 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:21.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:29:21.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:29:21.397 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:21.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:21.428 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:29:21.428 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:21.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:21.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:21 smithi119 ceph-mon[105288]: pgmap v1833: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 8083/163146 objects degraded (4.954%); 3168/163146 objects misplaced (1.942%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:29:21.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1290703638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:21.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:21 smithi119 ceph-mon[101432]: pgmap v1833: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 8083/163146 objects degraded (4.954%); 3168/163146 objects misplaced (1.942%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:29:21.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1290703638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:21.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:21.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:21.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:21.759 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:21.783 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:21 smithi145 ceph-mon[89359]: pgmap v1833: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 8083/163146 objects degraded (4.954%); 3168/163146 objects misplaced (1.942%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:29:21.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1290703638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:21.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:21.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:21.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:22.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:22.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:22.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:22.263 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 41 16 60657 60641 92.4267 74.875 0.0130767 0.0108142 2024-09-21T10:29:22.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:22.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:22.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:22.460 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:22.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:22.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:22.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:22.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.741 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:22.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:22.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:22.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:22.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:22.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:23.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:23.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:23.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:23.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:23.263 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 42 16 61758 61742 91.8639 68.8125 0.00510405 0.0108792 2024-09-21T10:29:23.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:23.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:23.403 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:23 smithi145 ceph-mon[89359]: pgmap v1834: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 8083/170610 objects degraded (4.738%); 2793/170610 objects misplaced (1.637%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:29:23.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:23.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:23.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:23 smithi119 ceph-mon[101432]: pgmap v1834: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 8083/170610 objects degraded (4.738%); 2793/170610 objects misplaced (1.637%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:29:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:23 smithi119 ceph-mon[105288]: pgmap v1834: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 74 MiB/s wr, 1.18k op/s; 8083/170610 objects degraded (4.738%); 2793/170610 objects misplaced (1.637%); 2.8 MiB/s, 45 objects/s recovering 2024-09-21T10:29:23.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:23.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:23.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:23.805 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:23.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:23.946 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:23.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:23.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:24.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:24.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:24.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:24.208 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:24.263 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 43 16 62949 62933 91.4586 74.4375 0.0107312 0.0109284 2024-09-21T10:29:24.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:24.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:24.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:24.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:24.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:24.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:29:24] "GET /metrics HTTP/1.1" 200 36914 "" "Prometheus/2.51.0" 2024-09-21T10:29:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:24.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:24.718 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:29:24.718 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:29:24.718 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:29:24.718 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:29:24.718 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:29:24.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:24.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:24.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:24.884 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:24.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:24.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:25.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:25.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:25.146 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:25.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:25.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 44 16 64154 64138 91.0879 75.3125 0.0151975 0.0109724 2024-09-21T10:29:25.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:25.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:25.426 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:25 smithi145 ceph-mon[89359]: pgmap v1835: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s; 8083/177099 objects degraded (4.564%); 2793/177099 objects misplaced (1.577%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:29:25.426 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:25 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/177099 objects degraded (4.564%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:25.427 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:25.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:29:25.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:25.573 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:25 smithi119 ceph-mon[101432]: pgmap v1835: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s; 8083/177099 objects degraded (4.564%); 2793/177099 objects misplaced (1.577%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:29:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:25 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/177099 objects degraded (4.564%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:25 smithi119 ceph-mon[105288]: pgmap v1835: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s; 8083/177099 objects degraded (4.564%); 2793/177099 objects misplaced (1.577%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:29:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:25 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/177099 objects degraded (4.564%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:25.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:25.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:25.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:25.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:25.859 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:29:25.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:25.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:25.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:26.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:26.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:26.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:26.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:26.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 45 16 65282 65266 90.6303 70.5 0.00918998 0.0110272 2024-09-21T10:29:26.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:26.385 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:29:26.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:26.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:26.524 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/9216528' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:26 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/9216528' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:26.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:26 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:26.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:26.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:26.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:26.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/9216528' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:26 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:29:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:26.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:29:26.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:26.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:27.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:27.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:27.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:27.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:27.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 46 16 66351 66335 90.1125 66.8125 0.0115033 0.0110911 2024-09-21T10:29:27.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:27.340 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:27.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:27.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:27.500 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:27 smithi145 ceph-mon[89359]: pgmap v1836: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 8083/184845 objects degraded (4.373%); 2793/184845 objects misplaced (1.511%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:29:27.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:27 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:27.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:27 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:29:27.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:27 smithi145 ceph-mon[89359]: osdmap e1251: 8 total, 8 up, 7 in 2024-09-21T10:29:27.508 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:27.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:27.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[101432]: pgmap v1836: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 8083/184845 objects degraded (4.373%); 2793/184845 objects misplaced (1.511%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:29:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:29:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[101432]: osdmap e1251: 8 total, 8 up, 7 in 2024-09-21T10:29:27.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[105288]: pgmap v1836: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.8 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 8083/184845 objects degraded (4.373%); 2793/184845 objects misplaced (1.511%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:29:27.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:27.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:29:27.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:27 smithi119 ceph-mon[105288]: osdmap e1251: 8 total, 8 up, 7 in 2024-09-21T10:29:27.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247717 45.1 45:8d06fae3:::benchmark_data_smithi119_168706_object117547:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.871543+0000", 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.90492439800001, 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.082356224, 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.790 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247717 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871543+0000", 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871543+0000", 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871541+0000", 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871547+0000", 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871558+0000", 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1029000000000001e-05 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871564+0000", 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7540000000000003e-06 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871625+0000", 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1367999999999999e-05 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871635+0000", 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0404000000000001e-05 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.943799+0000", 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0721633340000001 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.943935+0000", 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013609700000000001 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.944030+0000", 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4760999999999996e-05 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953473+0000", 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094437940000000001 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953655+0000", 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018141899999999999 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953705+0000", 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0417000000000002e-05 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953741+0000", 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.578e-05 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953775+0000", 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4335000000000002e-05 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953899+0000", 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012356199999999999 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247775 45.1 45:8c7efd63:::benchmark_data_smithi119_168706_object117605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.928030+0000", 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.848437297, 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0314164690000001, 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247775 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.797 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928030+0000", 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928030+0000", 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928028+0000", 2024-09-21T10:29:27.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928036+0000", 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0330000000000008e-06 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928050+0000", 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3959e-05 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928056+0000", 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2929999999999997e-06 2024-09-21T10:29:27.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928137+0000", 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1064999999999997e-05 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928149+0000", 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1309e-05 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.946541+0000", 2024-09-21T10:29:27.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183918620000001 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.946625+0000", 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4000999999999998e-05 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.946695+0000", 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0251000000000002e-05 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953666+0000", 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069715849999999998 2024-09-21T10:29:27.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953945+0000", 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027831900000000002 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953974+0000", 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9479999999999999e-05 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959272+0000", 2024-09-21T10:29:27.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052979000000000004 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959326+0000", 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4128000000000001e-05 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959446+0000", 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011993999999999999 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.803 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247780 45.1 45:87c042b8:::benchmark_data_smithi119_168706_object117610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.930239+0000", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.846228266, 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.029423786, 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247780 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930239+0000", 2024-09-21T10:29:27.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930239+0000", 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930237+0000", 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930246+0000", 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1410000000000005e-06 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930263+0000", 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7158000000000001e-05 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930269+0000", 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9290000000000003e-06 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930320+0000", 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1496999999999998e-05 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930330+0000", 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4609999999999997e-06 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.950055+0000", 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.019725067 2024-09-21T10:29:27.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.950204+0000", 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000149574 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.950334+0000", 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129231 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953712+0000", 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003378579 2024-09-21T10:29:27.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954001+0000", 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028842400000000002 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954035+0000", 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4539999999999998e-05 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959552+0000", 2024-09-21T10:29:27.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055170239999999997 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959601+0000", 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8526000000000003e-05 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959663+0000", 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1914000000000004e-05 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247802 45.1 45:83bacf6a:::benchmark_data_smithi119_168706_object117632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.962464+0000", 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.814002702, 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.003117987, 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247802 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962464+0000", 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962464+0000", 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962463+0000", 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.812 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962472+0000", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5240000000000006e-06 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962486+0000", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3929e-05 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962492+0000", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9689999999999999e-06 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.813 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962546+0000", 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4026000000000001e-05 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962557+0000", 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1168e-05 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954060+0000", 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99150294900000002 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.814 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954155+0000", 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5129000000000003e-05 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954236+0000", 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0541e-05 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959459+0000", 2024-09-21T10:29:27.815 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052234550000000001 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959500+0000", 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0667999999999998e-05 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965432+0000", 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059324970000000001 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965490+0000", 2024-09-21T10:29:27.816 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7763000000000002e-05 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965523+0000", 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2475999999999999e-05 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965582+0000", 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9605000000000003e-05 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.817 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247853 45.1 45:80a3af5b:::benchmark_data_smithi119_168706_object117683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:45.036160+0000", 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.74030685300001, 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93760746900000003, 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247853 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.818 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036160+0000", 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036160+0000", 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036159+0000", 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.819 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036165+0000", 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3790000000000003e-06 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036176+0000", 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0701e-05 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036180+0000", 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.549e-06 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.820 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036250+0000", 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0099999999999996e-05 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036265+0000", 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5021999999999999e-05 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.956746+0000", 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.920480717 2024-09-21T10:29:27.821 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.956864+0000", 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117983 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.956958+0000", 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3844000000000005e-05 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959507+0000", 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025486380000000002 2024-09-21T10:29:27.822 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965385+0000", 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058785820000000003 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965642+0000", 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000256934 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973614+0000", 2024-09-21T10:29:27.823 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007971512 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973664+0000", 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0219999999999997e-05 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973768+0000", 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103717 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.824 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247870 45.1 45:8e84670f:::benchmark_data_smithi119_168706_object117700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:45.058481+0000", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.717985945, 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92971519400000002, 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247870 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058481+0000", 2024-09-21T10:29:27.825 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058481+0000", 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058479+0000", 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058488+0000", 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3729999999999993e-06 2024-09-21T10:29:27.826 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058507+0000", 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9595000000000001e-05 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058514+0000", 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4690000000000002e-06 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058571+0000", 2024-09-21T10:29:27.827 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7747999999999997e-05 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058582+0000", 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0713999999999999e-05 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959687+0000", 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.90110485399999996 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.828 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.829 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959788+0000", 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100725 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959872+0000", 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4790000000000006e-05 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965438+0000", 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055659860000000002 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.830 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965466+0000", 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8061000000000001e-05 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973522+0000", 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080556440000000007 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988008+0000", 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014486357 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.831 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988102+0000", 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3778999999999998e-05 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988196+0000", 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3980000000000005e-05 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247882 45.1 45:8dc2ee87:::benchmark_data_smithi119_168706_object117712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:45.077462+0000", 2024-09-21T10:29:27.832 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.69900478700001, 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.91105475700000005, 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247882 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077462+0000", 2024-09-21T10:29:27.833 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077462+0000", 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077461+0000", 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.834 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077469+0000", 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3420000000000008e-06 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077484+0000", 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4609e-05 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077490+0000", 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3910000000000003e-06 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.835 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077540+0000", 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9641999999999997e-05 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077557+0000", 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7564e-05 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.962124+0000", 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.88456663099999999 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.836 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.962246+0000", 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012238200000000001 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.962380+0000", 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013418600000000001 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965589+0000", 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032087869999999998 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.837 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965616+0000", 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.652e-05 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973581+0000", 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079651059999999996 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988331+0000", 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014750535 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.838 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988406+0000", 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4847999999999999e-05 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988517+0000", 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110826 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250064 45.1 45:811e61af:::benchmark_data_smithi119_168706_object119894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1224)", 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.889381+0000", 2024-09-21T10:29:27.839 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.887086471999993, 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0805241029999999, 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250064 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889381+0000", 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.840 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889381+0000", 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889379+0000", 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889387+0000", 2024-09-21T10:29:27.841 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.255e-06 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889398+0000", 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1972e-05 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889405+0000", 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9990000000000002e-06 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889466+0000", 2024-09-21T10:29:27.842 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0007000000000003e-05 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889582+0000", 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116004 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889691+0000", 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010945700000000001 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.843 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.893378+0000", 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036874310000000001 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.894674+0000", 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012952809999999999 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.894806+0000", 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132228 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.844 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896376+0000", 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015699990000000001 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896387+0000", 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0861e-05 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.969848+0000", 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.073460954 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.845 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.969905+0000", 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6904000000000003e-05 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250069 45.1 45:88025a9d:::benchmark_data_smithi119_168706_object119899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1224)", 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.895697+0000", 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.880769951000005, 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0901574789999999, 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.846 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250069 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895697+0000", 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895697+0000", 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895695+0000", 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895703+0000", 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2629999999999997e-06 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895714+0000", 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1314e-05 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.848 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895720+0000", 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5509999999999999e-06 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896438+0000", 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000718923 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896447+0000", 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0189999999999995e-06 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.849 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.970354+0000", 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.073906721 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.970448+0000", 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3754999999999999e-05 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.970548+0000", 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100361 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.850 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981595+0000", 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01104657 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985700+0000", 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041046839999999999 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985752+0000", 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2450000000000001e-05 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.851 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985800+0000", 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8191e-05 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985855+0000", 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4317000000000002e-05 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250099 45.1 45:8e8c7d4b:::benchmark_data_smithi119_168706_object119929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.852 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.936855+0000", 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.83961214, 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.049148078, 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250099 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936855+0000", 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936855+0000", 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936853+0000", 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936860+0000", 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6030000000000001e-06 2024-09-21T10:29:27.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936872+0000", 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.224e-05 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936878+0000", 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8629999999999996e-06 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936935+0000", 2024-09-21T10:29:27.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7578000000000003e-05 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936945+0000", 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6220000000000004e-06 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.973862+0000", 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.036916766 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.973977+0000", 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115418 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.974098+0000", 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012074999999999999 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985644+0000", 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011546307 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985896+0000", 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025206000000000002 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985931+0000", 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4465999999999998e-05 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985969+0000", 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7996e-05 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986003+0000", 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4224000000000003e-05 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250104 45.1 45:82dcfcfc:::benchmark_data_smithi119_168706_object119934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.947258+0000", 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.829208765000004, 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0388857600000001, 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.859 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250104 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947258+0000", 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947258+0000", 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947257+0000", 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947263+0000", 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7169999999999996e-06 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947273+0000", 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0069e-05 2024-09-21T10:29:27.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947279+0000", 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1940000000000003e-06 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947338+0000", 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8733999999999997e-05 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947348+0000", 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0385e-05 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.977450+0000", 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.030102267 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.977577+0000", 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012686699999999999 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.977695+0000", 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117941 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985864+0000", 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081691890000000003 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986039+0000", 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017473099999999999 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986074+0000", 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4434000000000003e-05 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986109+0000", 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5167999999999999e-05 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986144+0000", 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5281000000000003e-05 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250115 45.1 45:8f62f6be:::benchmark_data_smithi119_168706_object119945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.955841+0000", 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.820625543000006, 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.030431326, 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250115 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955841+0000", 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955841+0000", 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955840+0000", 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:27.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955847+0000", 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7849999999999992e-06 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955862+0000", 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4928e-05 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955869+0000", 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.63e-06 2024-09-21T10:29:27.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955926+0000", 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7465e-05 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955937+0000", 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0512e-05 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981618+0000", 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0256806570000001 2024-09-21T10:29:27.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981750+0000", 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132218 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981868+0000", 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117701 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986012+0000", 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004144483 2024-09-21T10:29:27.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986171+0000", 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015925499999999999 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986206+0000", 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4699000000000001e-05 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986245+0000", 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8682e-05 2024-09-21T10:29:27.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986273+0000", 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8072000000000002e-05 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250129 45.1 45:8156bc2a:::benchmark_data_smithi119_168706_object119959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.978102+0000", 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.798364939999999, 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0242735489999999, 2024-09-21T10:29:27.872 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250129 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978102+0000", 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978102+0000", 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978100+0000", 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978109+0000", 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2269999999999993e-06 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978123+0000", 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3934000000000001e-05 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978128+0000", 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9610000000000002e-06 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978188+0000", 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9478999999999998e-05 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978198+0000", 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0084999999999999e-05 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986300+0000", 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0081023689999999 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986412+0000", 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112009 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.987281+0000", 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00086866799999999996 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002213+0000", 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014931411 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002243+0000", 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0434999999999999e-05 2024-09-21T10:29:27.877 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002283+0000", 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0145000000000003e-05 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002341+0000", 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7686999999999998e-05 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002376+0000", 2024-09-21T10:29:27.878 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4832000000000001e-05 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250140 45.1 45:8f92095e:::benchmark_data_smithi119_168706_object119970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.994583+0000", 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.781883966999999, 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.012726478, 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.879 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250140 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994583+0000", 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994583+0000", 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.880 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994582+0000", 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994588+0000", 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3419999999999996e-06 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994600+0000", 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.171e-05 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.881 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.882 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.882 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994604+0000", 2024-09-21T10:29:27.882 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2910000000000001e-06 2024-09-21T10:29:27.882 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.882 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.883 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.883 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994652+0000", 2024-09-21T10:29:27.883 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7491000000000003e-05 2024-09-21T10:29:27.883 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.883 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994660+0000", 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1459999999999996e-06 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.991345+0000", 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99668568599999996 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.991462+0000", 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116952 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.884 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.991583+0000", 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120221 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002277+0000", 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010693945999999999 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004725+0000", 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024486170000000002 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.885 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007227+0000", 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002502251 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007271+0000", 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4005999999999998e-05 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007310+0000", 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8090000000000003e-05 2024-09-21T10:29:27.886 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250143 45.1 45:87c2933b:::benchmark_data_smithi119_168706_object119973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.994838+0000", 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.781629203999998, 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0229272229999999, 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.887 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250143 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994838+0000", 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994838+0000", 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.888 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994837+0000", 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994851+0000", 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.361e-05 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994860+0000", 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4569999999999994e-06 2024-09-21T10:29:27.889 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994864+0000", 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0400000000000003e-06 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994901+0000", 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6758000000000003e-05 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994910+0000", 2024-09-21T10:29:27.890 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7630000000000006e-06 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.995056+0000", 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001458059999999 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.995155+0000", 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9439999999999997e-05 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.995260+0000", 2024-09-21T10:29:27.891 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010448 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002424+0000", 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071644669999999999 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004731+0000", 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023075130000000002 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017679+0000", 2024-09-21T10:29:27.892 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012947157000000001 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017727+0000", 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8367e-05 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017765+0000", 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.799e-05 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.893 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250157 45.1 45:89d7408f:::benchmark_data_smithi119_168706_object119987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.012165+0000", 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.764302009999994, 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.011180382, 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250157 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.894 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012165+0000", 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012165+0000", 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012163+0000", 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.895 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012171+0000", 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012183+0000", 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2571999999999999e-05 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012189+0000", 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7429999999999999e-06 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.896 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012297+0000", 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108449 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012312+0000", 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4783000000000001e-05 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.998447+0000", 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98613498200000005 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.897 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.998564+0000", 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116338 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.998700+0000", 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136563 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017640+0000", 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018939837000000001 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.898 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017842+0000", 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000202461 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.023253+0000", 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054105079999999996 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.023304+0000", 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0963e-05 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.899 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.023345+0000", 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1523000000000002e-05 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250165 45.1 45:80acb05c:::benchmark_data_smithi119_168706_object119995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.024689+0000", 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.751778283999997, 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0049197569999999, 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.900 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250165 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024689+0000", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024689+0000", 2024-09-21T10:29:27.901 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024686+0000", 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024693+0000", 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9380000000000003e-06 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024704+0000", 2024-09-21T10:29:27.902 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024709+0000", 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3950000000000003e-06 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024747+0000", 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8676e-05 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024754+0000", 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9689999999999997e-06 2024-09-21T10:29:27.903 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002491+0000", 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97773674899999996 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002560+0000", 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9281000000000003e-05 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002626+0000", 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6241000000000005e-05 2024-09-21T10:29:27.904 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017774+0000", 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015147282999999999 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.022962+0000", 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051881219999999999 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.905 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029514+0000", 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065526150000000003 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029569+0000", 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4413000000000002e-05 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029608+0000", 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.906 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250168 45.1 45:89fc89dc:::benchmark_data_smithi119_168706_object119998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.027310+0000", 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.749157483999994, 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.002409452, 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250168 2024-09-21T10:29:27.907 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027310+0000", 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027310+0000", 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.908 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027308+0000", 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027314+0000", 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4700000000000001e-06 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027324+0000", 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.046e-05 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.909 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027330+0000", 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7790000000000001e-06 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027374+0000", 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4058999999999999e-05 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027381+0000", 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0060000000000003e-06 2024-09-21T10:29:27.910 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004774+0000", 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97739246800000001 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004905+0000", 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013101599999999999 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.911 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.005007+0000", 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010286800000000001 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017780+0000", 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012772451000000001 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.024728+0000", 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069478860000000003 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.912 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029647+0000", 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0049195369999999999 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029683+0000", 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5939000000000003e-05 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029719+0000", 2024-09-21T10:29:27.913 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5651000000000001e-05 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250187 45.1 45:8e1f7a06:::benchmark_data_smithi119_168706_object120017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.051926+0000", 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.724541075000005, 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98551227200000002, 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.914 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250187 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051926+0000", 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051926+0000", 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.915 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051924+0000", 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051930+0000", 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.916 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051940+0000", 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7550000000000006e-06 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051946+0000", 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1090000000000003e-06 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051992+0000", 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5760000000000002e-05 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.917 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051999+0000", 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.545e-06 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007338+0000", 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.955339406 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007429+0000", 2024-09-21T10:29:27.918 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1113000000000007e-05 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007525+0000", 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5538999999999994e-05 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017785+0000", 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010260686999999999 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.919 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029758+0000", 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01197295 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037352+0000", 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0075938009999999998 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037401+0000", 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9054999999999997e-05 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.920 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037438+0000", 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7085000000000002e-05 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250204 45.1 45:81b5fc70:::benchmark_data_smithi119_168706_object120034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.076035+0000", 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 98.700431523999995, 2024-09-21T10:29:27.921 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96149175200000003, 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250204 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076035+0000", 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.922 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076035+0000", 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076033+0000", 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076041+0000", 2024-09-21T10:29:27.923 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4780000000000002e-06 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076055+0000", 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4419999999999999e-05 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076061+0000", 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.896e-06 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.924 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076117+0000", 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5529999999999999e-05 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076126+0000", 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8030000000000008e-06 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.010934+0000", 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93480773500000003 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:29:27.925 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.011035+0000", 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010054 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.011406+0000", 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037157200000000003 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017791+0000", 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006384514 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.926 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029786+0000", 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011995653 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037467+0000", 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076803369999999998 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037498+0000", 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1451000000000001e-05 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:29:27.927 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037527+0000", 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9031e-05 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:29:27.928 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:29:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:27.929 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:27.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:28.029 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:28.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.068 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:28.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:28.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:28.206 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:28.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 47 16 67484 67468 89.7017 70.8125 0.00912085 0.0111425 2024-09-21T10:29:28.307 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:28.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:28.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:28.437 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:28.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:28.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:28.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:28.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:28.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:28.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:28.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:28.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:28.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:28.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:28.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:29.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:29.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:29:29.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:29.142 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:29.243 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:29.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 48 16 68613 68597 89.3029 70.5625 0.0160648 0.0111904 2024-09-21T10:29:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:29.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:29.354 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:29:29.354 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:29:29.354 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:29:29.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:29:29.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:29:29.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:29 smithi119 ceph-mon[101432]: pgmap v1838: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 8083/191358 objects degraded (4.224%); 2338/191358 objects misplaced (1.222%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:29:29.425 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:29 smithi119 ceph-mon[105288]: pgmap v1838: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 8083/191358 objects degraded (4.224%); 2338/191358 objects misplaced (1.222%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:29:29.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:29.524 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:29.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:29.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:29.710 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:29.721 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:29:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:29.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:29.761 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:29:29.761 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:29.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:29 smithi145 ceph-mon[89359]: pgmap v1838: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s; 8083/191358 objects degraded (4.224%); 2338/191358 objects misplaced (1.222%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:29:29.800 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:29:29.800 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:29.835 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:29:29.835 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:29.864 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:29:29.865 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:29.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:29.901 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:29:29.902 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:29.904 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:29.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:29.930 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:29:29.930 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:29.959 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:29:29.959 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:30.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:30.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:30.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:30.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:30.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 49 16 69508 69492 88.6219 55.9375 0.0116586 0.0112774 2024-09-21T10:29:30.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:30.312 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:30.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:30.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:30.453 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:30 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/191358 objects degraded (4.224%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:30 smithi145 ceph-mon[89359]: pgmap v1839: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/196716 objects degraded (4.109%); 2338/196716 objects misplaced (1.189%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:29:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1262427970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:29:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:29:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:30.554 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:30.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:30.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:30.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:30.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:30.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:30.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:30.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:30 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/191358 objects degraded (4.224%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:30.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:30 smithi119 ceph-mon[105288]: pgmap v1839: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/196716 objects degraded (4.109%); 2338/196716 objects misplaced (1.189%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:29:30.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1262427970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:30.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:30 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/191358 objects degraded (4.224%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:30.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:30 smithi119 ceph-mon[101432]: pgmap v1839: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/196716 objects degraded (4.109%); 2338/196716 objects misplaced (1.189%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:29:30.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1262427970' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:30.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:30.983 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:30.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:31.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:31.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:31.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:31.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:31.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:31.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:31.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 50 16 70646 70630 88.2719 71.125 0.0254903 0.0113232 2024-09-21T10:29:31.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:31.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:31.475 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:31.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:31.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:31.644 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:31.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:31.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.782 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:31.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:31.869 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:31.869 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:31.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:31.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:31.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:31.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:32.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:32.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:32.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:32.189 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:32.265 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 51 16 71744 71728 87.8866 68.625 0.0150161 0.0113722 2024-09-21T10:29:32.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:32.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:32.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:32.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:32.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:32.605 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:32.703 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:32 smithi119 ceph-mon[101432]: pgmap v1840: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 8083/204939 objects degraded (3.944%); 1818/204939 objects misplaced (0.887%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:32 smithi119 ceph-mon[105288]: pgmap v1840: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 8083/204939 objects degraded (3.944%); 1818/204939 objects misplaced (0.887%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:32.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:32.750 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:32.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:32.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:32.873 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:32.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:32.974 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:33.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:33.022 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:32 smithi145 ceph-mon[89359]: pgmap v1840: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 71 MiB/s wr, 1.14k op/s; 8083/204939 objects degraded (3.944%); 1818/204939 objects misplaced (0.887%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:33.110 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:33.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:33.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:33.267 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 52 15 72801 72786 87.4655 66.125 0.00780192 0.0114275 2024-09-21T10:29:33.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:33.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:33.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:33.413 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:33.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:29:33.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:33.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:33.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:33.667 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:33.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:33.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:33.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:33.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:33.967 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:33.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:34.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:34.096 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:34.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:34.225 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:34.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:34.267 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 53 16 73795 73779 86.9861 62.0625 0.0371187 0.0114905 2024-09-21T10:29:34.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:34.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.367 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:34.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:34.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:34.495 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:34.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:34.622 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:34.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:34.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:34 smithi119 ceph-mon[101432]: pgmap v1841: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/210801 objects degraded (3.834%); 1818/210801 objects misplaced (0.862%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:34 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/210801 objects degraded (3.834%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:34.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:29:34] "GET /metrics HTTP/1.1" 200 36917 "" "Prometheus/2.51.0" 2024-09-21T10:29:34.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:34 smithi119 ceph-mon[105288]: pgmap v1841: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/210801 objects degraded (3.834%); 1818/210801 objects misplaced (0.862%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:34.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:34 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/210801 objects degraded (3.834%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:34.874 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:34.898 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:34.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:35.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:35.026 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:34 smithi145 ceph-mon[89359]: pgmap v1841: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s; 8083/210801 objects degraded (3.834%); 1818/210801 objects misplaced (0.862%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:34 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/210801 objects degraded (3.834%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:35.127 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:35.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:35.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:35.267 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 54 16 74677 74661 86.3961 55.125 0.0178935 0.0115662 2024-09-21T10:29:35.282 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:35.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:35.421 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:35.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:35.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:35.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:35.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:35.690 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:35.791 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:35.816 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:35.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:35.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:35.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:36.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:36.091 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:36.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:36.245 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:36.268 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 55 16 75596 75580 85.8681 57.4375 0.0109584 0.0116399 2024-09-21T10:29:36.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:36.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:36.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:36.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:36.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:36.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:36.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:36.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:36.810 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:36.879 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:36 smithi119 ceph-mon[101432]: pgmap v1842: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s; 8083/217692 objects degraded (3.713%); 1818/217692 objects misplaced (0.835%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:36.880 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:36 smithi119 ceph-mon[105288]: pgmap v1842: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s; 8083/217692 objects degraded (3.713%); 1818/217692 objects misplaced (0.835%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:36.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:36.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:36.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:36 smithi145 ceph-mon[89359]: pgmap v1842: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 68 MiB/s wr, 1.09k op/s; 8083/217692 objects degraded (3.713%); 1818/217692 objects misplaced (0.835%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:37.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:37.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:37.114 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:29:37.116 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:29:37.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:37.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:37.268 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 56 16 76528 76512 85.3749 58.25 0.0131307 0.0117067 2024-09-21T10:29:37.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:37.348 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:37.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:37.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:37.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:37.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:37.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:37.657 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:29:37.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:37.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:37.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:37.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:37.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:37.896 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1842,"stamp":"2024-09-21T10:29:35.555608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4754434419,"num_objects":72564,"num_object_clones":0,"num_object_copies":217692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8083,"num_objects_misplaced":1818,"num_objects_unfound":0,"num_objects_dirty":72564,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":72614,"num_write_kb":4642547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2124,"num_bytes_recovered":137745611,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73695,"ondisk_log_size":73695,"up":297,"acting":292,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":303,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16304120,"kb_used_data":14191872,"kb_used_omap":729,"kb_used_meta":2111462,"kb_avail":639911944,"statfs":{"total":671965249536,"available":655269830656,"internally_reserved":0,"allocated":14532476928,"data_stored":14497184716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":747440,"internal_metadata":2162137168},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":717553664,"num_objects":10949,"num_object_clones":0,"num_object_copies":32847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-975,"num_objects_unfound":0,"num_objects_dirty":10949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10949,"num_write_kb":700736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":31981568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007917"},"pg_stats":[{"pgid":"46.f","version":"1251'4506","reported_seq":4528,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:32.264365+0000","last_change":"2024-09-21T10:28:29.312961+0000","last_active":"2024-09-21T10:29:32.264365+0000","last_peered":"2024-09-21T10:29:32.264365+0000","last_clean":"2024-09-21T10:29:32.264365+0000","last_became_active":"2024-09-21T10:28:29.312714+0000","last_became_peered":"2024-09-21T10:28:29.312714+0000","last_unstale":"2024-09-21T10:29:32.264365+0000","last_undegraded":"2024-09-21T10:29:32.264365+0000","last_fullsized":"2024-09-21T10:29:32.264365+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4506,"log_dups_size":0,"ondisk_log_size":4506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:21:02.479803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295305216,"num_objects":4506,"num_object_clones":0,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4506,"num_write_kb":288384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.e","version":"1251'4650","reported_seq":4689,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:34.797746+0000","last_change":"2024-09-21T10:28:29.308128+0000","last_active":"2024-09-21T10:29:34.797746+0000","last_peered":"2024-09-21T10:29:34.797746+0000","last_clean":"2024-09-21T10:29:34.797746+0000","last_became_active":"2024-09-21T10:28:29.307818+0000","last_became_peered":"2024-09-21T10:28:29.307818+0000","last_unstale":"2024-09-21T10:29:34.797746+0000","last_undegraded":"2024-09-21T10:29:34.797746+0000","last_fullsized":"2024-09-21T10:29:34.797746+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:09:36.193258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304742400,"num_objects":4650,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4650,"num_write_kb":297600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.d","version":"1251'4548","reported_seq":4587,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:32.958959+0000","last_change":"2024-09-21T10:28:29.310826+0000","last_active":"2024-09-21T10:29:32.958959+0000","last_peered":"2024-09-21T10:29:32.958959+0000","last_clean":"2024-09-21T10:29:32.958959+0000","last_became_active":"2024-09-21T10:28:29.310534+0000","last_became_peered":"2024-09-21T10:28:29.310534+0000","last_unstale":"2024-09-21T10:29:32.958959+0000","last_undegraded":"2024-09-21T10:29:32.958959+0000","last_fullsized":"2024-09-21T10:29:32.958959+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:08.140475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.c","version":"1251'4625","reported_seq":4664,"reported_epoch":1251,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:34.804977+0000","last_change":"2024-09-21T10:28:59.762600+0000","last_active":"2024-09-21T10:29:34.804977+0000","last_peered":"2024-09-21T10:29:34.804977+0000","last_clean":"2024-09-21T10:28:55.798572+0000","last_became_active":"2024-09-21T10:28:59.748020+0000","last_became_peered":"2024-09-21T10:28:59.748020+0000","last_unstale":"2024-09-21T10:29:34.804977+0000","last_undegraded":"2024-09-21T10:28:59.725349+0000","last_fullsized":"2024-09-21T10:28:59.723310+0000","mapping_epoch":1245,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303104000,"num_objects":4625,"num_object_clones":0,"num_object_copies":13875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4625,"num_write_kb":296000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":2136},{"shards":"3,7","objects":2489}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.b","version":"1251'4432","reported_seq":4470,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:33.029226+0000","last_change":"2024-09-21T10:28:29.312942+0000","last_active":"2024-09-21T10:29:33.029226+0000","last_peered":"2024-09-21T10:29:33.029226+0000","last_clean":"2024-09-21T10:29:33.029226+0000","last_became_active":"2024-09-21T10:28:29.312596+0000","last_became_peered":"2024-09-21T10:28:29.312596+0000","last_unstale":"2024-09-21T10:29:33.029226+0000","last_undegraded":"2024-09-21T10:29:33.029226+0000","last_fullsized":"2024-09-21T10:29:33.029226+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:19:57.217872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290455552,"num_objects":4432,"num_object_clones":0,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4432,"num_write_kb":283648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.a","version":"1251'4616","reported_seq":4638,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:33.676152+0000","last_change":"2024-09-21T10:28:29.309191+0000","last_active":"2024-09-21T10:29:33.676152+0000","last_peered":"2024-09-21T10:29:33.676152+0000","last_clean":"2024-09-21T10:29:33.676152+0000","last_became_active":"2024-09-21T10:28:29.308876+0000","last_became_peered":"2024-09-21T10:28:29.308876+0000","last_unstale":"2024-09-21T10:29:33.676152+0000","last_undegraded":"2024-09-21T10:29:33.676152+0000","last_fullsized":"2024-09-21T10:29:33.676152+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:38.414144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302514176,"num_objects":4616,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4616,"num_write_kb":295424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.9","version":"1251'4519","reported_seq":4558,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:33.023065+0000","last_change":"2024-09-21T10:28:29.309362+0000","last_active":"2024-09-21T10:29:33.023065+0000","last_peered":"2024-09-21T10:29:33.023065+0000","last_clean":"2024-09-21T10:29:33.023065+0000","last_became_active":"2024-09-21T10:28:29.309079+0000","last_became_peered":"2024-09-21T10:28:29.309079+0000","last_unstale":"2024-09-21T10:29:33.023065+0000","last_undegraded":"2024-09-21T10:29:33.023065+0000","last_fullsized":"2024-09-21T10:29:33.023065+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4519,"log_dups_size":0,"ondisk_log_size":4519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:57:22.010069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296157184,"num_objects":4519,"num_object_clones":0,"num_object_copies":13557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4519,"num_write_kb":289216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.8","version":"1251'4411","reported_seq":4450,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:30.727460+0000","last_change":"2024-09-21T10:28:29.310126+0000","last_active":"2024-09-21T10:29:30.727460+0000","last_peered":"2024-09-21T10:29:30.727460+0000","last_clean":"2024-09-21T10:29:30.727460+0000","last_became_active":"2024-09-21T10:28:29.309688+0000","last_became_peered":"2024-09-21T10:28:29.309688+0000","last_unstale":"2024-09-21T10:29:30.727460+0000","last_undegraded":"2024-09-21T10:29:30.727460+0000","last_fullsized":"2024-09-21T10:29:30.727460+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4411,"log_dups_size":0,"ondisk_log_size":4411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:59.367638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289079296,"num_objects":4411,"num_object_clones":0,"num_object_copies":13233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":282304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.7","version":"1251'4701","reported_seq":4740,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:34.797147+0000","last_change":"2024-09-21T10:28:29.314114+0000","last_active":"2024-09-21T10:29:34.797147+0000","last_peered":"2024-09-21T10:29:34.797147+0000","last_clean":"2024-09-21T10:29:34.797147+0000","last_became_active":"2024-09-21T10:28:29.313435+0000","last_became_peered":"2024-09-21T10:28:29.313435+0000","last_unstale":"2024-09-21T10:29:34.797147+0000","last_undegraded":"2024-09-21T10:29:34.797147+0000","last_fullsized":"2024-09-21T10:29:34.797147+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4701,"log_dups_size":0,"ondisk_log_size":4701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:04.845295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308084736,"num_objects":4701,"num_object_clones":0,"num_object_copies":14103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4701,"num_write_kb":300864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.6","version":"1251'4483","reported_seq":4505,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:32.265990+0000","last_change":"2024-09-21T10:28:29.313102+0000","last_active":"2024-09-21T10:29:32.265990+0000","last_peered":"2024-09-21T10:29:32.265990+0000","last_clean":"2024-09-21T10:29:32.265990+0000","last_became_active":"2024-09-21T10:28:29.312807+0000","last_became_peered":"2024-09-21T10:28:29.312807+0000","last_unstale":"2024-09-21T10:29:32.265990+0000","last_undegraded":"2024-09-21T10:29:32.265990+0000","last_fullsized":"2024-09-21T10:29:32.265990+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4483,"log_dups_size":0,"ondisk_log_size":4483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:53:47.070266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293797888,"num_objects":4483,"num_object_clones":0,"num_object_copies":13449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4483,"num_write_kb":286912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.5","version":"1251'4674","reported_seq":4717,"reported_epoch":1251,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:34.802859+0000","last_change":"2024-09-21T10:28:57.888253+0000","last_active":"2024-09-21T10:29:34.802859+0000","last_peered":"2024-09-21T10:29:34.802859+0000","last_clean":"2024-09-21T10:28:55.800958+0000","last_became_active":"2024-09-21T10:28:57.884783+0000","last_became_peered":"2024-09-21T10:28:57.884783+0000","last_unstale":"2024-09-21T10:29:34.802859+0000","last_undegraded":"2024-09-21T10:28:57.881309+0000","last_fullsized":"2024-09-21T10:28:57.807670+0000","mapping_epoch":1245,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306315264,"num_objects":4674,"num_object_clones":0,"num_object_copies":14022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4674,"num_write_kb":299136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4","objects":2564},{"shards":"3,4,6","objects":2110}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.4","version":"1251'4534","reported_seq":4556,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:33.685135+0000","last_change":"2024-09-21T10:28:29.310707+0000","last_active":"2024-09-21T10:29:33.685135+0000","last_peered":"2024-09-21T10:29:33.685135+0000","last_clean":"2024-09-21T10:29:33.685135+0000","last_became_active":"2024-09-21T10:28:29.310429+0000","last_became_peered":"2024-09-21T10:28:29.310429+0000","last_unstale":"2024-09-21T10:29:33.685135+0000","last_undegraded":"2024-09-21T10:29:33.685135+0000","last_fullsized":"2024-09-21T10:29:33.685135+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4534,"log_dups_size":0,"ondisk_log_size":4534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:40:14.041852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297140224,"num_objects":4534,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4534,"num_write_kb":290176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.3","version":"1251'4419","reported_seq":4457,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:30.743497+0000","last_change":"2024-09-21T10:28:29.314070+0000","last_active":"2024-09-21T10:29:30.743497+0000","last_peered":"2024-09-21T10:29:30.743497+0000","last_clean":"2024-09-21T10:29:30.743497+0000","last_became_active":"2024-09-21T10:28:29.313767+0000","last_became_peered":"2024-09-21T10:28:29.313767+0000","last_unstale":"2024-09-21T10:29:30.743497+0000","last_undegraded":"2024-09-21T10:29:30.743497+0000","last_fullsized":"2024-09-21T10:29:30.743497+0000","mapping_epoch":1237,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4419,"log_dups_size":0,"ondisk_log_size":4419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:03.859098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289603584,"num_objects":4419,"num_object_clones":0,"num_object_copies":13257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4419,"num_write_kb":282816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.2","version":"1251'4468","reported_seq":6334,"reported_epoch":1251,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:32.263165+0000","last_change":"2024-09-21T10:29:04.343746+0000","last_active":"2024-09-21T10:29:32.263165+0000","last_peered":"2024-09-21T10:29:32.263165+0000","last_clean":"2024-09-21T10:28:55.794029+0000","last_became_active":"2024-09-21T10:28:57.880742+0000","last_became_peered":"2024-09-21T10:28:57.880742+0000","last_unstale":"2024-09-21T10:29:32.263165+0000","last_undegraded":"2024-09-21T10:29:04.343113+0000","last_fullsized":"2024-09-21T10:28:57.807383+0000","mapping_epoch":1245,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4468,"log_dups_size":0,"ondisk_log_size":4468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:25.125355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292814848,"num_objects":4468,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1291,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4468,"num_write_kb":285952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":912,"num_bytes_recovered":59768832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":1291},{"shards":"2,6","objects":3177}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.1","version":"1251'4345","reported_seq":7939,"reported_epoch":1251,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:29:30.753256+0000","last_change":"2024-09-21T10:29:04.349107+0000","last_active":"2024-09-21T10:29:30.753256+0000","last_peered":"2024-09-21T10:29:30.753256+0000","last_clean":"2024-09-21T10:28:55.792832+0000","last_became_active":"2024-09-21T10:28:57.914395+0000","last_became_peered":"2024-09-21T10:28:57.914395+0000","last_unstale":"2024-09-21T10:29:30.753256+0000","last_undegraded":"2024-09-21T10:29:30.753256+0000","last_fullsized":"2024-09-21T10:28:57.808492+0000","mapping_epoch":1245,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284753920,"num_objects":4345,"num_object_clones":0,"num_object_copies":13035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1818,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":278080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":77791232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":909},{"shards":"1,4","objects":3436}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.0","version":"1251'4606","reported_seq":4646,"reported_epoch":1251,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:29:34.827758+0000","last_change":"2024-09-21T10:29:05.409568+0000","last_active":"2024-09-21T10:29:34.827758+0000","last_peered":"2024-09-21T10:29:34.827758+0000","last_clean":"2024-09-21T10:29:03.902926+0000","last_became_active":"2024-09-21T10:29:05.405810+0000","last_became_peered":"2024-09-21T10:29:05.405810+0000","last_unstale":"2024-09-21T10:29:34.827758+0000","last_undegraded":"2024-09-21T10:29:05.360593+0000","last_fullsized":"2024-09-21T10:29:05.342786+0000","mapping_epoch":1248,"log_start":"0'0","ondisk_log_start":"0'0","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301858816,"num_objects":4606,"num_object_clones":0,"num_object_copies":13818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":294784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1],"avail_no_missing":["2","0","1"],"object_location_counts":[{"shards":"0,1,2","objects":2546},{"shards":"1,2","objects":2060}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":42,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.379406+0000","last_change":"2024-09-21T10:28:56.823783+0000","last_active":"2024-09-21T10:29:26.379406+0000","last_peered":"2024-09-21T10:29:26.379406+0000","last_clean":"2024-09-21T10:29:26.379406+0000","last_became_active":"2024-09-21T10:28:56.815410+0000","last_became_peered":"2024-09-21T10:28:56.815410+0000","last_unstale":"2024-09-21T10:29:26.379406+0000","last_undegraded":"2024-09-21T10:29:26.379406+0000","last_fullsized":"2024-09-21T10:29:26.379406+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:34.577514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":570,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1228,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.358628+0000","last_change":"2024-09-21T10:28:37.587114+0000","last_active":"2024-09-21T10:29:06.358628+0000","last_peered":"2024-09-21T10:29:06.358628+0000","last_clean":"2024-09-21T10:29:06.358628+0000","last_became_active":"2024-09-21T10:28:37.586686+0000","last_became_peered":"2024-09-21T10:28:37.586686+0000","last_unstale":"2024-09-21T10:29:06.358628+0000","last_undegraded":"2024-09-21T10:29:06.358628+0000","last_fullsized":"2024-09-21T10:29:06.358628+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:41:45.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1818,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.373232+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:29:26.373232+0000","last_peered":"2024-09-21T10:29:26.373232+0000","last_clean":"2024-09-21T10:29:26.373232+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:29:26.373232+0000","last_undegraded":"2024-09-21T10:29:26.373232+0000","last_fullsized":"2024-09-21T10:29:26.373232+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1837,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372819+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:29:26.372819+0000","last_peered":"2024-09-21T10:29:26.372819+0000","last_clean":"2024-09-21T10:29:26.372819+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:29:26.372819+0000","last_undegraded":"2024-09-21T10:29:26.372819+0000","last_fullsized":"2024-09-21T10:29:26.372819+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":62,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357334+0000","last_change":"2024-09-21T10:27:20.323228+0000","last_active":"2024-09-21T10:29:06.357334+0000","last_peered":"2024-09-21T10:29:06.357334+0000","last_clean":"2024-09-21T10:29:06.357334+0000","last_became_active":"2024-09-21T10:27:20.322808+0000","last_became_peered":"2024-09-21T10:27:20.322808+0000","last_unstale":"2024-09-21T10:29:06.357334+0000","last_undegraded":"2024-09-21T10:29:06.357334+0000","last_fullsized":"2024-09-21T10:29:06.357334+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:12:27.259692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1214,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.356827+0000","last_change":"2024-09-21T10:27:19.333245+0000","last_active":"2024-09-21T10:29:06.356827+0000","last_peered":"2024-09-21T10:29:06.356827+0000","last_clean":"2024-09-21T10:29:06.356827+0000","last_became_active":"2024-09-21T10:27:19.332965+0000","last_became_peered":"2024-09-21T10:27:19.332965+0000","last_unstale":"2024-09-21T10:29:06.356827+0000","last_undegraded":"2024-09-21T10:29:06.356827+0000","last_fullsized":"2024-09-21T10:29:06.356827+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:04.828777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":62,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357100+0000","last_change":"2024-09-21T10:27:20.324804+0000","last_active":"2024-09-21T10:29:06.357100+0000","last_peered":"2024-09-21T10:29:06.357100+0000","last_clean":"2024-09-21T10:29:06.357100+0000","last_became_active":"2024-09-21T10:27:20.324499+0000","last_became_peered":"2024-09-21T10:27:20.324499+0000","last_unstale":"2024-09-21T10:29:06.357100+0000","last_undegraded":"2024-09-21T10:29:06.357100+0000","last_fullsized":"2024-09-21T10:29:06.357100+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:15.209140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1214,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357384+0000","last_change":"2024-09-21T10:27:19.370409+0000","last_active":"2024-09-21T10:29:06.357384+0000","last_peered":"2024-09-21T10:29:06.357384+0000","last_clean":"2024-09-21T10:29:06.357384+0000","last_became_active":"2024-09-21T10:27:19.370096+0000","last_became_peered":"2024-09-21T10:27:19.370096+0000","last_unstale":"2024-09-21T10:29:06.357384+0000","last_undegraded":"2024-09-21T10:29:06.357384+0000","last_fullsized":"2024-09-21T10:29:06.357384+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:03.124925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":16,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357264+0000","last_change":"2024-09-21T10:28:56.813482+0000","last_active":"2024-09-21T10:29:06.357264+0000","last_peered":"2024-09-21T10:29:06.357264+0000","last_clean":"2024-09-21T10:29:06.357264+0000","last_became_active":"2024-09-21T10:28:56.813103+0000","last_became_peered":"2024-09-21T10:28:56.813103+0000","last_unstale":"2024-09-21T10:29:06.357264+0000","last_undegraded":"2024-09-21T10:29:06.357264+0000","last_fullsized":"2024-09-21T10:29:06.357264+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:20.438161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1762,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386673+0000","last_change":"2024-09-21T10:29:04.650051+0000","last_active":"2024-09-21T10:29:26.386673+0000","last_peered":"2024-09-21T10:29:26.386673+0000","last_clean":"2024-09-21T10:29:26.386673+0000","last_became_active":"2024-09-21T10:29:04.649705+0000","last_became_peered":"2024-09-21T10:29:04.649705+0000","last_unstale":"2024-09-21T10:29:26.386673+0000","last_undegraded":"2024-09-21T10:29:26.386673+0000","last_fullsized":"2024-09-21T10:29:26.386673+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:45.854900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1189,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.356857+0000","last_change":"2024-09-21T10:27:19.325859+0000","last_active":"2024-09-21T10:29:06.356857+0000","last_peered":"2024-09-21T10:29:06.356857+0000","last_clean":"2024-09-21T10:29:06.356857+0000","last_became_active":"2024-09-21T10:27:19.325540+0000","last_became_peered":"2024-09-21T10:27:19.325540+0000","last_unstale":"2024-09-21T10:29:06.356857+0000","last_undegraded":"2024-09-21T10:29:06.356857+0000","last_fullsized":"2024-09-21T10:29:06.356857+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:11.324592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1820,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.377075+0000","last_change":"2024-09-21T10:28:37.333977+0000","last_active":"2024-09-21T10:29:26.377075+0000","last_peered":"2024-09-21T10:29:26.377075+0000","last_clean":"2024-09-21T10:29:26.377075+0000","last_became_active":"2024-09-21T10:28:37.333630+0000","last_became_peered":"2024-09-21T10:28:37.333630+0000","last_unstale":"2024-09-21T10:29:26.377075+0000","last_undegraded":"2024-09-21T10:29:26.377075+0000","last_fullsized":"2024-09-21T10:29:26.377075+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:07.637543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1918,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.391773+0000","last_change":"2024-09-21T10:27:19.331712+0000","last_active":"2024-09-21T10:29:26.391773+0000","last_peered":"2024-09-21T10:29:26.391773+0000","last_clean":"2024-09-21T10:29:26.391773+0000","last_became_active":"2024-09-21T10:27:19.330959+0000","last_became_peered":"2024-09-21T10:27:19.330959+0000","last_unstale":"2024-09-21T10:29:26.391773+0000","last_undegraded":"2024-09-21T10:29:26.391773+0000","last_fullsized":"2024-09-21T10:29:26.391773+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:46.389484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1933,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.378018+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:29:26.378018+0000","last_peered":"2024-09-21T10:29:26.378018+0000","last_clean":"2024-09-21T10:29:26.378018+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:29:26.378018+0000","last_undegraded":"2024-09-21T10:29:26.378018+0000","last_fullsized":"2024-09-21T10:29:26.378018+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1845,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372259+0000","last_change":"2024-09-21T10:28:56.819014+0000","last_active":"2024-09-21T10:29:26.372259+0000","last_peered":"2024-09-21T10:29:26.372259+0000","last_clean":"2024-09-21T10:29:26.372259+0000","last_became_active":"2024-09-21T10:28:56.818640+0000","last_became_peered":"2024-09-21T10:28:56.818640+0000","last_unstale":"2024-09-21T10:29:26.372259+0000","last_undegraded":"2024-09-21T10:29:26.372259+0000","last_fullsized":"2024-09-21T10:29:26.372259+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:49.759102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1199,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.368027+0000","last_change":"2024-09-21T10:26:56.957568+0000","last_active":"2024-09-21T10:29:06.368027+0000","last_peered":"2024-09-21T10:29:06.368027+0000","last_clean":"2024-09-21T10:29:06.368027+0000","last_became_active":"2024-09-21T10:26:56.957191+0000","last_became_peered":"2024-09-21T10:26:56.957191+0000","last_unstale":"2024-09-21T10:29:06.368027+0000","last_undegraded":"2024-09-21T10:29:06.368027+0000","last_fullsized":"2024-09-21T10:29:06.368027+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:11:04.630815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1831,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372427+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:29:26.372427+0000","last_peered":"2024-09-21T10:29:26.372427+0000","last_clean":"2024-09-21T10:29:26.372427+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:29:26.372427+0000","last_undegraded":"2024-09-21T10:29:26.372427+0000","last_fullsized":"2024-09-21T10:29:26.372427+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1937,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.378091+0000","last_change":"2024-09-21T10:26:56.956418+0000","last_active":"2024-09-21T10:29:26.378091+0000","last_peered":"2024-09-21T10:29:26.378091+0000","last_clean":"2024-09-21T10:29:26.378091+0000","last_became_active":"2024-09-21T10:26:56.956129+0000","last_became_peered":"2024-09-21T10:26:56.956129+0000","last_unstale":"2024-09-21T10:29:26.378091+0000","last_undegraded":"2024-09-21T10:29:26.378091+0000","last_fullsized":"2024-09-21T10:29:26.378091+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:27.517048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1814,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372664+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:29:26.372664+0000","last_peered":"2024-09-21T10:29:26.372664+0000","last_clean":"2024-09-21T10:29:26.372664+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:29:26.372664+0000","last_undegraded":"2024-09-21T10:29:26.372664+0000","last_fullsized":"2024-09-21T10:29:26.372664+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":551,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360196+0000","last_change":"2024-09-21T10:26:56.955932+0000","last_active":"2024-09-21T10:29:06.360196+0000","last_peered":"2024-09-21T10:29:06.360196+0000","last_clean":"2024-09-21T10:29:06.360196+0000","last_became_active":"2024-09-21T10:26:56.955446+0000","last_became_peered":"2024-09-21T10:26:56.955446+0000","last_unstale":"2024-09-21T10:29:06.360196+0000","last_undegraded":"2024-09-21T10:29:06.360196+0000","last_fullsized":"2024-09-21T10:29:06.360196+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:27:15.329971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1462,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.387198+0000","last_change":"2024-09-21T10:27:19.327772+0000","last_active":"2024-09-21T10:29:26.387198+0000","last_peered":"2024-09-21T10:29:26.387198+0000","last_clean":"2024-09-21T10:29:26.387198+0000","last_became_active":"2024-09-21T10:27:19.327467+0000","last_became_peered":"2024-09-21T10:27:19.327467+0000","last_unstale":"2024-09-21T10:29:26.387198+0000","last_undegraded":"2024-09-21T10:29:26.387198+0000","last_fullsized":"2024-09-21T10:29:26.387198+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:28:38.639278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1917,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386339+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:29:26.386339+0000","last_peered":"2024-09-21T10:29:26.386339+0000","last_clean":"2024-09-21T10:29:26.386339+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:29:26.386339+0000","last_undegraded":"2024-09-21T10:29:26.386339+0000","last_fullsized":"2024-09-21T10:29:26.386339+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1224,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357020+0000","last_change":"2024-09-21T10:27:19.330694+0000","last_active":"2024-09-21T10:29:06.357020+0000","last_peered":"2024-09-21T10:29:06.357020+0000","last_clean":"2024-09-21T10:29:06.357020+0000","last_became_active":"2024-09-21T10:27:19.329455+0000","last_became_peered":"2024-09-21T10:27:19.329455+0000","last_unstale":"2024-09-21T10:29:06.357020+0000","last_undegraded":"2024-09-21T10:29:06.357020+0000","last_fullsized":"2024-09-21T10:29:06.357020+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:27.883673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1218,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357374+0000","last_change":"2024-09-21T10:27:19.371314+0000","last_active":"2024-09-21T10:29:06.357374+0000","last_peered":"2024-09-21T10:29:06.357374+0000","last_clean":"2024-09-21T10:29:06.357374+0000","last_became_active":"2024-09-21T10:27:19.370747+0000","last_became_peered":"2024-09-21T10:27:19.370747+0000","last_unstale":"2024-09-21T10:29:06.357374+0000","last_undegraded":"2024-09-21T10:29:06.357374+0000","last_fullsized":"2024-09-21T10:29:06.357374+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:00.522037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":56,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372775+0000","last_change":"2024-09-21T10:28:21.154220+0000","last_active":"2024-09-21T10:29:26.372775+0000","last_peered":"2024-09-21T10:29:26.372775+0000","last_clean":"2024-09-21T10:29:26.372775+0000","last_became_active":"2024-09-21T10:28:17.260219+0000","last_became_peered":"2024-09-21T10:28:17.260219+0000","last_unstale":"2024-09-21T10:29:26.372775+0000","last_undegraded":"2024-09-21T10:29:26.372775+0000","last_fullsized":"2024-09-21T10:29:26.372775+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:59:35.089040+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1199,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.356558+0000","last_change":"2024-09-21T10:27:20.428677+0000","last_active":"2024-09-21T10:29:06.356558+0000","last_peered":"2024-09-21T10:29:06.356558+0000","last_clean":"2024-09-21T10:29:06.356558+0000","last_became_active":"2024-09-21T10:27:20.428376+0000","last_became_peered":"2024-09-21T10:27:20.428376+0000","last_unstale":"2024-09-21T10:29:06.356558+0000","last_undegraded":"2024-09-21T10:29:06.356558+0000","last_fullsized":"2024-09-21T10:29:06.356558+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:31.411176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1205,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.368003+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:29:06.368003+0000","last_peered":"2024-09-21T10:29:06.368003+0000","last_clean":"2024-09-21T10:29:06.368003+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:29:06.368003+0000","last_undegraded":"2024-09-21T10:29:06.368003+0000","last_fullsized":"2024-09-21T10:29:06.368003+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":53,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386605+0000","last_change":"2024-09-21T10:29:04.342115+0000","last_active":"2024-09-21T10:29:26.386605+0000","last_peered":"2024-09-21T10:29:26.386605+0000","last_clean":"2024-09-21T10:29:26.386605+0000","last_became_active":"2024-09-21T10:29:04.340202+0000","last_became_peered":"2024-09-21T10:29:04.340202+0000","last_unstale":"2024-09-21T10:29:26.386605+0000","last_undegraded":"2024-09-21T10:29:26.386605+0000","last_fullsized":"2024-09-21T10:29:26.386605+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:52.282454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":36,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357262+0000","last_change":"2024-09-21T10:28:22.295064+0000","last_active":"2024-09-21T10:29:06.357262+0000","last_peered":"2024-09-21T10:29:06.357262+0000","last_clean":"2024-09-21T10:29:06.357262+0000","last_became_active":"2024-09-21T10:28:17.260564+0000","last_became_peered":"2024-09-21T10:28:17.260564+0000","last_unstale":"2024-09-21T10:29:06.357262+0000","last_undegraded":"2024-09-21T10:29:06.357262+0000","last_fullsized":"2024-09-21T10:29:06.357262+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.295009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.295009+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:57.816702+0000","scrub_duration":0.0014921820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1829,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372343+0000","last_change":"2024-09-21T10:28:56.817623+0000","last_active":"2024-09-21T10:29:26.372343+0000","last_peered":"2024-09-21T10:29:26.372343+0000","last_clean":"2024-09-21T10:29:26.372343+0000","last_became_active":"2024-09-21T10:28:56.817337+0000","last_became_peered":"2024-09-21T10:28:56.817337+0000","last_unstale":"2024-09-21T10:29:26.372343+0000","last_undegraded":"2024-09-21T10:29:26.372343+0000","last_fullsized":"2024-09-21T10:29:26.372343+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:31.718364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1927,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386733+0000","last_change":"2024-09-21T10:27:19.326897+0000","last_active":"2024-09-21T10:29:26.386733+0000","last_peered":"2024-09-21T10:29:26.386733+0000","last_clean":"2024-09-21T10:29:26.386733+0000","last_became_active":"2024-09-21T10:27:19.324476+0000","last_became_peered":"2024-09-21T10:27:19.324476+0000","last_unstale":"2024-09-21T10:29:26.386733+0000","last_undegraded":"2024-09-21T10:29:26.386733+0000","last_fullsized":"2024-09-21T10:29:26.386733+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:57.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1813,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.373737+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:29:26.373737+0000","last_peered":"2024-09-21T10:29:26.373737+0000","last_clean":"2024-09-21T10:29:26.373737+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:29:26.373737+0000","last_undegraded":"2024-09-21T10:29:26.373737+0000","last_fullsized":"2024-09-21T10:29:26.373737+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":356,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.367853+0000","last_change":"2024-09-21T10:28:21.749325+0000","last_active":"2024-09-21T10:29:06.367853+0000","last_peered":"2024-09-21T10:29:06.367853+0000","last_clean":"2024-09-21T10:29:06.367853+0000","last_became_active":"2024-09-21T10:28:17.264767+0000","last_became_peered":"2024-09-21T10:28:17.264767+0000","last_unstale":"2024-09-21T10:29:06.367853+0000","last_undegraded":"2024-09-21T10:29:06.367853+0000","last_fullsized":"2024-09-21T10:29:06.367853+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.749265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.749265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:52.716016+0000","scrub_duration":0.0017801379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1929,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.382370+0000","last_change":"2024-09-21T10:26:57.026095+0000","last_active":"2024-09-21T10:29:26.382370+0000","last_peered":"2024-09-21T10:29:26.382370+0000","last_clean":"2024-09-21T10:29:26.382370+0000","last_became_active":"2024-09-21T10:26:56.958513+0000","last_became_peered":"2024-09-21T10:26:56.958513+0000","last_unstale":"2024-09-21T10:29:26.382370+0000","last_undegraded":"2024-09-21T10:29:26.382370+0000","last_fullsized":"2024-09-21T10:29:26.382370+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:23.065911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1248,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360107+0000","last_change":"2024-09-21T10:29:04.338405+0000","last_active":"2024-09-21T10:29:06.360107+0000","last_peered":"2024-09-21T10:29:06.360107+0000","last_clean":"2024-09-21T10:29:06.360107+0000","last_became_active":"2024-09-21T10:29:04.338028+0000","last_became_peered":"2024-09-21T10:29:04.338028+0000","last_unstale":"2024-09-21T10:29:06.360107+0000","last_undegraded":"2024-09-21T10:29:06.360107+0000","last_fullsized":"2024-09-21T10:29:06.360107+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:52.709092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1914,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386878+0000","last_change":"2024-09-21T10:26:57.025262+0000","last_active":"2024-09-21T10:29:26.386878+0000","last_peered":"2024-09-21T10:29:26.386878+0000","last_clean":"2024-09-21T10:29:26.386878+0000","last_became_active":"2024-09-21T10:26:57.024903+0000","last_became_peered":"2024-09-21T10:26:57.024903+0000","last_unstale":"2024-09-21T10:29:26.386878+0000","last_undegraded":"2024-09-21T10:29:26.386878+0000","last_fullsized":"2024-09-21T10:29:26.386878+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:40.642148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":356,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.367707+0000","last_change":"2024-09-21T10:28:29.591030+0000","last_active":"2024-09-21T10:29:06.367707+0000","last_peered":"2024-09-21T10:29:06.367707+0000","last_clean":"2024-09-21T10:29:06.367707+0000","last_became_active":"2024-09-21T10:28:17.266055+0000","last_became_peered":"2024-09-21T10:28:17.266055+0000","last_unstale":"2024-09-21T10:29:06.367707+0000","last_undegraded":"2024-09-21T10:29:06.367707+0000","last_fullsized":"2024-09-21T10:29:06.367707+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:29.590974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:29.590974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:40:17.176176+0000","scrub_duration":0.00139189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1912,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386471+0000","last_change":"2024-09-21T10:27:20.427309+0000","last_active":"2024-09-21T10:29:26.386471+0000","last_peered":"2024-09-21T10:29:26.386471+0000","last_clean":"2024-09-21T10:29:26.386471+0000","last_became_active":"2024-09-21T10:27:20.426992+0000","last_became_peered":"2024-09-21T10:27:20.426992+0000","last_unstale":"2024-09-21T10:29:26.386471+0000","last_undegraded":"2024-09-21T10:29:26.386471+0000","last_fullsized":"2024-09-21T10:29:26.386471+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:22:12.169478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1921,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.387042+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:29:26.387042+0000","last_peered":"2024-09-21T10:29:26.387042+0000","last_clean":"2024-09-21T10:29:26.387042+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:29:26.387042+0000","last_undegraded":"2024-09-21T10:29:26.387042+0000","last_fullsized":"2024-09-21T10:29:26.387042+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":724,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.376953+0000","last_change":"2024-09-21T10:26:56.958589+0000","last_active":"2024-09-21T10:29:26.376953+0000","last_peered":"2024-09-21T10:29:26.376953+0000","last_clean":"2024-09-21T10:29:26.376953+0000","last_became_active":"2024-09-21T10:26:56.957824+0000","last_became_peered":"2024-09-21T10:26:56.957824+0000","last_unstale":"2024-09-21T10:29:26.376953+0000","last_undegraded":"2024-09-21T10:29:26.376953+0000","last_fullsized":"2024-09-21T10:29:26.376953+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:14:41.032119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":362,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.377767+0000","last_change":"2024-09-21T10:28:17.266280+0000","last_active":"2024-09-21T10:29:26.377767+0000","last_peered":"2024-09-21T10:29:26.377767+0000","last_clean":"2024-09-21T10:29:26.377767+0000","last_became_active":"2024-09-21T10:28:17.265826+0000","last_became_peered":"2024-09-21T10:28:17.265826+0000","last_unstale":"2024-09-21T10:29:26.377767+0000","last_undegraded":"2024-09-21T10:29:26.377767+0000","last_fullsized":"2024-09-21T10:29:26.377767+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:30:18.941947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1918,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386964+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:29:26.386964+0000","last_peered":"2024-09-21T10:29:26.386964+0000","last_clean":"2024-09-21T10:29:26.386964+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:29:26.386964+0000","last_undegraded":"2024-09-21T10:29:26.386964+0000","last_fullsized":"2024-09-21T10:29:26.386964+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1219,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357331+0000","last_change":"2024-09-21T10:27:19.333723+0000","last_active":"2024-09-21T10:29:06.357331+0000","last_peered":"2024-09-21T10:29:06.357331+0000","last_clean":"2024-09-21T10:29:06.357331+0000","last_became_active":"2024-09-21T10:27:19.333492+0000","last_became_peered":"2024-09-21T10:27:19.333492+0000","last_unstale":"2024-09-21T10:29:06.357331+0000","last_undegraded":"2024-09-21T10:29:06.357331+0000","last_fullsized":"2024-09-21T10:29:06.357331+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:03.860673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":120,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386388+0000","last_change":"2024-09-21T10:26:57.033704+0000","last_active":"2024-09-21T10:29:26.386388+0000","last_peered":"2024-09-21T10:29:26.386388+0000","last_clean":"2024-09-21T10:29:26.386388+0000","last_became_active":"2024-09-21T10:26:56.958387+0000","last_became_peered":"2024-09-21T10:26:56.958387+0000","last_unstale":"2024-09-21T10:29:26.386388+0000","last_undegraded":"2024-09-21T10:29:26.386388+0000","last_fullsized":"2024-09-21T10:29:26.386388+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:33:36.081861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":1372,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.359879+0000","last_change":"2024-09-21T10:28:26.208348+0000","last_active":"2024-09-21T10:29:06.359879+0000","last_peered":"2024-09-21T10:29:06.359879+0000","last_clean":"2024-09-21T10:29:06.359879+0000","last_became_active":"2024-09-21T10:28:17.261142+0000","last_became_peered":"2024-09-21T10:28:17.261142+0000","last_unstale":"2024-09-21T10:29:06.359879+0000","last_undegraded":"2024-09-21T10:29:06.359879+0000","last_fullsized":"2024-09-21T10:29:06.359879+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:26.208290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:26.208290+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:42.090767+0000","scrub_duration":0.0015511240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1923,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.387457+0000","last_change":"2024-09-21T10:29:04.649991+0000","last_active":"2024-09-21T10:29:26.387457+0000","last_peered":"2024-09-21T10:29:26.387457+0000","last_clean":"2024-09-21T10:29:26.387457+0000","last_became_active":"2024-09-21T10:29:04.649698+0000","last_became_peered":"2024-09-21T10:29:04.649698+0000","last_unstale":"2024-09-21T10:29:26.387457+0000","last_undegraded":"2024-09-21T10:29:26.387457+0000","last_fullsized":"2024-09-21T10:29:26.387457+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:14:31.372900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1919,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.388271+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:29:26.388271+0000","last_peered":"2024-09-21T10:29:26.388271+0000","last_clean":"2024-09-21T10:29:26.388271+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:29:26.388271+0000","last_undegraded":"2024-09-21T10:29:26.388271+0000","last_fullsized":"2024-09-21T10:29:26.388271+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1850,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.379371+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:29:26.379371+0000","last_peered":"2024-09-21T10:29:26.379371+0000","last_clean":"2024-09-21T10:29:26.379371+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:29:26.379371+0000","last_undegraded":"2024-09-21T10:29:26.379371+0000","last_fullsized":"2024-09-21T10:29:26.379371+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":59,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386081+0000","last_change":"2024-09-21T10:28:56.821396+0000","last_active":"2024-09-21T10:29:26.386081+0000","last_peered":"2024-09-21T10:29:26.386081+0000","last_clean":"2024-09-21T10:29:26.386081+0000","last_became_active":"2024-09-21T10:28:56.820841+0000","last_became_peered":"2024-09-21T10:28:56.820841+0000","last_unstale":"2024-09-21T10:29:26.386081+0000","last_undegraded":"2024-09-21T10:29:26.386081+0000","last_fullsized":"2024-09-21T10:29:26.386081+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:57:52.560138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":702,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.386805+0000","last_change":"2024-09-21T10:28:56.817541+0000","last_active":"2024-09-21T10:29:26.386805+0000","last_peered":"2024-09-21T10:29:26.386805+0000","last_clean":"2024-09-21T10:29:26.386805+0000","last_became_active":"2024-09-21T10:28:56.817221+0000","last_became_peered":"2024-09-21T10:28:56.817221+0000","last_unstale":"2024-09-21T10:29:26.386805+0000","last_undegraded":"2024-09-21T10:29:26.386805+0000","last_fullsized":"2024-09-21T10:29:26.386805+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:51.568259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1944,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.377915+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:29:26.377915+0000","last_peered":"2024-09-21T10:29:26.377915+0000","last_clean":"2024-09-21T10:29:26.377915+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:29:26.377915+0000","last_undegraded":"2024-09-21T10:29:26.377915+0000","last_fullsized":"2024-09-21T10:29:26.377915+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1238,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360660+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:29:06.360660+0000","last_peered":"2024-09-21T10:29:06.360660+0000","last_clean":"2024-09-21T10:29:06.360660+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:29:06.360660+0000","last_undegraded":"2024-09-21T10:29:06.360660+0000","last_fullsized":"2024-09-21T10:29:06.360660+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1201,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.367616+0000","last_change":"2024-09-21T10:27:19.327425+0000","last_active":"2024-09-21T10:29:06.367616+0000","last_peered":"2024-09-21T10:29:06.367616+0000","last_clean":"2024-09-21T10:29:06.367616+0000","last_became_active":"2024-09-21T10:27:19.326641+0000","last_became_peered":"2024-09-21T10:27:19.326641+0000","last_unstale":"2024-09-21T10:29:06.367616+0000","last_undegraded":"2024-09-21T10:29:06.367616+0000","last_fullsized":"2024-09-21T10:29:06.367616+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:31:37.389673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1828,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372608+0000","last_change":"2024-09-21T10:27:20.323085+0000","last_active":"2024-09-21T10:29:26.372608+0000","last_peered":"2024-09-21T10:29:26.372608+0000","last_clean":"2024-09-21T10:29:26.372608+0000","last_became_active":"2024-09-21T10:27:20.322679+0000","last_became_peered":"2024-09-21T10:27:20.322679+0000","last_unstale":"2024-09-21T10:29:26.372608+0000","last_undegraded":"2024-09-21T10:29:26.372608+0000","last_fullsized":"2024-09-21T10:29:26.372608+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:10:30.641216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":115,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.381264+0000","last_change":"2024-09-21T10:26:57.033494+0000","last_active":"2024-09-21T10:29:26.381264+0000","last_peered":"2024-09-21T10:29:26.381264+0000","last_clean":"2024-09-21T10:29:26.381264+0000","last_became_active":"2024-09-21T10:26:56.957867+0000","last_became_peered":"2024-09-21T10:26:56.957867+0000","last_unstale":"2024-09-21T10:29:26.381264+0000","last_undegraded":"2024-09-21T10:29:26.381264+0000","last_fullsized":"2024-09-21T10:29:26.381264+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:32.453055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1373,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360607+0000","last_change":"2024-09-21T10:29:04.338929+0000","last_active":"2024-09-21T10:29:06.360607+0000","last_peered":"2024-09-21T10:29:06.360607+0000","last_clean":"2024-09-21T10:29:06.360607+0000","last_became_active":"2024-09-21T10:29:04.338528+0000","last_became_peered":"2024-09-21T10:29:04.338528+0000","last_unstale":"2024-09-21T10:29:06.360607+0000","last_undegraded":"2024-09-21T10:29:06.360607+0000","last_fullsized":"2024-09-21T10:29:06.360607+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.166696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.166696+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:35.196037+0000","scrub_duration":0.0013075529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":52,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.380100+0000","last_change":"2024-09-21T10:28:21.293677+0000","last_active":"2024-09-21T10:29:26.380100+0000","last_peered":"2024-09-21T10:29:26.380100+0000","last_clean":"2024-09-21T10:29:26.380100+0000","last_became_active":"2024-09-21T10:28:17.259227+0000","last_became_peered":"2024-09-21T10:28:17.259227+0000","last_unstale":"2024-09-21T10:29:26.380100+0000","last_undegraded":"2024-09-21T10:29:26.380100+0000","last_fullsized":"2024-09-21T10:29:26.380100+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:10:54.065590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1918,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.387374+0000","last_change":"2024-09-21T10:27:19.327581+0000","last_active":"2024-09-21T10:29:26.387374+0000","last_peered":"2024-09-21T10:29:26.387374+0000","last_clean":"2024-09-21T10:29:26.387374+0000","last_became_active":"2024-09-21T10:27:19.327252+0000","last_became_peered":"2024-09-21T10:27:19.327252+0000","last_unstale":"2024-09-21T10:29:26.387374+0000","last_undegraded":"2024-09-21T10:29:26.387374+0000","last_fullsized":"2024-09-21T10:29:26.387374+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:46.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1822,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372344+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:29:26.372344+0000","last_peered":"2024-09-21T10:29:26.372344+0000","last_clean":"2024-09-21T10:29:26.372344+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:29:26.372344+0000","last_undegraded":"2024-09-21T10:29:26.372344+0000","last_fullsized":"2024-09-21T10:29:26.372344+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1218,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.356802+0000","last_change":"2024-09-21T10:27:19.329566+0000","last_active":"2024-09-21T10:29:06.356802+0000","last_peered":"2024-09-21T10:29:06.356802+0000","last_clean":"2024-09-21T10:29:06.356802+0000","last_became_active":"2024-09-21T10:27:19.329307+0000","last_became_peered":"2024-09-21T10:27:19.329307+0000","last_unstale":"2024-09-21T10:29:06.356802+0000","last_undegraded":"2024-09-21T10:29:06.356802+0000","last_fullsized":"2024-09-21T10:29:06.356802+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:59.305906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1171,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.356997+0000","last_change":"2024-09-21T10:27:19.326055+0000","last_active":"2024-09-21T10:29:06.356997+0000","last_peered":"2024-09-21T10:29:06.356997+0000","last_clean":"2024-09-21T10:29:06.356997+0000","last_became_active":"2024-09-21T10:27:19.324778+0000","last_became_peered":"2024-09-21T10:27:19.324778+0000","last_unstale":"2024-09-21T10:29:06.356997+0000","last_undegraded":"2024-09-21T10:29:06.356997+0000","last_fullsized":"2024-09-21T10:29:06.356997+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:12.572967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":57,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.381505+0000","last_change":"2024-09-21T10:28:24.352395+0000","last_active":"2024-09-21T10:29:26.381505+0000","last_peered":"2024-09-21T10:29:26.381505+0000","last_clean":"2024-09-21T10:29:26.381505+0000","last_became_active":"2024-09-21T10:28:17.264784+0000","last_became_peered":"2024-09-21T10:28:17.264784+0000","last_unstale":"2024-09-21T10:29:26.381505+0000","last_undegraded":"2024-09-21T10:29:26.381505+0000","last_fullsized":"2024-09-21T10:29:26.381505+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:36:17.168517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1178,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357164+0000","last_change":"2024-09-21T10:27:20.323254+0000","last_active":"2024-09-21T10:29:06.357164+0000","last_peered":"2024-09-21T10:29:06.357164+0000","last_clean":"2024-09-21T10:29:06.357164+0000","last_became_active":"2024-09-21T10:27:20.322751+0000","last_became_peered":"2024-09-21T10:27:20.322751+0000","last_unstale":"2024-09-21T10:29:06.357164+0000","last_undegraded":"2024-09-21T10:29:06.357164+0000","last_fullsized":"2024-09-21T10:29:06.357164+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.909953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1207,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.367867+0000","last_change":"2024-09-21T10:27:19.326938+0000","last_active":"2024-09-21T10:29:06.367867+0000","last_peered":"2024-09-21T10:29:06.367867+0000","last_clean":"2024-09-21T10:29:06.367867+0000","last_became_active":"2024-09-21T10:27:19.326640+0000","last_became_peered":"2024-09-21T10:27:19.326640+0000","last_unstale":"2024-09-21T10:29:06.367867+0000","last_undegraded":"2024-09-21T10:29:06.367867+0000","last_fullsized":"2024-09-21T10:29:06.367867+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:24.413432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1372,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.359759+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:29:06.359759+0000","last_peered":"2024-09-21T10:29:06.359759+0000","last_clean":"2024-09-21T10:29:06.359759+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:29:06.359759+0000","last_undegraded":"2024-09-21T10:29:06.359759+0000","last_fullsized":"2024-09-21T10:29:06.359759+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1218,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.357840+0000","last_change":"2024-09-21T10:27:19.332150+0000","last_active":"2024-09-21T10:29:06.357840+0000","last_peered":"2024-09-21T10:29:06.357840+0000","last_clean":"2024-09-21T10:29:06.357840+0000","last_became_active":"2024-09-21T10:27:19.331901+0000","last_became_peered":"2024-09-21T10:27:19.331901+0000","last_unstale":"2024-09-21T10:29:06.357840+0000","last_undegraded":"2024-09-21T10:29:06.357840+0000","last_fullsized":"2024-09-21T10:29:06.357840+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:07.758296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":28,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372632+0000","last_change":"2024-09-21T10:28:56.834029+0000","last_active":"2024-09-21T10:29:26.372632+0000","last_peered":"2024-09-21T10:29:26.372632+0000","last_clean":"2024-09-21T10:29:26.372632+0000","last_became_active":"2024-09-21T10:28:56.817594+0000","last_became_peered":"2024-09-21T10:28:56.817594+0000","last_unstale":"2024-09-21T10:29:26.372632+0000","last_undegraded":"2024-09-21T10:29:26.372632+0000","last_fullsized":"2024-09-21T10:29:26.372632+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:44.035052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1943,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.378051+0000","last_change":"2024-09-21T10:27:19.370824+0000","last_active":"2024-09-21T10:29:26.378051+0000","last_peered":"2024-09-21T10:29:26.378051+0000","last_clean":"2024-09-21T10:29:26.378051+0000","last_became_active":"2024-09-21T10:27:19.370477+0000","last_became_peered":"2024-09-21T10:27:19.370477+0000","last_unstale":"2024-09-21T10:29:26.378051+0000","last_undegraded":"2024-09-21T10:29:26.378051+0000","last_fullsized":"2024-09-21T10:29:26.378051+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:17.020888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1259,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.373009+0000","last_change":"2024-09-21T10:26:56.956218+0000","last_active":"2024-09-21T10:29:26.373009+0000","last_peered":"2024-09-21T10:29:26.373009+0000","last_clean":"2024-09-21T10:29:26.373009+0000","last_became_active":"2024-09-21T10:26:56.955902+0000","last_became_peered":"2024-09-21T10:26:56.955902+0000","last_unstale":"2024-09-21T10:29:26.373009+0000","last_undegraded":"2024-09-21T10:29:26.373009+0000","last_fullsized":"2024-09-21T10:29:26.373009+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:25:08.908926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1189,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.367570+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:29:06.367570+0000","last_peered":"2024-09-21T10:29:06.367570+0000","last_clean":"2024-09-21T10:29:06.367570+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:29:06.367570+0000","last_undegraded":"2024-09-21T10:29:06.367570+0000","last_fullsized":"2024-09-21T10:29:06.367570+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1224,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360084+0000","last_change":"2024-09-21T10:26:57.016488+0000","last_active":"2024-09-21T10:29:06.360084+0000","last_peered":"2024-09-21T10:29:06.360084+0000","last_clean":"2024-09-21T10:29:06.360084+0000","last_became_active":"2024-09-21T10:26:56.955144+0000","last_became_peered":"2024-09-21T10:26:56.955144+0000","last_unstale":"2024-09-21T10:29:06.360084+0000","last_undegraded":"2024-09-21T10:29:06.360084+0000","last_fullsized":"2024-09-21T10:29:06.360084+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:29.887586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1214,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.356701+0000","last_change":"2024-09-21T10:27:19.327246+0000","last_active":"2024-09-21T10:29:06.356701+0000","last_peered":"2024-09-21T10:29:06.356701+0000","last_clean":"2024-09-21T10:29:06.356701+0000","last_became_active":"2024-09-21T10:27:19.326343+0000","last_became_peered":"2024-09-21T10:27:19.326343+0000","last_unstale":"2024-09-21T10:29:06.356701+0000","last_undegraded":"2024-09-21T10:29:06.356701+0000","last_fullsized":"2024-09-21T10:29:06.356701+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:37.654823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1228,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360252+0000","last_change":"2024-09-21T10:27:19.324762+0000","last_active":"2024-09-21T10:29:06.360252+0000","last_peered":"2024-09-21T10:29:06.360252+0000","last_clean":"2024-09-21T10:29:06.360252+0000","last_became_active":"2024-09-21T10:27:19.324341+0000","last_became_peered":"2024-09-21T10:27:19.324341+0000","last_unstale":"2024-09-21T10:29:06.360252+0000","last_undegraded":"2024-09-21T10:29:06.360252+0000","last_fullsized":"2024-09-21T10:29:06.360252+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:42.722645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1210,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360589+0000","last_change":"2024-09-21T10:27:20.322710+0000","last_active":"2024-09-21T10:29:06.360589+0000","last_peered":"2024-09-21T10:29:06.360589+0000","last_clean":"2024-09-21T10:29:06.360589+0000","last_became_active":"2024-09-21T10:27:20.322390+0000","last_became_peered":"2024-09-21T10:27:20.322390+0000","last_unstale":"2024-09-21T10:29:06.360589+0000","last_undegraded":"2024-09-21T10:29:06.360589+0000","last_fullsized":"2024-09-21T10:29:06.360589+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:22:56.725786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1241,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360219+0000","last_change":"2024-09-21T10:28:56.826222+0000","last_active":"2024-09-21T10:29:06.360219+0000","last_peered":"2024-09-21T10:29:06.360219+0000","last_clean":"2024-09-21T10:29:06.360219+0000","last_became_active":"2024-09-21T10:28:56.813187+0000","last_became_peered":"2024-09-21T10:28:56.813187+0000","last_unstale":"2024-09-21T10:29:06.360219+0000","last_undegraded":"2024-09-21T10:29:06.360219+0000","last_fullsized":"2024-09-21T10:29:06.360219+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:28.841806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1246,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.367510+0000","last_change":"2024-09-21T10:28:56.831886+0000","last_active":"2024-09-21T10:29:06.367510+0000","last_peered":"2024-09-21T10:29:06.367510+0000","last_clean":"2024-09-21T10:29:06.367510+0000","last_became_active":"2024-09-21T10:28:56.820623+0000","last_became_peered":"2024-09-21T10:28:56.820623+0000","last_unstale":"2024-09-21T10:29:06.367510+0000","last_undegraded":"2024-09-21T10:29:06.367510+0000","last_fullsized":"2024-09-21T10:29:06.367510+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:21.515011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1812,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.373528+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:29:26.373528+0000","last_peered":"2024-09-21T10:29:26.373528+0000","last_clean":"2024-09-21T10:29:26.373528+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:29:26.373528+0000","last_undegraded":"2024-09-21T10:29:26.373528+0000","last_fullsized":"2024-09-21T10:29:26.373528+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1236,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.360044+0000","last_change":"2024-09-21T10:26:04.187372+0000","last_active":"2024-09-21T10:29:06.360044+0000","last_peered":"2024-09-21T10:29:06.360044+0000","last_clean":"2024-09-21T10:29:06.360044+0000","last_became_active":"2024-09-21T10:26:04.186026+0000","last_became_peered":"2024-09-21T10:26:04.186026+0000","last_unstale":"2024-09-21T10:29:06.360044+0000","last_undegraded":"2024-09-21T10:29:06.360044+0000","last_fullsized":"2024-09-21T10:29:06.360044+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:55.130410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1931,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.377830+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:29:26.377830+0000","last_peered":"2024-09-21T10:29:26.377830+0000","last_clean":"2024-09-21T10:29:26.377830+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:29:26.377830+0000","last_undegraded":"2024-09-21T10:29:26.377830+0000","last_fullsized":"2024-09-21T10:29:26.377830+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1931,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.377918+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:29:26.377918+0000","last_peered":"2024-09-21T10:29:26.377918+0000","last_clean":"2024-09-21T10:29:26.377918+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:29:26.377918+0000","last_undegraded":"2024-09-21T10:29:26.377918+0000","last_fullsized":"2024-09-21T10:29:26.377918+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":661,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.359991+0000","last_change":"2024-09-21T10:26:56.955563+0000","last_active":"2024-09-21T10:29:06.359991+0000","last_peered":"2024-09-21T10:29:06.359991+0000","last_clean":"2024-09-21T10:29:06.359991+0000","last_became_active":"2024-09-21T10:26:56.955148+0000","last_became_peered":"2024-09-21T10:26:56.955148+0000","last_unstale":"2024-09-21T10:29:06.359991+0000","last_undegraded":"2024-09-21T10:29:06.359991+0000","last_fullsized":"2024-09-21T10:29:06.359991+0000","mapping_epoch":1206,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1207,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:59.655293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1241,"reported_epoch":1250,"state":"active+clean","last_fresh":"2024-09-21T10:29:06.359942+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:29:06.359942+0000","last_peered":"2024-09-21T10:29:06.359942+0000","last_clean":"2024-09-21T10:29:06.359942+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:29:06.359942+0000","last_undegraded":"2024-09-21T10:29:06.359942+0000","last_fullsized":"2024-09-21T10:29:06.359942+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1820,"reported_epoch":1251,"state":"active+clean","last_fresh":"2024-09-21T10:29:26.372934+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:29:26.372934+0000","last_peered":"2024-09-21T10:29:26.372934+0000","last_clean":"2024-09-21T10:29:26.372934+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:29:26.372934+0000","last_undegraded":"2024-09-21T10:29:26.372934+0000","last_fullsized":"2024-09-21T10:29:26.372934+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":46,"num_pg":16,"stat_sum":{"num_bytes":4753784832,"num_objects":72537,"num_object_clones":0,"num_object_copies":217611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8083,"num_objects_misplaced":1818,"num_objects_unfound":0,"num_objects_dirty":72537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":72537,"num_write_kb":4642368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2101,"num_bytes_recovered":137691136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14550433792,"data_stored":14550433792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72537,"ondisk_log_size":72537,"up":48,"acting":43,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1100,"ondisk_log_size":1100,"up":33,"acting":33,"num_store_stats":7},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":786,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397920,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2535768,"kb_used_data":2225936,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91209384,"statfs":{"total":95995035648,"available":93398409216,"internally_reserved":0,"allocated":2279358464,"data_stored":2274311937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049579,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892252,"kb_used_data":1556244,"kb_used_omap":242,"kb_used_meta":335757,"kb_avail":91852900,"statfs":{"total":95995035648,"available":94057369600,"internally_reserved":0,"allocated":1593593856,"data_stored":1588544318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":343815391},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":158913790378,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395484,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1993816,"kb_used_data":1670288,"kb_used_omap":224,"kb_used_meta":323295,"kb_avail":91751336,"statfs":{"total":95995035648,"available":93953368064,"internally_reserved":0,"allocated":1710374912,"data_stored":1705328941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":331054985},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644538,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2396556,"kb_used_data":2079812,"kb_used_omap":67,"kb_used_meta":316668,"kb_avail":91348596,"statfs":{"total":95995035648,"available":93540962304,"internally_reserved":0,"allocated":2129727488,"data_stored":2124689103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":324268893},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527556,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2843340,"kb_used_data":2539076,"kb_used_omap":75,"kb_used_meta":304180,"kb_avail":90901812,"statfs":{"total":95995035648,"available":93083455488,"internally_reserved":0,"allocated":2600013824,"data_stored":2594975907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":311480474},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274708,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2232848,"kb_used_data":1958216,"kb_used_omap":51,"kb_used_meta":274572,"kb_avail":91512304,"statfs":{"total":95995035648,"available":93708599296,"internally_reserved":0,"allocated":2005213184,"data_stored":2000170172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":281162069},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353947,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2409540,"kb_used_data":2162300,"kb_used_omap":66,"kb_used_meta":247165,"kb_avail":91335612,"statfs":{"total":95995035648,"available":93527666688,"internally_reserved":0,"allocated":2214195200,"data_stored":2209164338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":253097284},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2225602560,"data_stored":2225602560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2016608256,"data_stored":2016608256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2600730624,"data_stored":2600730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2135949312,"data_stored":2135949312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1713569792,"data_stored":1713569792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1590886400,"data_stored":1590886400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2267086848,"data_stored":2267086848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:29:37.900 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:29:37.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-21T10:29:37.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:37.981 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:37 smithi119 ceph-mon[101432]: from='client.55474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:37.981 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2460118896' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:37.982 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:37 smithi119 ceph-mon[105288]: from='client.55474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:37.982 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2460118896' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:38.002 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:37 smithi145 ceph-mon[89359]: from='client.55474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:29:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2460118896' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:29:38.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:38.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:38.197 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:29:38.197 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:29:38.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:38.268 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 57 16 77513 77497 84.957 61.5625 0.0224371 0.0117644 2024-09-21T10:29:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.291 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:38.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:38.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:38.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:38.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:38.567 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:38.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:38.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:38.811 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:38.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:38.855 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[105288]: pgmap v1843: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 65 MiB/s wr, 1.05k op/s; 8083/223257 objects degraded (3.620%); 1254/223257 objects misplaced (0.562%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:38.855 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:29:38.855 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2460118896' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:38.855 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[105288]: osdmap e1252: 8 total, 8 up, 7 in 2024-09-21T10:29:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:38.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:38.857 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[101432]: pgmap v1843: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 65 MiB/s wr, 1.05k op/s; 8083/223257 objects degraded (3.620%); 1254/223257 objects misplaced (0.562%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:38.857 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:29:38.857 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2460118896' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:38.857 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:38 smithi119 ceph-mon[101432]: osdmap e1252: 8 total, 8 up, 7 in 2024-09-21T10:29:38.865 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:38.904 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:29:38.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:39.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:38 smithi145 ceph-mon[89359]: pgmap v1843: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 65 MiB/s wr, 1.05k op/s; 8083/223257 objects degraded (3.620%); 1254/223257 objects misplaced (0.562%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:29:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:38 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:29:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2460118896' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:29:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:38 smithi145 ceph-mon[89359]: osdmap e1252: 8 total, 8 up, 7 in 2024-09-21T10:29:39.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:39.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:39.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:39.268 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 58 16 78538 78522 84.5967 64.0625 0.0203879 0.0118149 2024-09-21T10:29:39.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:39.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:39.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:39.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:39.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:39.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:39.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:39.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:39.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/223257 objects degraded (3.620%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/223257 objects degraded (3.620%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:39.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:39.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/223257 objects degraded (3.620%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:40.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:40.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:40.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:40.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:40.268 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 59 16 79600 79584 84.2878 66.375 0.00953404 0.0118581 2024-09-21T10:29:40.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:40.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:40.492 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:40.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:29:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:29:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:40.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:40.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:40.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:40.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:40.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:40.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:41.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:41.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:40 smithi145 ceph-mon[89359]: pgmap v1845: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 8083/227895 objects degraded (3.547%); 1254/227895 objects misplaced (0.550%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:29:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.076 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:41.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:40 smithi119 ceph-mon[101432]: pgmap v1845: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 8083/227895 objects degraded (3.547%); 1254/227895 objects misplaced (0.550%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:29:41.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:40 smithi119 ceph-mon[105288]: pgmap v1845: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s; 8083/227895 objects degraded (3.547%); 1254/227895 objects misplaced (0.550%); 3.4 MiB/s, 54 objects/s recovering 2024-09-21T10:29:41.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.205 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:41.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:41.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:29:41.267538+0000 min lat: 0.00134105 max lat: 4.01826 avg lat: 0.0119125 2024-09-21T10:29:41.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:29:41.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 60 16 80578 80562 83.9017 61.125 0.00912758 0.0119125 2024-09-21T10:29:41.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:41.343 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:41.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:41.584 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:41.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:41.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:41.748 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:41.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:41.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:41.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:41.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:42.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:42.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:42.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:42.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:42.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 61 16 81500 81484 83.4709 57.625 0.0101067 0.0119746 2024-09-21T10:29:42.286 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:42.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:42.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:42.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:29:42.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:42.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:42.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:42.670 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:42.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:42.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:42 smithi119 ceph-mon[105288]: pgmap v1846: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 8083/235119 objects degraded (3.438%); 612/235119 objects misplaced (0.260%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:42 smithi119 ceph-mon[101432]: pgmap v1846: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 8083/235119 objects degraded (3.438%); 612/235119 objects misplaced (0.260%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:42.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:42.943 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:42.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:42.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:43.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:42 smithi145 ceph-mon[89359]: pgmap v1846: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 63 MiB/s wr, 1.00k op/s; 8083/235119 objects degraded (3.438%); 612/235119 objects misplaced (0.260%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:43.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:43.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:43.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:43.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:43.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 62 16 82386 82370 83.0177 55.375 0.0206711 0.0120401 2024-09-21T10:29:43.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:43.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:43.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:43.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:43.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:43.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:43.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:43.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:43.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:43.871 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:43.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:43.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:44.003 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:44.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:44.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:44.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:44.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 63 16 83378 83362 82.684 62 0.00997706 0.012088 2024-09-21T10:29:44.284 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:44.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:44.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:44.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:44.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:44.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:44.691 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:44.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:44.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:44.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:29:44] "GET /metrics HTTP/1.1" 200 36917 "" "Prometheus/2.51.0" 2024-09-21T10:29:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:44 smithi119 ceph-mon[105288]: pgmap v1847: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 62 MiB/s wr, 991 op/s; 8083/240552 objects degraded (3.360%); 612/240552 objects misplaced (0.254%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/240552 objects degraded (3.360%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:44.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:44 smithi119 ceph-mon[101432]: pgmap v1847: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 62 MiB/s wr, 991 op/s; 8083/240552 objects degraded (3.360%); 612/240552 objects misplaced (0.254%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:44.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/240552 objects degraded (3.360%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:44.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:44.945 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:44.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:44 smithi145 ceph-mon[89359]: pgmap v1847: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 62 MiB/s wr, 991 op/s; 8083/240552 objects degraded (3.360%); 612/240552 objects misplaced (0.254%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/240552 objects degraded (3.360%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:45.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:45.075 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:45.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:45.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:45.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 64 16 84249 84233 82.2426 54.4375 0.0141678 0.0121535 2024-09-21T10:29:45.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:45.332 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:45.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.470 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:45.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:45.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:45.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:45.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:45.734 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:45.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:45.869 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:45.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:45.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:45.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:46.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:46.116 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:46.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:46.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:46.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 65 16 84963 84947 81.6639 44.625 0.00373191 0.0121726 2024-09-21T10:29:46.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:46.364 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:46.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:46.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:46.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:46.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:46.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:46.742 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:46.777 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:46.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:46.867 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:46.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:46.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:46.945 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:46 smithi119 ceph-mon[105288]: pgmap v1848: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 61 MiB/s wr, 970 op/s; 8083/246819 objects degraded (3.275%); 612/246819 objects misplaced (0.248%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:46.946 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:46 smithi119 ceph-mon[105288]: osdmap e1253: 8 total, 8 up, 7 in 2024-09-21T10:29:46.946 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:46 smithi119 ceph-mon[101432]: pgmap v1848: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 61 MiB/s wr, 970 op/s; 8083/246819 objects degraded (3.275%); 612/246819 objects misplaced (0.248%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:46.946 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:46 smithi119 ceph-mon[101432]: osdmap e1253: 8 total, 8 up, 7 in 2024-09-21T10:29:47.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:46 smithi145 ceph-mon[89359]: pgmap v1848: 99 pgs: 1 active+recovering+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 61 MiB/s wr, 970 op/s; 8083/246819 objects degraded (3.275%); 612/246819 objects misplaced (0.248%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:29:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:46 smithi145 ceph-mon[89359]: osdmap e1253: 8 total, 8 up, 7 in 2024-09-21T10:29:47.049 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:47.099 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:29:47.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:47.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:47.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 66 16 85378 85362 80.8196 25.9375 0.00554972 0.0123549 2024-09-21T10:29:47.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:47.316 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:47.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:47.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:47.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:47.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:47.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:47.697 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:47.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:47.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:47.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:47.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:47.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:47.956 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:47 smithi119 ceph-mon[105288]: osdmap e1254: 8 total, 8 up, 7 in 2024-09-21T10:29:47.956 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:47.957 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:47 smithi119 ceph-mon[101432]: osdmap e1254: 8 total, 8 up, 7 in 2024-09-21T10:29:47.957 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:47.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:47 smithi145 ceph-mon[89359]: osdmap e1254: 8 total, 8 up, 7 in 2024-09-21T10:29:47.975 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:29:48.008 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:48.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:48.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:48.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:48.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 67 16 86143 86127 80.3269 47.8125 0.00871304 0.0124425 2024-09-21T10:29:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:48.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:48.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:48.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:48.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.596 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:48.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:48.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:48.725 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:48.827 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:48.862 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:48.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:48.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:48.999 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:48 smithi145 ceph-mon[89359]: pgmap v1851: 99 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 62 MiB/s wr, 988 op/s; 8083/251631 objects degraded (3.212%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:29:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:49.100 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:48 smithi119 ceph-mon[105288]: pgmap v1851: 99 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 62 MiB/s wr, 988 op/s; 8083/251631 objects degraded (3.212%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:29:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:48 smithi119 ceph-mon[101432]: pgmap v1851: 99 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 62 MiB/s wr, 988 op/s; 8083/251631 objects degraded (3.212%); 4.9 MiB/s, 78 objects/s recovering 2024-09-21T10:29:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:29:49.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:49.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:49.269 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:49.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 68 16 87179 87163 80.0978 64.75 0.0169943 0.0124789 2024-09-21T10:29:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:49.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:49.412 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:49.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:49.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:49.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:49.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:49.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:49.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.840 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:49.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:49 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8083/251631 objects degraded (3.212%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:49.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:49 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8083/251631 objects degraded (3.212%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:49.941 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:49.973 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:49.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:49 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8083/251631 objects degraded (3.212%), 4 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:50.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:50.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:50.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:50.256 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:50.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:50.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 69 16 88095 88079 79.7666 57.25 0.0149038 0.0125306 2024-09-21T10:29:50.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:50.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:50.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:50.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:29:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:29:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:29:50.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:50.665 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:50.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:50.809 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:50.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:50.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:50.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:51.029 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:50 smithi119 ceph-mon[105288]: pgmap v1852: 99 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s; 8083/255135 objects degraded (3.168%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:29:51.029 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:50 smithi119 ceph-mon[101432]: pgmap v1852: 99 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s; 8083/255135 objects degraded (3.168%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:29:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:50 smithi145 ceph-mon[89359]: pgmap v1852: 99 pgs: 1 peering, 4 active+recovery_wait+undersized+degraded+remapped, 94 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 52 MiB/s wr, 833 op/s; 8083/255135 objects degraded (3.168%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:29:51.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:51.084 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:51.186 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:51.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:51.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 70 16 89048 89032 79.478 59.5625 0.00686933 0.0125757 2024-09-21T10:29:51.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:51.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:29:51.405 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:29:51.406 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:29:51.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:51.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:51.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:51.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:51.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:51.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:51.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:51.928 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:51.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:52.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:51 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1160645665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:52.024 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:51 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1160645665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:52.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:52.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:52.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:52.192 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:52.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 71 16 90013 89997 79.208 60.3125 0.0259877 0.0126194 2024-09-21T10:29:52.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:52.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:51 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1160645665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:29:52.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:52.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:52.482 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:52.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:52.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:52.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:52.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:52.760 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:52.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:52.908 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:52.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:53.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:52 smithi145 ceph-mon[89359]: pgmap v1853: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s; 5947/261039 objects degraded (2.278%); 1824/261039 objects misplaced (0.699%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:53.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:52 smithi119 ceph-mon[105288]: pgmap v1853: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s; 5947/261039 objects degraded (2.278%); 1824/261039 objects misplaced (0.699%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:53.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:52 smithi119 ceph-mon[101432]: pgmap v1853: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s; 5947/261039 objects degraded (2.278%); 1824/261039 objects misplaced (0.699%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:53.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:53.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:53.269 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 72 16 90946 90930 78.9177 58.3125 0.0129667 0.0126653 2024-09-21T10:29:53.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:53.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:53.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:53.669 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.698 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:53.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:53.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:53.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:53.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:53.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:53.952 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 1 pg_num to 21 2024-09-21T10:29:53.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:53.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:54.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:54.107 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:54.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:54.225 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:54.225 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:54.225 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2513852467' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:29:54.225 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:29:54.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:54.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:54.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2513852467' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:29:54.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:53 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:29:54.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:54.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:54.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 73 16 91809 91793 78.5755 53.9375 0.00978155 0.0127211 2024-09-21T10:29:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:29:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2513852467' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:29:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:53 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-21T10:29:54.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:54.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:54.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:54.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:54.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:54.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:54.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:54.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.771 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:54.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:54.839 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:29:54] "GET /metrics HTTP/1.1" 200 36909 "" "Prometheus/2.51.0" 2024-09-21T10:29:54.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:54.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:54.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:54.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:54.960 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:29:55.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:55.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[105288]: pgmap v1854: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 5947/266412 objects degraded (2.232%); 1824/266412 objects misplaced (0.685%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T10:29:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[105288]: osdmap e1255: 8 total, 8 up, 7 in 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5947/266412 objects degraded (2.232%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/131278749' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[101432]: pgmap v1854: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 5947/266412 objects degraded (2.232%); 1824/266412 objects misplaced (0.685%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[101432]: osdmap e1255: 8 total, 8 up, 7 in 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5947/266412 objects degraded (2.232%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/131278749' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:55.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:55.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:55.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:55.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 74 16 92713 92697 78.2771 56.5 0.00899939 0.0127698 2024-09-21T10:29:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:54 smithi145 ceph-mon[89359]: pgmap v1854: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.4 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 51 MiB/s wr, 816 op/s; 5947/266412 objects degraded (2.232%); 1824/266412 objects misplaced (0.685%); 4.8 MiB/s, 77 objects/s recovering 2024-09-21T10:29:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-21T10:29:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:54 smithi145 ceph-mon[89359]: osdmap e1255: 8 total, 8 up, 7 in 2024-09-21T10:29:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:54 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5947/266412 objects degraded (2.232%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:29:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/131278749' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:29:55.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:55.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:55.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:55.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:55.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:55.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:55.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:55.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:55.856 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:55.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:55.988 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:55.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:56.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:56.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:56.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:56.219 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:56.253 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:56.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:56.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 75 16 93655 93639 78.0184 58.875 0.0118268 0.0128113 2024-09-21T10:29:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:55 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:29:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:55 smithi145 ceph-mon[89359]: osdmap e1256: 8 total, 8 up, 7 in 2024-09-21T10:29:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:55 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T10:29:56.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:56.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:29:56.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[105288]: osdmap e1256: 8 total, 8 up, 7 in 2024-09-21T10:29:56.312 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T10:29:56.312 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:29:56.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:29:56.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[101432]: osdmap e1256: 8 total, 8 up, 7 in 2024-09-21T10:29:56.313 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:55 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-21T10:29:56.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:56.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:56.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:56.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:56.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:56.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:56.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:56.824 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:56.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:29:56.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:56.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:56.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:56.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:57.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:29:57.107 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:29:57.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:57.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:57.248 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:56 smithi119 ceph-mon[101432]: pgmap v1857: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 871 op/s; 5947/272556 objects degraded (2.182%); 1589/272556 objects misplaced (0.583%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:29:57.248 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:56 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T10:29:57.248 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:56 smithi119 ceph-mon[101432]: osdmap e1257: 8 total, 8 up, 7 in 2024-09-21T10:29:57.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:56 smithi119 ceph-mon[105288]: pgmap v1857: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 871 op/s; 5947/272556 objects degraded (2.182%); 1589/272556 objects misplaced (0.583%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:29:57.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:56 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T10:29:57.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:56 smithi119 ceph-mon[105288]: osdmap e1257: 8 total, 8 up, 7 in 2024-09-21T10:29:57.254 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:29:57.254 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:29:57.254 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:29:57.254 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:29:57.254 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:29:57.271 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 76 16 94505 94489 77.6907 53.125 0.050796 0.0128653 2024-09-21T10:29:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:56 smithi145 ceph-mon[89359]: pgmap v1857: 99 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 5.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 54 MiB/s wr, 871 op/s; 5947/272556 objects degraded (2.182%); 1589/272556 objects misplaced (0.583%); 4.3 MiB/s, 68 objects/s recovering 2024-09-21T10:29:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:56 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-21T10:29:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:56 smithi145 ceph-mon[89359]: osdmap e1257: 8 total, 8 up, 7 in 2024-09-21T10:29:57.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:57.363 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:57.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:57.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:57.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:57.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:57.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:57.734 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:57.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:57.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:57.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:57.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:57.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:58.004 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:29:58.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:58.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:29:58.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:58.173 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:58.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 77 16 95436 95420 77.4374 58.1875 0.0151119 0.0129068 2024-09-21T10:29:58.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:58.278 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:58 smithi119 ceph-mon[101432]: osdmap e1258: 8 total, 8 up, 7 in 2024-09-21T10:29:58.278 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:58 smithi119 ceph-mon[105288]: osdmap e1258: 8 total, 8 up, 7 in 2024-09-21T10:29:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:58 smithi145 ceph-mon[89359]: osdmap e1258: 8 total, 8 up, 7 in 2024-09-21T10:29:58.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:58.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:29:58.439 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:29:58.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:58.582 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:58.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:29:58.726 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:29:58.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:58.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:58.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:29:58.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:58.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:29:59.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:59.120 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:59.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:59.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:59.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 78 16 96286 96270 77.1256 53.125 0.0195931 0.0129591 2024-09-21T10:29:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:29:59 smithi145 ceph-mon[89359]: pgmap v1860: 109 pgs: 8 unknown, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s; 5947/277533 objects degraded (2.143%); 1589/277533 objects misplaced (0.573%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:29:59.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:29:59 smithi119 ceph-mon[101432]: pgmap v1860: 109 pgs: 8 unknown, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s; 5947/277533 objects degraded (2.143%); 1589/277533 objects misplaced (0.573%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:29:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:29:59 smithi119 ceph-mon[105288]: pgmap v1860: 109 pgs: 8 unknown, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 97 active+clean; 5.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 58 MiB/s wr, 926 op/s; 5947/277533 objects degraded (2.143%); 1589/277533 objects misplaced (0.573%); 3.7 MiB/s, 58 objects/s recovering 2024-09-21T10:29:59.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:29:59.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:59.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:59.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:29:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:59.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:29:59.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:29:59.806 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:29:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:29:59.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:29:59.946 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:29:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:29:59.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:00.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:00.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:00.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:00.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:00.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 79 16 97122 97106 76.8107 52.25 0.00813408 0.0130133 2024-09-21T10:30:00.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:30:00.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:30:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:30:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded 2024-09-21T10:30:00.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T10:30:00.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: [WRN] PG_DEGRADED: Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded 2024-09-21T10:30:00.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: pg 46.0 is active+recovery_wait+undersized+degraded+remapped, acting [2,1] 2024-09-21T10:30:00.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: pg 46.2 is active+recovery_wait+undersized+degraded+remapped, acting [6,2] 2024-09-21T10:30:00.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:00 smithi145 ceph-mon[89359]: pg 46.5 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-09-21T10:30:00.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: [WRN] PG_DEGRADED: Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: pg 46.0 is active+recovery_wait+undersized+degraded+remapped, acting [2,1] 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: pg 46.2 is active+recovery_wait+undersized+degraded+remapped, acting [6,2] 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[101432]: pg 46.5 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: Health detail: HEALTH_WARN noscrub flag(s) set; Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded 2024-09-21T10:30:00.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-21T10:30:00.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: [WRN] PG_DEGRADED: Degraded data redundancy: 5947/277533 objects degraded (2.143%), 3 pgs degraded 2024-09-21T10:30:00.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: pg 46.0 is active+recovery_wait+undersized+degraded+remapped, acting [2,1] 2024-09-21T10:30:00.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: pg 46.2 is active+recovery_wait+undersized+degraded+remapped, acting [6,2] 2024-09-21T10:30:00.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:00 smithi119 ceph-mon[105288]: pg 46.5 is active+recovery_wait+undersized+degraded+remapped, acting [3,4] 2024-09-21T10:30:00.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:00.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:00.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:00.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:00.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:00.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:00.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:00.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:00.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:00.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:00.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:00.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:01.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:01.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:01.270 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:30:01.269498+0000 min lat: 0.00134105 max lat: 4.01826 avg lat: 0.0130678 2024-09-21T10:30:01.271 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:30:01.271 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 80 16 97941 97925 76.4904 51.1875 0.0191696 0.0130678 2024-09-21T10:30:01.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:01 smithi145 ceph-mon[89359]: pgmap v1861: 109 pgs: 4 unknown, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 51 MiB/s wr, 811 op/s; 5947/281811 objects degraded (2.110%); 1589/281811 objects misplaced (0.564%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:30:01.312 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:01.360 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:30:01.360 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:30:01.360 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-21T10:30:01.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd in 5 2024-09-21T10:30:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:01 smithi119 ceph-mon[101432]: pgmap v1861: 109 pgs: 4 unknown, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 51 MiB/s wr, 811 op/s; 5947/281811 objects degraded (2.110%); 1589/281811 objects misplaced (0.564%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:30:01.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:01 smithi119 ceph-mon[105288]: pgmap v1861: 109 pgs: 4 unknown, 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 101 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 51 MiB/s wr, 811 op/s; 5947/281811 objects degraded (2.110%); 1589/281811 objects misplaced (0.564%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:30:01.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:01.437 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:01.539 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:01.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:01.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:01.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:01.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:01.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:01.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:01.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:01.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:01.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:02.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:02.119 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:02.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:02.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:30:02.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:02.271 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 81 16 98829 98813 76.2312 55.5 0.0147997 0.0131112 2024-09-21T10:30:02.274 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:02.346 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:02.369 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:02.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:02.498 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:02.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:02.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:02.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:02.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:02.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:02.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:02.909 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:02.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:02.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:03.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:03.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:03.176 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:03.239 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:03.246 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:03.271 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 82 16 99669 99653 75.9412 52.5 0.0160333 0.013162 2024-09-21T10:30:03.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:03 smithi145 ceph-mon[89359]: pgmap v1862: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 50 MiB/s wr, 804 op/s; 5947/288489 objects degraded (2.061%); 1302/288489 objects misplaced (0.451%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:30:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:03.300 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:03.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:03 smithi119 ceph-mon[101432]: pgmap v1862: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 50 MiB/s wr, 804 op/s; 5947/288489 objects degraded (2.061%); 1302/288489 objects misplaced (0.451%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:30:03.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:03.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:03.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:03 smithi119 ceph-mon[105288]: pgmap v1862: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 5.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 50 MiB/s wr, 804 op/s; 5947/288489 objects degraded (2.061%); 1302/288489 objects misplaced (0.451%); 2.7 MiB/s, 43 objects/s recovering 2024-09-21T10:30:03.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:03.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:03.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:03.428 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:03.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:03.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:03.657 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:03.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:03.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:03.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:03.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:03.942 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:03.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:04.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:04.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:04.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:04.093 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:04.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:04.211 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:04.272 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 83 16 100553 100537 75.6918 55.25 0.0189628 0.0132049 2024-09-21T10:30:04.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:04.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:04.437 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:04.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:04.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:04.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:04.683 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:30:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:30:04] "GET /metrics HTTP/1.1" 200 36911 "" "Prometheus/2.51.0" 2024-09-21T10:30:04.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:04.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:04.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:04.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:04.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:04.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:04.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:04.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:05.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:05.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:05.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:05.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:05.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:05.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:05.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:05.272 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 84 16 101337 101321 75.3741 49 0.0249091 0.0132613 2024-09-21T10:30:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:05 smithi145 ceph-mon[89359]: pgmap v1863: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 864 op/s; 5947/293319 objects degraded (2.027%); 1302/293319 objects misplaced (0.444%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:30:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:05 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5947/293319 objects degraded (2.027%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:05.332 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:05 smithi119 ceph-mon[101432]: pgmap v1863: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 864 op/s; 5947/293319 objects degraded (2.027%); 1302/293319 objects misplaced (0.444%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:30:05.332 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:05 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5947/293319 objects degraded (2.027%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:05.332 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:05 smithi119 ceph-mon[105288]: pgmap v1863: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.0 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 54 MiB/s wr, 864 op/s; 5947/293319 objects degraded (2.027%); 1302/293319 objects misplaced (0.444%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:30:05.332 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:05 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5947/293319 objects degraded (2.027%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:05.357 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:05.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:05.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:05.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:05.506 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:05.607 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:05.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:05.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:05.783 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:05.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:05.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:06.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:06.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:06.139 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2889802356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:06.139 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2889802356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:06.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:06.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:06.271 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 85 16 102148 102132 75.0836 50.6875 0.0220142 0.0133139 2024-09-21T10:30:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2889802356' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:06.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:06.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:06.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:06.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:06.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:06.602 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:06.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:06.704 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:06.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:06.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:06.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:06.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:07.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:07.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:07.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:07.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:07.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 86 16 103076 103060 74.8839 58 0.00976036 0.0133483 2024-09-21T10:30:07.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:07.304 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:07 smithi119 ceph-mon[101432]: pgmap v1864: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 52 MiB/s wr, 826 op/s; 5947/298890 objects degraded (1.990%); 1050/298890 objects misplaced (0.351%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:30:07.304 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:07 smithi119 ceph-mon[105288]: pgmap v1864: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 52 MiB/s wr, 826 op/s; 5947/298890 objects degraded (1.990%); 1050/298890 objects misplaced (0.351%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:30:07.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:07.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:07.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:07 smithi145 ceph-mon[89359]: pgmap v1864: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 52 MiB/s wr, 826 op/s; 5947/298890 objects degraded (1.990%); 1050/298890 objects misplaced (0.351%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:30:07.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:07.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:07.708 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:07.810 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:07.849 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:07.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:07.977 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:07.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:08.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:08.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:08.222 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:08.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:08.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:08.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 87 16 103921 103905 74.6302 52.8125 0.0217848 0.0133927 2024-09-21T10:30:08.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:08.371 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:08.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:08.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:08.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:08.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:08.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:30:08.571+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1258 definitely_dead 0 2024-09-21T10:30:08.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:08.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:08.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:30:09.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:marked in osd.5. 2024-09-21T10:30:09.114 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:09.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:09.215 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:09.237 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 88 16 104624 104608 74.2814 43.9375 0.00412778 0.0134299 2024-09-21T10:30:09.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:09.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[101432]: pgmap v1865: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s; 5947/303696 objects degraded (1.958%); 1050/303696 objects misplaced (0.346%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4046067727' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2946586343' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[105288]: pgmap v1865: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s; 5947/303696 objects degraded (1.958%); 1050/303696 objects misplaced (0.346%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4046067727' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2946586343' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:30:09.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:09 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:30:09.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:09.483 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:09.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:09 smithi145 ceph-mon[89359]: pgmap v1865: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 56 MiB/s wr, 900 op/s; 5947/303696 objects degraded (1.958%); 1050/303696 objects misplaced (0.346%); 3.2 MiB/s, 50 objects/s recovering 2024-09-21T10:30:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4046067727' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:30:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2946586343' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:30:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:09 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:30:09.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:09.606 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:09.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:09.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:09.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:09.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:09.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:09.977 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:09.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:10.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:10.100 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:10.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:10.222 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:10.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:10.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 89 16 104624 104608 73.4469 0 - 0.0134299 2024-09-21T10:30:10.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:10.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4046067727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[105288]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5947/303696 objects degraded (1.958%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[105288]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:10.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4046067727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:10.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T10:30:10.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[101432]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-21T10:30:10.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5947/303696 objects degraded (1.958%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:10.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:10 smithi119 ceph-mon[101432]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-21T10:30:10.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.479 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:10.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:30:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4046067727' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:30:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T10:30:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-mon[89359]: osdmap e1259: 8 total, 8 up, 8 in 2024-09-21T10:30:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5947/303696 objects degraded (1.958%), 3 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:10.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:10 smithi145 ceph-mon[89359]: osdmap e1260: 8 total, 8 up, 8 in 2024-09-21T10:30:10.580 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:10.615 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:10.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:30:10.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:10.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:10.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:10.899 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-21T10:30:10.932 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:10.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:10.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:11.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:11.067 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:11.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:11.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total time run: 90.0114 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total writes made: 105098 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Write size: 65536 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object size: 65536 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Bandwidth (MB/sec): 72.9755 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Bandwidth: 40.3455 2024-09-21T10:30:11.273 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max bandwidth (MB/sec): 221.438 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average IOPS: 1167 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev IOPS: 645.54 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max IOPS: 3543 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min IOPS: 0 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average Latency(s): 0.013698 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Latency(s): 0.0491431 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max latency(s): 4.01826 2024-09-21T10:30:11.274 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min latency(s): 0.00134105 2024-09-21T10:30:11.280 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:30:11.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:11.359 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: pgmap v1868: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 50 MiB/s wr, 800 op/s; 5947/307719 objects degraded (1.933%); 1050/307719 objects misplaced (0.341%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.2 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6582 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.f starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6571 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.3 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6558 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.6 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6507 2024-09-21T10:30:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.8 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6502 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.1 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6508 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[105288]: 46.5 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6639 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: pgmap v1868: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 50 MiB/s wr, 800 op/s; 5947/307719 objects degraded (1.933%); 1050/307719 objects misplaced (0.341%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.2 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6582 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.f starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6571 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.3 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6558 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.6 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6507 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.8 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6502 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.1 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6508 2024-09-21T10:30:11.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:11 smithi119 ceph-mon[101432]: 46.5 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6639 2024-09-21T10:30:11.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:11.494 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: pgmap v1868: 109 pgs: 1 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 105 active+clean; 6.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 50 MiB/s wr, 800 op/s; 5947/307719 objects degraded (1.933%); 1050/307719 objects misplaced (0.341%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:30:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.2 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6582 2024-09-21T10:30:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: osdmap e1261: 8 total, 8 up, 8 in 2024-09-21T10:30:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.f starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6571 2024-09-21T10:30:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.3 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6558 2024-09-21T10:30:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.6 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6507 2024-09-21T10:30:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.8 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6502 2024-09-21T10:30:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.1 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6508 2024-09-21T10:30:11.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:11 smithi145 ceph-mon[89359]: 46.5 starting backfill to osd.5 from (0'0,0'0] MAX to 1258'6639 2024-09-21T10:30:11.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:11.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:11.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:11.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:11.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:11.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:11.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:12.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:12.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:12.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:12.160 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:12.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:12.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:12.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.336 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:12.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:12.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:12.481 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:12.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:12.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:12.610 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:12.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:12.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:12.741 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:12.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:12.866 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:12.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:12.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:12.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:13.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.039 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:13.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:13.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:13.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:13.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:13.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:13.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:13 smithi119 ceph-mon[101432]: pgmap v1870: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 49 MiB/s wr, 776 op/s; 3836/312873 objects degraded (1.226%); 26697/312873 objects misplaced (8.533%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:30:13.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:13 smithi119 ceph-mon[105288]: pgmap v1870: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 49 MiB/s wr, 776 op/s; 3836/312873 objects degraded (1.226%); 26697/312873 objects misplaced (8.533%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:30:13.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:13.438 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:13.539 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:13.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:13 smithi145 ceph-mon[89359]: pgmap v1870: 109 pgs: 3 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 100 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 49 MiB/s wr, 776 op/s; 3836/312873 objects degraded (1.226%); 26697/312873 objects misplaced (8.533%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:30:13.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:13.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:13.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:13.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:13.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:13.837 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:13.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:30:13.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:13.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:13.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:14.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.113 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:14.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:14.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:14.344 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:14.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:14.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:14.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:14.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:14.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:14.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.626 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:14.748 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:14.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:30:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:30:14] "GET /metrics HTTP/1.1" 200 36911 "" "Prometheus/2.51.0" 2024-09-21T10:30:14.872 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:14.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:14.995 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:14.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:15.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:15.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:15.226 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.260 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:15.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:15.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:15 smithi119 ceph-mon[101432]: pgmap v1871: 109 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 37 MiB/s wr, 662 op/s; 10314/313239 objects degraded (3.293%); 38521/313239 objects misplaced (12.298%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:30:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:15 smithi119 ceph-mon[101432]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-21T10:30:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:15 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10314/313239 objects degraded (3.293%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:15 smithi119 ceph-mon[105288]: pgmap v1871: 109 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 37 MiB/s wr, 662 op/s; 10314/313239 objects degraded (3.293%); 38521/313239 objects misplaced (12.298%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:30:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:15 smithi119 ceph-mon[105288]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-21T10:30:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:15 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10314/313239 objects degraded (3.293%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:15.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:15.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:15 smithi145 ceph-mon[89359]: pgmap v1871: 109 pgs: 1 active+recovery_wait+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 remapped+peering, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 170 B/s rd, 37 MiB/s wr, 662 op/s; 10314/313239 objects degraded (3.293%); 38521/313239 objects misplaced (12.298%); 5.1 MiB/s, 80 objects/s recovering 2024-09-21T10:30:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:15 smithi145 ceph-mon[89359]: osdmap e1262: 8 total, 8 up, 8 in 2024-09-21T10:30:15.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:15 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10314/313239 objects degraded (3.293%), 3 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:30:15.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:15.688 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:15.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:15.790 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:15.831 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:15.900 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:30:15.900 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:30:15.900 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:30:15.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T10:30:15.933 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:15.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:15.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:15.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:16.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:16.090 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:16.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:16.220 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:30:16.220 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:30:16.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:16 smithi119 ceph-mon[101432]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-21T10:30:16.222 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:16 smithi119 ceph-mon[105288]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-21T10:30:16.224 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:16.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.373 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:16.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:16.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:16.506 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:16 smithi145 ceph-mon[89359]: osdmap e1263: 8 total, 8 up, 8 in 2024-09-21T10:30:16.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:16.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:16.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:16.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:16.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:16.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:16.913 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:16.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:17.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:17.045 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:17.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.168 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:17.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:17.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:17.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:17.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:17 smithi119 ceph-mon[101432]: pgmap v1874: 109 pgs: 1 peering, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 170 B/s rd, 26 MiB/s wr, 664 op/s; 16415/310647 objects degraded (5.284%); 44813/310647 objects misplaced (14.426%); 14 MiB/s, 220 objects/s recovering 2024-09-21T10:30:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:17 smithi119 ceph-mon[105288]: pgmap v1874: 109 pgs: 1 peering, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 170 B/s rd, 26 MiB/s wr, 664 op/s; 16415/310647 objects degraded (5.284%); 44813/310647 objects misplaced (14.426%); 14 MiB/s, 220 objects/s recovering 2024-09-21T10:30:17.399 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:17.427 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:17.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:17.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:17 smithi145 ceph-mon[89359]: pgmap v1874: 109 pgs: 1 peering, 2 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 1 active+recovery_wait+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 170 B/s rd, 26 MiB/s wr, 664 op/s; 16415/310647 objects degraded (5.284%); 44813/310647 objects misplaced (14.426%); 14 MiB/s, 220 objects/s recovering 2024-09-21T10:30:17.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:17.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:17.709 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:17.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:17.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:17.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:17.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:17.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:17.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:18.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:18.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:18.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:18.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:18.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:18.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:18.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:18.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:18.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:18.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:18.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:18.800 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:18.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:18.938 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:18.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:19.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:19.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:19.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:19.192 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:19.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:19.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:19.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:19.427 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:19.447 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:19 smithi145 ceph-mon[89359]: pgmap v1875: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 142 B/s rd, 13 MiB/s wr, 589 op/s; 15124/307194 objects degraded (4.923%); 39514/307194 objects misplaced (12.863%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:30:19.465 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:19.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:19.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:19 smithi119 ceph-mon[105288]: pgmap v1875: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 142 B/s rd, 13 MiB/s wr, 589 op/s; 15124/307194 objects degraded (4.923%); 39514/307194 objects misplaced (12.863%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:30:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:19 smithi119 ceph-mon[101432]: pgmap v1875: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 142 B/s rd, 13 MiB/s wr, 589 op/s; 15124/307194 objects degraded (4.923%); 39514/307194 objects misplaced (12.863%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:30:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:19.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:30:19.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.767 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:19.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:19.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:19.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:19.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:20.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.082 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:20.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:20.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:20.232 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:20.262 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:20 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 15124/307194 objects degraded (4.923%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:20.263 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:20 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 15124/307194 objects degraded (4.923%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:20.335 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:20.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:20.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:20.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:20.520 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:30:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:30:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:20 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 15124/307194 objects degraded (4.923%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:20.622 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:20.648 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:20.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:20.800 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:20.902 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:20.943 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:20.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:21.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:21.074 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:21.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:21.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:21.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:21.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:21.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:21.389 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:21 smithi119 ceph-mon[105288]: pgmap v1876: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 6.5 MiB/s wr, 568 op/s; 15124/304236 objects degraded (4.971%); 39514/304236 objects misplaced (12.988%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T10:30:21.390 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:21 smithi119 ceph-mon[101432]: pgmap v1876: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 6.5 MiB/s wr, 568 op/s; 15124/304236 objects degraded (4.971%); 39514/304236 objects misplaced (12.988%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T10:30:21.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:21.488 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:21.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:21 smithi145 ceph-mon[89359]: pgmap v1876: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 peering, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 98 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 127 B/s rd, 6.5 MiB/s wr, 568 op/s; 15124/304236 objects degraded (4.971%); 39514/304236 objects misplaced (12.988%); 9.6 MiB/s, 153 objects/s recovering 2024-09-21T10:30:21.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:21.623 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:21.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:21.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:21.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:21.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:21.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:21.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:21.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:22.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:22.121 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:22.122 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:30:22.122 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:30:22.122 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:22.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:22.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:22.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:22.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:22.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2349145508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:22.558 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2349145508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2349145508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:22.661 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:22.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:22.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:22.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:22.824 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:22.871 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:22.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:22.957 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:22.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:23.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:23.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:23.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:23.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.269 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:23.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:23.370 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.399 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:23.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:23.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:23.529 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:23 smithi145 ceph-mon[89359]: pgmap v1877: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.5 MiB/s wr, 714 op/s; 14885/297984 objects degraded (4.995%); 38372/297984 objects misplaced (12.877%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:30:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:23 smithi119 ceph-mon[101432]: pgmap v1877: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.5 MiB/s wr, 714 op/s; 14885/297984 objects degraded (4.995%); 38372/297984 objects misplaced (12.877%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:30:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:23 smithi119 ceph-mon[105288]: pgmap v1877: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 2.5 MiB/s wr, 714 op/s; 14885/297984 objects degraded (4.995%); 38372/297984 objects misplaced (12.877%); 13 MiB/s, 211 objects/s recovering 2024-09-21T10:30:23.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:23.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:23.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.802 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:23.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:23.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:23.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:23.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:24.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:24.156 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:24.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:24.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:24.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:24.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:24 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:24.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:24 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:24.355 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:24 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:24.355 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:24 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:24.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:24.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:24.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:24 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:24 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:24.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:24.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:24.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:30:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:30:24] "GET /metrics HTTP/1.1" 200 36931 "" "Prometheus/2.51.0" 2024-09-21T10:30:24.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:24.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:24.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:24.851 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:24.883 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:24.887 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:24.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:24.995 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:24.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:25.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:25.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:25.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:25.284 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:25.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:25.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:25.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:25.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:25 smithi145 ceph-mon[89359]: pgmap v1878: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 MiB/s wr, 687 op/s; 14885/294501 objects degraded (5.054%); 37619/294501 objects misplaced (12.774%); 15 MiB/s, 233 objects/s recovering 2024-09-21T10:30:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:30:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:30:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:25 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 14885/294501 objects degraded (5.054%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:25.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[105288]: pgmap v1878: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 MiB/s wr, 687 op/s; 14885/294501 objects degraded (5.054%); 37619/294501 objects misplaced (12.774%); 15 MiB/s, 233 objects/s recovering 2024-09-21T10:30:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:30:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:30:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 14885/294501 objects degraded (5.054%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:25.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[101432]: pgmap v1878: 109 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.5 MiB/s wr, 687 op/s; 14885/294501 objects degraded (5.054%); 37619/294501 objects misplaced (12.774%); 15 MiB/s, 233 objects/s recovering 2024-09-21T10:30:25.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:30:25.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:30:25.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:30:25.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:25 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 14885/294501 objects degraded (5.054%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:25.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:25.707 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:25.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:30:25.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:25.914 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:25.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:26.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:26.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:26.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:26.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:26.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:26.333 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:26.434 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:26.470 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:26.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:26.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:26.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:26.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:26.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:26.884 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:26.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:26.985 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:27.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:27.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:27.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.303 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:27.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:27.403 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:27.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:27.531 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:27.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:27 smithi145 ceph-mon[89359]: pgmap v1879: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 440 KiB/s wr, 689 op/s; 14641/289404 objects degraded (5.059%); 37402/289404 objects misplaced (12.924%); 11 MiB/s, 176 objects/s recovering 2024-09-21T10:30:27.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:27 smithi119 ceph-mon[101432]: pgmap v1879: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 440 KiB/s wr, 689 op/s; 14641/289404 objects degraded (5.059%); 37402/289404 objects misplaced (12.924%); 11 MiB/s, 176 objects/s recovering 2024-09-21T10:30:27.555 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:27 smithi119 ceph-mon[105288]: pgmap v1879: 109 pgs: 1 active+recovery_wait+degraded+remapped, 1 active+recovering+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 4 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 440 KiB/s wr, 689 op/s; 14641/289404 objects degraded (5.059%); 37402/289404 objects misplaced (12.924%); 11 MiB/s, 176 objects/s recovering 2024-09-21T10:30:27.558 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:27.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:27.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:27.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:27.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:27.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:27.952 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:27.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:28.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:28.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:28.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.225 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:28.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:28.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:28.358 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:28.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:28.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:28.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:28.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:30:28.639 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:30:28.640 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:30:28.640 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:30:28.640 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:30:28.644 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:28.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:28.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:28.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:28.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:28.903 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:28.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:29.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:29.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:29.133 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:29.170 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:29.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:30:29.410 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:30:29.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:30:29.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:29.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:29 smithi145 ceph-mon[89359]: pgmap v1880: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 384 KiB/s wr, 723 op/s; 14641/285039 objects degraded (5.136%); 41674/285039 objects misplaced (14.620%); 12 MiB/s, 185 objects/s recovering 2024-09-21T10:30:29.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:29 smithi119 ceph-mon[105288]: pgmap v1880: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 384 KiB/s wr, 723 op/s; 14641/285039 objects degraded (5.136%); 41674/285039 objects misplaced (14.620%); 12 MiB/s, 185 objects/s recovering 2024-09-21T10:30:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:29 smithi119 ceph-mon[101432]: pgmap v1880: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 2 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.8 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 384 KiB/s wr, 723 op/s; 14641/285039 objects degraded (5.136%); 41674/285039 objects misplaced (14.620%); 12 MiB/s, 185 objects/s recovering 2024-09-21T10:30:29.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:29.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:29.678 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:30:29.678 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:118687 46.3 46:cc3b5aaf:::benchmark_data_smithi119_206743_object13586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1263)", 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:30:29.641560+0000", 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.020778555000000001, 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020810518, 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:30:29.679 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 118687 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641560+0000", 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641560+0000", 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.680 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641561+0000", 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.296e-06 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641566+0000", 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7779999999999998e-06 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:30:29.681 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641577+0000", 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1618e-05 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641585+0000", 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7880000000000007e-06 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.642990+0000", 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014052489999999999 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.682 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.643116+0000", 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012569799999999999 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.643237+0000", 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012134 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.650815+0000", 2024-09-21T10:30:29.683 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075772440000000003 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.651872+0000", 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010575000000000001 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:118688 46.8 46:192dc998:::benchmark_data_smithi119_206743_object13587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1263)", 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:30:29.641608+0000", 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.020730252000000001, 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020833203000000002, 2024-09-21T10:30:29.684 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 118688 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641608+0000", 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.685 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641608+0000", 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641608+0000", 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5000000000000003e-07 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641611+0000", 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.841e-06 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641619+0000", 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.641625+0000", 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9850000000000003e-06 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.643972+0000", 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023470159999999999 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.644081+0000", 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108825 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5,7", 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.644195+0000", 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114727 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.651858+0000", 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076630159999999999 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.651897+0000", 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8609000000000003e-05 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.651956+0000", 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8686e-05 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:118690 46.d e1263/1237 46:b0ebf60c:::benchmark_data_smithi119_206743_object13589:head v 1263'7472, mlcod=1263'7470)", 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:30:29.652119+0000", 2024-09-21T10:30:29.689 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.010218796, 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010385482, 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652119+0000", 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:30:29.690 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652119+0000", 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652120+0000", 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.54e-07 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652126+0000", 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3929999999999999e-06 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652136+0000", 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0712e-05 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652143+0000", 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1110000000000002e-06 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652323+0000", 2024-09-21T10:30:29.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017923400000000001 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652344+0000", 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1455e-05 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:118689 46.0 e1263/1248)", 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:30:29.652172+0000", 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.010166417000000001, 2024-09-21T10:30:29.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010371292000000001, 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652172+0000", 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652172+0000", 2024-09-21T10:30:29.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652172+0000", 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7899999999999999e-07 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652176+0000", 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9670000000000003e-06 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652184+0000", 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5399999999999998e-06 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652188+0000", 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5730000000000001e-06 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36246.0:118688 46.8 e1263/1260)", 2024-09-21T10:30:29.696 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:30:29.652215+0000", 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.010122928, 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010358282999999999, 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652215+0000", 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652215+0000", 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:30:29.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652216+0000", 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0900000000000002e-07 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652219+0000", 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3900000000000002e-06 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652227+0000", 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7689999999999996e-06 2024-09-21T10:30:29.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:30:29.652232+0000", 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1270000000000002e-06 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 5 2024-09-21T10:30:29.699 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:30:29.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:29.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:29.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:29.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:29.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:30.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:30.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:30.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:30.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:30.271 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:30.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:30.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:30.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:30.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:30.542 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:30 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 14641/285039 objects degraded (5.136%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:30.542 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:30:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:30:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:30.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:30.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:30 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 14641/285039 objects degraded (5.136%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:30.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:30 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 14641/285039 objects degraded (5.136%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:30.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:30.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:30.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:30.840 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:30.942 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:30.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:30.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:31.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.114 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:31.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:31.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:31.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:31.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:31.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:31.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:31.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:31.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:31 smithi119 ceph-mon[101432]: pgmap v1881: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 8859/281481 objects degraded (3.147%); 47408/281481 objects misplaced (16.842%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T10:30:31.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:31 smithi119 ceph-mon[105288]: pgmap v1881: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 8859/281481 objects degraded (3.147%); 47408/281481 objects misplaced (16.842%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T10:30:31.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:31.652 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:31 smithi145 ceph-mon[89359]: pgmap v1881: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 1 active+recovery_wait+undersized+degraded+remapped, 99 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 713 op/s; 8859/281481 objects degraded (3.147%); 47408/281481 objects misplaced (16.842%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T10:30:31.669 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:31.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:31.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:31.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:31.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:30:31.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:31.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:31.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:32.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:30:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=cleanup t=2024-09-21T10:30:31.650618828Z level=info msg="Completed cleanup jobs" duration=12.982213ms 2024-09-21T10:30:32.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.099 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:32.200 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:32.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:32.264 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:32.302 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:32.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:32.356 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:32.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:32.485 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:32.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:32.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:32.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:32.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:32.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:32.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:32.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:32.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:33.027 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:33.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:33.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:33.163 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:33.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:33.265 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:33.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:33.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:33.426 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:33.472 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:33 smithi119 ceph-mon[105288]: pgmap v1882: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 8625/275424 objects degraded (3.132%); 46315/275424 objects misplaced (16.816%); 12 MiB/s, 195 objects/s recovering 2024-09-21T10:30:33.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:33.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:33 smithi119 ceph-mon[101432]: pgmap v1882: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 8625/275424 objects degraded (3.132%); 46315/275424 objects misplaced (16.816%); 12 MiB/s, 195 objects/s recovering 2024-09-21T10:30:33.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:33.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:33.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:33.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:33.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:33.738 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:33 smithi145 ceph-mon[89359]: pgmap v1882: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 799 op/s; 8625/275424 objects degraded (3.132%); 46315/275424 objects misplaced (16.816%); 12 MiB/s, 195 objects/s recovering 2024-09-21T10:30:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:33.796 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:33.832 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:33.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:33.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.002 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:34.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:34.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:34.145 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:34.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:34.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:34.425 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:34.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:34.560 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:34.662 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:34.690 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:34.791 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:34.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:30:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:30:34] "GET /metrics HTTP/1.1" 200 36921 "" "Prometheus/2.51.0" 2024-09-21T10:30:34.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:34.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:35.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.097 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:35.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:35.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:35.223 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:35.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:35.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:35.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:35.478 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:35.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:35.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:35.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:35 smithi119 ceph-mon[105288]: pgmap v1883: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 8750/272202 objects degraded (3.215%); 45701/272202 objects misplaced (16.789%); 10 MiB/s, 162 objects/s recovering 2024-09-21T10:30:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:35 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8750/272202 objects degraded (3.215%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:35 smithi119 ceph-mon[101432]: pgmap v1883: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 8750/272202 objects degraded (3.215%); 45701/272202 objects misplaced (16.789%); 10 MiB/s, 162 objects/s recovering 2024-09-21T10:30:35.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:35 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8750/272202 objects degraded (3.215%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:35.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:35.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:35 smithi145 ceph-mon[89359]: pgmap v1883: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 8750/272202 objects degraded (3.215%); 45701/272202 objects misplaced (16.789%); 10 MiB/s, 162 objects/s recovering 2024-09-21T10:30:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:35 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8750/272202 objects degraded (3.215%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:35.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:35.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:35.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:35.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:36.110 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:36.145 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:36.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:36.286 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:36.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:36.416 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:36.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:36.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:36.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:36.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:36.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:36.806 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:36.907 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:36.935 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:36.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:36.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:37.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:37.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:37.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:37.186 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:37.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:37.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:37.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:37.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:37.462 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:37.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:37.605 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:37 smithi119 ceph-mon[105288]: pgmap v1884: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 8529/267447 objects degraded (3.189%); 45468/267447 objects misplaced (17.001%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T10:30:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:37 smithi119 ceph-mon[101432]: pgmap v1884: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 8529/267447 objects degraded (3.189%); 45468/267447 objects misplaced (17.001%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T10:30:37.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:37.743 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:37 smithi145 ceph-mon[89359]: pgmap v1884: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 751 op/s; 8529/267447 objects degraded (3.189%); 45468/267447 objects misplaced (17.001%); 9.8 MiB/s, 156 objects/s recovering 2024-09-21T10:30:37.844 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:37.872 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:37.974 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:37.994 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:37.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:38.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:38.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:38.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:38.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:38.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.402 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:38.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:38.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:38.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:38.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:38.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:38.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:38.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:38.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:38.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:38.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:39.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:39.059 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:39.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:39.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.188 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:39.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:39.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:39.317 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:39.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:39.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:39.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:39.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:39.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:39 smithi119 ceph-mon[101432]: pgmap v1885: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 8529/263610 objects degraded (3.235%); 44189/263610 objects misplaced (16.763%); 7.4 MiB/s, 118 objects/s recovering 2024-09-21T10:30:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:39 smithi119 ceph-mon[105288]: pgmap v1885: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 8529/263610 objects degraded (3.235%); 44189/263610 objects misplaced (16.763%); 7.4 MiB/s, 118 objects/s recovering 2024-09-21T10:30:39.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:39.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:39 smithi145 ceph-mon[89359]: pgmap v1885: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 716 op/s; 8529/263610 objects degraded (3.235%); 44189/263610 objects misplaced (16.763%); 7.4 MiB/s, 118 objects/s recovering 2024-09-21T10:30:39.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:39.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:39.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:39.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:39.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:40.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:40.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:40.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:40.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:40.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:40.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:40.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:40.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:40.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:40 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8529/263610 objects degraded (3.235%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:30:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:30:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:40.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:40 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8529/263610 objects degraded (3.235%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:40 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8529/263610 objects degraded (3.235%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.637 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:40.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:40.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:40.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:40.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:40.926 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:40.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:41.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:41.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:41.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:41.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:41.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:41.340 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:41.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:41.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:41.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.613 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:41.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:41 smithi119 ceph-mon[105288]: pgmap v1886: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 8662/260721 objects degraded (3.322%); 43965/260721 objects misplaced (16.863%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:30:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:41 smithi119 ceph-mon[101432]: pgmap v1886: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 8662/260721 objects degraded (3.322%); 43965/260721 objects misplaced (16.863%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:30:41.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:41.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:41 smithi145 ceph-mon[89359]: pgmap v1886: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 675 op/s; 8662/260721 objects degraded (3.322%); 43965/260721 objects misplaced (16.863%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:30:41.856 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:41.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:41.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:41.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:42.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:42.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:42.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:42.152 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:42.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:42.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:42.387 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:42.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:42.513 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:42.538 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:42.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:42.783 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:42.810 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:42.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:42.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:42.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:43.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:43.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:43.193 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:43.223 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:43.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:43.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:43.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:43.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:43.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:43.589 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:30:43.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:43 smithi119 ceph-mon[105288]: pgmap v1887: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 8458/255717 objects degraded (3.308%); 42958/255717 objects misplaced (16.799%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T10:30:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1723786246' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:43 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:43 smithi119 ceph-mon[101432]: pgmap v1887: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 8458/255717 objects degraded (3.308%); 42958/255717 objects misplaced (16.799%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T10:30:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1723786246' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:43.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:43 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:43.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:43.743 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:43 smithi145 ceph-mon[89359]: pgmap v1887: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 715 op/s; 8458/255717 objects degraded (3.308%); 42958/255717 objects misplaced (16.799%); 8.1 MiB/s, 129 objects/s recovering 2024-09-21T10:30:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1723786246' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:43 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:30:43.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:43.871 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:43.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:43.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:44.010 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:44.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:44.147 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:44.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:44.269 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:44.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:44.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:44.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:44.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:44.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:44.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:30:44] "GET /metrics HTTP/1.1" 200 36921 "" "Prometheus/2.51.0" 2024-09-21T10:30:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:30:44.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-mon[105288]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-21T10:30:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:44.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:30:44.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:44 smithi119 ceph-mon[101432]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-21T10:30:44.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:44.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:44.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:44 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:30:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:44 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:30:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:44 smithi145 ceph-mon[89359]: osdmap e1264: 8 total, 8 up, 8 in 2024-09-21T10:30:44.895 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:44.925 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:44.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:45.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:45.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:45.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:45.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:45.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:45.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:45.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:45.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:45.606 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:45 smithi119 ceph-mon[105288]: pgmap v1889: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 8577/252543 objects degraded (3.396%); 42323/252543 objects misplaced (16.759%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:30:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:45 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8458/255717 objects degraded (3.308%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:45 smithi119 ceph-mon[101432]: pgmap v1889: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 8577/252543 objects degraded (3.396%); 42323/252543 objects misplaced (16.759%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:30:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:45 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8458/255717 objects degraded (3.308%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:45.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:45.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:45 smithi145 ceph-mon[89359]: pgmap v1889: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 655 op/s; 8577/252543 objects degraded (3.396%); 42323/252543 objects misplaced (16.759%); 7.0 MiB/s, 111 objects/s recovering 2024-09-21T10:30:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:45 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8458/255717 objects degraded (3.308%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:45.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:45.869 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:45.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:45.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:46.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:46.147 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:46.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:46.276 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:46.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:46.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:46.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:46.543 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:46.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:46.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:46.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:46.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:46.823 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:46.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:46.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:46.954 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:46.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:46.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:47.041 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:30:47.042 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:30:47.042 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:30:47.042 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:30:47.042 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:30:47.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:47.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:47.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:47.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:47.337 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:47.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:47 smithi119 ceph-mon[101432]: pgmap v1890: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 8389/248319 objects degraded (3.378%); 42083/248319 objects misplaced (16.947%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:30:47.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:47.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:47.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:47 smithi119 ceph-mon[105288]: pgmap v1890: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 8389/248319 objects degraded (3.378%); 42083/248319 objects misplaced (16.947%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:30:47.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:47.657 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:47.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:47.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:47.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:47 smithi145 ceph-mon[89359]: pgmap v1890: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 637 op/s; 8389/248319 objects degraded (3.378%); 42083/248319 objects misplaced (16.947%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:30:47.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:47.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:47.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:48.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:48.049 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:48.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:48.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:48.185 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:48.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:48.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:48.369 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:30:48.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:48.445 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:48.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:48.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:48.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:48.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:48.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.747 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:48.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:30:48.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:48.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:48.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:48.915 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:30:48.915 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:30:48.939 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:48.950 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:30:48.950 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:30:48.982 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:30:48.982 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:30:49.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:49.022 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:30:49.022 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:30:49.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:49.055 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:30:49.055 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:30:49.087 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:30:49.087 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:30:49.122 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:30:49.123 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:30:49.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:49.152 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:30:49.152 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:30:49.176 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:49.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:49.305 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:49.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:49.407 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:49.439 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:49.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:49.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:49.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:49.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:49 smithi119 ceph-mon[105288]: pgmap v1891: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 8389/244248 objects degraded (3.435%); 40726/244248 objects misplaced (16.674%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:30:49.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1713237748' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:49.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:49 smithi119 ceph-mon[101432]: pgmap v1891: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 8389/244248 objects degraded (3.435%); 40726/244248 objects misplaced (16.674%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:30:49.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1713237748' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:49.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:49.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:49 smithi145 ceph-mon[89359]: pgmap v1891: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 645 op/s; 8389/244248 objects degraded (3.435%); 40726/244248 objects misplaced (16.674%); 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:30:49.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1713237748' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:30:49.709 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:49.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:49.844 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:49.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:30:49.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:49.980 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:49.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:30:50.047 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:30:50.048 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:30:50.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.126 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:50.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:50.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:50.259 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:50.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:50.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:50.363 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:30:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:30:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:30:50.394 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:50.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:50.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:50.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:50 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8389/244248 objects degraded (3.435%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:50 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8389/244248 objects degraded (3.435%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:50.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:50.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:50.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:50.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:50 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8389/244248 objects degraded (3.435%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:50.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:50.825 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:30:50.826 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:30:50.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-21T10:30:50.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:50.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:50.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:51.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:51.090 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:51.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:51.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:51.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:51.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:51.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:51.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:51.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:51 smithi119 ceph-mon[101432]: pgmap v1892: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 8547/240693 objects degraded (3.551%); 40484/240693 objects misplaced (16.820%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:30:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:51 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:30:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:51 smithi119 ceph-mon[105288]: pgmap v1892: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 8547/240693 objects degraded (3.551%); 40484/240693 objects misplaced (16.820%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:30:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:51 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:30:51.625 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:51.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:51.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:51.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:51.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:51 smithi145 ceph-mon[89359]: pgmap v1892: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 667 op/s; 8547/240693 objects degraded (3.551%); 40484/240693 objects misplaced (16.820%); 6.8 MiB/s, 108 objects/s recovering 2024-09-21T10:30:51.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:51 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:30:51.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:51.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:51.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:52.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:52.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:52.092 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:52.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:52.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:52.267 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.298 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:52.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:52.399 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:52.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:52.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:52.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:52.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:52 smithi119 ceph-mon[105288]: pgmap v1893: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 8322/234405 objects degraded (3.550%); 39251/234405 objects misplaced (16.745%); 7.1 MiB/s, 114 objects/s recovering 2024-09-21T10:30:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:52 smithi119 ceph-mon[101432]: pgmap v1893: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 8322/234405 objects degraded (3.550%); 39251/234405 objects misplaced (16.745%); 7.1 MiB/s, 114 objects/s recovering 2024-09-21T10:30:52.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:52.691 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:52.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:52.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:52 smithi145 ceph-mon[89359]: pgmap v1893: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 710 op/s; 8322/234405 objects degraded (3.550%); 39251/234405 objects misplaced (16.745%); 7.1 MiB/s, 114 objects/s recovering 2024-09-21T10:30:52.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:52.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:52.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:52.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:52.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:52.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:53.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:53.105 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:53.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:53.206 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:53.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:53.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:53.363 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:30:53.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:30:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.390 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:53.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:53.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:53.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:53.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:53.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:53.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:53.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:53.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:53.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:53.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:54.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:54.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:54.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:54.237 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:54.339 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:54.366 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:54.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:54.467 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:54.507 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:54.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:54.640 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:54.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:54.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:54.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:30:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:30:54] "GET /metrics HTTP/1.1" 200 36918 "" "Prometheus/2.51.0" 2024-09-21T10:30:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:54 smithi119 ceph-mon[101432]: pgmap v1894: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 8468/230958 objects degraded (3.666%); 38541/230958 objects misplaced (16.687%); 6.9 MiB/s, 111 objects/s recovering 2024-09-21T10:30:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:54 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8468/230958 objects degraded (3.666%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:54 smithi119 ceph-mon[105288]: pgmap v1894: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 8468/230958 objects degraded (3.666%); 38541/230958 objects misplaced (16.687%); 6.9 MiB/s, 111 objects/s recovering 2024-09-21T10:30:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:54 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8468/230958 objects degraded (3.666%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:54.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:54.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:54.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:54 smithi145 ceph-mon[89359]: pgmap v1894: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 709 op/s; 8468/230958 objects degraded (3.666%); 38541/230958 objects misplaced (16.687%); 6.9 MiB/s, 111 objects/s recovering 2024-09-21T10:30:55.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:54 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8468/230958 objects degraded (3.666%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:30:55.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:55.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:55.200 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:55.228 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:55.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:55.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:55.364 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:55.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:55.507 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:55.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:55.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:55.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:55.739 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:55.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.769 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:55.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:55.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:55.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:55.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:55.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:56.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:56.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:56.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:56.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.185 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:56.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:56.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:56.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:56.427 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:56.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:56.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:56.594 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:56.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.727 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:56.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:56.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:56.849 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:56.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:56.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:56.989 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:56.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:56 smithi145 ceph-mon[89359]: pgmap v1895: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 8250/226074 objects degraded (3.649%); 38180/226074 objects misplaced (16.888%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:30:57.091 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:57.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:56 smithi119 ceph-mon[101432]: pgmap v1895: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 8250/226074 objects degraded (3.649%); 38180/226074 objects misplaced (16.888%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:30:57.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:56 smithi119 ceph-mon[105288]: pgmap v1895: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 8250/226074 objects degraded (3.649%); 38180/226074 objects misplaced (16.888%); 8.9 MiB/s, 142 objects/s recovering 2024-09-21T10:30:57.121 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:57.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:57.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:57.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:57.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:57.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:30:57.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:57.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:30:57.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:57.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:57.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:57.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:57.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:30:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:57.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:57.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:30:58.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:58.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:58.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:58.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:58.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:58.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:58.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:58.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:58.419 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.460 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:58.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:58.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:58.583 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:58.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:58.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:30:58.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:58.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:30:58.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:58.851 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:58.953 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:58.983 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:58.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:58 smithi145 ceph-mon[89359]: pgmap v1896: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 8250/221868 objects degraded (3.718%); 36778/221868 objects misplaced (16.577%); 6.7 MiB/s, 106 objects/s recovering 2024-09-21T10:30:59.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:58 smithi119 ceph-mon[105288]: pgmap v1896: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 8250/221868 objects degraded (3.718%); 36778/221868 objects misplaced (16.577%); 6.7 MiB/s, 106 objects/s recovering 2024-09-21T10:30:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:58 smithi119 ceph-mon[101432]: pgmap v1896: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 734 op/s; 8250/221868 objects degraded (3.718%); 36778/221868 objects misplaced (16.577%); 6.7 MiB/s, 106 objects/s recovering 2024-09-21T10:30:59.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:30:59.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:30:59.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:30:59.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:59.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:59.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:59.499 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:30:59.530 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:30:59.632 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:30:59.658 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:30:59.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:30:59.781 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:30:59.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:30:59.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:30:59.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:30:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:30:59.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:00.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:00.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:00.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:59 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:31:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:30:59 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8250/221868 objects degraded (3.718%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:31:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:59 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:31:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:30:59 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8250/221868 objects degraded (3.718%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:31:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:59 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:31:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:30:59 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8250/221868 objects degraded (3.718%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:31:00.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:00.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:00.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:00.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:00.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:00.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:00.420 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:00.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:00.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:31:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:31:00.215Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:00.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:00.645 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:31:00.646 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:31:00.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:00.682 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:00.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:00.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:00.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:00.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:00.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:01.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:00 smithi145 ceph-mon[89359]: pgmap v1897: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 8389/218307 objects degraded (3.843%); 36550/218307 objects misplaced (16.742%); 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:31:01.059 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:00 smithi119 ceph-mon[105288]: pgmap v1897: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 8389/218307 objects degraded (3.843%); 36550/218307 objects misplaced (16.742%); 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:31:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:00 smithi119 ceph-mon[101432]: pgmap v1897: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 720 op/s; 8389/218307 objects degraded (3.843%); 36550/218307 objects misplaced (16.742%); 7.1 MiB/s, 113 objects/s recovering 2024-09-21T10:31:01.161 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:01.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:01.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:01.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:01.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:01.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:01.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:01.428 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:01.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:01.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:01.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:01.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:01.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:01.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:01.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:01.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:01.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:01 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:31:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:01 smithi145 ceph-mon[89359]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-21T10:31:02.055 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:02.085 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:02.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:01 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:31:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:01 smithi119 ceph-mon[101432]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-21T10:31:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:01 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:31:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:01 smithi119 ceph-mon[105288]: osdmap e1265: 8 total, 8 up, 8 in 2024-09-21T10:31:02.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:02.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:02.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:02.324 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:02.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:02.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:02.495 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:02.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:02.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:02.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:02.768 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:02.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:02.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:02.904 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:02.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:03.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:03.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:02 smithi145 ceph-mon[89359]: 1.10 scrub starts 2024-09-21T10:31:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:02 smithi145 ceph-mon[89359]: pgmap v1899: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 8089/212829 objects degraded (3.801%); 35471/212829 objects misplaced (16.666%); 7.1 MiB/s, 127 objects/s recovering 2024-09-21T10:31:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:02 smithi145 ceph-mon[89359]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-21T10:31:03.048 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:02 smithi119 ceph-mon[101432]: 1.10 scrub starts 2024-09-21T10:31:03.048 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:02 smithi119 ceph-mon[101432]: pgmap v1899: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 8089/212829 objects degraded (3.801%); 35471/212829 objects misplaced (16.666%); 7.1 MiB/s, 127 objects/s recovering 2024-09-21T10:31:03.048 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:02 smithi119 ceph-mon[101432]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-21T10:31:03.048 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:02 smithi119 ceph-mon[105288]: 1.10 scrub starts 2024-09-21T10:31:03.049 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:02 smithi119 ceph-mon[105288]: pgmap v1899: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.3 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 718 op/s; 8089/212829 objects degraded (3.801%); 35471/212829 objects misplaced (16.666%); 7.1 MiB/s, 127 objects/s recovering 2024-09-21T10:31:03.049 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:02 smithi119 ceph-mon[105288]: osdmap e1266: 8 total, 8 up, 8 in 2024-09-21T10:31:03.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:03.180 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:03.281 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:03.314 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:03.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:03.456 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:03.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:03.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:03.702 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:03.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:03.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:03.875 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:03.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:03.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:04.024 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:04.098 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:04.098 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:04.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.178 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:04.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:04.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:04.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:04.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:04.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:04.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:04.573 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:04.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:04.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:04.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:04 smithi119 ceph-mon[101432]: pgmap v1901: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 8127/210159 objects degraded (3.867%); 34941/210159 objects misplaced (16.626%); 4.9 MiB/s, 95 objects/s recovering 2024-09-21T10:31:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:04 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8127/210159 objects degraded (3.867%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:31:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:31:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:31:04] "GET /metrics HTTP/1.1" 200 36923 "" "Prometheus/2.51.0" 2024-09-21T10:31:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:04 smithi119 ceph-mon[105288]: pgmap v1901: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 8127/210159 objects degraded (3.867%); 34941/210159 objects misplaced (16.626%); 4.9 MiB/s, 95 objects/s recovering 2024-09-21T10:31:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:04 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8127/210159 objects degraded (3.867%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:31:04.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:04.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:04.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:04 smithi145 ceph-mon[89359]: pgmap v1901: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 662 op/s; 8127/210159 objects degraded (3.867%); 34941/210159 objects misplaced (16.626%); 4.9 MiB/s, 95 objects/s recovering 2024-09-21T10:31:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:04 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8127/210159 objects degraded (3.867%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:31:05.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:05.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:05.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:05.249 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:05.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:05.383 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:05.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:05.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:05.617 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:05.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:05.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:05.773 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:05.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:05.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:05.914 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:05.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:05.991 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:05 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:31:05.991 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:05 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:31:06.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:06.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:31:06.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:06.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:05 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:31:06.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:06.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:06.229 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:06.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:06.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:06.518 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:06.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:06.620 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:06.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:06.757 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:31:06.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:06.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:06.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:06 smithi119 ceph-mon[105288]: pgmap v1902: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 8107/206199 objects degraded (3.932%); 34759/206199 objects misplaced (16.857%); 4.9 MiB/s, 97 objects/s recovering 2024-09-21T10:31:06.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2869139095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:06 smithi119 ceph-mon[101432]: pgmap v1902: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 8107/206199 objects degraded (3.932%); 34759/206199 objects misplaced (16.857%); 4.9 MiB/s, 97 objects/s recovering 2024-09-21T10:31:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2869139095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:06.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:06.922 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:06.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:07.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:06 smithi145 ceph-mon[89359]: pgmap v1902: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 652 op/s; 8107/206199 objects degraded (3.932%); 34759/206199 objects misplaced (16.857%); 4.9 MiB/s, 97 objects/s recovering 2024-09-21T10:31:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2869139095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:07.059 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:07.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:07.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:07.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:07.314 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:31:07.314 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:31:07.314 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:31:07.315 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:31:07.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:07.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:07.456 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:07.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:07.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:07.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:07.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:07.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:07.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:07.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:07.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:07.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:07 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:31:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2869139095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:07 smithi145 ceph-mon[89359]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-21T10:31:08.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:07 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:31:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2869139095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:07 smithi119 ceph-mon[105288]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-21T10:31:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:07 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:31:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2869139095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:07 smithi119 ceph-mon[101432]: osdmap e1267: 8 total, 8 up, 8 in 2024-09-21T10:31:08.136 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:08.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:08.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:08.272 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:08.373 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:08.406 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:08.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:08.535 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:08.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:08.675 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:08.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:08.790 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:31:08.791 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:31:08.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:31:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:08.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:08.961 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:08.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:09.034 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:08 smithi119 ceph-mon[105288]: pgmap v1904: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 8107/202239 objects degraded (4.009%); 33439/202239 objects misplaced (16.534%); 1.4 MiB/s, 25 objects/s recovering 2024-09-21T10:31:09.034 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:08 smithi119 ceph-mon[101432]: pgmap v1904: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 8107/202239 objects degraded (4.009%); 33439/202239 objects misplaced (16.534%); 1.4 MiB/s, 25 objects/s recovering 2024-09-21T10:31:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:08 smithi145 ceph-mon[89359]: pgmap v1904: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 634 op/s; 8107/202239 objects degraded (4.009%); 33439/202239 objects misplaced (16.534%); 1.4 MiB/s, 25 objects/s recovering 2024-09-21T10:31:09.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:09.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:09.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:09.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:09.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:09.306 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:09.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.366 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:09.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:09.467 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:09.496 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:09.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:09.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:09.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:09.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:09.789 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:09.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:31:09.892 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:09.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:09.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:09.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:10.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:09 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8107/202239 objects degraded (4.009%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:31:10.064 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:10.177 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:09 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8107/202239 objects degraded (4.009%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:31:10.177 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:09 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8107/202239 objects degraded (4.009%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:31:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.179 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:10.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:10.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.318 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:10.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:10.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:10.442 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:10.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:10.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:31:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:31:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:10.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:10.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:10.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:10.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:10.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:10.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:10.976 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:10.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:10.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:10 smithi145 ceph-mon[89359]: pgmap v1905: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 586 op/s; 8219/198753 objects degraded (4.135%); 33233/198753 objects misplaced (16.721%); 1.9 MiB/s, 33 objects/s recovering 2024-09-21T10:31:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:10 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:31:11.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:11.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:10 smithi119 ceph-mon[105288]: pgmap v1905: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 586 op/s; 8219/198753 objects degraded (4.135%); 33233/198753 objects misplaced (16.721%); 1.9 MiB/s, 33 objects/s recovering 2024-09-21T10:31:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:10 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:31:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:10 smithi119 ceph-mon[101432]: pgmap v1905: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 4.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 586 op/s; 8219/198753 objects degraded (4.135%); 33233/198753 objects misplaced (16.721%); 1.9 MiB/s, 33 objects/s recovering 2024-09-21T10:31:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:10 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:31:11.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:11.244 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:11.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:11.383 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:11.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:11.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:11.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:11.648 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:11.750 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:11.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:11.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:11.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:11.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:11.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:12.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:12.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:12.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:12.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:12.178 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:12.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:12.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:12.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:12.457 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:12.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:12.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:12.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:12.724 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:12.826 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:12.847 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:12.949 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:12.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:12.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:12 smithi145 ceph-mon[89359]: pgmap v1906: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 8219/192153 objects degraded (4.277%); 32005/192153 objects misplaced (16.656%); 1.6 MiB/s, 27 objects/s recovering 2024-09-21T10:31:13.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:13.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:12 smithi119 ceph-mon[105288]: pgmap v1906: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 8219/192153 objects degraded (4.277%); 32005/192153 objects misplaced (16.656%); 1.6 MiB/s, 27 objects/s recovering 2024-09-21T10:31:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:12 smithi119 ceph-mon[101432]: pgmap v1906: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 697 op/s; 8219/192153 objects degraded (4.277%); 32005/192153 objects misplaced (16.656%); 1.6 MiB/s, 27 objects/s recovering 2024-09-21T10:31:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.122 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:13.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:13.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:13.249 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:13.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:13.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.522 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:13.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:13.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:13.645 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:13.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:13.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:13.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:13.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:13.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:13.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:14.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:14.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.050 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:14.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:14.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:14.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:14.281 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:14.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:14.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:14.412 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:14.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:14.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:14.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:14.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:14.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:14.705 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:14.794 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:31:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:31:14] "GET /metrics HTTP/1.1" 200 36923 "" "Prometheus/2.51.0" 2024-09-21T10:31:14.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:14.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:14.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:14.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:15.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:14 smithi145 ceph-mon[89359]: pgmap v1907: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 8337/188634 objects degraded (4.420%); 31310/188634 objects misplaced (16.598%); 1.2 MiB/s, 21 objects/s recovering 2024-09-21T10:31:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:14 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8337/188634 objects degraded (4.420%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:15.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:14 smithi119 ceph-mon[105288]: pgmap v1907: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 8337/188634 objects degraded (4.420%); 31310/188634 objects misplaced (16.598%); 1.2 MiB/s, 21 objects/s recovering 2024-09-21T10:31:15.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:14 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8337/188634 objects degraded (4.420%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:15.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:14 smithi119 ceph-mon[101432]: pgmap v1907: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 717 op/s; 8337/188634 objects degraded (4.420%); 31310/188634 objects misplaced (16.598%); 1.2 MiB/s, 21 objects/s recovering 2024-09-21T10:31:15.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:14 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8337/188634 objects degraded (4.420%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:15.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:15.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:15.234 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:15.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:15.314 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:31:15.314 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:31:15.321 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:31:15.321 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:31:15.359 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:31:15.359 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:31:15.362 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:15.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:15.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:15.394 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:31:15.394 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:31:15.434 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:31:15.434 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:31:15.463 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:31:15.463 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:31:15.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:15.498 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:31:15.498 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:31:15.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:15.531 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:31:15.532 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:31:15.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:15.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:15.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:15.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:15.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:15.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:15.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:16.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:16.024 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3801912393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:16.025 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3801912393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:16.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:16.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:16.211 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:16.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3801912393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:16.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:16.352 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:16.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:16.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:16.613 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:16.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:16.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:16.851 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:16.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:16.877 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:16.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:16.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:17.007 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:17.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:16 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:31:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:16 smithi119 ceph-mon[105288]: pgmap v1908: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 8337/183762 objects degraded (4.537%); 31056/183762 objects misplaced (16.900%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:16 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:31:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:16 smithi119 ceph-mon[101432]: pgmap v1908: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 8337/183762 objects degraded (4.537%); 31056/183762 objects misplaced (16.900%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:17.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:17.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:17.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:16 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:31:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:16 smithi145 ceph-mon[89359]: pgmap v1908: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 8337/183762 objects degraded (4.537%); 31056/183762 objects misplaced (16.900%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:17.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:17.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:17.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:17.516 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:17.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:17.640 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:17.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:17.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:17.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:17.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:17.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:17.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:18.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:18.029 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:18.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:18.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:18.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:18.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:18.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:18.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:18.425 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:18.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:18.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:18.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:18.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:18.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:18.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:18.823 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:18.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:18.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:18.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:18 smithi145 ceph-mon[89359]: pgmap v1909: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 8337/179442 objects degraded (4.646%); 29616/179442 objects misplaced (16.504%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:19.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:19.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:18 smithi119 ceph-mon[105288]: pgmap v1909: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 8337/179442 objects degraded (4.646%); 29616/179442 objects misplaced (16.504%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:18 smithi119 ceph-mon[101432]: pgmap v1909: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.6 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 8337/179442 objects degraded (4.646%); 29616/179442 objects misplaced (16.504%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:19.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:19.246 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:19.348 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:19.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:19.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:19.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:19.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:19.633 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:19.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:19.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:153843 46.a e1267/1237 46:59605dd9:::benchmark_data_smithi119_206743_object48742:head v 1267'9606, mlcod=1267'9604)", 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.661432+0000", 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.019423073999999999, 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.019450920999999999, 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.686 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.661432+0000", 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.661432+0000", 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.661432+0000", 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1200000000000002e-07 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.687 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.661438+0000", 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1909999999999999e-06 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.661446+0000", 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3790000000000006e-06 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.665920+0000", 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004473828 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668990+0000", 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030705580000000001 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.669009+0000", 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8791999999999999e-05 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:153845 46.7 46:e134c308:::benchmark_data_smithi119_206743_object48744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.668085+0000", 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012769282999999999, 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012855623, 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153845 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.690 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668085+0000", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668085+0000", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668087+0000", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4959999999999999e-06 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668093+0000", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6420000000000004e-06 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668118+0000", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5159000000000001e-05 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668126+0000", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3780000000000001e-06 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668185+0000", 2024-09-21T10:31:19.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8650000000000003e-05 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668306+0000", 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120902 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.668431+0000", 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000125621 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.677047+0000", 2024-09-21T10:31:19.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.008615836 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.677889+0000", 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00084199800000000005 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:153848 46.e 46:73b4573e:::benchmark_data_smithi119_206743_object48747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.671923+0000", 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0089318169999999999, 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090885389999999996, 2024-09-21T10:31:19.694 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153848 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671923+0000", 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671923+0000", 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671925+0000", 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6160000000000001e-06 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671930+0000", 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.631e-06 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671944+0000", 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3961000000000001e-05 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671953+0000", 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6789999999999994e-06 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.674278+0000", 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023255369999999999 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.674477+0000", 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000198906 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.674694+0000", 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000216892 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:153849 46.0 46:05b0d63f:::benchmark_data_smithi119_206743_object48748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.671973+0000", 2024-09-21T10:31:19.698 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0088821400000000002, 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090967010000000004, 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153849 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671973+0000", 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671973+0000", 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671973+0000", 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5900000000000002e-07 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671975+0000", 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2910000000000002e-06 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671982+0000", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.37e-06 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.671987+0000", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1309999999999997e-06 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.672789+0000", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080235899999999999 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:31:19.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.672917+0000", 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000127818 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,4", 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.673038+0000", 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012058899999999999 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.678969+0000", 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059308920000000001 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.702 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:153851 46.a e1267/1237)", 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.676405+0000", 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0044495619999999998, 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047206619999999996, 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.676405+0000", 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.703 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.676405+0000", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.676407+0000", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3540000000000001e-06 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.676412+0000", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3689999999999998e-06 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.676424+0000", 2024-09-21T10:31:19.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2279999999999999e-05 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.676431+0000", 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0600000000000002e-06 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36246.0:153848 46.e e1267/1260)", 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.679892+0000", 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00096237199999999997, 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012678749999999999, 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.705 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679892+0000", 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679892+0000", 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679893+0000", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4e-07 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679902+0000", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3920000000000003e-06 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679916+0000", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4280999999999999e-05 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679923+0000", 2024-09-21T10:31:19.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6019999999999996e-06 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:153852 46.0 46.b28db750 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.679946+0000", 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00090865900000000003, 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001249338, 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:31:19.708 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153852 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679946+0000", 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679946+0000", 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.709 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679948+0000", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.576e-06 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679951+0000", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2490000000000001e-06 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679959+0000", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2849999999999995e-06 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.679965+0000", 2024-09-21T10:31:19.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3020000000000002e-06 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.681164+0000", 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001199669 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:153854 46.a e1267/1237)", 2024-09-21T10:31:19.711 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.680563+0000", 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00029185399999999997, 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00067748199999999998, 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680563+0000", 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680563+0000", 2024-09-21T10:31:19.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680564+0000", 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8599999999999998e-07 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680569+0000", 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7359999999999998e-06 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680582+0000", 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2319e-05 2024-09-21T10:31:19.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680587+0000", 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8289999999999996e-06 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:153840 46.b e1267/1260 46:d34010fe:::benchmark_data_smithi119_206743_object48739:head v 1267'9475, mlcod=1250'2965)", 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.680613+0000", 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.000242024, 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000664586, 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.714 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680613+0000", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680613+0000", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680614+0000", 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5799999999999998e-07 2024-09-21T10:31:19.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680621+0000", 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2869999999999999e-06 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680636+0000", 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4932e-05 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680643+0000", 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6810000000000001e-06 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680832+0000", 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018998299999999999 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.680885+0000", 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2673000000000002e-05 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:153838 46.2 e1267/1260 46:40c54d03:::benchmark_data_smithi119_206743_object48737:head v 1267'9685, mlcod=1267'9683)", 2024-09-21T10:31:19.717 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:19.651827+0000", 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.02902749, 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.029496495000000001, 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.651827+0000", 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.651827+0000", 2024-09-21T10:31:19.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.651828+0000", 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6399999999999998e-07 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.651831+0000", 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8449999999999996e-06 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.651838+0000", 2024-09-21T10:31:19.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6850000000000004e-06 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.651843+0000", 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3599999999999998e-06 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.654935+0000", 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003092956 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:19.654958+0000", 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2038000000000001e-05 2024-09-21T10:31:19.720 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.721 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:31:19.721 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.721 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:31:19.721 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:31:19.721 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 10 2024-09-21T10:31:19.721 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:31:19.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:19.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:19.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:19.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:19.921 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:19.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:20.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8337/179442 objects degraded (4.646%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.051 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:20.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8337/179442 objects degraded (4.646%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8337/179442 objects degraded (4.646%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:20.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:20.188 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:20.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:20.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:20.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:20.425 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:20.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:31:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:31:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:20.555 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.581 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:20.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:20.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:20.709 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:20.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:20.844 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:20.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:20.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:20.977 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:20.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:20 smithi145 ceph-mon[89359]: pgmap v1910: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 8460/175941 objects degraded (4.808%); 29385/175941 objects misplaced (16.702%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:21.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:20 smithi119 ceph-mon[101432]: pgmap v1910: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 8460/175941 objects degraded (4.808%); 29385/175941 objects misplaced (16.702%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:20 smithi119 ceph-mon[105288]: pgmap v1910: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 730 op/s; 8460/175941 objects degraded (4.808%); 29385/175941 objects misplaced (16.702%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:21.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:21.173 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:31:21.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:31:21.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:21.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:21.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:21.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:21.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:21.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:21.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:21.679 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:21.742 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:31:21.742 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:31:21.742 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:31:21.742 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:31:21.742 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:31:21.742 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:31:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:31:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:31:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:31:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:31:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:31:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:31:21.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:21.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:21.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:21.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:21.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:22.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:22.102 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:22.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:22.240 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:22.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:22.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:22.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:22.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:22.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:22.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:22.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:22.632 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:22.733 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:22.766 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:22.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:22.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:22.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:23.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:23.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:23.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:22 smithi145 ceph-mon[89359]: pgmap v1911: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 8460/169458 objects degraded (4.992%); 28172/169458 objects misplaced (16.625%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:22 smithi119 ceph-mon[101432]: pgmap v1911: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 8460/169458 objects degraded (4.992%); 28172/169458 objects misplaced (16.625%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:23.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:22 smithi119 ceph-mon[105288]: pgmap v1911: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 813 op/s; 8460/169458 objects degraded (4.992%); 28172/169458 objects misplaced (16.625%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:23.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:23.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:23.265 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:23.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:23.397 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:23.425 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:23.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.568 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:23.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:23.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:23.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:23.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:23.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:23.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:24.085 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:24.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:24.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:24.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.255 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:24.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:24.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:24.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:24.482 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:24.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:24.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:24.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:24.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:24.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:24.796 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:31:24] "GET /metrics HTTP/1.1" 200 36920 "" "Prometheus/2.51.0" 2024-09-21T10:31:24.797 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[101432]: pgmap v1912: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 722 op/s; 8554/166119 objects degraded (5.149%); 27524/166119 objects misplaced (16.569%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:24.797 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[101432]: 1.10 scrub starts 2024-09-21T10:31:24.797 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8554/166119 objects degraded (5.149%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:24.797 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:31:24.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:24.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:24.906 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:24.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:24.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:25.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:25.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[105288]: pgmap v1912: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 722 op/s; 8554/166119 objects degraded (5.149%); 27524/166119 objects misplaced (16.569%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:25.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[105288]: 1.10 scrub starts 2024-09-21T10:31:25.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8554/166119 objects degraded (5.149%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:25.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:24 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:31:25.091 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:25.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:31:25.129 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:31:25.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-21T10:31:25.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:25.237 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:24 smithi145 ceph-mon[89359]: pgmap v1912: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 722 op/s; 8554/166119 objects degraded (5.149%); 27524/166119 objects misplaced (16.569%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:25.237 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:24 smithi145 ceph-mon[89359]: 1.10 scrub starts 2024-09-21T10:31:25.237 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:24 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8554/166119 objects degraded (5.149%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:25.237 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:24 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:31:25.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:25.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:25.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:25.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:25.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:25.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:25.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:25.661 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:25.675 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:25.743 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:25.825 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:25 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:31:25.825 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:25.825 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:25.825 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:25 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:31:25.826 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:25.826 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:25.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:25.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:25.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:25.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:26.009 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:26.111 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:26.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:25 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:31:26.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:26.137 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:26.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:26.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:26.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:26.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:26.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:26.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:26.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:26.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:26.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:26.829 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:26.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:26.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:26.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:27.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:26 smithi145 ceph-mon[89359]: pgmap v1913: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8554/161166 objects degraded (5.308%); 27285/161166 objects misplaced (16.930%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:27.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:27.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:26 smithi119 ceph-mon[105288]: pgmap v1913: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8554/161166 objects degraded (5.308%); 27285/161166 objects misplaced (16.930%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:26 smithi119 ceph-mon[101432]: pgmap v1913: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 8554/161166 objects degraded (5.308%); 27285/161166 objects misplaced (16.930%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:27.211 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.249 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:27.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:27.350 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:27.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:27.478 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:27.488 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:27.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:27.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:27.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:27.760 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:27.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:27.837 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:31:27.906 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:27.948 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:27.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:28.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:28.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:28.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/352994646' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:28.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:27 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/352994646' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:28.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:27 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:28.188 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:28.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/352994646' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:27 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:31:28.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:28.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.369 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:28.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:28.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:28.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:28.611 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:28.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:28.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:28.790 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:28.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:28.892 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:28.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:28.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:29.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:28 smithi145 ceph-mon[89359]: pgmap v1914: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 8554/156972 objects degraded (5.449%); 25887/156972 objects misplaced (16.491%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:28 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:28 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:31:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:28 smithi145 ceph-mon[89359]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-21T10:31:29.057 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[105288]: pgmap v1914: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 8554/156972 objects degraded (5.449%); 25887/156972 objects misplaced (16.491%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:31:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[105288]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-21T10:31:29.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[101432]: pgmap v1914: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 743 op/s; 8554/156972 objects degraded (5.449%); 25887/156972 objects misplaced (16.491%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:29.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:29.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:31:29.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:28 smithi119 ceph-mon[101432]: osdmap e1268: 8 total, 8 up, 8 in 2024-09-21T10:31:29.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:29.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:29.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:29.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:29.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:29.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:29.454 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:29.556 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:29.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:29.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:29.721 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:29.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:29.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:29.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:29.987 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:29.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8554/156972 objects degraded (5.449%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:30.088 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:30.088 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8554/156972 objects degraded (5.449%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:30.088 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8554/156972 objects degraded (5.449%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:30.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:30.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:30.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:30.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:30.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:30.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:30.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:31:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:31:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:30.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.681 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:30.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:30.782 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:30.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:30.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:30.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:30.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:31.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:31.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:31 smithi119 ceph-mon[101432]: pgmap v1916: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 8728/153402 objects degraded (5.690%); 25637/153402 objects misplaced (16.712%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:31 smithi119 ceph-mon[105288]: pgmap v1916: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 8728/153402 objects degraded (5.690%); 25637/153402 objects misplaced (16.712%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:31 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:31 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:31.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.256 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:31.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:31 smithi145 ceph-mon[89359]: pgmap v1916: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 8728/153402 objects degraded (5.690%); 25637/153402 objects misplaced (16.712%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:31.357 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:31.386 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:31.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:31.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:31.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:31.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:31.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.796 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:31.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:31.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:31.933 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:31.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:32.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.057 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:32.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:32.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:32.184 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:32.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:32.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:32.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:32.448 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:32.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:32.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:32.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:31:32.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:32.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:32.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:32.949 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:32.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:32.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:33.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:33 smithi119 ceph-mon[101432]: pgmap v1917: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 8728/146874 objects degraded (5.943%); 24407/146874 objects misplaced (16.618%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:33.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:33.043 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:31:33.043 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:31:33.044 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:31:33.044 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:31:33.044 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:31:33.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:33.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:33.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:33.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:33 smithi145 ceph-mon[89359]: pgmap v1917: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 8728/146874 objects degraded (5.943%); 24407/146874 objects misplaced (16.618%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:33.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:33 smithi119 ceph-mon[105288]: pgmap v1917: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 8728/146874 objects degraded (5.943%); 24407/146874 objects misplaced (16.618%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:31:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:33.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:33.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:33.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:33.620 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:33.658 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:33.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:33.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:33.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:33.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:33.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:34.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:34.057 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:34.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:34.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:34.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:34.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:34.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:34.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:34.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:34.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:34.750 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:34.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:34.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:31:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:31:34] "GET /metrics HTTP/1.1" 200 36919 "" "Prometheus/2.51.0" 2024-09-21T10:31:34.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:34.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:34.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:35.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:35.121 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:35.148 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:35.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:35.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:35 smithi145 ceph-mon[89359]: pgmap v1918: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 8848/142044 objects degraded (6.229%); 23670/142044 objects misplaced (16.664%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:35 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8848/142044 objects degraded (6.229%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:35 smithi119 ceph-mon[105288]: pgmap v1918: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 8848/142044 objects degraded (6.229%); 23670/142044 objects misplaced (16.664%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:35 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8848/142044 objects degraded (6.229%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:35 smithi119 ceph-mon[101432]: pgmap v1918: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 801 op/s; 8848/142044 objects degraded (6.229%); 23670/142044 objects misplaced (16.664%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:35.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:35 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8848/142044 objects degraded (6.229%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:35.373 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:35.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:35.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:35.540 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:35.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:35.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:35.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:35.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:35.818 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:35.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:35.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:35.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:36.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:36.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:36.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:36.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:36.224 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:36.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:36.342 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:31:36.342 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:31:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:31:36.342 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:31:36.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:36.377 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:31:36.377 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:31:36.410 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:31:36.410 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:31:36.439 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:31:36.440 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:31:36.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:36.480 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:31:36.480 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:31:36.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:36.498 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:31:36.498 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:31:36.513 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:31:36.513 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:31:36.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:36.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:36.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:36.769 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:36.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:36.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:36.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:37.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:37.045 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:37.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:37.178 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:37.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:37.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:37 smithi145 ceph-mon[89359]: pgmap v1919: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 8848/138948 objects degraded (6.368%); 23454/138948 objects misplaced (16.880%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/547050678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:37.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:37 smithi119 ceph-mon[101432]: pgmap v1919: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 8848/138948 objects degraded (6.368%); 23454/138948 objects misplaced (16.880%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/547050678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:37 smithi119 ceph-mon[105288]: pgmap v1919: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 8848/138948 objects degraded (6.368%); 23454/138948 objects misplaced (16.880%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/547050678' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:31:37.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:37.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:37.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:37.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:37.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:37.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:37.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.851 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:37.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:37.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:37.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:37.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:38.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:38.104 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:38.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:38.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:38.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:38.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:38.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:38.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:38.624 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:38.726 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:38.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:38.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:38.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:38.900 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:38.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:39.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:39.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:39.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:39.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:39 smithi145 ceph-mon[89359]: pgmap v1920: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 8848/135177 objects degraded (6.545%); 22198/135177 objects misplaced (16.421%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:39 smithi119 ceph-mon[105288]: pgmap v1920: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 8848/135177 objects degraded (6.545%); 22198/135177 objects misplaced (16.421%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:39 smithi119 ceph-mon[101432]: pgmap v1920: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 726 op/s; 8848/135177 objects degraded (6.545%); 22198/135177 objects misplaced (16.421%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:31:39.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:39.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:39.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:39.587 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:39.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:39.723 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:39.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:39.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:39.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:39.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:39.976 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:39.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:40.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:40.112 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:40.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:40.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:40 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8848/135177 objects degraded (6.545%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:40 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8848/135177 objects degraded (6.545%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:40.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:40.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:31:40.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:40.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:40.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:31:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:31:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:40 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8848/135177 objects degraded (6.545%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:40.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:40.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:40.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:40.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:40.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:40.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:40.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:40.933 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:40.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:40.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:41.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:41.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:41.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:41.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:41.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:41.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:41.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:41 smithi119 ceph-mon[101432]: pgmap v1921: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 8934/132129 objects degraded (6.762%); 21985/132129 objects misplaced (16.639%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:41 smithi119 ceph-mon[105288]: pgmap v1921: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 8934/132129 objects degraded (6.762%); 21985/132129 objects misplaced (16.639%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:41.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:41.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.496 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:41.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:41 smithi145 ceph-mon[89359]: pgmap v1921: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 702 op/s; 8934/132129 objects degraded (6.762%); 21985/132129 objects misplaced (16.639%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:31:41.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:41.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:41.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:41.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:41.897 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:41.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:41.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:42.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:42.072 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:31:42.073 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:31:42.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:42.176 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:42.278 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:42.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:42.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:42.451 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:42.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:42.594 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:42.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:42.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:42.725 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:42.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:31:42.856 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:31:42.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T10:31:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:42.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:42.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:42.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:43.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:43.063 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:43 smithi119 ceph-mon[101432]: pgmap v1922: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 8934/126480 objects degraded (7.064%); 20941/126480 objects misplaced (16.557%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:31:43.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:43.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:43.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:43.312 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:43.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:43 smithi119 ceph-mon[105288]: pgmap v1922: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 8934/126480 objects degraded (7.064%); 20941/126480 objects misplaced (16.557%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:31:43.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:43.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:43.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:43.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:43 smithi145 ceph-mon[89359]: pgmap v1922: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 747 op/s; 8934/126480 objects degraded (7.064%); 20941/126480 objects misplaced (16.557%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:31:43.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:43.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:43.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:43.744 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:43.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:43.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:43.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:31:43.942 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:43.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:43.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:44.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:44.100 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:44.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:44.228 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:44.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:44.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:44.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:44.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:44.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:44.587 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:31:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:31:44] "GET /metrics HTTP/1.1" 200 36919 "" "Prometheus/2.51.0" 2024-09-21T10:31:44.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:44.655 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:44.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:44.797 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:44.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:44.898 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:44.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:44.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:45.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:45.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:45 smithi119 ceph-mon[105288]: pgmap v1923: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 9064/121452 objects degraded (7.463%); 20254/121452 objects misplaced (16.677%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:31:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:45 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9064/121452 objects degraded (7.463%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:45 smithi119 ceph-mon[101432]: pgmap v1923: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 9064/121452 objects degraded (7.463%); 20254/121452 objects misplaced (16.677%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:31:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:45 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9064/121452 objects degraded (7.463%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:45.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:45.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:45.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:45.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:45.345 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:45.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:45.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:45.467 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:45.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:45 smithi145 ceph-mon[89359]: pgmap v1923: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 705 op/s; 9064/121452 objects degraded (7.463%); 20254/121452 objects misplaced (16.677%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:31:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:45 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9064/121452 objects degraded (7.463%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:45.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:45.603 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:45.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:45.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:45.745 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:45.847 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:45.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:45.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:45.985 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:46.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:46.142 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:46.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.276 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:46.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/869855394' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/869855394' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:46.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:46.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:46.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:46.531 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:46.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:46.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/869855394' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:31:46.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:46.668 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:46.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:46.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:46.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:46.936 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:46.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:47.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:47.067 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:47.095 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:47.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:47.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:47.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:47.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[101432]: pgmap v1924: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 9064/117657 objects degraded (7.704%); 20001/117657 objects misplaced (16.999%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/869855394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[101432]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-21T10:31:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[105288]: pgmap v1924: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 9064/117657 objects degraded (7.704%); 20001/117657 objects misplaced (16.999%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/869855394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:47.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:47 smithi119 ceph-mon[105288]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-21T10:31:47.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:47.471 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:47 smithi145 ceph-mon[89359]: pgmap v1924: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 676 op/s; 9064/117657 objects degraded (7.704%); 20001/117657 objects misplaced (16.999%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:31:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:47 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:31:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/869855394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:31:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:47 smithi145 ceph-mon[89359]: osdmap e1269: 8 total, 8 up, 8 in 2024-09-21T10:31:47.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:47.608 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:47.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:47.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:47.746 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:47.847 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:47.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:47.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:47.976 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:48.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:48.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:48.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:48.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:48.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:48.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:48.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:48.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:48.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:48.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:48.542 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:31:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:31:48.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:48.669 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:48.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:48.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:48.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:48.949 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:48.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:48.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:49.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.082 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:49.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:49.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:49.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:49.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:49.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:49.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:49 smithi119 ceph-mon[101432]: pgmap v1926: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 9064/113427 objects degraded (7.991%); 18591/113427 objects misplaced (16.390%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:31:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:49 smithi119 ceph-mon[105288]: pgmap v1926: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 9064/113427 objects degraded (7.991%); 18591/113427 objects misplaced (16.390%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:31:49.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:49 smithi145 ceph-mon[89359]: pgmap v1926: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 724 op/s; 9064/113427 objects degraded (7.991%); 18591/113427 objects misplaced (16.390%); 1.4 MiB/s, 21 objects/s recovering 2024-09-21T10:31:49.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:49.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:49.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:49.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:49.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:49.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:49.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:49.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:49.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:49.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:49.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:50.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:50.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:50.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:50.201 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:50.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:50.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:50 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9064/113427 objects degraded (7.991%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:50.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:50.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:50.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:50 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9064/113427 objects degraded (7.991%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:50.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:50 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9064/113427 objects degraded (7.991%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:50.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:50.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:31:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:31:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:31:50.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:50.607 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:50.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:50.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:50.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:50.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:50.887 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:50.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:50.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:51.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:51.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:51.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:51.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:51.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.288 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:51.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:51 smithi119 ceph-mon[101432]: pgmap v1927: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 9177/109935 objects degraded (8.348%); 18375/109935 objects misplaced (16.714%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:51 smithi119 ceph-mon[105288]: pgmap v1927: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 9177/109935 objects degraded (8.348%); 18375/109935 objects misplaced (16.714%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:51.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:51.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:51.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:51 smithi145 ceph-mon[89359]: pgmap v1927: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 739 op/s; 9177/109935 objects degraded (8.348%); 18375/109935 objects misplaced (16.714%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:51.558 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:51.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:51.688 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:51.790 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.814 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:51.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:51.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:51.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:51.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:52.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:52.071 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:52.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:52.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:52.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.342 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:52.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:52.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:52.479 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:52.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:52.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:52.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:52.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:52.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:52.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:52.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:52.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:53.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:53 smithi119 ceph-mon[105288]: pgmap v1928: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 9177/103491 objects degraded (8.867%); 17202/103491 objects misplaced (16.622%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:53.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:53 smithi119 ceph-mon[101432]: pgmap v1928: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 9177/103491 objects degraded (8.867%); 17202/103491 objects misplaced (16.622%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:53.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:53.177 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:53.278 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:53.314 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:53.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:53.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:53 smithi145 ceph-mon[89359]: pgmap v1928: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 766 op/s; 9177/103491 objects degraded (8.867%); 17202/103491 objects misplaced (16.622%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:31:53.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.580 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:53.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:53.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:53.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:53.793 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:31:53.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:53.849 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:53.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:53.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:53.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:54.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:54.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:54.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:54.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:54.362 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.394 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:54.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:54.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:54.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:54.611 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:31:54.611 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:31:54.611 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:31:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:31:54] "GET /metrics HTTP/1.1" 200 36919 "" "Prometheus/2.51.0" 2024-09-21T10:31:54.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:54.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:54.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:54.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:54.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:54.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:54.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:55.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:55.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:55.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:55 smithi119 ceph-mon[105288]: pgmap v1929: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 9322/98565 objects degraded (9.458%); 16508/98565 objects misplaced (16.748%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:31:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:55 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9322/98565 objects degraded (9.458%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:55 smithi119 ceph-mon[101432]: pgmap v1929: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 9322/98565 objects degraded (9.458%); 16508/98565 objects misplaced (16.748%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:31:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:55 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9322/98565 objects degraded (9.458%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:55.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:55.194 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:55.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:55.295 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:55.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:55.398 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:31:55.398 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:31:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:31:55.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:55.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:55.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:31:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:55 smithi145 ceph-mon[89359]: pgmap v1929: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 2.0 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 9322/98565 objects degraded (9.458%); 16508/98565 objects misplaced (16.748%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:31:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:55 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9322/98565 objects degraded (9.458%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:31:55.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:55.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:55.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:55.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:55.827 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:55.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:55.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:55.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:55.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:56.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:56.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:56.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:56.355 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:56.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:56.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:56.520 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:56.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:56.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:56.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:56.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:56.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:56.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:56.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:56.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:56.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:56.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:31:57.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:31:57.052 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:57.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:31:57.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:57.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:57.291 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:57 smithi119 ceph-mon[101432]: pgmap v1930: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 9322/95244 objects degraded (9.787%); 16292/95244 objects misplaced (17.106%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:31:57.291 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:57 smithi119 ceph-mon[105288]: pgmap v1930: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 9322/95244 objects degraded (9.787%); 16292/95244 objects misplaced (17.106%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:31:57.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:57.364 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:57.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:57.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:57.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:57 smithi145 ceph-mon[89359]: pgmap v1930: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.9 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 9322/95244 objects degraded (9.787%); 16292/95244 objects misplaced (17.106%); 1.3 MiB/s, 21 objects/s recovering 2024-09-21T10:31:57.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:57.636 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:57.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:57.768 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:57.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:57.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:57.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:58.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:58.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:58.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:58.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:58.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:58.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:31:58.301 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:31:58.403 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:58.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:58.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:58.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:58.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:58.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:58.832 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:58.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:58.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:58.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:58.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:31:59.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:31:59.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:31:59.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:31:59.222 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:31:59.324 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:31:59.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:31:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:31:59 smithi119 ceph-mon[105288]: pgmap v1931: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 9322/90975 objects degraded (10.247%); 14869/90975 objects misplaced (16.344%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:31:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:31:59 smithi119 ceph-mon[101432]: pgmap v1931: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 9322/90975 objects degraded (10.247%); 14869/90975 objects misplaced (16.344%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:31:59.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:31:59 smithi145 ceph-mon[89359]: pgmap v1931: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.8 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 692 op/s; 9322/90975 objects degraded (10.247%); 14869/90975 objects misplaced (16.344%); 1.2 MiB/s, 18 objects/s recovering 2024-09-21T10:31:59.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:31:59.625 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:31:59.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:31:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:31:59.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:31:59.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:31:59.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:31:59.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:31:59.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:00.017 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:00.050 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:32:00.125 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:32:00.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T10:32:00.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:00.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:00.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:00.334 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:00.350 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9322/90975 objects degraded (10.247%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:00.350 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9322/90975 objects degraded (10.247%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:00.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:00.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:00.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9322/90975 objects degraded (10.247%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:32:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:32:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:00.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:00.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.626 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:00.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:00.642 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:00.724 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:32:00.724 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:32:00.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:00.768 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:32:00.768 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:32:00.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:00.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:00.803 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:32:00.803 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:32:00.832 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:32:00.832 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:32:00.862 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:32:00.862 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:32:00.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:00.899 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:32:00.899 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:32:00.907 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:00.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:00.933 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:32:00.933 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:32:00.971 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:32:00.972 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:32:01.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:01.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:01.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:01.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:01.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:01 smithi145 ceph-mon[89359]: pgmap v1932: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.8 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 9478/87150 objects degraded (10.876%); 14618/87150 objects misplaced (16.773%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:32:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/784110501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:01.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:01.341 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:01 smithi119 ceph-mon[105288]: pgmap v1932: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.8 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 9478/87150 objects degraded (10.876%); 14618/87150 objects misplaced (16.773%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:32:01.341 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/784110501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:01.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:01 smithi119 ceph-mon[101432]: pgmap v1932: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.8 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 729 op/s; 9478/87150 objects degraded (10.876%); 14618/87150 objects misplaced (16.773%); 1.6 MiB/s, 25 objects/s recovering 2024-09-21T10:32:01.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/784110501' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:01.425 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:01.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:01.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:01.594 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:01.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:01.729 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:01.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:01.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:01.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:01.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:01.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:01.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:02.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:02.109 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:02.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:02.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:02.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:02.274 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:02.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:02.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:02.501 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:02.528 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:02.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:02.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:02.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:02.796 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:02.898 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:02.931 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:02.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:03.033 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:03.061 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:03.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:03.170 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:03.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:03.290 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:03.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:03.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:03.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:03.456 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:03.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:03.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[105288]: pgmap v1933: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 9478/80706 objects degraded (11.744%); 13414/80706 objects misplaced (16.621%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[101432]: pgmap v1933: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 9478/80706 objects degraded (11.744%); 13414/80706 objects misplaced (16.621%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:03.606 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:03.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:03.741 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:03 smithi145 ceph-mon[89359]: pgmap v1933: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.6 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 811 op/s; 9478/80706 objects degraded (11.744%); 13414/80706 objects misplaced (16.621%); 1.1 MiB/s, 16 objects/s recovering 2024-09-21T10:32:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:03.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:03.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:03.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:03.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:04.018 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:04.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:04.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:04.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:04.287 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:04.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:04.412 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:04.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:04.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:04.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:32:04] "GET /metrics HTTP/1.1" 200 36908 "" "Prometheus/2.51.0" 2024-09-21T10:32:04.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:04.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:04.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:04.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:04.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:04.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:04.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:04.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:04.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:32:04.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:32:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:32:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:04.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:04.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:04.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:04.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:05.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:05.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:05.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:05.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:05.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:05.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:05.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:05.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:05 smithi119 ceph-mon[101432]: pgmap v1934: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 9651/75600 objects degraded (12.766%); 12682/75600 objects misplaced (16.775%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:32:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:05 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9478/80706 objects degraded (11.744%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:05 smithi119 ceph-mon[105288]: pgmap v1934: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 9651/75600 objects degraded (12.766%); 12682/75600 objects misplaced (16.775%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:32:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:05 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9478/80706 objects degraded (11.744%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:05.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:05.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:05 smithi145 ceph-mon[89359]: pgmap v1934: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.5 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 774 op/s; 9651/75600 objects degraded (12.766%); 12682/75600 objects misplaced (16.775%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:32:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:05 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9478/80706 objects degraded (11.744%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:05.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:05.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:05.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:05.990 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:05.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:06.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:06.121 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:06.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:06.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:06.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:06.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:06.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:06.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:06.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:06.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:06.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:06.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:06.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:06.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:06.925 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:06.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:07.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:07.052 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:07.154 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:07.186 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:07.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:07.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:07.312 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:07.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:07.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:07.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:07.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:07 smithi119 ceph-mon[101432]: pgmap v1935: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 9651/72042 objects degraded (13.396%); 12418/72042 objects misplaced (17.237%); 986 KiB/s, 15 objects/s recovering 2024-09-21T10:32:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:07 smithi119 ceph-mon[105288]: pgmap v1935: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 9651/72042 objects degraded (13.396%); 12418/72042 objects misplaced (17.237%); 986 KiB/s, 15 objects/s recovering 2024-09-21T10:32:07.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:07.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:07 smithi145 ceph-mon[89359]: pgmap v1935: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.5 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 736 op/s; 9651/72042 objects degraded (13.396%); 12418/72042 objects misplaced (17.237%); 986 KiB/s, 15 objects/s recovering 2024-09-21T10:32:07.806 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:07.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:07.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:07.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:07.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:08.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:08.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:08.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:08.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:08.235 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:08.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:08.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:08.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:08.495 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:08.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:08.623 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:08.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:08.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:08.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:08.861 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:08.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:08.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:08.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.018 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:09.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:09.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:09.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:09.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:09.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:09.275 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:09.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:09.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:09.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:09.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:09 smithi119 ceph-mon[101432]: pgmap v1936: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 9651/67782 objects degraded (14.238%); 10998/67782 objects misplaced (16.226%); 986 KiB/s, 15 objects/s recovering 2024-09-21T10:32:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:09 smithi119 ceph-mon[105288]: pgmap v1936: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 9651/67782 objects degraded (14.238%); 10998/67782 objects misplaced (16.226%); 986 KiB/s, 15 objects/s recovering 2024-09-21T10:32:09.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:09.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:09.771 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:09 smithi145 ceph-mon[89359]: pgmap v1936: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.4 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 762 op/s; 9651/67782 objects degraded (14.238%); 10998/67782 objects misplaced (16.226%); 986 KiB/s, 15 objects/s recovering 2024-09-21T10:32:09.801 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:09.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:09.943 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:09.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:10.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:10.073 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:10.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:10.208 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:10.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:10.339 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:10.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:10.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:10.493 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:10 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9651/67782 objects degraded (14.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:10.493 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:10 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9651/67782 objects degraded (14.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:32:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:32:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:10 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9651/67782 objects degraded (14.238%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:10.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:10.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:10.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:10.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:10.844 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:10.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:10.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:10.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:11.010 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:11.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:11.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.154 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:11.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:11.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:11.293 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:11.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:11.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:11.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:11.564 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:11.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:11 smithi119 ceph-mon[105288]: pgmap v1937: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 9753/64632 objects degraded (15.090%); 10781/64632 objects misplaced (16.681%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:32:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:11 smithi119 ceph-mon[101432]: pgmap v1937: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 9753/64632 objects degraded (15.090%); 10781/64632 objects misplaced (16.681%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:32:11.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:11.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:11.793 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:11 smithi145 ceph-mon[89359]: pgmap v1937: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.3 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 0 B/s wr, 731 op/s; 9753/64632 objects degraded (15.090%); 10781/64632 objects misplaced (16.681%); 1.6 MiB/s, 24 objects/s recovering 2024-09-21T10:32:11.822 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:11.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:11.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:11.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:11.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:12.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:12.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:12.193 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:12.224 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:12.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:12.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:12.378 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:32:12.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:12.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:12.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:12.630 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:12.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:12.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3422741494' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:12 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3422741494' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:12 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3422741494' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:12 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:32:12.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:12.897 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:12.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:12.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:12.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:13.029 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:13.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:13.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:13.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:13.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:13.395 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:13.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:13.471 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[105288]: pgmap v1938: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 9753/58521 objects degraded (16.666%); 9659/58521 objects misplaced (16.505%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:32:13.471 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:32:13.471 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:32:13.471 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[105288]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-21T10:32:13.471 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:32:13.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[101432]: pgmap v1938: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 9753/58521 objects degraded (16.666%); 9659/58521 objects misplaced (16.505%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:32:13.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:32:13.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:32:13.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[101432]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-21T10:32:13.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:13 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:32:13.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:13.555 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:13.657 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:13.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:13.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:13.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:13 smithi145 ceph-mon[89359]: pgmap v1938: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.2 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 795 op/s; 9753/58521 objects degraded (16.666%); 9659/58521 objects misplaced (16.505%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:32:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:13 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:32:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:13 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:32:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:13 smithi145 ceph-mon[89359]: osdmap e1270: 8 total, 8 up, 8 in 2024-09-21T10:32:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:13 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:32:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.824 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:13.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:13.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:13.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:13.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:14.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:14.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:14.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:14.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:14.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:14.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:14.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:14.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:14.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:14.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:14.654 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:32:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:32:14] "GET /metrics HTTP/1.1" 200 36908 "" "Prometheus/2.51.0" 2024-09-21T10:32:14.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:14.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:14.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:14.898 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:32:14.898 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:32:14.898 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.898 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36246.0:194628 46.9 e1270/1237 46:9ef79bd3:::benchmark_data_smithi119_206743_object89527:head v 1270'12078, mlcod=1270'12075)", 2024-09-21T10:32:14.898 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:32:14.699454+0000", 2024-09-21T10:32:14.898 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.001518758, 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00154823, 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699454+0000", 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699454+0000", 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:14.899 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699454+0000", 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2500000000000004e-07 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699459+0000", 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8640000000000004e-06 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699474+0000", 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5235999999999999e-05 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.900 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699483+0000", 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1629999999999996e-06 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699676+0000", 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019317999999999999 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:14.901 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:14.699755+0000", 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9325000000000004e-05 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 1 2024-09-21T10:32:14.902 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:32:14.915 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:14.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:14.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:15.017 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:15.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.055 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:15.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:15.156 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:15.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:15.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:15.324 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:15.426 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:15.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:15.532 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:15 smithi119 ceph-mon[101432]: pgmap v1940: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 9753/55758 objects degraded (17.492%); 9201/55758 objects misplaced (16.502%); 736 KiB/s, 11 objects/s recovering 2024-09-21T10:32:15.532 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:15 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9753/55758 objects degraded (17.492%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:15.532 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:15 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:32:15.533 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:15 smithi119 ceph-mon[105288]: pgmap v1940: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 9753/55758 objects degraded (17.492%); 9201/55758 objects misplaced (16.502%); 736 KiB/s, 11 objects/s recovering 2024-09-21T10:32:15.533 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:15 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9753/55758 objects degraded (17.492%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:15.533 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:15 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:32:15.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:15.575 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:32:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:15.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:15.729 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:15 smithi145 ceph-mon[89359]: pgmap v1940: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.1 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 661 op/s; 9753/55758 objects degraded (17.492%); 9201/55758 objects misplaced (16.502%); 736 KiB/s, 11 objects/s recovering 2024-09-21T10:32:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:15 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9753/55758 objects degraded (17.492%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:15 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:32:15.811 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:15.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:32:15.905 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:32:15.906 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:32:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:15.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:15.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:16.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:16.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:16.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:16.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:16.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:16.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:16.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:32:16.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:16.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:16.558 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:16.590 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:16.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:16.819 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:16.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:16.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:17.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.097 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:17.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:17.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:17.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:17.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:17.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:17.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:17.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:17.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:17.626 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:17.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:17.764 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:17 smithi145 ceph-mon[89359]: pgmap v1941: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 9886/49977 objects degraded (19.781%); 8720/49977 objects misplaced (17.448%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:17.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:17.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:17 smithi119 ceph-mon[101432]: pgmap v1941: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 9886/49977 objects degraded (19.781%); 8720/49977 objects misplaced (17.448%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:17 smithi119 ceph-mon[105288]: pgmap v1941: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.0 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 9886/49977 objects degraded (19.781%); 8720/49977 objects misplaced (17.448%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:17.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:17.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:17.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:18.013 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:18.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:18.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:18.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:18.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:18.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:18.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:18.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:18.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:18.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:18.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:18.672 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:18.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:18.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:18.809 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:18.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:18.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:18.911 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:18.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:18.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:19.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:19.090 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:19.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:19.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:19.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:19.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:19.373 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:19.475 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:19.486 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:32:19.486 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:32:19.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T10:32:19.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:19.603 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:19.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:19.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:19 smithi145 ceph-mon[89359]: pgmap v1942: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 948 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 9886/45573 objects degraded (21.693%); 7252/45573 objects misplaced (15.913%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:19.695 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:19 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:32:19.712 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:19 smithi119 ceph-mon[101432]: pgmap v1942: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 948 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 9886/45573 objects degraded (21.693%); 7252/45573 objects misplaced (15.913%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:19.712 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:19 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:32:19.713 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:19 smithi119 ceph-mon[105288]: pgmap v1942: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 948 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 740 op/s; 9886/45573 objects degraded (21.693%); 7252/45573 objects misplaced (15.913%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:19.713 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:19 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:32:19.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:19.761 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:19.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:19.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:19.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:19.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:19.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:20.019 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:20.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:20.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:20.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:20.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:20.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:20.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:20.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:20.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:20.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:20 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9886/45573 objects degraded (21.693%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:32:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:32:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:20.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:20.627 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:20.627 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:20.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.689 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:20.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:20.726 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:20 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9886/45573 objects degraded (21.693%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:20.726 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:20 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9886/45573 objects degraded (21.693%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:20.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:20.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:20.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:20.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:20.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:20.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:21.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:21.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:21.224 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:21.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:21.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:21.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:21.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:21.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:21.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:21.728 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:21 smithi119 ceph-mon[101432]: pgmap v1943: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 876 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 9999/42102 objects degraded (23.749%); 6998/42102 objects misplaced (16.622%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T10:32:21.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:21 smithi119 ceph-mon[105288]: pgmap v1943: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 876 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 9999/42102 objects degraded (23.749%); 6998/42102 objects misplaced (16.622%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T10:32:21.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:21 smithi145 ceph-mon[89359]: pgmap v1943: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 876 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 750 op/s; 9999/42102 objects degraded (23.749%); 6998/42102 objects misplaced (16.622%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T10:32:21.822 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:21.916 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:32:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:32:21.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:21.949 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:32:21.949 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:32:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:21.961 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:21.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:21.988 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:32:21.988 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:32:22.021 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:32:22.021 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:32:22.052 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:32:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:32:22.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:22.082 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:32:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:32:22.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:22.111 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:32:22.111 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:32:22.144 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:32:22.144 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:32:22.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:22.249 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:22.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:22.385 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:22.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:22.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:22 smithi119 ceph-mon[101432]: 1.10 scrub starts 2024-09-21T10:32:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/505906715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:22 smithi119 ceph-mon[105288]: 1.10 scrub starts 2024-09-21T10:32:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/505906715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:22.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:22.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:22.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:22 smithi145 ceph-mon[89359]: 1.10 scrub starts 2024-09-21T10:32:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/505906715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:22.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:22.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:22.952 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:22.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:22.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:23.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:23.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:23.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:23.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:23.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:23.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:23.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:23.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:23.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:23.500 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:23.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:23 smithi119 ceph-mon[105288]: pgmap v1944: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 747 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 9999/35928 objects degraded (27.831%); 5851/35928 objects misplaced (16.285%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T10:32:23.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:23 smithi119 ceph-mon[101432]: pgmap v1944: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 747 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 9999/35928 objects degraded (27.831%); 5851/35928 objects misplaced (16.285%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T10:32:23.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:23.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.639 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:23.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:23.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:23.774 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:23 smithi145 ceph-mon[89359]: pgmap v1944: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 747 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 9999/35928 objects degraded (27.831%); 5851/35928 objects misplaced (16.285%); 1.6 MiB/s, 26 objects/s recovering 2024-09-21T10:32:23.875 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:23.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:23.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:24.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:24.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:24.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:24.181 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:24.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:24.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:24.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:24.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:24.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:24.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:24.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:24.670 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:24 smithi119 ceph-mon[101432]: pgmap v1945: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 687 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 9999/33030 objects degraded (30.272%); 5410/33030 objects misplaced (16.379%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:24.670 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:32:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:32:24] "GET /metrics HTTP/1.1" 200 36915 "" "Prometheus/2.51.0" 2024-09-21T10:32:24.671 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:24 smithi119 ceph-mon[105288]: pgmap v1945: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 687 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 9999/33030 objects degraded (30.272%); 5410/33030 objects misplaced (16.379%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:24.726 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:24.760 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:24 smithi145 ceph-mon[89359]: pgmap v1945: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 687 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 714 op/s; 9999/33030 objects degraded (30.272%); 5410/33030 objects misplaced (16.379%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:24.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:24.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:24.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:24.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:25.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:25.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:25.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:25.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.303 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:25.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:25.404 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:25.441 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:25.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:25.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:25.580 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:25.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:25.687 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:25 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9999/33030 objects degraded (30.272%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:25.687 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:25 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9999/33030 objects degraded (30.272%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:25.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:25 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9999/33030 objects degraded (30.272%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:25.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:25.850 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:25.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:25.980 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:25.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:26.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:26.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:26.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:26.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:26.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:26.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:26.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:26.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:26.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:26.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:26.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:32:26.713 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:32:26.714 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:32:26.714 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:32:26.714 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:32:26.714 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:32:26.714 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:32:26.714 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:26 smithi119 ceph-mon[101432]: pgmap v1946: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 558 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 10143/26826 objects degraded (37.810%); 4859/26826 objects misplaced (18.113%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T10:32:26.714 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:26 smithi119 ceph-mon[105288]: pgmap v1946: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 558 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 10143/26826 objects degraded (37.810%); 4859/26826 objects misplaced (18.113%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T10:32:26.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:26 smithi145 ceph-mon[89359]: pgmap v1946: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 558 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 803 op/s; 10143/26826 objects degraded (37.810%); 4859/26826 objects misplaced (18.113%); 1.9 MiB/s, 31 objects/s recovering 2024-09-21T10:32:26.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:26.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:26.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:26.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:27.031 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:27.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:27.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:27.198 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:27.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:27.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:27.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:27.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:32:27.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:27.456 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:27.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.589 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:27.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:27.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:27.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:27.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:27.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:27.870 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:27.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:27.972 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:28.003 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:28.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:28.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.259 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:28.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:28.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:28.394 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:28.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:28.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:28.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:28.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:28.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:28.807 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:28.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:28.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:28.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:29.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:28 smithi145 ceph-mon[89359]: pgmap v1947: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 463 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 10143/22296 objects degraded (45.492%); 3349/22296 objects misplaced (15.021%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:32:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:28 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:32:29.070 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:28 smithi119 ceph-mon[101432]: pgmap v1947: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 463 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 10143/22296 objects degraded (45.492%); 3349/22296 objects misplaced (15.021%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:32:29.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:28 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:32:29.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:28 smithi119 ceph-mon[105288]: pgmap v1947: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 463 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 768 op/s; 10143/22296 objects degraded (45.492%); 3349/22296 objects misplaced (15.021%); 1.3 MiB/s, 20 objects/s recovering 2024-09-21T10:32:29.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:28 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:32:29.172 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:29.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:29.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:29.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:29.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:29.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:29.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:29.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:29.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:29.596 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:29.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:29.739 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:29.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:29.871 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:29.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:29.994 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:29.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:29.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:29 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:32:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10143/22296 objects degraded (45.492%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:30.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:29 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:32:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10143/22296 objects degraded (45.492%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:29 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:32:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10143/22296 objects degraded (45.492%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:30.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:30.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:30.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:30.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:30.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:30.488 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:30.524 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:32:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:32:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:30.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:30.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:30.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:30.799 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:30.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:30.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:30.923 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:30.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:30.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:31.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:30 smithi145 ceph-mon[89359]: pgmap v1948: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 384 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 10252/18486 objects degraded (55.458%); 3110/18486 objects misplaced (16.824%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:32:31.057 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:31.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:30 smithi119 ceph-mon[105288]: pgmap v1948: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 384 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 10252/18486 objects degraded (55.458%); 3110/18486 objects misplaced (16.824%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:32:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:30 smithi119 ceph-mon[101432]: pgmap v1948: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 384 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 752 op/s; 10252/18486 objects degraded (55.458%); 3110/18486 objects misplaced (16.824%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:32:31.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:31.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:31.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:31.324 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:31.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:31.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:31.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:31.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:32:31.629 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:32:31.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:31.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:31.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:31.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:31 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:32:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:31 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:32:31.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:31.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:31.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:32.012 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:32.166 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:31 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:32:32.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.312 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:32.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:32.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:32.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:32.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:32.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:32.674 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:32.699 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:32.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:32.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:32.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:32.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:33.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:32 smithi145 ceph-mon[89359]: pgmap v1949: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 254 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 10252/12255 objects degraded (83.656%); 1952/12255 objects misplaced (15.928%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:32:33.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1944172607' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:33.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:33.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:33.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:32 smithi119 ceph-mon[105288]: pgmap v1949: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 254 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 10252/12255 objects degraded (83.656%); 1952/12255 objects misplaced (15.928%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:32:33.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1944172607' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:33.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:32 smithi119 ceph-mon[101432]: pgmap v1949: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 254 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 828 op/s; 10252/12255 objects degraded (83.656%); 1952/12255 objects misplaced (15.928%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:32:33.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1944172607' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:32:33.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:33.217 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:33.318 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:33.340 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:33.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:33.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:33.463 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:32:33.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:33.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:33.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:33.702 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:33.741 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:32:33.741 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:32:33.744 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:33.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:33.877 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:33.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:33.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:34.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:33 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:32:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1944172607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:33 smithi145 ceph-mon[89359]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-21T10:32:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:34.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:34.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1944172607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[105288]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-21T10:32:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:34.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:34.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1944172607' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:32:34.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[101432]: osdmap e1271: 8 total, 8 up, 8 in 2024-09-21T10:32:34.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:32:34.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:34.147 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:34.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:34.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:34.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:34.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:34.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:34.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:34.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.672 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:34.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:34.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:34.781 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:32:34] "GET /metrics HTTP/1.1" 200 36912 "" "Prometheus/2.51.0" 2024-09-21T10:32:34.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-mon[101432]: pgmap v1951: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 195 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 10252/9432 objects degraded (108.694%); 1455/9432 objects misplaced (15.426%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:34.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-mon[101432]: 1.10 scrub starts 2024-09-21T10:32:34.783 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10252/9432 objects degraded (108.694%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:34.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:34.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:34.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:34.940 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:32:34.940 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:32:34.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:34.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:34 smithi145 ceph-mon[89359]: pgmap v1951: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 195 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 10252/9432 objects degraded (108.694%); 1455/9432 objects misplaced (15.426%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:34 smithi145 ceph-mon[89359]: 1.10 scrub starts 2024-09-21T10:32:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:34 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10252/9432 objects degraded (108.694%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:35.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:35.087 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-mon[105288]: pgmap v1951: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 195 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 786 op/s; 10252/9432 objects degraded (108.694%); 1455/9432 objects misplaced (15.426%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:32:35.087 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-mon[105288]: 1.10 scrub starts 2024-09-21T10:32:35.087 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:34 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10252/9432 objects degraded (108.694%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:35.089 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:35.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:35.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:35.222 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:35.277 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Removed 105098 objects 2024-09-21T10:32:35.277 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Clean up completed and total clean up time :143.997 2024-09-21T10:32:35.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:35.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:35.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:35.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:35.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:35.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:35.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:35.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:35.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:35.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:35.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:35.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:35.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:35.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:35.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:36.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:36.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:36.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:36.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:36.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:36.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:36.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:36.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:36.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:36.451 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:36.523 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:36.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:32:36.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:36.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:36.630 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:32:36.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:32:36.630 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-09-21T10:32:36.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:32:36.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:36.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:36.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:36.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:36.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:36.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:32:36.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:36.983 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:36.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:36 smithi145 ceph-mon[89359]: pgmap v1952: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 89 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 10293/4332 objects degraded (237.604%); 1003/4332 objects misplaced (23.153%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:32:37.076 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:36 smithi119 ceph-mon[105288]: pgmap v1952: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 89 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 10293/4332 objects degraded (237.604%); 1003/4332 objects misplaced (23.153%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:32:37.076 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:36 smithi119 ceph-mon[101432]: pgmap v1952: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 89 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 749 op/s; 10293/4332 objects degraded (237.604%); 1003/4332 objects misplaced (23.153%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:32:37.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:37.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:37.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:37.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:37.330 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:37.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:37.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:37.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:37.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:37.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:37.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:37.699 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:37.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:37.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:37.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:37 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:32:37.795 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:37 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:32:37.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:37.844 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:37.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:37.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:37.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:37 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:32:38.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:38.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:38.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:38.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:38.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:38.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:38.357 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:38.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:38.462 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:38.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:38.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:38.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:38.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:38.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:38.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:38.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:38.959 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:38 smithi119 ceph-mon[101432]: pgmap v1953: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 27 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 10293/1347 objects degraded (764.143%); 8/1347 objects misplaced (0.594%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:32:38.960 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:38 smithi119 ceph-mon[105288]: pgmap v1953: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 27 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 10293/1347 objects degraded (764.143%); 8/1347 objects misplaced (0.594%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:32:38.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:38.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:39.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:39.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:39.185 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:39.207 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:38 smithi145 ceph-mon[89359]: pgmap v1953: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 27 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 698 op/s; 10293/1347 objects degraded (764.143%); 8/1347 objects misplaced (0.594%); 2.0 MiB/s, 32 objects/s recovering 2024-09-21T10:32:39.308 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:39.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:39.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:39.453 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:39.555 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:39.575 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:39.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:39.698 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:39.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:39.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:39.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:39.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:39.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:39.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:40.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:40.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10293/1347 objects degraded (764.143%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:40.053 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10293/1347 objects degraded (764.143%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:40.066 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:40.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:40.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10293/1347 objects degraded (764.143%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:40.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:40.312 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:40.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:40.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:40.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:40.510 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:32:40.510 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:32:40.532 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:32:40.532 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:32:40.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:32:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:32:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:40.549 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:32:40.549 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:32:40.560 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:40.570 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:32:40.570 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:32:40.586 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:32:40.586 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:32:40.607 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:32:40.607 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:32:40.626 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:32:40.627 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:32:40.643 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:32:40.643 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:32:40.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:40.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:40.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:40.807 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:40.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:40.926 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:40 smithi119 ceph-mon[101432]: pgmap v1954: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.6 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 9762/126 objects degraded (7747.619%); 7/126 objects misplaced (5.556%); 1.2 MiB/s, 72 objects/s recovering 2024-09-21T10:32:40.927 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2171302780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:40.927 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:40 smithi119 ceph-mon[105288]: pgmap v1954: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.6 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 9762/126 objects degraded (7747.619%); 7/126 objects misplaced (5.556%); 1.2 MiB/s, 72 objects/s recovering 2024-09-21T10:32:40.927 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2171302780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:40.930 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:40.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:41.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:41.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:41.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:41.176 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:41.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:40 smithi145 ceph-mon[89359]: pgmap v1954: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 1.6 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 611 op/s; 9762/126 objects degraded (7747.619%); 7/126 objects misplaced (5.556%); 1.2 MiB/s, 72 objects/s recovering 2024-09-21T10:32:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2171302780' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:32:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:41.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:41.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:41.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:41.547 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:41.578 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:41.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:41.670 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:41.771 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:41.792 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:41.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:41.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:41.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:42.017 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:42.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:42.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:42.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:42.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:42.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:42.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:42.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:42.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:42.530 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:42.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:42.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:42.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:42.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:42.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:42.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:42.899 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:42.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:43.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:43.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:43.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:42 smithi119 ceph-mon[105288]: pgmap v1955: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 405 op/s; 9762/81 objects degraded (12051.852%); 1.2 MiB/s, 72 objects/s recovering 2024-09-21T10:32:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:42 smithi119 ceph-mon[101432]: pgmap v1955: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 405 op/s; 9762/81 objects degraded (12051.852%); 1.2 MiB/s, 72 objects/s recovering 2024-09-21T10:32:43.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:43.145 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:43.246 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.271 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:43.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:42 smithi145 ceph-mon[89359]: pgmap v1955: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 405 op/s; 9762/81 objects degraded (12051.852%); 1.2 MiB/s, 72 objects/s recovering 2024-09-21T10:32:43.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:43.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:43.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:43.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:43.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:43.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:43.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:43.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:43.760 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:43.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:43.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:43.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:43.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:44.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:44.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:44.107 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:44.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:44.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:44.251 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:44.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:44.374 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:44.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:44.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:44.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:44.599 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:44.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:44.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:44.743 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:44.803 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:32:44] "GET /metrics HTTP/1.1" 200 36912 "" "Prometheus/2.51.0" 2024-09-21T10:32:44.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:44.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:44.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:44.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:44.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:44.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:45.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:45.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:32:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-mon[105288]: pgmap v1956: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 326 op/s; 9762/81 objects degraded (12051.852%); 1.1 MiB/s, 66 objects/s recovering 2024-09-21T10:32:45.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9762/81 objects degraded (12051.852%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:32:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-mon[101432]: pgmap v1956: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 326 op/s; 9762/81 objects degraded (12051.852%); 1.1 MiB/s, 66 objects/s recovering 2024-09-21T10:32:45.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9762/81 objects degraded (12051.852%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:45.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:45.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:44 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:32:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:44 smithi145 ceph-mon[89359]: pgmap v1956: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 326 op/s; 9762/81 objects degraded (12051.852%); 1.1 MiB/s, 66 objects/s recovering 2024-09-21T10:32:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9762/81 objects degraded (12051.852%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:45.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:45.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:45.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:45.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:45.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:45.606 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:45.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:45.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:45.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:45.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:45.851 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:45.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:45.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:45.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:46.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:46.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:46.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:46.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:46.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:46.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:46.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:46.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:46.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:46.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:46.588 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:46.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:46.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:46.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:46.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:46.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:46.935 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:46.956 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:46.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:47.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:47.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:46 smithi119 ceph-mon[101432]: pgmap v1957: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 259 op/s; 8923/81 objects degraded (11016.049%); 1.0 MiB/s, 130 objects/s recovering 2024-09-21T10:32:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:46 smithi119 ceph-mon[105288]: pgmap v1957: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 259 op/s; 8923/81 objects degraded (11016.049%); 1.0 MiB/s, 130 objects/s recovering 2024-09-21T10:32:47.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:47.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:46 smithi145 ceph-mon[89359]: pgmap v1957: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 259 op/s; 8923/81 objects degraded (11016.049%); 1.0 MiB/s, 130 objects/s recovering 2024-09-21T10:32:47.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:47.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:47.448 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:47.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:47.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:47.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:47.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:47.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:47.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:47.816 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:47.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:47.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:47.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:47.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:48.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:48.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:48.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:48.184 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:48.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:48.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:32:48.307 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:48.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:48.430 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:48.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:48.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:48.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:48.676 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:48.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:48.798 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:48.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:48.921 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:48.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:49.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:49.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:48 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:32:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:48 smithi119 ceph-mon[101432]: pgmap v1958: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 118 op/s; 8923/81 objects degraded (11016.049%); 11 KiB/s, 114 objects/s recovering 2024-09-21T10:32:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:48 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:32:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:48 smithi119 ceph-mon[105288]: pgmap v1958: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 118 op/s; 8923/81 objects degraded (11016.049%); 11 KiB/s, 114 objects/s recovering 2024-09-21T10:32:49.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:49.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:49.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:48 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:32:49.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:48 smithi145 ceph-mon[89359]: pgmap v1958: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 118 op/s; 8923/81 objects degraded (11016.049%); 11 KiB/s, 114 objects/s recovering 2024-09-21T10:32:49.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:49.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:49.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:32:49.390 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:49.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:49.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:49.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:49.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:49.655 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:49.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:49.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:49.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:49.901 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:49.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:49.962 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:49 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8923/81 objects degraded (11016.049%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:49.962 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:49 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8923/81 objects degraded (11016.049%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:50.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:50.024 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:50.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:50.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:50.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:49 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8923/81 objects degraded (11016.049%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:50.249 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:32:50.250 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:32:50.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:50.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:50.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:50.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:50.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:50.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:32:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:32:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:32:50.622 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:50.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:50.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:50.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:50.769 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:50.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:50.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:50.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:50.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:51.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:51.054 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155683 45.d 45:babfc7b3:::benchmark_data_smithi119_168706_object25513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1184)", 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.002573+0000", 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 435.04900495200002, 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0413941360000001, 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.055 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155683 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002573+0000", 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002573+0000", 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002575+0000", 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5850000000000001e-06 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002580+0000", 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3859999999999998e-06 2024-09-21T10:32:51.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002591+0000", 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1117e-05 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002598+0000", 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9500000000000004e-06 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002701+0000", 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102567 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002827+0000", 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012616999999999999 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.002972+0000", 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014517299999999999 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.007550+0000", 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045776310000000004 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.008112+0000", 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00056214199999999996 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054647+0000", 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.046535188999999998 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054656+0000", 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1560000000000001e-06 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.043912+0000", 2024-09-21T10:32:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98925529000000001 2024-09-21T10:32:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.043967+0000", 2024-09-21T10:32:51.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5779999999999998e-05 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155690 45.d 45:b8328bf4:::benchmark_data_smithi119_168706_object25520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1184)", 2024-09-21T10:32:51.063 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 435.03783870199999, 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0306722479999999, 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155690 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:32:51.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013740+0000", 2024-09-21T10:32:51.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4799999999999998e-07 2024-09-21T10:32:51.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013743+0000", 2024-09-21T10:32:51.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0630000000000002e-06 2024-09-21T10:32:51.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013751+0000", 2024-09-21T10:32:51.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4240000000000004e-06 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013756+0000", 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.013891+0000", 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013522999999999999 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.014000+0000", 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109286 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.014107+0000", 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106871 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.031183+0000", 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017076131000000001 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054681+0000", 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.023497977999999999 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.054687+0000", 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9440000000000002e-06 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044372+0000", 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98968437799999998 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044412+0000", 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0054e-05 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:155711 45.d 45:bba85108:::benchmark_data_smithi119_168706_object25541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1185)", 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 434.99223673, 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98960547399999998, 2024-09-21T10:32:51.073 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.074 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.074 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.074 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155711 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:32:51.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059342+0000", 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9400000000000004e-07 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059346+0000", 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.608e-06 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059356+0000", 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7459999999999995e-06 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059363+0000", 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.754e-06 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059439+0000", 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6777e-05 2024-09-21T10:32:51.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:36.059448+0000", 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9230000000000008e-06 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044470+0000", 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98502151400000004 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044559+0000", 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9623999999999994e-05 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0", 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.044648+0000", 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8647999999999996e-05 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.047982+0000", 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003334301 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.048818+0000", 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00083602399999999999 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.048868+0000", 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0108999999999997e-05 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:25:37.048947+0000", 2024-09-21T10:32:51.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8552000000000003e-05 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178862 45.d 45:b26a92ab:::benchmark_data_smithi119_168706_object48692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.165994+0000", 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 407.88558380199999, 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.116675195, 2024-09-21T10:32:51.081 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178862 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.165994+0000", 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.165994+0000", 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.165996+0000", 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5510000000000001e-06 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.166002+0000", 2024-09-21T10:32:51.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8510000000000004e-06 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.166015+0000", 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2659e-05 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.166023+0000", 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0339999999999997e-06 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.182261+0000", 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.016238890999999998 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.182380+0000", 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011845099999999999 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0", 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.182494+0000", 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114459 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.085 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190731+0000", 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082361599999999993 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190743+0000", 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2019e-05 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180045+0000", 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98930244700000003 2024-09-21T10:32:51.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180057+0000", 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2182e-05 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.267705+0000", 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0876482219999999 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for missing object", 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.267857+0000", 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015175400000000001 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282522+0000", 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014665025 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282670+0000", 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014749000000000001 2024-09-21T10:32:51.088 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178872 45.d 45:b9c4f220:::benchmark_data_smithi119_168706_object48702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1190)", 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.178058+0000", 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 407.87352057099997, 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.104982814, 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.089 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178872 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178058+0000", 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178058+0000", 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178059+0000", 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2079999999999999e-06 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178063+0000", 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5719999999999996e-06 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178074+0000", 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.045e-05 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.178081+0000", 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7490000000000001e-06 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190779+0000", 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012698397 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.190792+0000", 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3397000000000001e-05 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180088+0000", 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.989295857 2024-09-21T10:32:51.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180096+0000", 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0439999999999996e-06 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.268140+0000", 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.088043756 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.268261+0000", 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121038 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.268400+0000", 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013852999999999999 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.281948+0000", 2024-09-21T10:32:51.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013548098999999999 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282452+0000", 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050397899999999999 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282742+0000", 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029014699999999999 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282915+0000", 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017263700000000001 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283040+0000", 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012595400000000001 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178880 45.d 45:bd281a79:::benchmark_data_smithi119_168706_object48710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:32:51.097 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.195342+0000", 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 407.85623603099998, 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0879248580000001, 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178880 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195342+0000", 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195342+0000", 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195344+0000", 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5260000000000001e-06 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195348+0000", 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.76e-06 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195364+0000", 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6892e-05 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195372+0000", 2024-09-21T10:32:51.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195874+0000", 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00050212599999999998 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.195884+0000", 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9860000000000006e-06 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180139+0000", 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98425486299999998 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180147+0000", 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2349999999999999e-06 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.272917+0000", 2024-09-21T10:32:51.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.092769825 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.273062+0000", 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014444600000000001 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.273222+0000", 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015990500000000001 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.281992+0000", 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087703120000000006 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.282490+0000", 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049801700000000001 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283051+0000", 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00056076699999999999 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283136+0000", 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5342999999999999e-05 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283267+0000", 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013099499999999999 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:178912 45.d 45:b9c12902:::benchmark_data_smithi119_168706_object48742:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1191)", 2024-09-21T10:32:51.105 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:03.235809+0000", 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 407.81576940999997, 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0489738179999999, 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 178912 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235809+0000", 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235809+0000", 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235810+0000", 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9900000000000009e-07 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235813+0000", 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4740000000000001e-06 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235823+0000", 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9599999999999995e-06 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235828+0000", 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.203e-06 2024-09-21T10:32:51.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235890+0000", 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1362e-05 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:03.235901+0000", 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0788000000000001e-05 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180176+0000", 2024-09-21T10:32:51.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.94427550999999998 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:04.180184+0000", 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2130000000000008e-06 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.277238+0000", 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0970531889999999 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.277354+0000", 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116568 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.277518+0000", 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016381499999999999 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.283276+0000", 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057580340000000004 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284465+0000", 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011892070000000001 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284546+0000", 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0364000000000005e-05 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284630+0000", 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4439999999999998e-05 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:05.284783+0000", 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015269200000000001 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209128 45.d 45:b002702d:::benchmark_data_smithi119_168706_object78958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.17444619000003, 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.028074398, 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.113 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209128 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877132+0000", 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9999999999999998e-07 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877135+0000", 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4449999999999999e-06 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877141+0000", 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3230000000000003e-06 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877146+0000", 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.578e-06 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877233+0000", 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6698000000000002e-05 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.877244+0000", 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1333e-05 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.901205+0000", 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.023961245 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.901302+0000", 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6879999999999994e-05 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:50 smithi119 ceph-mon[105288]: pgmap v1959: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 35 op/s; 7866/81 objects degraded (9711.111%); 11 KiB/s, 202 objects/s recovering 2024-09-21T10:32:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:50 smithi119 ceph-mon[105288]: from='client.46349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:50 smithi119 ceph-mon[101432]: pgmap v1959: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 35 op/s; 7866/81 objects degraded (9711.111%); 11 KiB/s, 202 objects/s recovering 2024-09-21T10:32:51.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:50 smithi119 ceph-mon[101432]: from='client.46349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:51.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.901404+0000", 2024-09-21T10:32:51.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010234399999999999 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905038+0000", 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036335450000000002 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905099+0000", 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1057999999999997e-05 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905113+0000", 2024-09-21T10:32:51.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4389999999999999e-05 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905167+0000", 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3831e-05 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905206+0000", 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9328e-05 2024-09-21T10:32:51.121 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209152 45.d 45:b5404417:::benchmark_data_smithi119_168706_object78982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.904264+0000", 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.147314177, 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.008213383, 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.122 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209152 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904264+0000", 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904264+0000", 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904265+0000", 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5199999999999995e-07 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904269+0000", 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9509999999999999e-06 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904279+0000", 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9150000000000008e-06 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904286+0000", 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4449999999999999e-06 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904368+0000", 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1626999999999996e-05 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.904378+0000", 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7759999999999992e-06 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905241+0000", 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.000863007 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905313+0000", 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2460999999999996e-05 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.905383+0000", 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0304999999999999e-05 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912371+0000", 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069873690000000002 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912405+0000", 2024-09-21T10:32:51.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4709999999999998e-05 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912414+0000", 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1920000000000005e-06 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912444+0000", 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9844000000000001e-05 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912477+0000", 2024-09-21T10:32:51.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3929e-05 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209157 45.d 45:b1a7ba1c:::benchmark_data_smithi119_168706_object78987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.913010+0000", 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.13856850000002, 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99960250799999995, 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.129 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209157 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913010+0000", 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913010+0000", 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913011+0000", 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.384e-06 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913015+0000", 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.245e-06 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913024+0000", 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.653e-06 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913030+0000", 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4700000000000001e-06 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913113+0000", 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3172999999999998e-05 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.913122+0000", 2024-09-21T10:32:51.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3659999999999998e-06 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.908526+0000", 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99540369799999995 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.908645+0000", 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118922 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.908772+0000", 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012708399999999999 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912517+0000", 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037448450000000001 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912556+0000", 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9308999999999999e-05 2024-09-21T10:32:51.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912563+0000", 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.17e-06 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912587+0000", 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4009e-05 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912612+0000", 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5179999999999999e-05 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209163 45.d 45:b689418d:::benchmark_data_smithi119_168706_object78993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.919351+0000", 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.132227059, 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0055039210000001, 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.136 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209163 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919351+0000", 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919351+0000", 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919352+0000", 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0720000000000001e-06 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919357+0000", 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5859999999999998e-06 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919366+0000", 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1919999999999994e-06 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919373+0000", 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7510000000000002e-06 2024-09-21T10:32:51.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919434+0000", 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1178000000000003e-05 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.919443+0000", 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5790000000000004e-06 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912643+0000", 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99320071899999995 2024-09-21T10:32:51.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912761+0000", 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000117687 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.912902+0000", 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014121700000000001 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924550+0000", 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011648304999999999 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924605+0000", 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4215000000000002e-05 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924752+0000", 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014749899999999999 2024-09-21T10:32:51.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924808+0000", 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5566999999999998e-05 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924855+0000", 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7354000000000001e-05 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209174 45.d 45:b582510b:::benchmark_data_smithi119_168706_object79004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.926123+0000", 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.12545566799997, 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99885655200000001, 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209174 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926123+0000", 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926123+0000", 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926124+0000", 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6300000000000001e-06 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926130+0000", 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3489999999999999e-06 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926147+0000", 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6952999999999998e-05 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926155+0000", 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8380000000000008e-06 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926222+0000", 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6861999999999998e-05 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.926233+0000", 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1249e-05 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.916796+0000", 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99056217499999999 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.916925+0000", 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129627 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.917050+0000", 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012504399999999999 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924704+0000", 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076538480000000004 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924718+0000", 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3835000000000001e-05 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924902+0000", 2024-09-21T10:32:51.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018354499999999999 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924945+0000", 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3322999999999998e-05 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924979+0000", 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4273999999999997e-05 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209223 45.d 45:bad164a9:::benchmark_data_smithi119_168706_object79053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:45.988555+0000", 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.06302323199998, 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.94998562099999995, 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209223 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.151 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988555+0000", 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988555+0000", 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988556+0000", 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1960000000000001e-06 2024-09-21T10:32:51.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988562+0000", 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6160000000000001e-06 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988571+0000", 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5259999999999999e-06 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988578+0000", 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.657e-06 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988641+0000", 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2915000000000003e-05 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:45.988651+0000", 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0458999999999999e-05 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.920862+0000", 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93221024399999997 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.920977+0000", 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011578699999999999 2024-09-21T10:32:51.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.921096+0000", 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011887099999999999 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.924868+0000", 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0037720840000000002 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925022+0000", 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015386999999999999 2024-09-21T10:32:51.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938445+0000", 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013423091999999999 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938500+0000", 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4416999999999998e-05 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938541+0000", 2024-09-21T10:32:51.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0886999999999997e-05 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209244 45.d 45:b1a7562f:::benchmark_data_smithi119_168706_object79074:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:46.008462+0000", 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.04311631000002, 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93854806899999998, 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.158 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209244 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008462+0000", 2024-09-21T10:32:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008462+0000", 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008463+0000", 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.31e-06 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008468+0000", 2024-09-21T10:32:51.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5800000000000002e-06 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008477+0000", 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0839999999999998e-06 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008483+0000", 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5300000000000002e-06 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008554+0000", 2024-09-21T10:32:51.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0437000000000003e-05 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.008564+0000", 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0229e-05 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925049+0000", 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.916484462 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925168+0000", 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011900000000000001 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.925306+0000", 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013886599999999999 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.937933+0000", 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012626818 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938352+0000", 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0004191 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.946903+0000", 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085509980000000006 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.946966+0000", 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2939000000000002e-05 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.947010+0000", 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3716e-05 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:209252 45.d 45:bc201d98:::benchmark_data_smithi119_168706_object79082:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1203)", 2024-09-21T10:32:51.165 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:26:46.017131+0000", 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 365.034446839, 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93874006399999999, 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 209252 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017131+0000", 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017131+0000", 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017133+0000", 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7999999999999999e-06 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017139+0000", 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6450000000000002e-06 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017156+0000", 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6719999999999999e-05 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017164+0000", 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1210000000000007e-06 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017298+0000", 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013479 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.017333+0000", 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4885000000000002e-05 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938885+0000", 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.92155175899999997 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.938993+0000", 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107836 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:32:51.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.939110+0000", 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116883 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.947020+0000", 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079101529999999996 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.947200+0000", 2024-09-21T10:32:51.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000180058 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.955756+0000", 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0085562299999999997 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.955807+0000", 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0748000000000003e-05 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:26:46.955871+0000", 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4436000000000004e-05 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248855 45.6 45:68fea44d:::benchmark_data_smithi119_168706_object118685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1222)", 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.870752+0000", 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 304.18082651399999, 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0490298819999999, 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.173 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248855 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870752+0000", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870752+0000", 2024-09-21T10:32:51.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870753+0000", 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5340000000000001e-06 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870758+0000", 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0849999999999996e-06 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870771+0000", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2322e-05 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870778+0000", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0809999999999996e-06 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870833+0000", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5318000000000001e-05 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.870964+0000", 2024-09-21T10:32:51.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013073800000000001 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.871080+0000", 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000115803 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.879130+0000", 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080503699999999994 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.882842+0000", 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003711733 2024-09-21T10:32:51.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.886807+0000", 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039652749999999999 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.886816+0000", 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5169999999999999e-06 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.919735+0000", 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0329197619999999 2024-09-21T10:32:51.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.919782+0000", 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6344000000000002e-05 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248869 45.6 45:63b23490:::benchmark_data_smithi119_168706_object118699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.885958+0000", 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 304.16562024400002, 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0414795889999999, 2024-09-21T10:32:51.179 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248869 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885958+0000", 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885958+0000", 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885960+0000", 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.579e-06 2024-09-21T10:32:51.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885964+0000", 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4120000000000003e-06 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885975+0000", 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1449e-05 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.885984+0000", 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0579999999999998e-06 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.901089+0000", 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015105578999999999 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.901098+0000", 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9879999999999993e-06 2024-09-21T10:32:51.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.920124+0000", 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0190259349999999 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.920202+0000", 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7834000000000001e-05 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:32:51.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.920288+0000", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6397999999999995e-05 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927179+0000", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068907489999999998 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927310+0000", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013100300000000001 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927355+0000", 2024-09-21T10:32:51.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5219000000000002e-05 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927398+0000", 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2404000000000003e-05 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927438+0000", 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9981999999999998e-05 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248896 45.6 45:6c795fd8:::benchmark_data_smithi119_168706_object118726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 304.13647766100001, 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.012468983, 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248896 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915101+0000", 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7000000000000004e-07 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915105+0000", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7390000000000001e-06 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915113+0000", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1969999999999996e-06 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915120+0000", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4729999999999997e-06 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915251+0000", 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013153199999999999 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.915258+0000", 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8260000000000004e-06 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.923776+0000", 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0085175230000001 2024-09-21T10:32:51.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.923864+0000", 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8670000000000003e-05 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.923941+0000", 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7124000000000002e-05 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927348+0000", 2024-09-21T10:32:51.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003406203 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927444+0000", 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5995000000000005e-05 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927485+0000", 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1341000000000003e-05 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927534+0000", 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9212999999999998e-05 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927570+0000", 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5376999999999997e-05 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248913 45.6 45:673d7454:::benchmark_data_smithi119_168706_object118743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.933139+0000", 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 304.11843943500003, 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0031919119999999, 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248913 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933139+0000", 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933139+0000", 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933140+0000", 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3149999999999999e-06 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933145+0000", 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9099999999999996e-06 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933161+0000", 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5722999999999998e-05 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933169+0000", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7430000000000002e-06 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933225+0000", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6651999999999999e-05 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.933234+0000", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9889999999999998e-06 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927630+0000", 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99439544599999996 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.927735+0000", 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105407 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.928257+0000", 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052212700000000005 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936069+0000", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078119030000000002 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936108+0000", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9035000000000001e-05 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936229+0000", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012085400000000001 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936282+0000", 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3511999999999999e-05 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936331+0000", 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8296e-05 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:248931 45.6 45:633ee5d8:::benchmark_data_smithi119_168706_object118761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1223)", 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:46.949798+0000", 2024-09-21T10:32:51.200 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 304.10178054099998, 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99733362000000003, 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 248931 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949798+0000", 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949798+0000", 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949799+0000", 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.55e-06 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949805+0000", 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3700000000000003e-06 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949817+0000", 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2244e-05 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949824+0000", 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4070000000000004e-06 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949984+0000", 2024-09-21T10:32:51.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015923399999999999 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:46.949992+0000", 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6629999999999999e-06 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.932278+0000", 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98228570400000004 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.932400+0000", 2024-09-21T10:32:51.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012246699999999999 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.932535+0000", 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013443400000000001 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.936215+0000", 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036803069999999998 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.940108+0000", 2024-09-21T10:32:51.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003892923 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.947038+0000", 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069303389999999998 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.947086+0000", 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8074000000000001e-05 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:32:51.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:47.947131+0000", 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4904000000000002e-05 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:32:51.207 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:32:51.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:51.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:50 smithi145 ceph-mon[89359]: pgmap v1959: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 35 op/s; 7866/81 objects degraded (9711.111%); 11 KiB/s, 202 objects/s recovering 2024-09-21T10:32:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:50 smithi145 ceph-mon[89359]: from='client.46349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:32:51.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:51.351 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:51.453 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:51.474 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:51.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:51.576 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:51.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:51.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:51.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:51.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:51.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:51.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:51.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:51.966 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:51.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:52.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:52.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:52.094 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":1959,"stamp":"2024-09-21T10:32:49.634519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649587,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":28,"num_read_kb":69,"num_write":210275,"num_write_kb":6726452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16299,"num_bytes_recovered":897571179,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117168,"ondisk_log_size":117168,"up":327,"acting":325,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":347,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4257284,"kb_used_data":1519484,"kb_used_omap":731,"kb_used_meta":2736996,"kb_avail":745703932,"statfs":{"total":767960285184,"available":763600826368,"internally_reserved":0,"allocated":1555951616,"data_stored":1514502661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":749144,"internal_metadata":2802684328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-27590680,"num_objects":-422,"num_object_clones":0,"num_object_copies":-1266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2427,"num_objects_misplaced":-8,"num_objects_unfound":0,"num_objects_dirty":-422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":422,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2428,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"46.f","version":"1271'13224","reported_seq":13286,"reported_epoch":1271,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:32:35.271840+0000","last_change":"2024-09-21T10:30:24.917718+0000","last_active":"2024-09-21T10:32:35.271840+0000","last_peered":"2024-09-21T10:32:35.271840+0000","last_clean":"2024-09-21T10:30:09.059598+0000","last_became_active":"2024-09-21T10:30:10.560688+0000","last_became_peered":"2024-09-21T10:30:10.560688+0000","last_unstale":"2024-09-21T10:32:35.271840+0000","last_undegraded":"2024-09-21T10:32:35.271840+0000","last_fullsized":"2024-09-21T10:32:35.271840+0000","mapping_epoch":1260,"log_start":"1258'6000","ondisk_log_start":"1258'6000","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":7224,"log_dups_size":0,"ondisk_log_size":7224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:45.273019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13224,"num_write_kb":423168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,3],"avail_no_missing":["6","0","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.e","version":"1271'13158","reported_seq":13241,"reported_epoch":1271,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:32:35.212532+0000","last_change":"2024-09-21T10:30:10.671356+0000","last_active":"2024-09-21T10:32:35.212532+0000","last_peered":"2024-09-21T10:32:35.212532+0000","last_clean":"2024-09-21T10:30:09.033647+0000","last_became_active":"2024-09-21T10:30:10.656973+0000","last_became_peered":"2024-09-21T10:30:10.656973+0000","last_unstale":"2024-09-21T10:32:35.212532+0000","last_undegraded":"2024-09-21T10:30:10.434191+0000","last_fullsized":"2024-09-21T10:30:10.397601+0000","mapping_epoch":1260,"log_start":"1252'5200","ondisk_log_start":"1252'5200","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":7958,"log_dups_size":0,"ondisk_log_size":7958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6553,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13158,"num_write_kb":421056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,7","objects":-6553}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.d","version":"1271'13262","reported_seq":13343,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:35.268470+0000","last_change":"2024-09-21T10:28:29.310826+0000","last_active":"2024-09-21T10:32:35.268470+0000","last_peered":"2024-09-21T10:32:35.268470+0000","last_clean":"2024-09-21T10:32:35.268470+0000","last_became_active":"2024-09-21T10:28:29.310534+0000","last_became_peered":"2024-09-21T10:28:29.310534+0000","last_unstale":"2024-09-21T10:32:35.268470+0000","last_undegraded":"2024-09-21T10:32:35.268470+0000","last_fullsized":"2024-09-21T10:32:35.268470+0000","mapping_epoch":1237,"log_start":"1250'3200","ondisk_log_start":"1250'3200","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:24:08.140475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13262,"num_write_kb":424321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.c","version":"1271'13138","reported_seq":17502,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:35.194620+0000","last_change":"2024-09-21T10:30:15.115030+0000","last_active":"2024-09-21T10:32:35.194620+0000","last_peered":"2024-09-21T10:32:35.194620+0000","last_clean":"2024-09-21T10:32:35.194620+0000","last_became_active":"2024-09-21T10:30:15.105160+0000","last_became_peered":"2024-09-21T10:30:15.105160+0000","last_unstale":"2024-09-21T10:32:35.194620+0000","last_undegraded":"2024-09-21T10:32:35.194620+0000","last_fullsized":"2024-09-21T10:32:35.194620+0000","mapping_epoch":1262,"log_start":"1263'7653","ondisk_log_start":"1263'7653","created":1237,"last_epoch_clean":1263,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:44.628036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13138,"num_write_kb":420416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2137,"num_bytes_recovered":140050432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.b","version":"1271'13016","reported_seq":23429,"reported_epoch":1271,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:32:48.721421+0000","last_change":"2024-09-21T10:30:24.922158+0000","last_active":"2024-09-21T10:32:48.721421+0000","last_peered":"2024-09-21T10:32:48.721421+0000","last_clean":"2024-09-21T10:30:09.030714+0000","last_became_active":"2024-09-21T10:30:10.670430+0000","last_became_peered":"2024-09-21T10:30:10.670430+0000","last_unstale":"2024-09-21T10:32:48.721421+0000","last_undegraded":"2024-09-21T10:30:10.437513+0000","last_fullsized":"2024-09-21T10:30:10.402440+0000","mapping_epoch":1260,"log_start":"1250'3000","ondisk_log_start":"1250'3000","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13016,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5165,"num_bytes_recovered":179503104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2","objects":-1313}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.a","version":"1271'13098","reported_seq":13140,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:35.213829+0000","last_change":"2024-09-21T10:28:29.309191+0000","last_active":"2024-09-21T10:32:35.213829+0000","last_peered":"2024-09-21T10:32:35.213829+0000","last_clean":"2024-09-21T10:32:35.213829+0000","last_became_active":"2024-09-21T10:28:29.308876+0000","last_became_peered":"2024-09-21T10:28:29.308876+0000","last_unstale":"2024-09-21T10:32:35.213829+0000","last_undegraded":"2024-09-21T10:32:35.213829+0000","last_fullsized":"2024-09-21T10:32:35.213829+0000","mapping_epoch":1237,"log_start":"1263'6900","ondisk_log_start":"1263'6900","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":6198,"log_dups_size":0,"ondisk_log_size":6198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:38.414144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13098,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.9","version":"1271'13048","reported_seq":13128,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:35.269427+0000","last_change":"2024-09-21T10:28:29.309362+0000","last_active":"2024-09-21T10:32:35.269427+0000","last_peered":"2024-09-21T10:32:35.269427+0000","last_clean":"2024-09-21T10:32:35.269427+0000","last_became_active":"2024-09-21T10:28:29.309079+0000","last_became_peered":"2024-09-21T10:28:29.309079+0000","last_unstale":"2024-09-21T10:32:35.269427+0000","last_undegraded":"2024-09-21T10:32:35.269427+0000","last_fullsized":"2024-09-21T10:32:35.269427+0000","mapping_epoch":1237,"log_start":"1250'3000","ondisk_log_start":"1250'3000","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:57:22.010069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13048,"num_write_kb":417536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"46.8","version":"1271'13052","reported_seq":13136,"reported_epoch":1271,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:32:35.273137+0000","last_change":"2024-09-21T10:30:10.599969+0000","last_active":"2024-09-21T10:32:35.273137+0000","last_peered":"2024-09-21T10:32:35.273137+0000","last_clean":"2024-09-21T10:30:09.077894+0000","last_became_active":"2024-09-21T10:30:10.592537+0000","last_became_peered":"2024-09-21T10:30:10.592537+0000","last_unstale":"2024-09-21T10:32:35.273137+0000","last_undegraded":"2024-09-21T10:32:35.273137+0000","last_fullsized":"2024-09-21T10:32:35.273137+0000","mapping_epoch":1260,"log_start":"1263'7269","ondisk_log_start":"1263'7269","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":5783,"log_dups_size":0,"ondisk_log_size":5783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13052,"num_write_kb":417664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,4,7],"avail_no_missing":["1","4","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.7","version":"1271'13186","reported_seq":13266,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:35.235765+0000","last_change":"2024-09-21T10:28:29.314114+0000","last_active":"2024-09-21T10:32:35.235765+0000","last_peered":"2024-09-21T10:32:35.235765+0000","last_clean":"2024-09-21T10:32:35.235765+0000","last_became_active":"2024-09-21T10:28:29.313435+0000","last_became_peered":"2024-09-21T10:28:29.313435+0000","last_unstale":"2024-09-21T10:32:35.235765+0000","last_undegraded":"2024-09-21T10:32:35.235765+0000","last_fullsized":"2024-09-21T10:32:35.235765+0000","mapping_epoch":1237,"log_start":"1252'5200","ondisk_log_start":"1252'5200","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":7986,"log_dups_size":0,"ondisk_log_size":7986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:04.845295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13186,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"46.6","version":"1271'13084","reported_seq":13142,"reported_epoch":1271,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:32:35.211640+0000","last_change":"2024-09-21T10:30:10.724206+0000","last_active":"2024-09-21T10:32:35.211640+0000","last_peered":"2024-09-21T10:32:35.211640+0000","last_clean":"2024-09-21T10:30:09.027563+0000","last_became_active":"2024-09-21T10:30:10.711376+0000","last_became_peered":"2024-09-21T10:30:10.711376+0000","last_unstale":"2024-09-21T10:32:35.211640+0000","last_undegraded":"2024-09-21T10:32:35.211640+0000","last_fullsized":"2024-09-21T10:32:35.211640+0000","mapping_epoch":1260,"log_start":"1263'7238","ondisk_log_start":"1263'7238","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":5846,"log_dups_size":0,"ondisk_log_size":5846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13084,"num_write_kb":418688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[1,6,3],"avail_no_missing":["1","3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.5","version":"1271'13340","reported_seq":17652,"reported_epoch":1271,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:32:35.266737+0000","last_change":"2024-09-21T10:30:59.241385+0000","last_active":"2024-09-21T10:32:35.266737+0000","last_peered":"2024-09-21T10:32:35.266737+0000","last_clean":"2024-09-21T10:28:55.800958+0000","last_became_active":"2024-09-21T10:30:10.550952+0000","last_became_peered":"2024-09-21T10:30:10.550952+0000","last_unstale":"2024-09-21T10:32:35.266737+0000","last_undegraded":"2024-09-21T10:32:35.266737+0000","last_fullsized":"2024-09-21T10:32:35.266737+0000","mapping_epoch":1260,"log_start":"1263'7863","ondisk_log_start":"1263'7863","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":5477,"log_dups_size":0,"ondisk_log_size":5477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13340,"num_write_kb":426880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2112,"num_bytes_recovered":138412032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,4,2],"avail_no_missing":["3","2","4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"46.4","version":"1271'13202","reported_seq":13244,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:35.225637+0000","last_change":"2024-09-21T10:28:29.310707+0000","last_active":"2024-09-21T10:32:35.225637+0000","last_peered":"2024-09-21T10:32:35.225637+0000","last_clean":"2024-09-21T10:32:35.225637+0000","last_became_active":"2024-09-21T10:28:29.310429+0000","last_became_peered":"2024-09-21T10:28:29.310429+0000","last_unstale":"2024-09-21T10:32:35.225637+0000","last_undegraded":"2024-09-21T10:32:35.225637+0000","last_fullsized":"2024-09-21T10:32:35.225637+0000","mapping_epoch":1237,"log_start":"1263'7000","ondisk_log_start":"1263'7000","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":6202,"log_dups_size":0,"ondisk_log_size":6202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:40:14.041852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13202,"num_write_kb":422464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"46.3","version":"1271'13178","reported_seq":13261,"reported_epoch":1271,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:32:35.263873+0000","last_change":"2024-09-21T10:30:10.665973+0000","last_active":"2024-09-21T10:32:35.263873+0000","last_peered":"2024-09-21T10:32:35.263873+0000","last_clean":"2024-09-21T10:30:09.042214+0000","last_became_active":"2024-09-21T10:30:10.661509+0000","last_became_peered":"2024-09-21T10:30:10.661509+0000","last_unstale":"2024-09-21T10:32:35.263873+0000","last_undegraded":"2024-09-21T10:32:35.263873+0000","last_fullsized":"2024-09-21T10:32:35.263873+0000","mapping_epoch":1260,"log_start":"1263'7403","ondisk_log_start":"1263'7403","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13178,"num_write_kb":421696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,0,2],"avail_no_missing":["1","0","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.2","version":"1271'13220","reported_seq":17705,"reported_epoch":1271,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:32:35.227713+0000","last_change":"2024-09-21T10:30:24.901717+0000","last_active":"2024-09-21T10:32:35.227713+0000","last_peered":"2024-09-21T10:32:35.227713+0000","last_clean":"2024-09-21T10:28:55.794029+0000","last_became_active":"2024-09-21T10:30:10.555664+0000","last_became_peered":"2024-09-21T10:30:10.555664+0000","last_unstale":"2024-09-21T10:32:35.227713+0000","last_undegraded":"2024-09-21T10:32:35.227713+0000","last_fullsized":"2024-09-21T10:32:35.227713+0000","mapping_epoch":1260,"log_start":"1258'6000","ondisk_log_start":"1258'6000","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":7220,"log_dups_size":0,"ondisk_log_size":7220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:05.319262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13220,"num_write_kb":423040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2203,"num_bytes_recovered":144375808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[6,2,7],"avail_no_missing":["6","0","2","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":6,"purged_snaps":[]},{"pgid":"46.1","version":"1271'13080","reported_seq":19457,"reported_epoch":1271,"state":"active+recovery_wait+remapped","last_fresh":"2024-09-21T10:32:35.233244+0000","last_change":"2024-09-21T10:30:10.597034+0000","last_active":"2024-09-21T10:32:35.233244+0000","last_peered":"2024-09-21T10:32:35.233244+0000","last_clean":"2024-09-21T10:30:09.060878+0000","last_became_active":"2024-09-21T10:30:10.585434+0000","last_became_peered":"2024-09-21T10:30:10.585434+0000","last_unstale":"2024-09-21T10:32:35.233244+0000","last_undegraded":"2024-09-21T10:32:35.233244+0000","last_fullsized":"2024-09-21T10:32:35.233244+0000","mapping_epoch":1260,"log_start":"1263'7262","ondisk_log_start":"1263'7262","created":1237,"last_epoch_clean":1254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13080,"num_write_kb":418560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2097,"num_bytes_recovered":137428992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[1,7,4],"avail_no_missing":["1","4","5","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"46.0","version":"1271'12912","reported_seq":18097,"reported_epoch":1271,"state":"active+recovery_wait","last_fresh":"2024-09-21T10:32:35.258533+0000","last_change":"2024-09-21T10:31:01.755811+0000","last_active":"2024-09-21T10:32:35.258533+0000","last_peered":"2024-09-21T10:32:35.258533+0000","last_clean":"2024-09-21T10:29:03.902926+0000","last_became_active":"2024-09-21T10:31:01.732347+0000","last_became_peered":"2024-09-21T10:31:01.732347+0000","last_unstale":"2024-09-21T10:32:35.258533+0000","last_undegraded":"2024-09-21T10:32:35.258533+0000","last_fullsized":"2024-09-21T10:32:35.258533+0000","mapping_epoch":1265,"log_start":"1252'5000","ondisk_log_start":"1252'5000","created":1237,"last_epoch_clean":1238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:28:28.287892+0000","last_clean_scrub_stamp":"2024-09-21T10:28:28.287892+0000","objects_scrubbed":0,"log_size":7912,"log_dups_size":0,"ondisk_log_size":7912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12912,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2547,"num_bytes_recovered":156958720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":["2","0","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":27,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.392085+0000","last_change":"2024-09-21T10:30:09.420642+0000","last_active":"2024-09-21T10:32:12.392085+0000","last_peered":"2024-09-21T10:32:12.392085+0000","last_clean":"2024-09-21T10:32:12.392085+0000","last_became_active":"2024-09-21T10:30:09.407144+0000","last_became_peered":"2024-09-21T10:30:09.407144+0000","last_unstale":"2024-09-21T10:32:12.392085+0000","last_undegraded":"2024-09-21T10:32:12.392085+0000","last_fullsized":"2024-09-21T10:32:12.392085+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:49.017939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1248,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.373945+0000","last_change":"2024-09-21T10:28:37.587114+0000","last_active":"2024-09-21T10:32:12.373945+0000","last_peered":"2024-09-21T10:32:12.373945+0000","last_clean":"2024-09-21T10:32:12.373945+0000","last_became_active":"2024-09-21T10:28:37.586686+0000","last_became_peered":"2024-09-21T10:28:37.586686+0000","last_unstale":"2024-09-21T10:32:12.373945+0000","last_undegraded":"2024-09-21T10:32:12.373945+0000","last_fullsized":"2024-09-21T10:32:12.373945+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:41:45.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1859,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.731781+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:32:32.731781+0000","last_peered":"2024-09-21T10:32:32.731781+0000","last_clean":"2024-09-21T10:32:32.731781+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:32:32.731781+0000","last_undegraded":"2024-09-21T10:32:32.731781+0000","last_fullsized":"2024-09-21T10:32:32.731781+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1878,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.725914+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:32:32.725914+0000","last_peered":"2024-09-21T10:32:32.725914+0000","last_clean":"2024-09-21T10:32:32.725914+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:32:32.725914+0000","last_undegraded":"2024-09-21T10:32:32.725914+0000","last_fullsized":"2024-09-21T10:32:32.725914+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":25,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391881+0000","last_change":"2024-09-21T10:30:09.427552+0000","last_active":"2024-09-21T10:32:12.391881+0000","last_peered":"2024-09-21T10:32:12.391881+0000","last_clean":"2024-09-21T10:32:12.391881+0000","last_became_active":"2024-09-21T10:30:09.412897+0000","last_became_peered":"2024-09-21T10:30:09.412897+0000","last_unstale":"2024-09-21T10:32:12.391881+0000","last_undegraded":"2024-09-21T10:32:12.391881+0000","last_fullsized":"2024-09-21T10:32:12.391881+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:49.300103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1234,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374173+0000","last_change":"2024-09-21T10:27:19.333245+0000","last_active":"2024-09-21T10:32:12.374173+0000","last_peered":"2024-09-21T10:32:12.374173+0000","last_clean":"2024-09-21T10:32:12.374173+0000","last_became_active":"2024-09-21T10:27:19.332965+0000","last_became_peered":"2024-09-21T10:27:19.332965+0000","last_unstale":"2024-09-21T10:32:12.374173+0000","last_undegraded":"2024-09-21T10:32:12.374173+0000","last_fullsized":"2024-09-21T10:32:12.374173+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:04.828777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":25,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391762+0000","last_change":"2024-09-21T10:30:09.425518+0000","last_active":"2024-09-21T10:32:12.391762+0000","last_peered":"2024-09-21T10:32:12.391762+0000","last_clean":"2024-09-21T10:32:12.391762+0000","last_became_active":"2024-09-21T10:30:09.412102+0000","last_became_peered":"2024-09-21T10:30:09.412102+0000","last_unstale":"2024-09-21T10:32:12.391762+0000","last_undegraded":"2024-09-21T10:32:12.391762+0000","last_fullsized":"2024-09-21T10:32:12.391762+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:44.848838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1234,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374460+0000","last_change":"2024-09-21T10:27:19.370409+0000","last_active":"2024-09-21T10:32:12.374460+0000","last_peered":"2024-09-21T10:32:12.374460+0000","last_clean":"2024-09-21T10:32:12.374460+0000","last_became_active":"2024-09-21T10:27:19.370096+0000","last_became_peered":"2024-09-21T10:27:19.370096+0000","last_unstale":"2024-09-21T10:32:12.374460+0000","last_undegraded":"2024-09-21T10:32:12.374460+0000","last_fullsized":"2024-09-21T10:32:12.374460+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:03.124925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":36,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374124+0000","last_change":"2024-09-21T10:28:56.813482+0000","last_active":"2024-09-21T10:32:12.374124+0000","last_peered":"2024-09-21T10:32:12.374124+0000","last_clean":"2024-09-21T10:32:12.374124+0000","last_became_active":"2024-09-21T10:28:56.813103+0000","last_became_peered":"2024-09-21T10:28:56.813103+0000","last_unstale":"2024-09-21T10:32:12.374124+0000","last_undegraded":"2024-09-21T10:32:12.374124+0000","last_fullsized":"2024-09-21T10:32:12.374124+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:20.438161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1803,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735135+0000","last_change":"2024-09-21T10:29:04.650051+0000","last_active":"2024-09-21T10:32:32.735135+0000","last_peered":"2024-09-21T10:32:32.735135+0000","last_clean":"2024-09-21T10:32:32.735135+0000","last_became_active":"2024-09-21T10:29:04.649705+0000","last_became_peered":"2024-09-21T10:29:04.649705+0000","last_unstale":"2024-09-21T10:32:32.735135+0000","last_undegraded":"2024-09-21T10:32:32.735135+0000","last_fullsized":"2024-09-21T10:32:32.735135+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:45.854900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1209,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374257+0000","last_change":"2024-09-21T10:27:19.325859+0000","last_active":"2024-09-21T10:32:12.374257+0000","last_peered":"2024-09-21T10:32:12.374257+0000","last_clean":"2024-09-21T10:32:12.374257+0000","last_became_active":"2024-09-21T10:27:19.325540+0000","last_became_peered":"2024-09-21T10:27:19.325540+0000","last_unstale":"2024-09-21T10:32:12.374257+0000","last_undegraded":"2024-09-21T10:32:12.374257+0000","last_fullsized":"2024-09-21T10:32:12.374257+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:11.324592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1861,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.727912+0000","last_change":"2024-09-21T10:28:37.333977+0000","last_active":"2024-09-21T10:32:32.727912+0000","last_peered":"2024-09-21T10:32:32.727912+0000","last_clean":"2024-09-21T10:32:32.727912+0000","last_became_active":"2024-09-21T10:28:37.333630+0000","last_became_peered":"2024-09-21T10:28:37.333630+0000","last_unstale":"2024-09-21T10:32:32.727912+0000","last_undegraded":"2024-09-21T10:32:32.727912+0000","last_fullsized":"2024-09-21T10:32:32.727912+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:07.637543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1959,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.734904+0000","last_change":"2024-09-21T10:27:19.331712+0000","last_active":"2024-09-21T10:32:32.734904+0000","last_peered":"2024-09-21T10:32:32.734904+0000","last_clean":"2024-09-21T10:32:32.734904+0000","last_became_active":"2024-09-21T10:27:19.330959+0000","last_became_peered":"2024-09-21T10:27:19.330959+0000","last_unstale":"2024-09-21T10:32:32.734904+0000","last_undegraded":"2024-09-21T10:32:32.734904+0000","last_fullsized":"2024-09-21T10:32:32.734904+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:46.389484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1975,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735572+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:32:32.735572+0000","last_peered":"2024-09-21T10:32:32.735572+0000","last_clean":"2024-09-21T10:32:32.735572+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:32:32.735572+0000","last_undegraded":"2024-09-21T10:32:32.735572+0000","last_fullsized":"2024-09-21T10:32:32.735572+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1886,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.723935+0000","last_change":"2024-09-21T10:28:56.819014+0000","last_active":"2024-09-21T10:32:32.723935+0000","last_peered":"2024-09-21T10:32:32.723935+0000","last_clean":"2024-09-21T10:32:32.723935+0000","last_became_active":"2024-09-21T10:28:56.818640+0000","last_became_peered":"2024-09-21T10:28:56.818640+0000","last_unstale":"2024-09-21T10:32:32.723935+0000","last_undegraded":"2024-09-21T10:32:32.723935+0000","last_fullsized":"2024-09-21T10:32:32.723935+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:49.759102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1229,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.426256+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:32:12.426256+0000","last_peered":"2024-09-21T10:32:12.426256+0000","last_clean":"2024-09-21T10:32:12.426256+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:32:12.426256+0000","last_undegraded":"2024-09-21T10:32:12.426256+0000","last_fullsized":"2024-09-21T10:32:12.426256+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1873,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.724160+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:32:32.724160+0000","last_peered":"2024-09-21T10:32:32.724160+0000","last_clean":"2024-09-21T10:32:32.724160+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:32:32.724160+0000","last_undegraded":"2024-09-21T10:32:32.724160+0000","last_fullsized":"2024-09-21T10:32:32.724160+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1986,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735804+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:32:32.735804+0000","last_peered":"2024-09-21T10:32:32.735804+0000","last_clean":"2024-09-21T10:32:32.735804+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:32:32.735804+0000","last_undegraded":"2024-09-21T10:32:32.735804+0000","last_fullsized":"2024-09-21T10:32:32.735804+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":112,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.726697+0000","last_change":"2024-09-21T10:30:09.407298+0000","last_active":"2024-09-21T10:32:32.726697+0000","last_peered":"2024-09-21T10:32:32.726697+0000","last_clean":"2024-09-21T10:32:32.726697+0000","last_became_active":"2024-09-21T10:30:09.406898+0000","last_became_peered":"2024-09-21T10:30:09.406898+0000","last_unstale":"2024-09-21T10:32:32.726697+0000","last_undegraded":"2024-09-21T10:32:32.726697+0000","last_fullsized":"2024-09-21T10:32:32.726697+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1856,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.726051+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:32:32.726051+0000","last_peered":"2024-09-21T10:32:32.726051+0000","last_clean":"2024-09-21T10:32:32.726051+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:32:32.726051+0000","last_undegraded":"2024-09-21T10:32:32.726051+0000","last_fullsized":"2024-09-21T10:32:32.726051+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391970+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:32:12.391970+0000","last_peered":"2024-09-21T10:32:12.391970+0000","last_clean":"2024-09-21T10:32:12.391970+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:32:12.391970+0000","last_undegraded":"2024-09-21T10:32:12.391970+0000","last_fullsized":"2024-09-21T10:32:12.391970+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1503,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.739055+0000","last_change":"2024-09-21T10:27:19.327772+0000","last_active":"2024-09-21T10:32:32.739055+0000","last_peered":"2024-09-21T10:32:32.739055+0000","last_clean":"2024-09-21T10:32:32.739055+0000","last_became_active":"2024-09-21T10:27:19.327467+0000","last_became_peered":"2024-09-21T10:27:19.327467+0000","last_unstale":"2024-09-21T10:32:32.739055+0000","last_undegraded":"2024-09-21T10:32:32.739055+0000","last_fullsized":"2024-09-21T10:32:32.739055+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:28:38.639278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1959,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736491+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:32:32.736491+0000","last_peered":"2024-09-21T10:32:32.736491+0000","last_clean":"2024-09-21T10:32:32.736491+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:32:32.736491+0000","last_undegraded":"2024-09-21T10:32:32.736491+0000","last_fullsized":"2024-09-21T10:32:32.736491+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1244,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374375+0000","last_change":"2024-09-21T10:27:19.330694+0000","last_active":"2024-09-21T10:32:12.374375+0000","last_peered":"2024-09-21T10:32:12.374375+0000","last_clean":"2024-09-21T10:32:12.374375+0000","last_became_active":"2024-09-21T10:27:19.329455+0000","last_became_peered":"2024-09-21T10:27:19.329455+0000","last_unstale":"2024-09-21T10:32:12.374375+0000","last_undegraded":"2024-09-21T10:32:12.374375+0000","last_fullsized":"2024-09-21T10:32:12.374375+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:27.883673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1238,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374588+0000","last_change":"2024-09-21T10:27:19.371314+0000","last_active":"2024-09-21T10:32:12.374588+0000","last_peered":"2024-09-21T10:32:12.374588+0000","last_clean":"2024-09-21T10:32:12.374588+0000","last_became_active":"2024-09-21T10:27:19.370747+0000","last_became_peered":"2024-09-21T10:27:19.370747+0000","last_unstale":"2024-09-21T10:32:12.374588+0000","last_undegraded":"2024-09-21T10:32:12.374588+0000","last_fullsized":"2024-09-21T10:32:12.374588+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:00.522037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":113,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.726981+0000","last_change":"2024-09-21T10:30:09.413781+0000","last_active":"2024-09-21T10:32:32.726981+0000","last_peered":"2024-09-21T10:32:32.726981+0000","last_clean":"2024-09-21T10:32:32.726981+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:32:32.726981+0000","last_undegraded":"2024-09-21T10:32:32.726981+0000","last_fullsized":"2024-09-21T10:32:32.726981+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1229,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.373923+0000","last_change":"2024-09-21T10:30:09.423426+0000","last_active":"2024-09-21T10:32:12.373923+0000","last_peered":"2024-09-21T10:32:12.373923+0000","last_clean":"2024-09-21T10:32:12.373923+0000","last_became_active":"2024-09-21T10:30:09.422262+0000","last_became_peered":"2024-09-21T10:30:09.422262+0000","last_unstale":"2024-09-21T10:32:12.373923+0000","last_undegraded":"2024-09-21T10:32:12.373923+0000","last_fullsized":"2024-09-21T10:32:12.373923+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:25.701866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1225,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.426148+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:32:12.426148+0000","last_peered":"2024-09-21T10:32:12.426148+0000","last_clean":"2024-09-21T10:32:12.426148+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:32:12.426148+0000","last_undegraded":"2024-09-21T10:32:12.426148+0000","last_fullsized":"2024-09-21T10:32:12.426148+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":94,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735600+0000","last_change":"2024-09-21T10:29:04.342115+0000","last_active":"2024-09-21T10:32:32.735600+0000","last_peered":"2024-09-21T10:32:32.735600+0000","last_clean":"2024-09-21T10:32:32.735600+0000","last_became_active":"2024-09-21T10:29:04.340202+0000","last_became_peered":"2024-09-21T10:29:04.340202+0000","last_unstale":"2024-09-21T10:32:32.735600+0000","last_undegraded":"2024-09-21T10:32:32.735600+0000","last_fullsized":"2024-09-21T10:32:32.735600+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:52.282454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":66,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374661+0000","last_change":"2024-09-21T10:29:56.991958+0000","last_active":"2024-09-21T10:32:12.374661+0000","last_peered":"2024-09-21T10:32:12.374661+0000","last_clean":"2024-09-21T10:32:12.374661+0000","last_became_active":"2024-09-21T10:29:56.991594+0000","last_became_peered":"2024-09-21T10:29:56.991594+0000","last_unstale":"2024-09-21T10:32:12.374661+0000","last_undegraded":"2024-09-21T10:32:12.374661+0000","last_fullsized":"2024-09-21T10:32:12.374661+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.295009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.295009+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014921820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1870,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.724018+0000","last_change":"2024-09-21T10:28:56.817623+0000","last_active":"2024-09-21T10:32:32.724018+0000","last_peered":"2024-09-21T10:32:32.724018+0000","last_clean":"2024-09-21T10:32:32.724018+0000","last_became_active":"2024-09-21T10:28:56.817337+0000","last_became_peered":"2024-09-21T10:28:56.817337+0000","last_unstale":"2024-09-21T10:32:32.724018+0000","last_undegraded":"2024-09-21T10:32:32.724018+0000","last_fullsized":"2024-09-21T10:32:32.724018+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:31.718364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1969,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735831+0000","last_change":"2024-09-21T10:27:19.326897+0000","last_active":"2024-09-21T10:32:32.735831+0000","last_peered":"2024-09-21T10:32:32.735831+0000","last_clean":"2024-09-21T10:32:32.735831+0000","last_became_active":"2024-09-21T10:27:19.324476+0000","last_became_peered":"2024-09-21T10:27:19.324476+0000","last_unstale":"2024-09-21T10:32:32.735831+0000","last_undegraded":"2024-09-21T10:32:32.735831+0000","last_fullsized":"2024-09-21T10:32:32.735831+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:57.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1854,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.725565+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:32:32.725565+0000","last_peered":"2024-09-21T10:32:32.725565+0000","last_clean":"2024-09-21T10:32:32.725565+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:32:32.725565+0000","last_undegraded":"2024-09-21T10:32:32.725565+0000","last_fullsized":"2024-09-21T10:32:32.725565+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":386,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.426075+0000","last_change":"2024-09-21T10:29:57.002238+0000","last_active":"2024-09-21T10:32:12.426075+0000","last_peered":"2024-09-21T10:32:12.426075+0000","last_clean":"2024-09-21T10:32:12.426075+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:32:12.426075+0000","last_undegraded":"2024-09-21T10:32:12.426075+0000","last_fullsized":"2024-09-21T10:32:12.426075+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.749265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.749265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017801379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1977,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.743661+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:32:32.743661+0000","last_peered":"2024-09-21T10:32:32.743661+0000","last_clean":"2024-09-21T10:32:32.743661+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:32:32.743661+0000","last_undegraded":"2024-09-21T10:32:32.743661+0000","last_fullsized":"2024-09-21T10:32:32.743661+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1268,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375448+0000","last_change":"2024-09-21T10:29:04.338405+0000","last_active":"2024-09-21T10:32:12.375448+0000","last_peered":"2024-09-21T10:32:12.375448+0000","last_clean":"2024-09-21T10:32:12.375448+0000","last_became_active":"2024-09-21T10:29:04.338028+0000","last_became_peered":"2024-09-21T10:29:04.338028+0000","last_unstale":"2024-09-21T10:32:12.375448+0000","last_undegraded":"2024-09-21T10:32:12.375448+0000","last_fullsized":"2024-09-21T10:32:12.375448+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:52.709092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1963,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736119+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:32:32.736119+0000","last_peered":"2024-09-21T10:32:32.736119+0000","last_clean":"2024-09-21T10:32:32.736119+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:32:32.736119+0000","last_undegraded":"2024-09-21T10:32:32.736119+0000","last_fullsized":"2024-09-21T10:32:32.736119+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":386,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.426076+0000","last_change":"2024-09-21T10:29:57.002608+0000","last_active":"2024-09-21T10:32:12.426076+0000","last_peered":"2024-09-21T10:32:12.426076+0000","last_clean":"2024-09-21T10:32:12.426076+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:32:12.426076+0000","last_undegraded":"2024-09-21T10:32:12.426076+0000","last_fullsized":"2024-09-21T10:32:12.426076+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:29.590974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:29.590974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00139189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1961,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736628+0000","last_change":"2024-09-21T10:30:09.408634+0000","last_active":"2024-09-21T10:32:32.736628+0000","last_peered":"2024-09-21T10:32:32.736628+0000","last_clean":"2024-09-21T10:32:32.736628+0000","last_became_active":"2024-09-21T10:30:09.408316+0000","last_became_peered":"2024-09-21T10:30:09.408316+0000","last_unstale":"2024-09-21T10:32:32.736628+0000","last_undegraded":"2024-09-21T10:32:32.736628+0000","last_fullsized":"2024-09-21T10:32:32.736628+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:36.075653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1963,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736286+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:32:32.736286+0000","last_peered":"2024-09-21T10:32:32.736286+0000","last_clean":"2024-09-21T10:32:32.736286+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:32:32.736286+0000","last_undegraded":"2024-09-21T10:32:32.736286+0000","last_fullsized":"2024-09-21T10:32:32.736286+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391736+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:32:12.391736+0000","last_peered":"2024-09-21T10:32:12.391736+0000","last_clean":"2024-09-21T10:32:12.391736+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:32:12.391736+0000","last_undegraded":"2024-09-21T10:32:12.391736+0000","last_fullsized":"2024-09-21T10:32:12.391736+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391455+0000","last_change":"2024-09-21T10:30:09.402364+0000","last_active":"2024-09-21T10:32:12.391455+0000","last_peered":"2024-09-21T10:32:12.391455+0000","last_clean":"2024-09-21T10:32:12.391455+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:32:12.391455+0000","last_undegraded":"2024-09-21T10:32:12.391455+0000","last_fullsized":"2024-09-21T10:32:12.391455+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1960,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736207+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:32:32.736207+0000","last_peered":"2024-09-21T10:32:32.736207+0000","last_clean":"2024-09-21T10:32:32.736207+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:32:32.736207+0000","last_undegraded":"2024-09-21T10:32:32.736207+0000","last_fullsized":"2024-09-21T10:32:32.736207+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1239,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374373+0000","last_change":"2024-09-21T10:27:19.333723+0000","last_active":"2024-09-21T10:32:12.374373+0000","last_peered":"2024-09-21T10:32:12.374373+0000","last_clean":"2024-09-21T10:32:12.374373+0000","last_became_active":"2024-09-21T10:27:19.333492+0000","last_became_peered":"2024-09-21T10:27:19.333492+0000","last_unstale":"2024-09-21T10:32:12.374373+0000","last_undegraded":"2024-09-21T10:32:12.374373+0000","last_fullsized":"2024-09-21T10:32:12.374373+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:03.860673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":25,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391732+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:32:12.391732+0000","last_peered":"2024-09-21T10:32:12.391732+0000","last_clean":"2024-09-21T10:32:12.391732+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:32:12.391732+0000","last_undegraded":"2024-09-21T10:32:12.391732+0000","last_fullsized":"2024-09-21T10:32:12.391732+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":1417,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:43.518930+0000","last_change":"2024-09-21T10:32:43.518930+0000","last_active":"2024-09-21T10:32:43.518930+0000","last_peered":"2024-09-21T10:32:43.518930+0000","last_clean":"2024-09-21T10:32:43.518930+0000","last_became_active":"2024-09-21T10:29:57.002088+0000","last_became_peered":"2024-09-21T10:29:57.002088+0000","last_unstale":"2024-09-21T10:32:43.518930+0000","last_undegraded":"2024-09-21T10:32:43.518930+0000","last_fullsized":"2024-09-21T10:32:43.518930+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:26.208290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:26.208290+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015511240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1972,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.738601+0000","last_change":"2024-09-21T10:30:09.415060+0000","last_active":"2024-09-21T10:32:32.738601+0000","last_peered":"2024-09-21T10:32:32.738601+0000","last_clean":"2024-09-21T10:32:32.738601+0000","last_became_active":"2024-09-21T10:30:09.414412+0000","last_became_peered":"2024-09-21T10:30:09.414412+0000","last_unstale":"2024-09-21T10:32:32.738601+0000","last_undegraded":"2024-09-21T10:32:32.738601+0000","last_fullsized":"2024-09-21T10:32:32.738601+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:25:36.187303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1961,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736681+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:32:32.736681+0000","last_peered":"2024-09-21T10:32:32.736681+0000","last_clean":"2024-09-21T10:32:32.736681+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:32:32.736681+0000","last_undegraded":"2024-09-21T10:32:32.736681+0000","last_fullsized":"2024-09-21T10:32:32.736681+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1891,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.726329+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:32:32.726329+0000","last_peered":"2024-09-21T10:32:32.726329+0000","last_clean":"2024-09-21T10:32:32.726329+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:32:32.726329+0000","last_undegraded":"2024-09-21T10:32:32.726329+0000","last_fullsized":"2024-09-21T10:32:32.726329+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":116,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.737014+0000","last_change":"2024-09-21T10:30:09.415904+0000","last_active":"2024-09-21T10:32:32.737014+0000","last_peered":"2024-09-21T10:32:32.737014+0000","last_clean":"2024-09-21T10:32:32.737014+0000","last_became_active":"2024-09-21T10:30:09.415524+0000","last_became_peered":"2024-09-21T10:30:09.415524+0000","last_unstale":"2024-09-21T10:32:32.737014+0000","last_undegraded":"2024-09-21T10:32:32.737014+0000","last_fullsized":"2024-09-21T10:32:32.737014+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":743,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735931+0000","last_change":"2024-09-21T10:28:56.817541+0000","last_active":"2024-09-21T10:32:32.735931+0000","last_peered":"2024-09-21T10:32:32.735931+0000","last_clean":"2024-09-21T10:32:32.735931+0000","last_became_active":"2024-09-21T10:28:56.817221+0000","last_became_peered":"2024-09-21T10:28:56.817221+0000","last_unstale":"2024-09-21T10:32:32.735931+0000","last_undegraded":"2024-09-21T10:32:32.735931+0000","last_fullsized":"2024-09-21T10:32:32.735931+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:51.568259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1985,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735002+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:32:32.735002+0000","last_peered":"2024-09-21T10:32:32.735002+0000","last_clean":"2024-09-21T10:32:32.735002+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:32:32.735002+0000","last_undegraded":"2024-09-21T10:32:32.735002+0000","last_fullsized":"2024-09-21T10:32:32.735002+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1258,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375141+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:32:12.375141+0000","last_peered":"2024-09-21T10:32:12.375141+0000","last_clean":"2024-09-21T10:32:12.375141+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:32:12.375141+0000","last_undegraded":"2024-09-21T10:32:12.375141+0000","last_fullsized":"2024-09-21T10:32:12.375141+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1221,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.425808+0000","last_change":"2024-09-21T10:27:19.327425+0000","last_active":"2024-09-21T10:32:12.425808+0000","last_peered":"2024-09-21T10:32:12.425808+0000","last_clean":"2024-09-21T10:32:12.425808+0000","last_became_active":"2024-09-21T10:27:19.326641+0000","last_became_peered":"2024-09-21T10:27:19.326641+0000","last_unstale":"2024-09-21T10:32:12.425808+0000","last_undegraded":"2024-09-21T10:32:12.425808+0000","last_fullsized":"2024-09-21T10:32:12.425808+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:31:37.389673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1877,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.725876+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:32:32.725876+0000","last_peered":"2024-09-21T10:32:32.725876+0000","last_clean":"2024-09-21T10:32:32.725876+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:32:32.725876+0000","last_undegraded":"2024-09-21T10:32:32.725876+0000","last_fullsized":"2024-09-21T10:32:32.725876+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":25,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.392167+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:32:12.392167+0000","last_peered":"2024-09-21T10:32:12.392167+0000","last_clean":"2024-09-21T10:32:12.392167+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:32:12.392167+0000","last_undegraded":"2024-09-21T10:32:12.392167+0000","last_fullsized":"2024-09-21T10:32:12.392167+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1421,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:36.732720+0000","last_change":"2024-09-21T10:32:36.732720+0000","last_active":"2024-09-21T10:32:36.732720+0000","last_peered":"2024-09-21T10:32:36.732720+0000","last_clean":"2024-09-21T10:32:36.732720+0000","last_became_active":"2024-09-21T10:29:57.005035+0000","last_became_peered":"2024-09-21T10:29:57.005035+0000","last_unstale":"2024-09-21T10:32:36.732720+0000","last_undegraded":"2024-09-21T10:32:36.732720+0000","last_fullsized":"2024-09-21T10:32:36.732720+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.166696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.166696+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013075529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391826+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:32:12.391826+0000","last_peered":"2024-09-21T10:32:12.391826+0000","last_clean":"2024-09-21T10:32:12.391826+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:32:12.391826+0000","last_undegraded":"2024-09-21T10:32:12.391826+0000","last_fullsized":"2024-09-21T10:32:12.391826+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1959,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.738510+0000","last_change":"2024-09-21T10:27:19.327581+0000","last_active":"2024-09-21T10:32:32.738510+0000","last_peered":"2024-09-21T10:32:32.738510+0000","last_clean":"2024-09-21T10:32:32.738510+0000","last_became_active":"2024-09-21T10:27:19.327252+0000","last_became_peered":"2024-09-21T10:27:19.327252+0000","last_unstale":"2024-09-21T10:32:32.738510+0000","last_undegraded":"2024-09-21T10:32:32.738510+0000","last_fullsized":"2024-09-21T10:32:32.738510+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:46.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1863,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.724088+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:32:32.724088+0000","last_peered":"2024-09-21T10:32:32.724088+0000","last_clean":"2024-09-21T10:32:32.724088+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:32:32.724088+0000","last_undegraded":"2024-09-21T10:32:32.724088+0000","last_fullsized":"2024-09-21T10:32:32.724088+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":115,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736906+0000","last_change":"2024-09-21T10:30:09.415536+0000","last_active":"2024-09-21T10:32:32.736906+0000","last_peered":"2024-09-21T10:32:32.736906+0000","last_clean":"2024-09-21T10:32:32.736906+0000","last_became_active":"2024-09-21T10:30:09.414550+0000","last_became_peered":"2024-09-21T10:30:09.414550+0000","last_unstale":"2024-09-21T10:32:32.736906+0000","last_undegraded":"2024-09-21T10:32:32.736906+0000","last_fullsized":"2024-09-21T10:32:32.736906+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1238,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374536+0000","last_change":"2024-09-21T10:27:19.329566+0000","last_active":"2024-09-21T10:32:12.374536+0000","last_peered":"2024-09-21T10:32:12.374536+0000","last_clean":"2024-09-21T10:32:12.374536+0000","last_became_active":"2024-09-21T10:27:19.329307+0000","last_became_peered":"2024-09-21T10:27:19.329307+0000","last_unstale":"2024-09-21T10:32:12.374536+0000","last_undegraded":"2024-09-21T10:32:12.374536+0000","last_fullsized":"2024-09-21T10:32:12.374536+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:59.305906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1191,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.373971+0000","last_change":"2024-09-21T10:27:19.326055+0000","last_active":"2024-09-21T10:32:12.373971+0000","last_peered":"2024-09-21T10:32:12.373971+0000","last_clean":"2024-09-21T10:32:12.373971+0000","last_became_active":"2024-09-21T10:27:19.324778+0000","last_became_peered":"2024-09-21T10:27:19.324778+0000","last_unstale":"2024-09-21T10:32:12.373971+0000","last_undegraded":"2024-09-21T10:32:12.373971+0000","last_fullsized":"2024-09-21T10:32:12.373971+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:12.572967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":25,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391496+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:32:12.391496+0000","last_peered":"2024-09-21T10:32:12.391496+0000","last_clean":"2024-09-21T10:32:12.391496+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:32:12.391496+0000","last_undegraded":"2024-09-21T10:32:12.391496+0000","last_fullsized":"2024-09-21T10:32:12.391496+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1198,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374493+0000","last_change":"2024-09-21T10:27:20.323254+0000","last_active":"2024-09-21T10:32:12.374493+0000","last_peered":"2024-09-21T10:32:12.374493+0000","last_clean":"2024-09-21T10:32:12.374493+0000","last_became_active":"2024-09-21T10:27:20.322751+0000","last_became_peered":"2024-09-21T10:27:20.322751+0000","last_unstale":"2024-09-21T10:32:12.374493+0000","last_undegraded":"2024-09-21T10:32:12.374493+0000","last_fullsized":"2024-09-21T10:32:12.374493+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.909953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1227,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.425933+0000","last_change":"2024-09-21T10:27:19.326938+0000","last_active":"2024-09-21T10:32:12.425933+0000","last_peered":"2024-09-21T10:32:12.425933+0000","last_clean":"2024-09-21T10:32:12.425933+0000","last_became_active":"2024-09-21T10:27:19.326640+0000","last_became_peered":"2024-09-21T10:27:19.326640+0000","last_unstale":"2024-09-21T10:32:12.425933+0000","last_undegraded":"2024-09-21T10:32:12.425933+0000","last_fullsized":"2024-09-21T10:32:12.425933+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:24.413432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1392,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374905+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:32:12.374905+0000","last_peered":"2024-09-21T10:32:12.374905+0000","last_clean":"2024-09-21T10:32:12.374905+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:32:12.374905+0000","last_undegraded":"2024-09-21T10:32:12.374905+0000","last_fullsized":"2024-09-21T10:32:12.374905+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1238,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374286+0000","last_change":"2024-09-21T10:27:19.332150+0000","last_active":"2024-09-21T10:32:12.374286+0000","last_peered":"2024-09-21T10:32:12.374286+0000","last_clean":"2024-09-21T10:32:12.374286+0000","last_became_active":"2024-09-21T10:27:19.331901+0000","last_became_peered":"2024-09-21T10:27:19.331901+0000","last_unstale":"2024-09-21T10:32:12.374286+0000","last_undegraded":"2024-09-21T10:32:12.374286+0000","last_fullsized":"2024-09-21T10:32:12.374286+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:07.758296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":69,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.725259+0000","last_change":"2024-09-21T10:28:56.834029+0000","last_active":"2024-09-21T10:32:32.725259+0000","last_peered":"2024-09-21T10:32:32.725259+0000","last_clean":"2024-09-21T10:32:32.725259+0000","last_became_active":"2024-09-21T10:28:56.817594+0000","last_became_peered":"2024-09-21T10:28:56.817594+0000","last_unstale":"2024-09-21T10:32:32.725259+0000","last_undegraded":"2024-09-21T10:32:32.725259+0000","last_fullsized":"2024-09-21T10:32:32.725259+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:44.035052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":385,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.425753+0000","last_change":"2024-09-21T10:29:57.010696+0000","last_active":"2024-09-21T10:32:12.425753+0000","last_peered":"2024-09-21T10:32:12.425753+0000","last_clean":"2024-09-21T10:32:12.425753+0000","last_became_active":"2024-09-21T10:29:57.010333+0000","last_became_peered":"2024-09-21T10:29:57.010333+0000","last_unstale":"2024-09-21T10:32:12.425753+0000","last_undegraded":"2024-09-21T10:32:12.425753+0000","last_fullsized":"2024-09-21T10:32:12.425753+0000","mapping_epoch":1257,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.749265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.749265+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017801379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1984,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.743521+0000","last_change":"2024-09-21T10:27:19.370824+0000","last_active":"2024-09-21T10:32:32.743521+0000","last_peered":"2024-09-21T10:32:32.743521+0000","last_clean":"2024-09-21T10:32:32.743521+0000","last_became_active":"2024-09-21T10:27:19.370477+0000","last_became_peered":"2024-09-21T10:27:19.370477+0000","last_unstale":"2024-09-21T10:32:32.743521+0000","last_undegraded":"2024-09-21T10:32:32.743521+0000","last_fullsized":"2024-09-21T10:32:32.743521+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:17.020888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391721+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:32:12.391721+0000","last_peered":"2024-09-21T10:32:12.391721+0000","last_clean":"2024-09-21T10:32:12.391721+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:32:12.391721+0000","last_undegraded":"2024-09-21T10:32:12.391721+0000","last_fullsized":"2024-09-21T10:32:12.391721+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":385,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.426199+0000","last_change":"2024-09-21T10:29:57.015850+0000","last_active":"2024-09-21T10:32:12.426199+0000","last_peered":"2024-09-21T10:32:12.426199+0000","last_clean":"2024-09-21T10:32:12.426199+0000","last_became_active":"2024-09-21T10:29:57.015464+0000","last_became_peered":"2024-09-21T10:29:57.015464+0000","last_unstale":"2024-09-21T10:32:12.426199+0000","last_undegraded":"2024-09-21T10:32:12.426199+0000","last_fullsized":"2024-09-21T10:32:12.426199+0000","mapping_epoch":1257,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:29.590974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:29.590974+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00139189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1209,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.425778+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:32:12.425778+0000","last_peered":"2024-09-21T10:32:12.425778+0000","last_clean":"2024-09-21T10:32:12.425778+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:32:12.425778+0000","last_undegraded":"2024-09-21T10:32:12.425778+0000","last_fullsized":"2024-09-21T10:32:12.425778+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1258,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375365+0000","last_change":"2024-09-21T10:30:09.415150+0000","last_active":"2024-09-21T10:32:12.375365+0000","last_peered":"2024-09-21T10:32:12.375365+0000","last_clean":"2024-09-21T10:32:12.375365+0000","last_became_active":"2024-09-21T10:30:09.402385+0000","last_became_peered":"2024-09-21T10:30:09.402385+0000","last_unstale":"2024-09-21T10:32:12.375365+0000","last_undegraded":"2024-09-21T10:32:12.375365+0000","last_fullsized":"2024-09-21T10:32:12.375365+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:53.975042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":117,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.726793+0000","last_change":"2024-09-21T10:30:09.432860+0000","last_active":"2024-09-21T10:32:32.726793+0000","last_peered":"2024-09-21T10:32:32.726793+0000","last_clean":"2024-09-21T10:32:32.726793+0000","last_became_active":"2024-09-21T10:30:09.406860+0000","last_became_peered":"2024-09-21T10:30:09.406860+0000","last_unstale":"2024-09-21T10:32:32.726793+0000","last_undegraded":"2024-09-21T10:32:32.726793+0000","last_fullsized":"2024-09-21T10:32:32.726793+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1234,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374041+0000","last_change":"2024-09-21T10:27:19.327246+0000","last_active":"2024-09-21T10:32:12.374041+0000","last_peered":"2024-09-21T10:32:12.374041+0000","last_clean":"2024-09-21T10:32:12.374041+0000","last_became_active":"2024-09-21T10:27:19.326343+0000","last_became_peered":"2024-09-21T10:27:19.326343+0000","last_unstale":"2024-09-21T10:32:12.374041+0000","last_undegraded":"2024-09-21T10:32:12.374041+0000","last_fullsized":"2024-09-21T10:32:12.374041+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:37.654823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1248,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375565+0000","last_change":"2024-09-21T10:27:19.324762+0000","last_active":"2024-09-21T10:32:12.375565+0000","last_peered":"2024-09-21T10:32:12.375565+0000","last_clean":"2024-09-21T10:32:12.375565+0000","last_became_active":"2024-09-21T10:27:19.324341+0000","last_became_peered":"2024-09-21T10:27:19.324341+0000","last_unstale":"2024-09-21T10:32:12.375565+0000","last_undegraded":"2024-09-21T10:32:12.375565+0000","last_fullsized":"2024-09-21T10:32:12.375565+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:42.722645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":65,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.374331+0000","last_change":"2024-09-21T10:29:57.001141+0000","last_active":"2024-09-21T10:32:12.374331+0000","last_peered":"2024-09-21T10:32:12.374331+0000","last_clean":"2024-09-21T10:32:12.374331+0000","last_became_active":"2024-09-21T10:29:57.000888+0000","last_became_peered":"2024-09-21T10:29:57.000888+0000","last_unstale":"2024-09-21T10:32:12.374331+0000","last_undegraded":"2024-09-21T10:32:12.374331+0000","last_fullsized":"2024-09-21T10:32:12.374331+0000","mapping_epoch":1257,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.295009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.295009+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014921820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391481+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:32:12.391481+0000","last_peered":"2024-09-21T10:32:12.391481+0000","last_clean":"2024-09-21T10:32:12.391481+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:32:12.391481+0000","last_undegraded":"2024-09-21T10:32:12.391481+0000","last_fullsized":"2024-09-21T10:32:12.391481+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1261,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375437+0000","last_change":"2024-09-21T10:28:56.826222+0000","last_active":"2024-09-21T10:32:12.375437+0000","last_peered":"2024-09-21T10:32:12.375437+0000","last_clean":"2024-09-21T10:32:12.375437+0000","last_became_active":"2024-09-21T10:28:56.813187+0000","last_became_peered":"2024-09-21T10:28:56.813187+0000","last_unstale":"2024-09-21T10:32:12.375437+0000","last_undegraded":"2024-09-21T10:32:12.375437+0000","last_fullsized":"2024-09-21T10:32:12.375437+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:28.841806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":1416,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:47.531496+0000","last_change":"2024-09-21T10:32:47.531496+0000","last_active":"2024-09-21T10:32:47.531496+0000","last_peered":"2024-09-21T10:32:47.531496+0000","last_clean":"2024-09-21T10:32:47.531496+0000","last_became_active":"2024-09-21T10:29:57.003088+0000","last_became_peered":"2024-09-21T10:29:57.003088+0000","last_unstale":"2024-09-21T10:32:47.531496+0000","last_undegraded":"2024-09-21T10:32:47.531496+0000","last_fullsized":"2024-09-21T10:32:47.531496+0000","mapping_epoch":1257,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:26.208290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:26.208290+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015511240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1286,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.425833+0000","last_change":"2024-09-21T10:30:09.424136+0000","last_active":"2024-09-21T10:32:12.425833+0000","last_peered":"2024-09-21T10:32:12.425833+0000","last_clean":"2024-09-21T10:32:12.425833+0000","last_became_active":"2024-09-21T10:30:09.414027+0000","last_became_peered":"2024-09-21T10:30:09.414027+0000","last_unstale":"2024-09-21T10:32:12.425833+0000","last_undegraded":"2024-09-21T10:32:12.425833+0000","last_fullsized":"2024-09-21T10:32:12.425833+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:28.506487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1853,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.725383+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:32:32.725383+0000","last_peered":"2024-09-21T10:32:32.725383+0000","last_clean":"2024-09-21T10:32:32.725383+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:32:32.725383+0000","last_undegraded":"2024-09-21T10:32:32.725383+0000","last_fullsized":"2024-09-21T10:32:32.725383+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":115,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.736885+0000","last_change":"2024-09-21T10:30:09.415905+0000","last_active":"2024-09-21T10:32:32.736885+0000","last_peered":"2024-09-21T10:32:32.736885+0000","last_clean":"2024-09-21T10:32:32.736885+0000","last_became_active":"2024-09-21T10:30:09.414616+0000","last_became_peered":"2024-09-21T10:30:09.414616+0000","last_unstale":"2024-09-21T10:32:32.736885+0000","last_undegraded":"2024-09-21T10:32:32.736885+0000","last_fullsized":"2024-09-21T10:32:32.736885+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1256,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375012+0000","last_change":"2024-09-21T10:26:04.187372+0000","last_active":"2024-09-21T10:32:12.375012+0000","last_peered":"2024-09-21T10:32:12.375012+0000","last_clean":"2024-09-21T10:32:12.375012+0000","last_became_active":"2024-09-21T10:26:04.186026+0000","last_became_peered":"2024-09-21T10:26:04.186026+0000","last_unstale":"2024-09-21T10:32:12.375012+0000","last_undegraded":"2024-09-21T10:32:12.375012+0000","last_fullsized":"2024-09-21T10:32:12.375012+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:55.130410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1973,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735525+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:32:32.735525+0000","last_peered":"2024-09-21T10:32:32.735525+0000","last_clean":"2024-09-21T10:32:32.735525+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:32:32.735525+0000","last_undegraded":"2024-09-21T10:32:32.735525+0000","last_fullsized":"2024-09-21T10:32:32.735525+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":1414,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:33.705596+0000","last_change":"2024-09-21T10:32:33.705596+0000","last_active":"2024-09-21T10:32:33.705596+0000","last_peered":"2024-09-21T10:32:33.705596+0000","last_clean":"2024-09-21T10:32:33.705596+0000","last_became_active":"2024-09-21T10:29:57.015843+0000","last_became_peered":"2024-09-21T10:29:57.015843+0000","last_unstale":"2024-09-21T10:32:33.705596+0000","last_undegraded":"2024-09-21T10:32:33.705596+0000","last_fullsized":"2024-09-21T10:32:33.705596+0000","mapping_epoch":1257,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.166696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.166696+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013075529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1972,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.735191+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:32:32.735191+0000","last_peered":"2024-09-21T10:32:32.735191+0000","last_clean":"2024-09-21T10:32:32.735191+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:32:32.735191+0000","last_undegraded":"2024-09-21T10:32:32.735191+0000","last_fullsized":"2024-09-21T10:32:32.735191+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391803+0000","last_change":"2024-09-21T10:30:09.405230+0000","last_active":"2024-09-21T10:32:12.391803+0000","last_peered":"2024-09-21T10:32:12.391803+0000","last_clean":"2024-09-21T10:32:12.391803+0000","last_became_active":"2024-09-21T10:30:09.404929+0000","last_became_peered":"2024-09-21T10:30:09.404929+0000","last_unstale":"2024-09-21T10:32:12.391803+0000","last_undegraded":"2024-09-21T10:32:12.391803+0000","last_fullsized":"2024-09-21T10:32:12.391803+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:01.454535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":21,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.391544+0000","last_change":"2024-09-21T10:30:09.403909+0000","last_active":"2024-09-21T10:32:12.391544+0000","last_peered":"2024-09-21T10:32:12.391544+0000","last_clean":"2024-09-21T10:32:12.391544+0000","last_became_active":"2024-09-21T10:30:09.402657+0000","last_became_peered":"2024-09-21T10:30:09.402657+0000","last_unstale":"2024-09-21T10:32:12.391544+0000","last_undegraded":"2024-09-21T10:32:12.391544+0000","last_fullsized":"2024-09-21T10:32:12.391544+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T09:53:13.106674+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1261,"reported_epoch":1270,"state":"active+clean","last_fresh":"2024-09-21T10:32:12.375647+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:32:12.375647+0000","last_peered":"2024-09-21T10:32:12.375647+0000","last_clean":"2024-09-21T10:32:12.375647+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:32:12.375647+0000","last_undegraded":"2024-09-21T10:32:12.375647+0000","last_fullsized":"2024-09-21T10:32:12.375647+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1862,"reported_epoch":1271,"state":"active+clean","last_fresh":"2024-09-21T10:32:32.727061+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:32:32.727061+0000","last_peered":"2024-09-21T10:32:32.727061+0000","last_clean":"2024-09-21T10:32:32.727061+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:32:32.727061+0000","last_undegraded":"2024-09-21T10:32:32.727061+0000","last_fullsized":"2024-09-21T10:32:32.727061+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":46,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7866,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210198,"num_write_kb":6726273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16264,"num_bytes_recovered":896925696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1440874496,"data_stored":1440874496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115010,"ondisk_log_size":115010,"up":48,"acting":46,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":644038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":68,"num_write":14,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":644038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1426,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":16660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397959,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":319832,"kb_used_data":14160,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93425320,"statfs":{"total":95995035648,"available":95667527680,"internally_reserved":0,"allocated":14499840,"data_stored":9312576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049618,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":627416,"kb_used_data":283088,"kb_used_omap":242,"kb_used_meta":344077,"kb_avail":93117736,"statfs":{"total":95995035648,"available":95352561664,"internally_reserved":0,"allocated":289882112,"data_stored":284695389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":352335071},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790417,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324240,"kb_used_data":14408,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93420912,"statfs":{"total":95995035648,"available":95663013888,"internally_reserved":0,"allocated":14753792,"data_stored":9571053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395523,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":739416,"kb_used_data":428364,"kb_used_omap":224,"kb_used_meta":310815,"kb_avail":93005736,"statfs":{"total":95995035648,"available":95237873664,"internally_reserved":0,"allocated":438644736,"data_stored":433461554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":318275465},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644576,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":818576,"kb_used_data":433796,"kb_used_omap":67,"kb_used_meta":384700,"kb_avail":92926576,"statfs":{"total":95995035648,"available":95156813824,"internally_reserved":0,"allocated":444207104,"data_stored":439027982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":393933661},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527594,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430796,"kb_used_data":14084,"kb_used_omap":75,"kb_used_meta":416628,"kb_avail":93314356,"statfs":{"total":95995035648,"available":95553900544,"internally_reserved":0,"allocated":14422016,"data_stored":9243362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":426627226},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274747,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":373712,"kb_used_data":13832,"kb_used_omap":51,"kb_used_meta":359820,"kb_avail":93371440,"statfs":{"total":95995035648,"available":95612354560,"internally_reserved":0,"allocated":14163968,"data_stored":8980219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":368456021},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353986,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":623296,"kb_used_data":317752,"kb_used_omap":66,"kb_used_meta":305469,"kb_avail":93121856,"statfs":{"total":95995035648,"available":95356780544,"internally_reserved":0,"allocated":325378048,"data_stored":320210526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":312800580},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":311230464,"data_stored":311230464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":429457408,"data_stored":429457408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":424476672,"data_stored":424476672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":275709952,"data_stored":275709952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":46,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:32:52.099 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:32:52.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_1 pg_num 26 2024-09-21T10:32:52.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:52.214 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:52.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:52.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:52.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:52.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:52.460 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:52.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:52.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:52.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:52.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:52.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:52.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:52.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:52.834 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:32:52.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:52.951 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:52 smithi119 ceph-mon[105288]: pgmap v1960: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 7866/81 objects degraded (9711.111%); 0 B/s, 157 objects/s recovering 2024-09-21T10:32:52.951 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2746240794' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:32:52.951 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:52 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:32:52.951 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:52 smithi119 ceph-mon[101432]: pgmap v1960: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 7866/81 objects degraded (9711.111%); 0 B/s, 157 objects/s recovering 2024-09-21T10:32:52.951 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2746240794' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:32:52.951 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:52 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:32:52.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:52.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:52.974 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:32:53.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.078 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:53.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:53.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:53.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:53.256 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:52 smithi145 ceph-mon[89359]: pgmap v1960: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 0 B/s wr, 1 op/s; 7866/81 objects degraded (9711.111%); 0 B/s, 157 objects/s recovering 2024-09-21T10:32:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2746240794' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:32:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:52 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:32:53.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:53.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:53.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:53.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:53.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:53.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:53.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:53.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:53.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:53.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:53.822 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:53.894 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:53.907 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:32:53.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:53.946 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:53.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:53.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:54.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[105288]: 1.10 deep-scrub starts 2024-09-21T10:32:54.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:32:54.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:32:54.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[105288]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-21T10:32:54.019 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[101432]: 1.10 deep-scrub starts 2024-09-21T10:32:54.019 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:32:54.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:32:54.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:53 smithi119 ceph-mon[101432]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-21T10:32:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:53 smithi145 ceph-mon[89359]: 1.10 deep-scrub starts 2024-09-21T10:32:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:53 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:32:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:53 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:32:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:53 smithi145 ceph-mon[89359]: osdmap e1272: 8 total, 8 up, 8 in 2024-09-21T10:32:54.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:54.069 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:54.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:54.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:54.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:54.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:54.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:54.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:54.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:54.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:54.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:32:54] "GET /metrics HTTP/1.1" 200 36902 "" "Prometheus/2.51.0" 2024-09-21T10:32:54.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:54.688 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:54.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:54.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:54.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:54.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:54.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:54.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:54.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:54.948 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:54 smithi145 ceph-mon[89359]: pgmap v1962: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7866/81 objects degraded (9711.111%); 0 B/s, 189 objects/s recovering 2024-09-21T10:32:54.948 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:54 smithi145 ceph-mon[89359]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-21T10:32:54.948 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:54 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 7866/81 objects degraded (9711.111%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:55.038 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:32:55.060 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:32:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-mon[101432]: pgmap v1962: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7866/81 objects degraded (9711.111%); 0 B/s, 189 objects/s recovering 2024-09-21T10:32:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-mon[101432]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-21T10:32:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 7866/81 objects degraded (9711.111%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-mon[105288]: pgmap v1962: 109 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 7866/81 objects degraded (9711.111%); 0 B/s, 189 objects/s recovering 2024-09-21T10:32:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-mon[105288]: osdmap e1273: 8 total, 8 up, 8 in 2024-09-21T10:32:55.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:54 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 7866/81 objects degraded (9711.111%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:32:55.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:55.174 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:32:55.175 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:32:55.175 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:32:55.175 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:32:55.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:55.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:55.309 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:55.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:55.433 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:55.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:55.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:55.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.680 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:55.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:55.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:55.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:32:55.803 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:55.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:32:55.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:55.935 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:55.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:56.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:56.040 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:55 smithi119 ceph-mon[101432]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-21T10:32:56.040 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:55 smithi119 ceph-mon[105288]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-21T10:32:56.059 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:56.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:56.117 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:56.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:56.183 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:56.238 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:55 smithi145 ceph-mon[89359]: osdmap e1274: 8 total, 8 up, 8 in 2024-09-21T10:32:56.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:56.306 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:56.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:56.430 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:56.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:56.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:56.655 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:56.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:56.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:56.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:56.925 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:56.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:57.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:57.048 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:57.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:56 smithi119 ceph-mon[105288]: pgmap v1965: 109 pgs: 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 248 objects/s recovering 2024-09-21T10:32:57.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:56 smithi119 ceph-mon[101432]: pgmap v1965: 109 pgs: 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 248 objects/s recovering 2024-09-21T10:32:57.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:32:57.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:32:57.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:56 smithi145 ceph-mon[89359]: pgmap v1965: 109 pgs: 1 active+recovery_wait, 1 active+undersized+remapped, 2 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 248 objects/s recovering 2024-09-21T10:32:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:57.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:57.419 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:57.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:57.646 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:57.667 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:57.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:57.789 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:57.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:57.913 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:57.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:57.951 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:32:57.951 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:32:57.951 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:32:57.951 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:32:57.951 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:32:58.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:58.035 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:58.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:58.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:58.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:58.260 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:58.282 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:58.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:58.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:58.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:58.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:58.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:58.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:32:58.631 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:58.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:58.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:58.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:58.877 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:32:58.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:32:58.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:58.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:32:59.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:32:59.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:32:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:58 smithi119 ceph-mon[101432]: pgmap v1966: 109 pgs: 1 active+recovery_wait, 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 248 objects/s recovering 2024-09-21T10:32:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:58 smithi119 ceph-mon[105288]: pgmap v1966: 109 pgs: 1 active+recovery_wait, 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 248 objects/s recovering 2024-09-21T10:32:59.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:59.170 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:59.183 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:58 smithi145 ceph-mon[89359]: pgmap v1966: 109 pgs: 1 active+recovery_wait, 1 active+undersized+remapped, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 99 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 248 objects/s recovering 2024-09-21T10:32:59.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:59.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:59.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:32:59.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:32:59.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:32:59.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:32:59.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:59.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:32:59.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:32:59.787 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:32:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:32:59.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:32:59.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:32:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:32:59.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:00.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:00.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:00.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:32:59 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 6373/81 objects degraded (7867.901%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:33:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:32:59 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 6373/81 objects degraded (7867.901%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:33:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:32:59 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 6373/81 objects degraded (7867.901%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:33:00.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:00.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:00.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:00.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:00.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:00.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:00.445 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:33:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:33:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:00.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:00.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:00.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:00.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:00.651 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:206271 45.1 45:82987e0b:::benchmark_data_smithi119_168706_object76101:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:40.841044+0000", 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 379.83639823300001, 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.050415605, 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.684 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 206271 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841044+0000", 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841044+0000", 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841042+0000", 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841048+0000", 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1990000000000003e-06 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841064+0000", 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6398000000000001e-05 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841071+0000", 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1360000000000002e-06 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841131+0000", 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0254000000000003e-05 2024-09-21T10:33:00.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841138+0000", 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6650000000000003e-06 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.872528+0000", 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0313896119999999 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.876002+0000", 2024-09-21T10:33:00.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003474361 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0", 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.876098+0000", 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5988000000000003e-05 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.887217+0000", 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011118846999999999 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:00.689 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.891320+0000", 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041023659999999997 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.891370+0000", 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.99e-05 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.891459+0000", 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9635000000000005e-05 2024-09-21T10:33:00.690 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:206339 45.1 45:8cf2172c:::benchmark_data_smithi119_168706_object76169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:40.927717+0000", 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 379.74972476699998, 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96891153200000002, 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 206339 2024-09-21T10:33:00.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927717+0000", 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927717+0000", 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927716+0000", 2024-09-21T10:33:00.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927722+0000", 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5490000000000003e-06 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927734+0000", 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2136999999999999e-05 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927741+0000", 2024-09-21T10:33:00.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1620000000000002e-06 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927793+0000", 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1411e-05 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927802+0000", 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0489999999999993e-06 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.879226+0000", 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.95142409400000005 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.879342+0000", 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116457 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0", 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.879455+0000", 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011266 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:00.695 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.887223+0000", 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077681649999999996 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896538+0000", 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.009315149 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896573+0000", 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4255000000000003e-05 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.696 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896629+0000", 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5831000000000001e-05 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:206363 45.1 45:81c5df67:::benchmark_data_smithi119_168706_object76193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:40.966827+0000", 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 379.71061525900001, 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92998550800000002, 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:00.697 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 206363 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966827+0000", 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966827+0000", 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966825+0000", 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966834+0000", 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7299999999999994e-06 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966851+0000", 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7292999999999999e-05 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966858+0000", 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6830000000000002e-06 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966906+0000", 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8470999999999997e-05 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966916+0000", 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.005e-05 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.883112+0000", 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.91619621500000004 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.883215+0000", 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010313499999999999 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0", 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.883290+0000", 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4449000000000005e-05 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.887230+0000", 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039403370000000004 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896672+0000", 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094420649999999995 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896721+0000", 2024-09-21T10:33:00.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8977999999999998e-05 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896812+0000", 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0816999999999995e-05 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:131024 46.e e1264/1260 46:7941af9a:::benchmark_data_smithi119_206743_object25923:head v 1264'8210, mlcod=0'0)", 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:30:47.965402+0000", 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 132.71204020600001, 2024-09-21T10:33:00.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.080229320000000007, 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.965402+0000", 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.965402+0000", 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.965401+0000", 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.965411+0000", 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9080000000000007e-06 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.965427+0000", 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6430999999999999e-05 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.965434+0000", 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5459999999999997e-06 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.976450+0000", 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011016035 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.976465+0000", 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5566999999999999e-05 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:48.045593+0000", 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.069127560000000005 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:48.045631+0000", 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8083000000000001e-05 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:131032 46.e e1264/1260 46:7544838e:::benchmark_data_smithi119_206743_object25931:head v 1264'8211, mlcod=0'0)", 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:30:47.975654+0000", 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 132.70178763999999, 2024-09-21T10:33:00.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070005542000000004, 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.975654+0000", 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.975654+0000", 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.975653+0000", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.975664+0000", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1164e-05 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.975681+0000", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6543000000000001e-05 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.975688+0000", 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0190000000000001e-06 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.984067+0000", 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083794520000000008 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:47.984096+0000", 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8500000000000002e-05 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:48.045638+0000", 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.061541653000000002 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:30:48.045660+0000", 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2181e-05 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.711 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:152360 46.e e1267/1260 46:79bd107a:::benchmark_data_smithi119_206743_object47259:head v 1267'9535, mlcod=0'0)", 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:17.893794+0000", 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.78364733799999, 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.081087627999999995, 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893794+0000", 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893794+0000", 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893794+0000", 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893802+0000", 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7770000000000002e-06 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893813+0000", 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1753000000000001e-05 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893824+0000", 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1032999999999999e-05 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893871+0000", 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7076000000000001e-05 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.893895+0000", 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3951000000000002e-05 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.974848+0000", 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.080952283999999999 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.974882+0000", 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4464000000000001e-05 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:152356 46.6 e1267/1260 46:6770d26a:::benchmark_data_smithi119_206743_object47255:head v 1267'9546, mlcod=1267'9543)", 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:17.894097+0000", 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.783344767, 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.072481503000000003, 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.894097+0000", 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.716 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.894097+0000", 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.894096+0000", 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.894102+0000", 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.717 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.894111+0000", 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.738e-06 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.894115+0000", 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8680000000000001e-06 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.895034+0000", 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00091905399999999999 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.718 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.895053+0000", 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.876e-05 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.966525+0000", 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.071472656999999995 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.966579+0000", 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3270000000000003e-05 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.719 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:152361 46.e e1267/1260 46:74dd36d7:::benchmark_data_smithi119_206743_object47260:head v 1267'9536, mlcod=0'0)", 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:17.896977+0000", 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.780465226, 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.077922580000000005, 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.896977+0000", 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.720 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.896977+0000", 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.896976+0000", 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.896984+0000", 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3270000000000001e-06 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.721 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.896995+0000", 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1299e-05 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.897001+0000", 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.429e-06 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.911533+0000", 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014531792999999999 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.722 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.911560+0000", 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7180000000000001e-05 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.974886+0000", 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.063326428000000004 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.723 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:17.974899+0000", 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3030000000000001e-05 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:152470 46.e e1267/1260 46:7975be61:::benchmark_data_smithi119_206743_object47369:head v 1267'9554, mlcod=0'0)", 2024-09-21T10:33:00.724 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:18.062972+0000", 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.61447006, 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070825194999999994, 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.062972+0000", 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.062972+0000", 2024-09-21T10:33:00.725 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.062971+0000", 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.062979+0000", 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3750000000000003e-06 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.726 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.062998+0000", 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8479e-05 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.063003+0000", 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6760000000000004e-06 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.093020+0000", 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.030016953999999998 2024-09-21T10:33:00.727 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.093043+0000", 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2537000000000001e-05 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.133765+0000", 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.040722582 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.133797+0000", 2024-09-21T10:33:00.728 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1588000000000003e-05 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:152473 46.e e1267/1260 46:70e4f777:::benchmark_data_smithi119_206743_object47372:head v 1267'9556, mlcod=0'0)", 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:18.072769+0000", 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.604673049, 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070715429999999996, 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.729 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.072769+0000", 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.072769+0000", 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.072767+0000", 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.730 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.072776+0000", 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5429999999999994e-06 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.072794+0000", 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.804e-05 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.072801+0000", 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1019999999999999e-06 2024-09-21T10:33:00.731 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.110101+0000", 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.037299374000000003 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.110118+0000", 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7025e-05 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.732 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.143303+0000", 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.033185020000000003 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.143484+0000", 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000181602 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:152883 46.e e1267/1260 46:755dfd3e:::benchmark_data_smithi119_206743_object47782:head v 1267'9588, mlcod=0'0)", 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:18.522432+0000", 2024-09-21T10:33:00.733 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 102.155009656, 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.071420012000000005, 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.522432+0000", 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.522432+0000", 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.734 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.522431+0000", 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.522438+0000", 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6610000000000003e-06 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.522448+0000", 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0431000000000001e-05 2024-09-21T10:33:00.735 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.522454+0000", 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5589999999999996e-06 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.543593+0000", 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.021138475 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.543613+0000", 2024-09-21T10:33:00.736 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0905999999999999e-05 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.593799+0000", 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.050185281999999998 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:18.593852+0000", 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3437999999999999e-05 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.737 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:160046 46.b e1268/1260 46:db7e5b0a:::benchmark_data_smithi119_206743_object54945:head v 1268'9877, mlcod=1250'3353)", 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:28.060570+0000", 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 92.616871716999995, 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.084118878999999994, 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.060570+0000", 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.738 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.060570+0000", 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.060569+0000", 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.060577+0000", 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.52e-06 2024-09-21T10:33:00.739 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.060591+0000", 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4002000000000001e-05 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.060596+0000", 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.519e-06 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.076869+0000", 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016272300999999999 2024-09-21T10:33:00.740 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.076899+0000", 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9946000000000001e-05 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.144615+0000", 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.067715950999999996 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.144689+0000", 2024-09-21T10:33:00.741 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4428e-05 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:160319 46.e e1268/1260 46:76a84e0f:::benchmark_data_smithi119_206743_object55218:head v 1268'10061, mlcod=0'0)", 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:28.344191+0000", 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 92.333250786999997, 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.082557301999999999, 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.742 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.344191+0000", 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.344191+0000", 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.344190+0000", 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.743 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.344203+0000", 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2638e-05 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.344215+0000", 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2167000000000001e-05 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.344221+0000", 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4200000000000004e-06 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.744 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.367916+0000", 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023694719 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.367939+0000", 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2676000000000001e-05 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.426687+0000", 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.058748723000000003 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.745 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.426748+0000", 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0918000000000002e-05 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:160322 46.e e1268/1260 46:74001a6f:::benchmark_data_smithi119_206743_object55221:head v 1268'10062, mlcod=0'0)", 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:28.347469+0000", 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 92.329972751, 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.079336021000000007, 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.746 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.347469+0000", 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.347469+0000", 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.347469+0000", 2024-09-21T10:33:00.747 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.347472+0000", 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7050000000000001e-06 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.347478+0000", 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0530000000000003e-06 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.748 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.347482+0000", 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5889999999999999e-06 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.378322+0000", 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.030840163 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.378348+0000", 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5948e-05 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.749 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.426756+0000", 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.048407638000000003 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.426805+0000", 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9280999999999998e-05 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:160330 46.6 e1268/1260 46:6d44a533:::benchmark_data_smithi119_206743_object55229:head v 1268'10042, mlcod=1268'10038)", 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:28.351839+0000", 2024-09-21T10:33:00.750 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 92.325602670999999, 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.085112324000000003, 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.351839+0000", 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.351839+0000", 2024-09-21T10:33:00.751 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.351838+0000", 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.351847+0000", 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5930000000000006e-06 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.351859+0000", 2024-09-21T10:33:00.752 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1833e-05 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.351865+0000", 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0399999999999998e-06 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.388836+0000", 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.036971314999999998 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.753 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.388857+0000", 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0757000000000001e-05 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.436677+0000", 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047819970000000003 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.436951+0000", 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000274844 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.754 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:160331 46.e e1268/1260 46:765d810c:::benchmark_data_smithi119_206743_object55230:head v 1268'10063, mlcod=0'0)", 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:28.352498+0000", 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 92.324943429000001, 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.084535215999999996, 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.352498+0000", 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.755 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.352498+0000", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.352498+0000", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.352502+0000", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.698e-06 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.756 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.352507+0000", 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5840000000000002e-06 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.352510+0000", 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.05e-06 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.389354+0000", 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.036843766999999999 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.757 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.389376+0000", 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2209999999999999e-05 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.436954+0000", 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.047577913999999999 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:28.437034+0000", 2024-09-21T10:33:00.758 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9297999999999999e-05 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:161877 46.e e1268/1260 46:7851ede9:::benchmark_data_smithi119_206743_object56776:head v 1268'10162, mlcod=0'0)", 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:31:30.294241+0000", 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 90.383200337000005, 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.075117563999999998, 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.759 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.294241+0000", 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.294241+0000", 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.294241+0000", 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.760 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.294246+0000", 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3310000000000003e-06 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.294259+0000", 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2490999999999999e-05 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.294263+0000", 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7729999999999999e-06 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.761 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.313462+0000", 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.019198454 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.313493+0000", 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0797999999999999e-05 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.369264+0000", 2024-09-21T10:33:00.762 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.055771790000000002 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:31:30.369359+0000", 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4523000000000003e-05 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:187014 46.e e1269/1260 46:745bedc8:::benchmark_data_smithi119_206743_object81913:head v 1269'11741, mlcod=0'0)", 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:32:04.216747+0000", 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 56.460694531000001, 2024-09-21T10:33:00.763 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070781738999999996, 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.216747+0000", 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.216747+0000", 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.764 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.216747+0000", 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.216754+0000", 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0859999999999996e-06 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.216766+0000", 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2109999999999999e-05 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.765 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.216771+0000", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3079999999999998e-06 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.217427+0000", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000655559 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.217448+0000", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1302e-05 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.287488+0000", 2024-09-21T10:33:00.766 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070040184000000005 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:04.287529+0000", 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0884999999999999e-05 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:189892 46.e e1269/1260 46:763460fa:::benchmark_data_smithi119_206743_object84791:head v 1269'11951, mlcod=0'0)", 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:32:08.545398+0000", 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 52.132044227000002, 2024-09-21T10:33:00.767 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.071356979000000001, 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.545398+0000", 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.545398+0000", 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.768 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.545397+0000", 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.545403+0000", 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.404e-06 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.545414+0000", 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1273e-05 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.769 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.545419+0000", 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0010000000000001e-06 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.577763+0000", 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.032343021 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.577779+0000", 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6631999999999999e-05 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.770 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.616710+0000", 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.038931217999999997 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:08.616755+0000", 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4175999999999999e-05 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36246.0:192971 46.e e1270/1260 46:7e67d0c2:::benchmark_data_smithi119_206743_object87870:head v 1270'12111, mlcod=0'0)", 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:32:12.656481+0000", 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 48.020960324000001, 2024-09-21T10:33:00.771 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.074465025000000004, 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.656481+0000", 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.656481+0000", 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:00.772 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.656481+0000", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.656488+0000", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8279999999999997e-06 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.656501+0000", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3371e-05 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.656506+0000", 2024-09-21T10:33:00.773 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1780000000000002e-06 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.660390+0000", 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038840179999999999 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.660412+0000", 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1559000000000001e-05 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.730905+0000", 2024-09-21T10:33:00.774 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070493084999999997 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:32:12.730946+0000", 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1780999999999999e-05 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:00.775 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:33:00.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:00.797 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:00.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:00.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:00.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:01.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:00 smithi145 ceph-mon[89359]: pgmap v1967: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 219 objects/s recovering 2024-09-21T10:33:01.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:01.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:00 smithi119 ceph-mon[101432]: pgmap v1967: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 219 objects/s recovering 2024-09-21T10:33:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:00 smithi119 ceph-mon[105288]: pgmap v1967: 109 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 6373/81 objects degraded (7867.901%); 0 B/s, 219 objects/s recovering 2024-09-21T10:33:01.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:01.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:01.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:01.290 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:01.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:01.414 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:01.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.537 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:01.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:01.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:33:01.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:01.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:01.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:01.810 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:01.872 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:01.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:01.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:01.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:02.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:02.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:02.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:02.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:02.306 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:02.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:02.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:02.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:02.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:02.672 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:33:02.673 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:33:02.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:02.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:02.690 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:33:02.690 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:33:02.713 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:33:02.886 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:33:02.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:33:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:33:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:02.888 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:33:02.888 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:33:02.888 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:33:02.888 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:33:02.889 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:33:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:33:02.889 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:33:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:33:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 46 pg_num to 26 2024-09-21T10:33:02.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:02.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:02.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:03.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:03.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:02 smithi145 ceph-mon[89359]: pgmap v1968: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5946/81 objects degraded (7340.741%); 0 B/s, 239 objects/s recovering 2024-09-21T10:33:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2650738569' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:33:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:02 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:33:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/77603354' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[101432]: pgmap v1968: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5946/81 objects degraded (7340.741%); 0 B/s, 239 objects/s recovering 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2650738569' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/77603354' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[105288]: pgmap v1968: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5946/81 objects degraded (7340.741%); 0 B/s, 239 objects/s recovering 2024-09-21T10:33:03.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2650738569' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:33:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:33:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/77603354' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:03.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:03.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:03.158 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:03.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:03.280 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:03.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:03.404 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:03.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:03.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:03.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:03.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:03.644 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:03.644 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:03.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:03.753 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:03.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:03.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:03.907 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:03.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:04.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:03 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:33:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:03 smithi145 ceph-mon[89359]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-21T10:33:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:33:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:04.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:04.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:04.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:04.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:04.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:04.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:04.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[105288]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[101432]: osdmap e1275: 8 total, 8 up, 8 in 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:33:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:33:04.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:33:04.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:04.492 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:04.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:04.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:04.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:04.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:04.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:04.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:04.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:33:04] "GET /metrics HTTP/1.1" 200 36892 "" "Prometheus/2.51.0" 2024-09-21T10:33:04.966 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:04.988 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:04.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:04 smithi145 ceph-mon[89359]: pgmap v1970: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5946/81 objects degraded (7340.741%); 0 B/s, 68 objects/s recovering 2024-09-21T10:33:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:33:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:04 smithi145 ceph-mon[89359]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-21T10:33:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:04 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5946/81 objects degraded (7340.741%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:05.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:05.112 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:05.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:05.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:05.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:05.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:05.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[105288]: pgmap v1970: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5946/81 objects degraded (7340.741%); 0 B/s, 68 objects/s recovering 2024-09-21T10:33:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[105288]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5946/81 objects degraded (7340.741%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[101432]: pgmap v1970: 109 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 6 active+remapped+backfill_wait, 100 active+clean; 634 KiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 5946/81 objects degraded (7340.741%); 0 B/s, 68 objects/s recovering 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[101432]: osdmap e1276: 8 total, 8 up, 8 in 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:05.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:04 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5946/81 objects degraded (7340.741%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:05.396 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-21T10:33:05.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-21T10:33:05.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:05.483 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:05.584 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:05.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:05.707 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:05.729 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:05.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:05.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:05.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:05.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-21T10:33:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: 46.8 continuing backfill to osd.5 from (1263'7269,1271'13052] MIN to 1271'13052 2024-09-21T10:33:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: 46.5 continuing backfill to osd.5 from (1263'7863,1271'13340] MIN to 1271'13340 2024-09-21T10:33:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: 46.1 continuing backfill to osd.5 from (1263'7262,1271'13080] MIN to 1271'13080 2024-09-21T10:33:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: 46.6 continuing backfill to osd.5 from (1263'7238,1271'13084] MIN to 1271'13084 2024-09-21T10:33:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: 46.3 continuing backfill to osd.5 from (1263'7403,1271'13178] MIN to 1271'13178 2024-09-21T10:33:06.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:05 smithi145 ceph-mon[89359]: 46.2 continuing backfill to osd.5 from (1258'6000,1271'13220] MIN to 1271'13220 2024-09-21T10:33:06.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:06.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: 46.8 continuing backfill to osd.5 from (1263'7269,1271'13052] MIN to 1271'13052 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: 46.5 continuing backfill to osd.5 from (1263'7863,1271'13340] MIN to 1271'13340 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: 46.1 continuing backfill to osd.5 from (1263'7262,1271'13080] MIN to 1271'13080 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: 46.6 continuing backfill to osd.5 from (1263'7238,1271'13084] MIN to 1271'13084 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: 46.3 continuing backfill to osd.5 from (1263'7403,1271'13178] MIN to 1271'13178 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[101432]: 46.2 continuing backfill to osd.5 from (1258'6000,1271'13220] MIN to 1271'13220 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: osdmap e1277: 8 total, 8 up, 8 in 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: 46.8 continuing backfill to osd.5 from (1263'7269,1271'13052] MIN to 1271'13052 2024-09-21T10:33:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: 46.5 continuing backfill to osd.5 from (1263'7863,1271'13340] MIN to 1271'13340 2024-09-21T10:33:06.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: 46.1 continuing backfill to osd.5 from (1263'7262,1271'13080] MIN to 1271'13080 2024-09-21T10:33:06.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: 46.6 continuing backfill to osd.5 from (1263'7238,1271'13084] MIN to 1271'13084 2024-09-21T10:33:06.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: 46.3 continuing backfill to osd.5 from (1263'7403,1271'13178] MIN to 1271'13178 2024-09-21T10:33:06.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:05 smithi119 ceph-mon[105288]: 46.2 continuing backfill to osd.5 from (1258'6000,1271'13220] MIN to 1271'13220 2024-09-21T10:33:06.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:06.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:06.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:06.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:06.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:06.478 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:06.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:06.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:06.601 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:06.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:06.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:06.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:06.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:06.863 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:06.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:06.971 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:06.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:06.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:07.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:07.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:06 smithi119 ceph-mon[101432]: pgmap v1973: 119 pgs: 8 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 103 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%); 0 B/s, 137 objects/s recovering 2024-09-21T10:33:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:06 smithi119 ceph-mon[101432]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-21T10:33:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:06 smithi119 ceph-mon[105288]: pgmap v1973: 119 pgs: 8 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 103 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%); 0 B/s, 137 objects/s recovering 2024-09-21T10:33:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:06 smithi119 ceph-mon[105288]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-21T10:33:07.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:07.217 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:07.225 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:06 smithi145 ceph-mon[89359]: pgmap v1973: 119 pgs: 8 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 103 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%); 0 B/s, 137 objects/s recovering 2024-09-21T10:33:07.225 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:06 smithi145 ceph-mon[89359]: osdmap e1278: 8 total, 8 up, 8 in 2024-09-21T10:33:07.320 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:07.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:07.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:07.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:07.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:07.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:07.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:07.710 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:07.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:07.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:07.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:33:07.913 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:33:07.935 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:07.957 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:07.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:08.058 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:08.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: 46.12 continuing backfill to osd.5 from (1258'6000,1271'13220] MIN to 1271'13220 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: 46.18 continuing backfill to osd.5 from (1263'7269,1271'13052] MIN to 1271'13052 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: 46.11 continuing backfill to osd.5 from (1263'7262,1271'13080] MIN to 1271'13080 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: 46.13 continuing backfill to osd.5 from (1263'7403,1271'13178] MIN to 1271'13178 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: 46.16 continuing backfill to osd.5 from (1263'7238,1271'13084] MIN to 1271'13084 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: 46.15 continuing backfill to osd.5 from (1263'7863,1271'13340] MIN to 1271'13340 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3003083306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:08.109 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: 46.12 continuing backfill to osd.5 from (1258'6000,1271'13220] MIN to 1271'13220 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: 46.18 continuing backfill to osd.5 from (1263'7269,1271'13052] MIN to 1271'13052 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: 46.11 continuing backfill to osd.5 from (1263'7262,1271'13080] MIN to 1271'13080 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: 46.13 continuing backfill to osd.5 from (1263'7403,1271'13178] MIN to 1271'13178 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: 46.16 continuing backfill to osd.5 from (1263'7238,1271'13084] MIN to 1271'13084 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: 46.15 continuing backfill to osd.5 from (1263'7863,1271'13340] MIN to 1271'13340 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3003083306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: osdmap e1279: 8 total, 8 up, 8 in 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: 46.12 continuing backfill to osd.5 from (1258'6000,1271'13220] MIN to 1271'13220 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: 46.18 continuing backfill to osd.5 from (1263'7269,1271'13052] MIN to 1271'13052 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: 46.11 continuing backfill to osd.5 from (1263'7262,1271'13080] MIN to 1271'13080 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: 46.13 continuing backfill to osd.5 from (1263'7403,1271'13178] MIN to 1271'13178 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: 46.16 continuing backfill to osd.5 from (1263'7238,1271'13084] MIN to 1271'13084 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: 46.15 continuing backfill to osd.5 from (1263'7863,1271'13340] MIN to 1271'13340 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3003083306' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:08.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:08.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:08.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:08.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:08.328 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:08.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:08.381 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:08.382 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:08.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:08.452 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:08.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:08.575 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:08.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:08.698 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:08.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:08.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:08.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:08.944 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:08.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:08.987 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:08 smithi145 ceph-mon[89359]: pgmap v1976: 119 pgs: 4 remapped+peering, 4 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 103 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%) 2024-09-21T10:33:08.987 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:08 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:08.987 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:08 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:33:08.987 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:08 smithi145 ceph-mon[89359]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-21T10:33:09.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:33:09.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[105288]: pgmap v1976: 119 pgs: 4 remapped+peering, 4 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 103 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%) 2024-09-21T10:33:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:33:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[105288]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-21T10:33:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[101432]: pgmap v1976: 119 pgs: 4 remapped+peering, 4 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 103 active+clean; 634 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%) 2024-09-21T10:33:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:33:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:08 smithi119 ceph-mon[101432]: osdmap e1280: 8 total, 8 up, 8 in 2024-09-21T10:33:09.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:09.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:09.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:09.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:09.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:09.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:09.442 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:09.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:09.565 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:09.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:09.689 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:09.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:09.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:09.815 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:09.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:09.935 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:09.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:10.032 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:09 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5545/81 objects degraded (6845.679%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:10.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:10.057 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:10.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:10.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:09 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5545/81 objects degraded (6845.679%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:10.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:09 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5545/81 objects degraded (6845.679%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:10.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:10.180 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:10.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:10.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:33:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:33:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:10.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:10.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:10.397 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:33:10.398 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:33:10.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:10.430 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:10.531 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:10.554 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:10.656 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:10.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:10.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:10.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:10.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:10.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:10.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:10.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:11.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:11.048 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:11.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:10 smithi119 ceph-mon[101432]: pgmap v1978: 119 pgs: 4 remapped+peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%) 2024-09-21T10:33:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:10 smithi119 ceph-mon[105288]: pgmap v1978: 119 pgs: 4 remapped+peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%) 2024-09-21T10:33:11.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:11.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:11.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:11.184 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:10 smithi145 ceph-mon[89359]: pgmap v1978: 119 pgs: 4 remapped+peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5545/81 objects degraded (6845.679%) 2024-09-21T10:33:11.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:11.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:11.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:11.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:11.519 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:11.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:11.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:11.644 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:11.665 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:11.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:11.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:11.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:33:11.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:11.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:11.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:11.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:12.017 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:12.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:12.057 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:12.058 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:12.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:12.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:12.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:12.286 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:12.387 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:12.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:12.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:12.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:12.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:12.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:12.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:12.756 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:12.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:12.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:12.900 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:12.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:13.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:13.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:13.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:13.146 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:13.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:12 smithi145 ceph-mon[89359]: pgmap v1979: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5069/81 objects degraded (6258.025%); 0 B/s, 79 objects/s recovering 2024-09-21T10:33:13.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:33:13.247 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:13.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:12 smithi119 ceph-mon[105288]: pgmap v1979: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5069/81 objects degraded (6258.025%); 0 B/s, 79 objects/s recovering 2024-09-21T10:33:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:12 smithi119 ceph-mon[101432]: pgmap v1979: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5069/81 objects degraded (6258.025%); 0 B/s, 79 objects/s recovering 2024-09-21T10:33:13.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:13.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:13.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:13.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:13.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:13.622 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.644 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:13.745 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:13.766 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:13.821 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:13.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:13.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:13.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:13.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:14.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:14.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:14.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:14.237 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:14.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:14.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:14.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:14.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:14.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:14.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:14.627 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:14.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:14.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:14.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:14.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:33:14] "GET /metrics HTTP/1.1" 200 36892 "" "Prometheus/2.51.0" 2024-09-21T10:33:14.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:14.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:14.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:15.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:15.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:15.040 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:14 smithi145 ceph-mon[89359]: pgmap v1980: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5069/81 objects degraded (6258.025%); 0 B/s, 61 objects/s recovering 2024-09-21T10:33:15.041 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:15.041 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:15.041 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:14 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5069/81 objects degraded (6258.025%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:15.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[105288]: pgmap v1980: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5069/81 objects degraded (6258.025%); 0 B/s, 61 objects/s recovering 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5069/81 objects degraded (6258.025%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[101432]: pgmap v1980: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 5069/81 objects degraded (6258.025%); 0 B/s, 61 objects/s recovering 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:15.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:15.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:14 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5069/81 objects degraded (6258.025%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:15.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:15.225 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:15.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:15.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:15.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:15.473 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:15.495 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:15.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:15.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:15.636 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:15.636 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:15.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:15.741 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:15.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:15.864 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:15.966 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:15.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:15.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:16.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:16.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:16.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:16.232 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:16.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:33:16.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:16.354 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:16.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:16.478 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:16.576 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:16.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:16.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:16.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:16.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.724 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:16.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:16.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:16.847 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:16.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:16.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:16.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:17.003 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:16 smithi145 ceph-mon[89359]: pgmap v1981: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4520/81 objects degraded (5580.247%); 0 B/s, 117 objects/s recovering 2024-09-21T10:33:17.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:17.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:17.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:17.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:17.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:17.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:16 smithi119 ceph-mon[101432]: pgmap v1981: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4520/81 objects degraded (5580.247%); 0 B/s, 117 objects/s recovering 2024-09-21T10:33:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:16 smithi119 ceph-mon[105288]: pgmap v1981: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 4520/81 objects degraded (5580.247%); 0 B/s, 117 objects/s recovering 2024-09-21T10:33:17.454 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:17.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:17.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:17.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:17.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:17.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:17.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:17.848 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:17.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:17.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:17.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:18.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:18.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:18.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:18.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:18.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:18.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:18.342 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:18.369 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:18.369 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:18.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:18.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:18.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:18.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:18.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:18.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:18.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:18.833 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:18.908 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_1' removed 2024-09-21T10:33:18.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:18.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:18.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:19.008 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:18 smithi145 ceph-mon[89359]: pgmap v1982: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 21 MiB/s wr, 660 op/s; 4520/81 objects degraded (5580.247%); 6.9 MiB/s, 212 objects/s recovering 2024-09-21T10:33:19.008 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2141106778' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:33:19.008 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:18 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:33:19.031 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:33:19.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:19.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:19.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:19.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:19.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:19.277 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:19.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:19.323 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:18 smithi119 ceph-mon[105288]: pgmap v1982: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 21 MiB/s wr, 660 op/s; 4520/81 objects degraded (5580.247%); 6.9 MiB/s, 212 objects/s recovering 2024-09-21T10:33:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2141106778' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:33:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:18 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:33:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:18 smithi119 ceph-mon[101432]: pgmap v1982: 119 pgs: 1 active+recovering+undersized+degraded+remapped, 13 active+remapped+backfill_wait, 105 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 21 MiB/s wr, 660 op/s; 4520/81 objects degraded (5580.247%); 6.9 MiB/s, 212 objects/s recovering 2024-09-21T10:33:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2141106778' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:33:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:18 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:33:19.425 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:19.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:19.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:19.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:19.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:19.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:19.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:19.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:19.819 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:19.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:19.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:19.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:19 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:33:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:19 smithi145 ceph-mon[89359]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-21T10:33:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 4520/81 objects degraded (5580.247%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:19 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:33:20.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:20.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:33:20.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[105288]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-21T10:33:20.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 4520/81 objects degraded (5580.247%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:20.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:33:20.055 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:33:20.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[101432]: osdmap e1281: 8 total, 8 up, 8 in 2024-09-21T10:33:20.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 4520/81 objects degraded (5580.247%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:33:20.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:33:20.068 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:20.079 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:33:20.079 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:33:20.079 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-21T10:33:20.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_2 16 2024-09-21T10:33:20.097 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:20.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:20.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:20.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:20.312 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:20.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:20.418 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:33:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:33:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:20.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:20.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:20.560 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:20.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:20.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:20.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:20.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:20.807 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:20.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:20.930 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:20.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:21.013 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:20 smithi145 ceph-mon[89359]: pgmap v1984: 93 pgs: 93 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:33:21.013 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:20 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4520/81 objects degraded (5580.247%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:33:21.013 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:33:21.013 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:20 smithi145 ceph-mon[89359]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-21T10:33:21.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:21.044 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:21.045 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:21.052 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:21.150 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[105288]: pgmap v1984: 93 pgs: 93 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:33:21.150 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4520/81 objects degraded (5580.247%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:33:21.150 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:33:21.151 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[105288]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-21T10:33:21.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[101432]: pgmap v1984: 93 pgs: 93 active+clean; 634 KiB data, 3.9 GiB used, 711 GiB / 715 GiB avail 2024-09-21T10:33:21.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4520/81 objects degraded (5580.247%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:33:21.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:33:21.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:20 smithi119 ceph-mon[101432]: osdmap e1282: 8 total, 8 up, 8 in 2024-09-21T10:33:21.153 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:21.176 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:21.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:21.299 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:21.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:21.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:21.540 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:33:21.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:33:21.544 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:21.546 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:33:21.546 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:33:21.562 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:33:21.562 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:33:21.579 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:33:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:33:21.595 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:33:21.596 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:33:21.612 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:33:21.612 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:33:21.628 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:33:21.628 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:33:21.643 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:33:21.643 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:33:21.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:21.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:21.748 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:33:21.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:21.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:21.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:21.937 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:21.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:21.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:21.999 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:22.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:21 smithi145 ceph-mon[89359]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-21T10:33:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2451744855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:22.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:22.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:22.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:22.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:22.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:22.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:21 smithi119 ceph-mon[105288]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-21T10:33:22.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2451744855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:22.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:21 smithi119 ceph-mon[101432]: osdmap e1283: 8 total, 8 up, 8 in 2024-09-21T10:33:22.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2451744855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:22.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:22.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:22.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:22.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:22.655 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:22.676 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:22.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:22.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:22.799 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:22.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:22.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:22.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:22.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:22.994 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:22 smithi119 ceph-mon[101432]: pgmap v1987: 93 pgs: 1 activating, 2 peering, 90 active+clean; 634 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:22.994 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:22 smithi119 ceph-mon[105288]: pgmap v1987: 93 pgs: 1 activating, 2 peering, 90 active+clean; 634 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:23.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:23.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:23.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:23.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:23.198 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:22 smithi145 ceph-mon[89359]: pgmap v1987: 93 pgs: 1 activating, 2 peering, 90 active+clean; 634 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:23.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:23.291 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:23.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:23.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:23.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:23.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:23.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:23.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:23.701 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:33:23.701 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:33:23.701 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:33:23.701 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:33:23.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:23.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:23.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:23.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:23.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:23.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:23.911 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:33:24.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:24.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:24.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:24.156 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:24.239 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/710564982' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:33:24.239 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:33:24.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:24.263 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:33:24.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:24.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/710564982' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:33:24.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:33:24.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/710564982' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:33:24.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:33:24.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:24.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:24.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:24.507 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:24.525 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:24.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:33:24] "GET /metrics HTTP/1.1" 200 36912 "" "Prometheus/2.51.0" 2024-09-21T10:33:24.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:24.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:24.752 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:24.774 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:24.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:24.897 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:24.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:24.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:25.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:25.060 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[105288]: pgmap v1988: 93 pgs: 1 activating, 2 peering, 90 active+clean; 713 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 6 op/s 2024-09-21T10:33:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:33:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[105288]: osdmap e1284: 8 total, 8 up, 8 in 2024-09-21T10:33:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[105288]: 1.6 scrub starts 2024-09-21T10:33:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[105288]: 1.6 scrub ok 2024-09-21T10:33:25.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[101432]: pgmap v1988: 93 pgs: 1 activating, 2 peering, 90 active+clean; 713 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 6 op/s 2024-09-21T10:33:25.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:25.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:33:25.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[101432]: osdmap e1284: 8 total, 8 up, 8 in 2024-09-21T10:33:25.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[101432]: 1.6 scrub starts 2024-09-21T10:33:25.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:24 smithi119 ceph-mon[101432]: 1.6 scrub ok 2024-09-21T10:33:25.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:25.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:25.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:25.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:25.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:24 smithi145 ceph-mon[89359]: pgmap v1988: 93 pgs: 1 activating, 2 peering, 90 active+clean; 713 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 6 op/s 2024-09-21T10:33:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:24 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:24 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:33:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:24 smithi145 ceph-mon[89359]: osdmap e1284: 8 total, 8 up, 8 in 2024-09-21T10:33:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:24 smithi145 ceph-mon[89359]: 1.6 scrub starts 2024-09-21T10:33:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:24 smithi145 ceph-mon[89359]: 1.6 scrub ok 2024-09-21T10:33:25.370 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:25.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:25.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:25.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:25.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:25.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.640 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:25.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:25.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:25.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:25.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:25.887 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:25.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:25.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:26.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:26.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:33:26.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:26.134 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:26.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:26.243 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:25 smithi145 ceph-mon[89359]: 1.5 scrub starts 2024-09-21T10:33:26.243 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:25 smithi145 ceph-mon[89359]: 1.5 scrub ok 2024-09-21T10:33:26.256 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:26.258 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:33:26.258 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:33:26.258 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:33:26.258 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:33:26.258 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:33:26.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:25 smithi119 ceph-mon[101432]: 1.5 scrub starts 2024-09-21T10:33:26.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:25 smithi119 ceph-mon[101432]: 1.5 scrub ok 2024-09-21T10:33:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:25 smithi119 ceph-mon[105288]: 1.5 scrub starts 2024-09-21T10:33:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:25 smithi119 ceph-mon[105288]: 1.5 scrub ok 2024-09-21T10:33:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.380 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:26.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:26.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:26.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:26.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:26.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:26.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:26.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:26.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:33:26.855 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:26.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:26.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:26.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:33:26.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:27.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:27.036 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: 1.1 scrub starts 2024-09-21T10:33:27.036 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: 1.1 scrub ok 2024-09-21T10:33:27.036 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: 1.0 scrub starts 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: 1.0 scrub ok 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: 1.b scrub starts 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: 1.b scrub ok 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[101432]: pgmap v1990: 93 pgs: 1 activating, 92 active+clean; 713 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s; 66 KiB/s, 0 objects/s recovering 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: 1.1 scrub starts 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: 1.1 scrub ok 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: 1.0 scrub starts 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: 1.0 scrub ok 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: 1.b scrub starts 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: 1.b scrub ok 2024-09-21T10:33:27.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:26 smithi119 ceph-mon[105288]: pgmap v1990: 93 pgs: 1 activating, 92 active+clean; 713 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s; 66 KiB/s, 0 objects/s recovering 2024-09-21T10:33:27.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:27.142 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:27.200 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:33:27.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:27.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: 1.1 scrub starts 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: 1.1 scrub ok 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: 1.0 scrub starts 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: 1.0 scrub ok 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: 1.b scrub starts 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: 1.b scrub ok 2024-09-21T10:33:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:26 smithi145 ceph-mon[89359]: pgmap v1990: 93 pgs: 1 activating, 92 active+clean; 713 KiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 10 op/s; 66 KiB/s, 0 objects/s recovering 2024-09-21T10:33:27.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:27.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:27.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:27.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:27.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.635 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:27.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:27.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:27.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:27.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:27.881 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:27.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:27.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:27.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:28.004 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:28.013 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_2' created 2024-09-21T10:33:28.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:28.228 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:28.250 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:28.271 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:28.271 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: 1.2 scrub ok 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: 1.7 scrub starts 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: 1.7 scrub ok 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: 1.e scrub starts 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: 1.e scrub ok 2024-09-21T10:33:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2881221663' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:33:28.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: 1.2 scrub ok 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: 1.7 scrub starts 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: 1.7 scrub ok 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: 1.e scrub starts 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: 1.e scrub ok 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2881221663' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:33:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:33:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: 1.2 scrub ok 2024-09-21T10:33:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: 1.7 scrub starts 2024-09-21T10:33:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: 1.7 scrub ok 2024-09-21T10:33:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: 1.e scrub starts 2024-09-21T10:33:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: 1.e scrub ok 2024-09-21T10:33:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2881221663' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-21T10:33:28.373 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:28.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:28.495 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:28.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:28.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:28.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:28.719 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:28.741 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:28.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-21T10:33:28.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:28.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:28.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:28.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:28.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:28 smithi145 ceph-mon[89359]: 1.d scrub starts 2024-09-21T10:33:28.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:28 smithi145 ceph-mon[89359]: 1.d scrub ok 2024-09-21T10:33:28.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:28 smithi145 ceph-mon[89359]: pgmap v1991: 93 pgs: 93 active+clean; 713 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 8 op/s; 51 KiB/s, 0 objects/s recovering 2024-09-21T10:33:28.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:28 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2881221663' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:33:28.936 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:28 smithi145 ceph-mon[89359]: osdmap e1285: 8 total, 8 up, 8 in 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247717 45.1 45:8d06fae3:::benchmark_data_smithi119_168706_object117547:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.871543+0000", 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 344.06154404300003, 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.082356224, 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.941 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247717 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871543+0000", 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.942 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871543+0000", 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871541+0000", 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.943 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871547+0000", 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6590000000000004e-06 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871558+0000", 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1029000000000001e-05 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871564+0000", 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7540000000000003e-06 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.944 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871625+0000", 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1367999999999999e-05 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.871635+0000", 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0404000000000001e-05 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.943799+0000", 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0721633340000001 2024-09-21T10:33:28.945 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.943935+0000", 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013609700000000001 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.944030+0000", 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4760999999999996e-05 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.946 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953473+0000", 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094437940000000001 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953655+0000", 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018141899999999999 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953705+0000", 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0417000000000002e-05 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.947 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953741+0000", 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.578e-05 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953775+0000", 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4335000000000002e-05 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953899+0000", 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012356199999999999 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.948 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247775 45.1 45:8c7efd63:::benchmark_data_smithi119_168706_object117605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.928030+0000", 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 344.00505694200001, 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0314164690000001, 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.949 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247775 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928030+0000", 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928030+0000", 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.950 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928028+0000", 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928036+0000", 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0330000000000008e-06 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928050+0000", 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3959e-05 2024-09-21T10:33:28.951 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928056+0000", 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2929999999999997e-06 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928137+0000", 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1064999999999997e-05 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.928149+0000", 2024-09-21T10:33:28.952 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1309e-05 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.946541+0000", 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0183918620000001 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.946625+0000", 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4000999999999998e-05 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.953 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.946695+0000", 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0251000000000002e-05 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953666+0000", 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069715849999999998 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.954 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953945+0000", 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027831900000000002 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953974+0000", 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9479999999999999e-05 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959272+0000", 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052979000000000004 2024-09-21T10:33:28.955 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959326+0000", 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4128000000000001e-05 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959446+0000", 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011993999999999999 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.956 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247780 45.1 45:87c042b8:::benchmark_data_smithi119_168706_object117610:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.930239+0000", 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 344.002847911, 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.029423786, 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247780 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.957 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930239+0000", 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930239+0000", 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930237+0000", 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.958 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930246+0000", 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1410000000000005e-06 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930263+0000", 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7158000000000001e-05 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930269+0000", 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9290000000000003e-06 2024-09-21T10:33:28.959 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930320+0000", 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1496999999999998e-05 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.930330+0000", 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4609999999999997e-06 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.950055+0000", 2024-09-21T10:33:28.960 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.019725067 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.950204+0000", 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000149574 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.950334+0000", 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000129231 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.961 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.953712+0000", 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003378579 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954001+0000", 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028842400000000002 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954035+0000", 2024-09-21T10:33:28.962 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4539999999999998e-05 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959552+0000", 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055170239999999997 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959601+0000", 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8526000000000003e-05 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.963 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959663+0000", 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1914000000000004e-05 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247802 45.1 45:83bacf6a:::benchmark_data_smithi119_168706_object117632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:44.962464+0000", 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 343.97062234700002, 2024-09-21T10:33:28.964 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.003117987, 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247802 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962464+0000", 2024-09-21T10:33:28.965 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962464+0000", 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962463+0000", 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.966 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962472+0000", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5240000000000006e-06 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962486+0000", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3929e-05 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962492+0000", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9689999999999999e-06 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962546+0000", 2024-09-21T10:33:28.967 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4026000000000001e-05 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:44.962557+0000", 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1168e-05 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954060+0000", 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99150294900000002 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954155+0000", 2024-09-21T10:33:28.968 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5129000000000003e-05 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.954236+0000", 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0541e-05 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959459+0000", 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052234550000000001 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959500+0000", 2024-09-21T10:33:28.969 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0667999999999998e-05 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965432+0000", 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059324970000000001 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965490+0000", 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7763000000000002e-05 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.970 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965523+0000", 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2475999999999999e-05 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965582+0000", 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9605000000000003e-05 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.971 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247853 45.1 45:80a3af5b:::benchmark_data_smithi119_168706_object117683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:45.036160+0000", 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 343.89692649800003, 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93760746900000003, 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247853 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.972 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036160+0000", 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036160+0000", 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036159+0000", 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.973 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036165+0000", 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3790000000000003e-06 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036176+0000", 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0701e-05 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036180+0000", 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.549e-06 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.974 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036250+0000", 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0099999999999996e-05 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.036265+0000", 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5021999999999999e-05 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.956746+0000", 2024-09-21T10:33:28.975 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.920480717 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.956864+0000", 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117983 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.956958+0000", 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3844000000000005e-05 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959507+0000", 2024-09-21T10:33:28.976 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025486380000000002 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965385+0000", 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0058785820000000003 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965642+0000", 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000256934 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.977 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973614+0000", 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007971512 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973664+0000", 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0219999999999997e-05 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973768+0000", 2024-09-21T10:33:28.978 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103717 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247870 45.1 45:8e84670f:::benchmark_data_smithi119_168706_object117700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:45.058481+0000", 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 343.87460558999999, 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92971519400000002, 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.979 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247870 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058481+0000", 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058481+0000", 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.980 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058479+0000", 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058488+0000", 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3729999999999993e-06 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058507+0000", 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9595000000000001e-05 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.981 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058514+0000", 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4690000000000002e-06 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058571+0000", 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7747999999999997e-05 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.982 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.058582+0000", 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0713999999999999e-05 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959687+0000", 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.90110485399999996 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.983 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:28.984 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959788+0000", 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100725 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.959872+0000", 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4790000000000006e-05 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965438+0000", 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0055659860000000002 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.985 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965466+0000", 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8061000000000001e-05 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973522+0000", 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0080556440000000007 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.986 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988008+0000", 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014486357 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988102+0000", 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3778999999999998e-05 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988196+0000", 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3980000000000005e-05 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.987 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:247882 45.1 45:8dc2ee87:::benchmark_data_smithi119_168706_object117712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1221)", 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:45.077462+0000", 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 343.85562443200001, 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.91105475700000005, 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.988 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 247882 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077462+0000", 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077462+0000", 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.989 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077461+0000", 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077469+0000", 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3420000000000008e-06 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.990 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077484+0000", 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4609e-05 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077490+0000", 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3910000000000003e-06 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077540+0000", 2024-09-21T10:33:28.991 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9641999999999997e-05 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.077557+0000", 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7564e-05 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.962124+0000", 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.88456663099999999 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.992 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.962246+0000", 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012238200000000001 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.962380+0000", 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013418600000000001 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965589+0000", 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032087869999999998 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.993 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.965616+0000", 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.652e-05 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.973581+0000", 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0079651059999999996 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988331+0000", 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014750535 2024-09-21T10:33:28.994 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988406+0000", 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4847999999999999e-05 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:45.988517+0000", 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110826 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250064 45.1 45:811e61af:::benchmark_data_smithi119_168706_object119894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1224)", 2024-09-21T10:33:28.995 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.889381+0000", 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 340.043706117, 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0805241029999999, 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250064 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889381+0000", 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.996 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889381+0000", 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889379+0000", 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889387+0000", 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.255e-06 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.997 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889398+0000", 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1972e-05 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889405+0000", 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9990000000000002e-06 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889466+0000", 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0007000000000003e-05 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.998 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889582+0000", 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116004 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,5,6", 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.889691+0000", 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010945700000000001 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.893378+0000", 2024-09-21T10:33:28.999 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036874310000000001 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.894674+0000", 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012952809999999999 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.894806+0000", 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132228 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.000 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896376+0000", 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015699990000000001 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896387+0000", 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0861e-05 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.969848+0000", 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.073460954 2024-09-21T10:33:29.001 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.969905+0000", 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6904000000000003e-05 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250069 45.1 45:88025a9d:::benchmark_data_smithi119_168706_object119899:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1224)", 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.895697+0000", 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 340.03738959600003, 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0901574789999999, 2024-09-21T10:33:29.002 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250069 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895697+0000", 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.003 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895697+0000", 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895695+0000", 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895703+0000", 2024-09-21T10:33:29.004 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2629999999999997e-06 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895714+0000", 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1314e-05 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.895720+0000", 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5509999999999999e-06 2024-09-21T10:33:29.005 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896438+0000", 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000718923 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.896447+0000", 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0189999999999995e-06 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.006 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.970354+0000", 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.073906721 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.970448+0000", 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3754999999999999e-05 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.970548+0000", 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100361 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.007 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981595+0000", 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01104657 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985700+0000", 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041046839999999999 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985752+0000", 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2450000000000001e-05 2024-09-21T10:33:29.008 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985800+0000", 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8191e-05 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985855+0000", 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4317000000000002e-05 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.009 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250099 45.1 45:8e8c7d4b:::benchmark_data_smithi119_168706_object119929:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.936855+0000", 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.99623178500002, 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.049148078, 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250099 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.010 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936855+0000", 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936855+0000", 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936853+0000", 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.011 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936860+0000", 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6030000000000001e-06 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936872+0000", 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.224e-05 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936878+0000", 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8629999999999996e-06 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.012 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936935+0000", 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7578000000000003e-05 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.936945+0000", 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6220000000000004e-06 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.973862+0000", 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.036916766 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.013 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.973977+0000", 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115418 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.974098+0000", 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012074999999999999 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985644+0000", 2024-09-21T10:33:29.014 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011546307 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985896+0000", 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025206000000000002 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985931+0000", 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4465999999999998e-05 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.015 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985969+0000", 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7996e-05 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986003+0000", 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4224000000000003e-05 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250104 45.1 45:82dcfcfc:::benchmark_data_smithi119_168706_object119934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.947258+0000", 2024-09-21T10:33:29.016 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.98582841000001, 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0388857600000001, 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250104 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.017 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947258+0000", 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947258+0000", 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947257+0000", 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:29.018 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947263+0000", 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7169999999999996e-06 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947273+0000", 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0069e-05 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.019 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947279+0000", 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1940000000000003e-06 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947338+0000", 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8733999999999997e-05 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.020 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.947348+0000", 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0385e-05 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.977450+0000", 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.030102267 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.977577+0000", 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012686699999999999 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.021 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.977695+0000", 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117941 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.985864+0000", 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081691890000000003 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986039+0000", 2024-09-21T10:33:29.022 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017473099999999999 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986074+0000", 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4434000000000003e-05 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986109+0000", 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5167999999999999e-05 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986144+0000", 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5281000000000003e-05 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250115 45.1 45:8f62f6be:::benchmark_data_smithi119_168706_object119945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.024 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.955841+0000", 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.97724518799998, 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.030431326, 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250115 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955841+0000", 2024-09-21T10:33:29.025 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955841+0000", 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955840+0000", 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.026 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955847+0000", 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7849999999999992e-06 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955862+0000", 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4928e-05 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955869+0000", 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.63e-06 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.027 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955926+0000", 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7465e-05 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.955937+0000", 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0512e-05 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981618+0000", 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0256806570000001 2024-09-21T10:33:29.028 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981750+0000", 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000132218 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.981868+0000", 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117701 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986012+0000", 2024-09-21T10:33:29.029 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004144483 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986171+0000", 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015925499999999999 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986206+0000", 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4699000000000001e-05 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.030 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986245+0000", 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8682e-05 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986273+0000", 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8072000000000002e-05 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250129 45.1 45:8156bc2a:::benchmark_data_smithi119_168706_object119959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.031 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.978102+0000", 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.95498458499998, 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0242735489999999, 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250129 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978102+0000", 2024-09-21T10:33:29.032 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978102+0000", 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978100+0000", 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978109+0000", 2024-09-21T10:33:29.033 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2269999999999993e-06 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978123+0000", 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3934000000000001e-05 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978128+0000", 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9610000000000002e-06 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978188+0000", 2024-09-21T10:33:29.034 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9478999999999998e-05 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.978198+0000", 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0084999999999999e-05 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986300+0000", 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0081023689999999 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.035 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.986412+0000", 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112009 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.987281+0000", 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00086866799999999996 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002213+0000", 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014931411 2024-09-21T10:33:29.036 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002243+0000", 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0434999999999999e-05 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002283+0000", 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0145000000000003e-05 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002341+0000", 2024-09-21T10:33:29.037 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7686999999999998e-05 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002376+0000", 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4832000000000001e-05 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250140 45.1 45:8f92095e:::benchmark_data_smithi119_168706_object119970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.994583+0000", 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.93850361199998, 2024-09-21T10:33:29.038 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.012726478, 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250140 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994583+0000", 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.039 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994583+0000", 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994582+0000", 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994588+0000", 2024-09-21T10:33:29.040 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3419999999999996e-06 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994600+0000", 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.171e-05 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994604+0000", 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2910000000000001e-06 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.041 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994652+0000", 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7491000000000003e-05 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994660+0000", 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1459999999999996e-06 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.991345+0000", 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99668568599999996 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.042 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.991462+0000", 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116952 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.991583+0000", 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120221 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002277+0000", 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010693945999999999 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.043 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004725+0000", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024486170000000002 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007227+0000", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002502251 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007271+0000", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4005999999999998e-05 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007310+0000", 2024-09-21T10:33:29.044 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8090000000000003e-05 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250143 45.1 45:87c2933b:::benchmark_data_smithi119_168706_object119973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:48.994838+0000", 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.93824884899999, 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0229272229999999, 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250143 2024-09-21T10:33:29.045 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994838+0000", 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994838+0000", 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.046 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994837+0000", 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994851+0000", 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.361e-05 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994860+0000", 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4569999999999994e-06 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994864+0000", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0400000000000003e-06 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994901+0000", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6758000000000003e-05 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:48.994910+0000", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7630000000000006e-06 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.995056+0000", 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0001458059999999 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.995155+0000", 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9439999999999997e-05 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.995260+0000", 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010448 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.049 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002424+0000", 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071644669999999999 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004731+0000", 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023075130000000002 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017679+0000", 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012947157000000001 2024-09-21T10:33:29.050 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017727+0000", 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8367e-05 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017765+0000", 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.799e-05 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.051 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250157 45.1 45:89d7408f:::benchmark_data_smithi119_168706_object119987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.012165+0000", 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.92092165499997, 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.011180382, 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250157 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.052 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012165+0000", 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012165+0000", 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012163+0000", 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:29.053 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012171+0000", 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012183+0000", 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2571999999999999e-05 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.054 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012189+0000", 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7429999999999999e-06 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012297+0000", 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108449 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.012312+0000", 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4783000000000001e-05 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.055 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.998447+0000", 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98613498200000005 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.998564+0000", 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116338 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.998700+0000", 2024-09-21T10:33:29.056 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000136563 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017640+0000", 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018939837000000001 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017842+0000", 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000202461 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.057 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.023253+0000", 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054105079999999996 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.023304+0000", 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0963e-05 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.023345+0000", 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1523000000000002e-05 2024-09-21T10:33:29.058 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250165 45.1 45:80acb05c:::benchmark_data_smithi119_168706_object119995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.024689+0000", 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.90839792899999, 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0049197569999999, 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.059 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250165 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024689+0000", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024689+0000", 2024-09-21T10:33:29.060 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024686+0000", 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024693+0000", 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9380000000000003e-06 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.061 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024704+0000", 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0720000000000001e-05 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024709+0000", 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3950000000000003e-06 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024747+0000", 2024-09-21T10:33:29.062 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8676e-05 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.024754+0000", 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9689999999999997e-06 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002491+0000", 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97773674899999996 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.063 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002560+0000", 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9281000000000003e-05 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.002626+0000", 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6241000000000005e-05 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017774+0000", 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015147282999999999 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.064 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.022962+0000", 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051881219999999999 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029514+0000", 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065526150000000003 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029569+0000", 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4413000000000002e-05 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.065 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029608+0000", 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9567999999999998e-05 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250168 45.1 45:89fc89dc:::benchmark_data_smithi119_168706_object119998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.027310+0000", 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.905777129, 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.002409452, 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.066 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250168 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027310+0000", 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.067 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027310+0000", 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027308+0000", 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027314+0000", 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4700000000000001e-06 2024-09-21T10:33:29.068 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027324+0000", 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.046e-05 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.069 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027330+0000", 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7790000000000001e-06 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027374+0000", 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4058999999999999e-05 2024-09-21T10:33:29.070 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.027381+0000", 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0060000000000003e-06 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004774+0000", 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97739246800000001 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.071 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.004905+0000", 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013101599999999999 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.005007+0000", 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010286800000000001 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017780+0000", 2024-09-21T10:33:29.072 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012772451000000001 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.024728+0000", 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069478860000000003 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029647+0000", 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0049195369999999999 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.073 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029683+0000", 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5939000000000003e-05 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029719+0000", 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5651000000000001e-05 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.074 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250187 45.1 45:8e1f7a06:::benchmark_data_smithi119_168706_object120017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.051926+0000", 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.88116072000003, 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98551227200000002, 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250187 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.075 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051926+0000", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051926+0000", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051924+0000", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.076 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051930+0000", 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051940+0000", 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7550000000000006e-06 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051946+0000", 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1090000000000003e-06 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.077 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051992+0000", 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5760000000000002e-05 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.051999+0000", 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.545e-06 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007338+0000", 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.955339406 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.078 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007429+0000", 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1113000000000007e-05 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.007525+0000", 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5538999999999994e-05 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017785+0000", 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010260686999999999 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.079 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029758+0000", 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01197295 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037352+0000", 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0075938009999999998 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037401+0000", 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9054999999999997e-05 2024-09-21T10:33:29.080 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037438+0000", 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7085000000000002e-05 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:250204 45.1 45:81b5fc70:::benchmark_data_smithi119_168706_object120034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1225)", 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:49.076035+0000", 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 339.85705116899999, 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96149175200000003, 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:33:29.081 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 250204 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076035+0000", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076035+0000", 2024-09-21T10:33:29.082 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076033+0000", 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076041+0000", 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4780000000000002e-06 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.083 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076055+0000", 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4419999999999999e-05 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076061+0000", 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.896e-06 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076117+0000", 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5529999999999999e-05 2024-09-21T10:33:29.084 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:49.076126+0000", 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8030000000000008e-06 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.010934+0000", 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93480773500000003 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.011035+0000", 2024-09-21T10:33:29.085 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010054 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.011406+0000", 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037157200000000003 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.017791+0000", 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006384514 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.086 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.029786+0000", 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011995653 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037467+0000", 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076803369999999998 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037498+0000", 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1451000000000001e-05 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:33:29.087 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:50.037527+0000", 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9031e-05 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:33:29.088 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:33:29.089 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[101432]: 1.d scrub starts 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[101432]: 1.d scrub ok 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[101432]: pgmap v1991: 93 pgs: 93 active+clean; 713 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 8 op/s; 51 KiB/s, 0 objects/s recovering 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2881221663' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[101432]: osdmap e1285: 8 total, 8 up, 8 in 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[105288]: 1.d scrub starts 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[105288]: 1.d scrub ok 2024-09-21T10:33:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[105288]: pgmap v1991: 93 pgs: 93 active+clean; 713 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 29 KiB/s wr, 8 op/s; 51 KiB/s, 0 objects/s recovering 2024-09-21T10:33:29.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2881221663' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-21T10:33:29.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:28 smithi119 ceph-mon[105288]: osdmap e1285: 8 total, 8 up, 8 in 2024-09-21T10:33:29.191 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:29.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:29.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:29.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:29.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:29.458 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:29.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:29.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.581 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:29.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:29.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:29.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:29.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:29.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:29.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:33:29.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:29.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:29.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:30.032 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:29 smithi119 ceph-mon[101432]: osdmap e1286: 8 total, 8 up, 8 in 2024-09-21T10:33:30.032 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:29 smithi119 ceph-mon[105288]: osdmap e1286: 8 total, 8 up, 8 in 2024-09-21T10:33:30.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.075 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:30.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:30.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:30.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:29 smithi145 ceph-mon[89359]: osdmap e1286: 8 total, 8 up, 8 in 2024-09-21T10:33:30.298 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:30.320 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:30.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:30.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:33:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:30.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:30.565 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:30.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:30.688 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:30.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.814 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:30.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:30.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:30.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:30.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:30.985 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:31.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:31.060 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:31.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:31.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: 1.10 scrub starts 2024-09-21T10:33:31.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: 1.10 scrub ok 2024-09-21T10:33:31.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: pgmap v1994: 109 pgs: 8 creating+peering, 8 unknown, 93 active+clean; 634 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-09-21T10:33:31.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: osdmap e1287: 8 total, 8 up, 8 in 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: 1.10 scrub starts 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: 1.10 scrub ok 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: pgmap v1994: 109 pgs: 8 creating+peering, 8 unknown, 93 active+clean; 634 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: osdmap e1287: 8 total, 8 up, 8 in 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:33:31.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:33:31.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:30 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:31.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:31.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: 1.10 scrub starts 2024-09-21T10:33:31.186 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: 1.10 scrub ok 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: pgmap v1994: 109 pgs: 8 creating+peering, 8 unknown, 93 active+clean; 634 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 40 KiB/s, 0 objects/s recovering 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: osdmap e1287: 8 total, 8 up, 8 in 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:33:31.187 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:30 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:31.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:31.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:31.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:31.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:31.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:31.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:31.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:31.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:31.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:31.802 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:31.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:31.925 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:31.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:32.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:31 smithi145 ceph-mon[89359]: 1.11 scrub starts 2024-09-21T10:33:32.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:31 smithi145 ceph-mon[89359]: 1.11 scrub ok 2024-09-21T10:33:32.047 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:32.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:32.170 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:32.272 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:32.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:32.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:31 smithi119 ceph-mon[101432]: 1.11 scrub starts 2024-09-21T10:33:32.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:31 smithi119 ceph-mon[101432]: 1.11 scrub ok 2024-09-21T10:33:32.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:31 smithi119 ceph-mon[105288]: 1.11 scrub starts 2024-09-21T10:33:32.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:31 smithi119 ceph-mon[105288]: 1.11 scrub ok 2024-09-21T10:33:32.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:32.417 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:32.519 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:32.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:32.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:32.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:32.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:32.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:32.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:32.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:32.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:32.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:33.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:33.037 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:33.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:32 smithi119 ceph-mon[105288]: pgmap v1996: 109 pgs: 8 creating+peering, 2 unknown, 99 active+clean; 634 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:33.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:33.083 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:33.083 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:32 smithi119 ceph-mon[101432]: pgmap v1996: 109 pgs: 8 creating+peering, 2 unknown, 99 active+clean; 634 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:33.083 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:33.084 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:33.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:33.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:33.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:32 smithi145 ceph-mon[89359]: pgmap v1996: 109 pgs: 8 creating+peering, 2 unknown, 99 active+clean; 634 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:33.387 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:33.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:33.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:33.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.547 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:33.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:33.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:33.670 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:33.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:33.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:33.792 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:33.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:33.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:33.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:33.944 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-21T10:33:33.950 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:33:34.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:34.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2789567845' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:33:34.038 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4131565966' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:34.038 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:34.038 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2789567845' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:33:34.038 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4131565966' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:34.038 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:34.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:34.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:34.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:34.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2789567845' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:33:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4131565966' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:33:34.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:34.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:34.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:34.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:34.546 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:34.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:33:34] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:33:34.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:34.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:34.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:34.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:34.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:34.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:34.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:35.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:35.043 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[105288]: pgmap v1997: 109 pgs: 2 creating+peering, 107 active+clean; 608 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2789567845' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[105288]: osdmap e1288: 8 total, 8 up, 8 in 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[101432]: pgmap v1997: 109 pgs: 2 creating+peering, 107 active+clean; 608 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2789567845' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:33:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:34 smithi119 ceph-mon[101432]: osdmap e1288: 8 total, 8 up, 8 in 2024-09-21T10:33:35.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:35.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:35.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:35.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:34 smithi145 ceph-mon[89359]: pgmap v1997: 109 pgs: 2 creating+peering, 107 active+clean; 608 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:34 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:33:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2789567845' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:33:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:34 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:33:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:34 smithi145 ceph-mon[89359]: osdmap e1288: 8 total, 8 up, 8 in 2024-09-21T10:33:35.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:35.412 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:35.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:35.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:35.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:35.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:35.734 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-21T10:33:35.735 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-09-21T10:33:35.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:33:35.737 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:33:35.737 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:33:35.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:35.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:35.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:35.908 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:35.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:35.951 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:hints = 1 2024-09-21T10:33:35.951 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:33:35.952 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object prefix: benchmark_data_smithi119_238957 2024-09-21T10:33:35.952 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:33:35.952 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:33:36.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.052 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:36.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:36.153 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:36.186 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:36.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:35 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:33:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:35 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:33:36.320 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:36.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:35 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:33:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:35 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:33:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:35 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:33:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:35 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:33:36.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:36.456 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:36.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:36.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:36.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:36.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:36.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:36.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:36.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:36.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:36.952 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 1 16 3992 3976 248.453 248.5 0.00991878 0.00400552 2024-09-21T10:33:36.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:37.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:37.056 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:37.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:37.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:36 smithi119 ceph-mon[101432]: pgmap v1999: 109 pgs: 109 active+clean; 608 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:37.152 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:36 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]} : dispatch 2024-09-21T10:33:37.152 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:36 smithi119 ceph-mon[105288]: pgmap v1999: 109 pgs: 109 active+clean; 608 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:37.152 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:36 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]} : dispatch 2024-09-21T10:33:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.153 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:37.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:37.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.290 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:37.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:36 smithi145 ceph-mon[89359]: pgmap v1999: 109 pgs: 109 active+clean; 608 KiB data, 3.3 GiB used, 712 GiB / 715 GiB avail 2024-09-21T10:33:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:36 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]} : dispatch 2024-09-21T10:33:37.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:37.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:37.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:37.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:37.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:37.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:37.685 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:37.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:37.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:37.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:37.952 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 2 16 7345 7329 228.999 209.562 0.00506392 0.00435971 2024-09-21T10:33:37.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:37.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:38.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:38.091 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:38.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:38.224 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-21T10:33:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:38 smithi145 ceph-mon[89359]: osdmap e1289: 8 total, 8 up, 8 in 2024-09-21T10:33:38.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-21T10:33:38.325 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:38 smithi119 ceph-mon[105288]: osdmap e1289: 8 total, 8 up, 8 in 2024-09-21T10:33:38.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2024-09-21T10:33:38.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:38 smithi119 ceph-mon[101432]: osdmap e1289: 8 total, 8 up, 8 in 2024-09-21T10:33:38.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:38.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:38.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:38.492 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:38.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:33:38.575 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:33:38.575 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:33:38.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.575 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:9272 47.6 47:6c57d7cc:::benchmark_data_smithi119_238957_object9271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1290)", 2024-09-21T10:33:38.575 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:33:38.520416+0000", 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.010549892999999999, 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010585671, 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 9272 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.520416+0000", 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.520416+0000", 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.520418+0000", 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4410000000000001e-06 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.520475+0000", 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6929999999999999e-05 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.520484+0000", 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1980000000000007e-06 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.520491+0000", 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1189999999999999e-06 2024-09-21T10:33:38.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.521655+0000", 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001163704 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.521783+0000", 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012849399999999999 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.521939+0000", 2024-09-21T10:33:38.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015546900000000001 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.522835+0000", 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00089656000000000002 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.524612+0000", 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017768809999999999 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:33:38.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:9292 47.6 47:6e3e866e:::benchmark_data_smithi119_238957_object9291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1290)", 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:33:38.528604+0000", 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0023625669999999999, 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00248628, 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 9292 2024-09-21T10:33:38.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528604+0000", 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528604+0000", 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:33:38.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528606+0000", 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4770000000000002e-06 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528662+0000", 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6280999999999999e-05 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528686+0000", 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3385999999999999e-05 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528696+0000", 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0142999999999999e-05 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528764+0000", 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8308999999999999e-05 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.528979+0000", 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021474999999999999 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.529129+0000", 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014983800000000001 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:33:38.529930+0000", 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080160699999999997 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:33:38.585 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 2 2024-09-21T10:33:38.586 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:33:38.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:38.624 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:38.726 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:38.754 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:38.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:38.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:38.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:38.952 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 3 16 10464 10448 217.635 194.938 0.00503209 0.00458693 2024-09-21T10:33:39.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:39.050 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:39.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:39.103 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:39 smithi119 ceph-mon[105288]: pgmap v2001: 109 pgs: 109 active+clean; 49 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s wr, 98 op/s 2024-09-21T10:33:39.103 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:39 smithi119 ceph-mon[105288]: osdmap e1290: 8 total, 8 up, 8 in 2024-09-21T10:33:39.104 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/455206627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:39.104 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:39 smithi119 ceph-mon[101432]: pgmap v2001: 109 pgs: 109 active+clean; 49 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s wr, 98 op/s 2024-09-21T10:33:39.104 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:39 smithi119 ceph-mon[101432]: osdmap e1290: 8 total, 8 up, 8 in 2024-09-21T10:33:39.104 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/455206627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:39.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:39.184 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:39.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:39 smithi145 ceph-mon[89359]: pgmap v2001: 109 pgs: 109 active+clean; 49 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 MiB/s wr, 98 op/s 2024-09-21T10:33:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:39 smithi145 ceph-mon[89359]: osdmap e1290: 8 total, 8 up, 8 in 2024-09-21T10:33:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/455206627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:33:39.306 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:33:39.307 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:33:39.308 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:33:39.308 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:33:39.308 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:33:39.308 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:33:39.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:39.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:39.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:39.556 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:39.589 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:39.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:39.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:39.826 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:39.858 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:39.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 4 16 13794 13778 215.253 208.125 0.00648781 0.00463897 2024-09-21T10:33:39.961 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:39.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:39.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:40.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:40.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:33:40.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:40.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:40.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:33:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:33:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:40 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:33:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:40 smithi145 ceph-mon[89359]: 1.12 scrub ok 2024-09-21T10:33:40.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:40 smithi145 ceph-mon[89359]: 1.f scrub starts 2024-09-21T10:33:40.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:40 smithi145 ceph-mon[89359]: 1.f scrub ok 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[101432]: 1.12 scrub ok 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[101432]: 1.f scrub starts 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[101432]: 1.f scrub ok 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[105288]: 1.12 scrub ok 2024-09-21T10:33:40.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[105288]: 1.f scrub starts 2024-09-21T10:33:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:40 smithi119 ceph-mon[105288]: 1.f scrub ok 2024-09-21T10:33:40.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:40.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:40.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.558 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:40.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:40.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:40.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:40.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:40.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 5 16 16742 16726 209.035 184.25 0.00402779 0.00477814 2024-09-21T10:33:40.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:40.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:41.029 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:41.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:41.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:41.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:41.238 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:41 smithi145 ceph-mon[89359]: pgmap v2003: 109 pgs: 1 peering, 108 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:33:41.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:41 smithi119 ceph-mon[101432]: pgmap v2003: 109 pgs: 1 peering, 108 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:33:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:41 smithi119 ceph-mon[105288]: pgmap v2003: 109 pgs: 1 peering, 108 active+clean; 324 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:33:41.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:41.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:41.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:41.611 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:41.638 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:41.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:41.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:41.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:41.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:41.935 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:41.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:41.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 6 15 19542 19527 203.371 175.062 0.00448757 0.0049109 2024-09-21T10:33:42.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:42.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:42.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:42.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:42.344 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:42.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:42.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.485 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:42.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:42.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:42.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:42.718 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:42.750 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:42.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:42.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:42.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:42.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:42.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 7 16 21855 21839 194.959 144.5 0.00938672 0.00512377 2024-09-21T10:33:42.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:43.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:43.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:33:43.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:43.148 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:43.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:43.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:43 smithi119 ceph-mon[101432]: pgmap v2004: 109 pgs: 1 peering, 108 active+clean; 694 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:33:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:43 smithi119 ceph-mon[105288]: pgmap v2004: 109 pgs: 1 peering, 108 active+clean; 694 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:33:43.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:43.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:43.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:43.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:43 smithi145 ceph-mon[89359]: pgmap v2004: 109 pgs: 1 peering, 108 active+clean; 694 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-21T10:33:43.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:43.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:43.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:43.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:43.899 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:43.932 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:43.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:43.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 8 16 23894 23878 186.518 127.438 0.0021699 0.00534074 2024-09-21T10:33:44.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:44.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:44.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:44.196 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:44.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:44.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:44.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.459 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:44.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:44.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:44.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:44.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:44.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:44.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:33:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:33:44] "GET /metrics HTTP/1.1" 200 36823 "" "Prometheus/2.51.0" 2024-09-21T10:33:44.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 9 16 25673 25657 178.148 111.188 0.00771728 0.0056075 2024-09-21T10:33:44.957 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:44.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:44.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:45.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:45.115 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:45.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:45.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:45.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:45.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:45 smithi119 ceph-mon[101432]: pgmap v2005: 109 pgs: 109 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 142 MiB/s wr, 2.28k op/s 2024-09-21T10:33:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:45 smithi119 ceph-mon[105288]: pgmap v2005: 109 pgs: 109 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 142 MiB/s wr, 2.28k op/s 2024-09-21T10:33:45.374 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:45.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:45.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:45.498 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:45 smithi145 ceph-mon[89359]: pgmap v2005: 109 pgs: 109 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 142 MiB/s wr, 2.28k op/s 2024-09-21T10:33:45.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:45.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:45.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:45.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:45.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:45.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:45.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:45.902 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:45.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:45.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 10 16 27681 27665 172.882 125.5 0.00402343 0.00577798 2024-09-21T10:33:46.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:46.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:46.137 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:46.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:46.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:46.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:46.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:46.418 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:46.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:46.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:46.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:46.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:46.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:46.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:46.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:46.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:46.935 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:46.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:46.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 11 16 29118 29102 165.33 89.8125 0.00928167 0.00604246 2024-09-21T10:33:47.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:47.071 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:47.173 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:47.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:47.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:47.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:47.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:47 smithi119 ceph-mon[101432]: pgmap v2006: 109 pgs: 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 155 MiB/s wr, 2.47k op/s 2024-09-21T10:33:47.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:47 smithi119 ceph-mon[105288]: pgmap v2006: 109 pgs: 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 155 MiB/s wr, 2.47k op/s 2024-09-21T10:33:47.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:47.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:47 smithi145 ceph-mon[89359]: pgmap v2006: 109 pgs: 109 active+clean; 1.3 GiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 155 MiB/s wr, 2.47k op/s 2024-09-21T10:33:47.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:47.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:47.711 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:47.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:47.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:47.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:47.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:47.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 12 16 31255 31239 162.682 133.562 0.00560188 0.00614174 2024-09-21T10:33:47.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:48.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:48.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:48.146 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:48.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:48.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:48.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:48.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:48.511 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:33:48.513 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:33:48.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:48.558 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:48.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:48.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:48.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:48.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:48.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:48.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:48.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:48.954 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 13 16 33365 33349 160.311 131.875 0.00957674 0.00623271 2024-09-21T10:33:48.964 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:48.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:48 smithi145 ceph-mon[89359]: pgmap v2007: 109 pgs: 109 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-21T10:33:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:48 smithi145 ceph-mon[89359]: from='client.36522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:33:49.066 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:49.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:49.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[101432]: pgmap v2007: 109 pgs: 109 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-21T10:33:49.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:49.114 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:49.115 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[101432]: from='client.36522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:33:49.115 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[105288]: pgmap v2007: 109 pgs: 109 active+clean; 1.5 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 154 MiB/s wr, 2.46k op/s 2024-09-21T10:33:49.115 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:49.115 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:33:49.115 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:48 smithi119 ceph-mon[105288]: from='client.36522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:33:49.206 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.240 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:49.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:49.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.380 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:49.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:49.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:49.622 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:49.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:49.687 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2007,"stamp":"2024-09-21T10:33:47.650165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1661540301,"num_objects":25370,"num_object_clones":0,"num_object_copies":76110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25407,"num_write_kb":1622047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":919493,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27502,"ondisk_log_size":27502,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":332,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8159832,"kb_used_data":5285080,"kb_used_omap":731,"kb_used_meta":2873892,"kb_avail":741801384,"statfs":{"total":767960285184,"available":759604617216,"internally_reserved":0,"allocated":5411921920,"data_stored":5369361351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":749144,"internal_metadata":2942865832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1610651437,"num_objects":24576,"num_object_clones":0,"num_object_copies":73728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24576,"num_whiteouts":0,"num_read":-14,"num_read_kb":-34,"num_write":24570,"num_write_kb":1572854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003077"},"pg_stats":[{"pgid":"47.f","version":"1290'1841","reported_seq":1853,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:47.330967+0000","last_change":"2024-09-21T10:33:28.930280+0000","last_active":"2024-09-21T10:33:47.330967+0000","last_peered":"2024-09-21T10:33:47.330967+0000","last_clean":"2024-09-21T10:33:47.330967+0000","last_became_active":"2024-09-21T10:33:28.928885+0000","last_became_peered":"2024-09-21T10:33:28.928885+0000","last_unstale":"2024-09-21T10:33:47.330967+0000","last_undegraded":"2024-09-21T10:33:47.330967+0000","last_fullsized":"2024-09-21T10:33:47.330967+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1841,"log_dups_size":0,"ondisk_log_size":1841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:20.937916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":120651776,"num_objects":1841,"num_object_clones":0,"num_object_copies":5523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1841,"num_write_kb":117824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.e","version":"1290'1367","reported_seq":1380,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:43.408342+0000","last_change":"2024-09-21T10:33:28.932881+0000","last_active":"2024-09-21T10:33:43.408342+0000","last_peered":"2024-09-21T10:33:43.408342+0000","last_clean":"2024-09-21T10:33:43.408342+0000","last_became_active":"2024-09-21T10:33:28.932543+0000","last_became_peered":"2024-09-21T10:33:28.932543+0000","last_unstale":"2024-09-21T10:33:43.408342+0000","last_undegraded":"2024-09-21T10:33:43.408342+0000","last_fullsized":"2024-09-21T10:33:43.408342+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1367,"log_dups_size":0,"ondisk_log_size":1367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:30:10.930926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89587712,"num_objects":1367,"num_object_clones":0,"num_object_copies":4101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1367,"num_write_kb":87488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.d","version":"1290'1387","reported_seq":1400,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:42.953980+0000","last_change":"2024-09-21T10:33:29.381012+0000","last_active":"2024-09-21T10:33:42.953980+0000","last_peered":"2024-09-21T10:33:42.953980+0000","last_clean":"2024-09-21T10:33:42.953980+0000","last_became_active":"2024-09-21T10:33:29.380701+0000","last_became_peered":"2024-09-21T10:33:29.380701+0000","last_unstale":"2024-09-21T10:33:42.953980+0000","last_undegraded":"2024-09-21T10:33:42.953980+0000","last_fullsized":"2024-09-21T10:33:42.953980+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1387,"log_dups_size":0,"ondisk_log_size":1387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:26.203462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90898432,"num_objects":1387,"num_object_clones":0,"num_object_copies":4161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1387,"num_write_kb":88768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"47.c","version":"1290'1700","reported_seq":1721,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:46.043703+0000","last_change":"2024-09-21T10:33:28.929528+0000","last_active":"2024-09-21T10:33:46.043703+0000","last_peered":"2024-09-21T10:33:46.043703+0000","last_clean":"2024-09-21T10:33:46.043703+0000","last_became_active":"2024-09-21T10:33:28.929202+0000","last_became_peered":"2024-09-21T10:33:28.929202+0000","last_unstale":"2024-09-21T10:33:46.043703+0000","last_undegraded":"2024-09-21T10:33:46.043703+0000","last_fullsized":"2024-09-21T10:33:46.043703+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1700,"log_dups_size":0,"ondisk_log_size":1700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:14:17.406156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111411200,"num_objects":1700,"num_object_clones":0,"num_object_copies":5100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1700,"num_write_kb":108800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.b","version":"1290'1449","reported_seq":1462,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:43.396999+0000","last_change":"2024-09-21T10:33:28.930894+0000","last_active":"2024-09-21T10:33:43.396999+0000","last_peered":"2024-09-21T10:33:43.396999+0000","last_clean":"2024-09-21T10:33:43.396999+0000","last_became_active":"2024-09-21T10:33:28.930610+0000","last_became_peered":"2024-09-21T10:33:28.930610+0000","last_unstale":"2024-09-21T10:33:43.396999+0000","last_undegraded":"2024-09-21T10:33:43.396999+0000","last_fullsized":"2024-09-21T10:33:43.396999+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:14.319122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94961664,"num_objects":1449,"num_object_clones":0,"num_object_copies":4347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1449,"num_write_kb":92736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.a","version":"1290'1710","reported_seq":1731,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:46.055641+0000","last_change":"2024-09-21T10:33:29.379333+0000","last_active":"2024-09-21T10:33:46.055641+0000","last_peered":"2024-09-21T10:33:46.055641+0000","last_clean":"2024-09-21T10:33:46.055641+0000","last_became_active":"2024-09-21T10:33:29.379043+0000","last_became_peered":"2024-09-21T10:33:29.379043+0000","last_unstale":"2024-09-21T10:33:46.055641+0000","last_undegraded":"2024-09-21T10:33:46.055641+0000","last_fullsized":"2024-09-21T10:33:46.055641+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1710,"log_dups_size":0,"ondisk_log_size":1710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:52.910128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112066560,"num_objects":1710,"num_object_clones":0,"num_object_copies":5130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1710,"num_write_kb":109440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.9","version":"1290'1830","reported_seq":1843,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:47.321984+0000","last_change":"2024-09-21T10:33:28.929791+0000","last_active":"2024-09-21T10:33:47.321984+0000","last_peered":"2024-09-21T10:33:47.321984+0000","last_clean":"2024-09-21T10:33:47.321984+0000","last_became_active":"2024-09-21T10:33:28.929537+0000","last_became_peered":"2024-09-21T10:33:28.929537+0000","last_unstale":"2024-09-21T10:33:47.321984+0000","last_undegraded":"2024-09-21T10:33:47.321984+0000","last_fullsized":"2024-09-21T10:33:47.321984+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1830,"log_dups_size":0,"ondisk_log_size":1830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:50:55.620749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119930880,"num_objects":1830,"num_object_clones":0,"num_object_copies":5490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1830,"num_write_kb":117120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.8","version":"1290'1438","reported_seq":1451,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:42.967617+0000","last_change":"2024-09-21T10:33:28.930109+0000","last_active":"2024-09-21T10:33:42.967617+0000","last_peered":"2024-09-21T10:33:42.967617+0000","last_clean":"2024-09-21T10:33:42.967617+0000","last_became_active":"2024-09-21T10:33:28.929786+0000","last_became_peered":"2024-09-21T10:33:28.929786+0000","last_unstale":"2024-09-21T10:33:42.967617+0000","last_undegraded":"2024-09-21T10:33:42.967617+0000","last_fullsized":"2024-09-21T10:33:42.967617+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1438,"log_dups_size":0,"ondisk_log_size":1438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:37:14.621745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94240768,"num_objects":1438,"num_object_clones":0,"num_object_copies":4314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1438,"num_write_kb":92032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"47.7","version":"1290'1758","reported_seq":1779,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:46.052839+0000","last_change":"2024-09-21T10:33:29.379252+0000","last_active":"2024-09-21T10:33:46.052839+0000","last_peered":"2024-09-21T10:33:46.052839+0000","last_clean":"2024-09-21T10:33:46.052839+0000","last_became_active":"2024-09-21T10:33:29.378969+0000","last_became_peered":"2024-09-21T10:33:29.378969+0000","last_unstale":"2024-09-21T10:33:46.052839+0000","last_undegraded":"2024-09-21T10:33:46.052839+0000","last_fullsized":"2024-09-21T10:33:46.052839+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1758,"log_dups_size":0,"ondisk_log_size":1758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:37.174282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115212288,"num_objects":1758,"num_object_clones":0,"num_object_copies":5274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1758,"num_write_kb":112512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.6","version":"1290'1473","reported_seq":1494,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:43.721558+0000","last_change":"2024-09-21T10:33:29.379451+0000","last_active":"2024-09-21T10:33:43.721558+0000","last_peered":"2024-09-21T10:33:43.721558+0000","last_clean":"2024-09-21T10:33:43.721558+0000","last_became_active":"2024-09-21T10:33:29.379157+0000","last_became_peered":"2024-09-21T10:33:29.379157+0000","last_unstale":"2024-09-21T10:33:43.721558+0000","last_undegraded":"2024-09-21T10:33:43.721558+0000","last_fullsized":"2024-09-21T10:33:43.721558+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1473,"log_dups_size":0,"ondisk_log_size":1473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:18.919160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96534528,"num_objects":1473,"num_object_clones":0,"num_object_copies":4419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1473,"num_write_kb":94272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.5","version":"1290'1721","reported_seq":1742,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:45.062280+0000","last_change":"2024-09-21T10:33:28.930357+0000","last_active":"2024-09-21T10:33:45.062280+0000","last_peered":"2024-09-21T10:33:45.062280+0000","last_clean":"2024-09-21T10:33:45.062280+0000","last_became_active":"2024-09-21T10:33:28.930060+0000","last_became_peered":"2024-09-21T10:33:28.930060+0000","last_unstale":"2024-09-21T10:33:45.062280+0000","last_undegraded":"2024-09-21T10:33:45.062280+0000","last_fullsized":"2024-09-21T10:33:45.062280+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1721,"log_dups_size":0,"ondisk_log_size":1721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:46.160210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112787456,"num_objects":1721,"num_object_clones":0,"num_object_copies":5163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1721,"num_write_kb":110144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.4","version":"1290'1641","reported_seq":1662,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:45.071015+0000","last_change":"2024-09-21T10:33:28.930569+0000","last_active":"2024-09-21T10:33:45.071015+0000","last_peered":"2024-09-21T10:33:45.071015+0000","last_clean":"2024-09-21T10:33:45.071015+0000","last_became_active":"2024-09-21T10:33:28.930298+0000","last_became_peered":"2024-09-21T10:33:28.930298+0000","last_unstale":"2024-09-21T10:33:45.071015+0000","last_undegraded":"2024-09-21T10:33:45.071015+0000","last_fullsized":"2024-09-21T10:33:45.071015+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1641,"log_dups_size":0,"ondisk_log_size":1641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:27.498545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107544576,"num_objects":1641,"num_object_clones":0,"num_object_copies":4923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1641,"num_write_kb":105024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.3","version":"1290'1466","reported_seq":1487,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:43.708273+0000","last_change":"2024-09-21T10:33:29.380749+0000","last_active":"2024-09-21T10:33:43.708273+0000","last_peered":"2024-09-21T10:33:43.708273+0000","last_clean":"2024-09-21T10:33:43.708273+0000","last_became_active":"2024-09-21T10:33:29.379816+0000","last_became_peered":"2024-09-21T10:33:29.379816+0000","last_unstale":"2024-09-21T10:33:43.708273+0000","last_undegraded":"2024-09-21T10:33:43.708273+0000","last_fullsized":"2024-09-21T10:33:43.708273+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:51.982918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96075776,"num_objects":1466,"num_object_clones":0,"num_object_copies":4398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1466,"num_write_kb":93824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.2","version":"1290'1701","reported_seq":1722,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:46.055074+0000","last_change":"2024-09-21T10:33:28.929547+0000","last_active":"2024-09-21T10:33:46.055074+0000","last_peered":"2024-09-21T10:33:46.055074+0000","last_clean":"2024-09-21T10:33:46.055074+0000","last_became_active":"2024-09-21T10:33:28.929189+0000","last_became_peered":"2024-09-21T10:33:28.929189+0000","last_unstale":"2024-09-21T10:33:46.055074+0000","last_undegraded":"2024-09-21T10:33:46.055074+0000","last_fullsized":"2024-09-21T10:33:46.055074+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1701,"log_dups_size":0,"ondisk_log_size":1701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:43:57.094931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111476736,"num_objects":1701,"num_object_clones":0,"num_object_copies":5103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1701,"num_write_kb":108864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.1","version":"1290'1409","reported_seq":1422,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:43.491407+0000","last_change":"2024-09-21T10:33:28.930520+0000","last_active":"2024-09-21T10:33:43.491407+0000","last_peered":"2024-09-21T10:33:43.491407+0000","last_clean":"2024-09-21T10:33:43.491407+0000","last_became_active":"2024-09-21T10:33:28.930242+0000","last_became_peered":"2024-09-21T10:33:28.930242+0000","last_unstale":"2024-09-21T10:33:43.491407+0000","last_undegraded":"2024-09-21T10:33:43.491407+0000","last_fullsized":"2024-09-21T10:33:43.491407+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1409,"log_dups_size":0,"ondisk_log_size":1409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:39.781296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92340224,"num_objects":1409,"num_object_clones":0,"num_object_copies":4227,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1409,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1409,"num_write_kb":90176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"47.0","version":"1290'1453","reported_seq":1466,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:43.491480+0000","last_change":"2024-09-21T10:33:28.928618+0000","last_active":"2024-09-21T10:33:43.491480+0000","last_peered":"2024-09-21T10:33:43.491480+0000","last_clean":"2024-09-21T10:33:43.491480+0000","last_became_active":"2024-09-21T10:33:28.928317+0000","last_became_peered":"2024-09-21T10:33:28.928317+0000","last_unstale":"2024-09-21T10:33:43.491480+0000","last_undegraded":"2024-09-21T10:33:43.491480+0000","last_fullsized":"2024-09-21T10:33:43.491480+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:16:53.553956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95223808,"num_objects":1453,"num_object_clones":0,"num_object_copies":4359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1453,"num_write_kb":92992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":46,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008400+0000","last_change":"2024-09-21T10:30:09.420642+0000","last_active":"2024-09-21T10:33:37.008400+0000","last_peered":"2024-09-21T10:33:37.008400+0000","last_clean":"2024-09-21T10:33:37.008400+0000","last_became_active":"2024-09-21T10:30:09.407144+0000","last_became_peered":"2024-09-21T10:30:09.407144+0000","last_unstale":"2024-09-21T10:33:37.008400+0000","last_undegraded":"2024-09-21T10:33:37.008400+0000","last_fullsized":"2024-09-21T10:33:37.008400+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:49.017939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1267,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005605+0000","last_change":"2024-09-21T10:28:37.587114+0000","last_active":"2024-09-21T10:33:37.005605+0000","last_peered":"2024-09-21T10:33:37.005605+0000","last_clean":"2024-09-21T10:33:37.005605+0000","last_became_active":"2024-09-21T10:28:37.586686+0000","last_became_peered":"2024-09-21T10:28:37.586686+0000","last_unstale":"2024-09-21T10:33:37.005605+0000","last_undegraded":"2024-09-21T10:33:37.005605+0000","last_fullsized":"2024-09-21T10:33:37.005605+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:41:45.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1898,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021620+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:33:38.021620+0000","last_peered":"2024-09-21T10:33:38.021620+0000","last_clean":"2024-09-21T10:33:38.021620+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:33:38.021620+0000","last_undegraded":"2024-09-21T10:33:38.021620+0000","last_fullsized":"2024-09-21T10:33:38.021620+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1917,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.022121+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:33:38.022121+0000","last_peered":"2024-09-21T10:33:38.022121+0000","last_clean":"2024-09-21T10:33:38.022121+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:33:38.022121+0000","last_undegraded":"2024-09-21T10:33:38.022121+0000","last_fullsized":"2024-09-21T10:33:38.022121+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":44,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008358+0000","last_change":"2024-09-21T10:30:09.427552+0000","last_active":"2024-09-21T10:33:37.008358+0000","last_peered":"2024-09-21T10:33:37.008358+0000","last_clean":"2024-09-21T10:33:37.008358+0000","last_became_active":"2024-09-21T10:30:09.412897+0000","last_became_peered":"2024-09-21T10:30:09.412897+0000","last_unstale":"2024-09-21T10:33:37.008358+0000","last_undegraded":"2024-09-21T10:33:37.008358+0000","last_fullsized":"2024-09-21T10:33:37.008358+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:49.300103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1253,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005716+0000","last_change":"2024-09-21T10:27:19.333245+0000","last_active":"2024-09-21T10:33:37.005716+0000","last_peered":"2024-09-21T10:33:37.005716+0000","last_clean":"2024-09-21T10:33:37.005716+0000","last_became_active":"2024-09-21T10:27:19.332965+0000","last_became_peered":"2024-09-21T10:27:19.332965+0000","last_unstale":"2024-09-21T10:33:37.005716+0000","last_undegraded":"2024-09-21T10:33:37.005716+0000","last_fullsized":"2024-09-21T10:33:37.005716+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:04.828777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":44,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008253+0000","last_change":"2024-09-21T10:30:09.425518+0000","last_active":"2024-09-21T10:33:37.008253+0000","last_peered":"2024-09-21T10:33:37.008253+0000","last_clean":"2024-09-21T10:33:37.008253+0000","last_became_active":"2024-09-21T10:30:09.412102+0000","last_became_peered":"2024-09-21T10:30:09.412102+0000","last_unstale":"2024-09-21T10:33:37.008253+0000","last_undegraded":"2024-09-21T10:33:37.008253+0000","last_fullsized":"2024-09-21T10:33:37.008253+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:44.848838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1263,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.024141+0000","last_change":"2024-09-21T10:33:38.024141+0000","last_active":"2024-09-21T10:33:38.024141+0000","last_peered":"2024-09-21T10:33:38.024141+0000","last_clean":"2024-09-21T10:33:38.024141+0000","last_became_active":"2024-09-21T10:33:38.023837+0000","last_became_peered":"2024-09-21T10:33:38.023837+0000","last_unstale":"2024-09-21T10:33:38.024141+0000","last_undegraded":"2024-09-21T10:33:38.024141+0000","last_fullsized":"2024-09-21T10:33:38.024141+0000","mapping_epoch":1289,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:11.353781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":55,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005716+0000","last_change":"2024-09-21T10:28:56.813482+0000","last_active":"2024-09-21T10:33:37.005716+0000","last_peered":"2024-09-21T10:33:37.005716+0000","last_clean":"2024-09-21T10:33:37.005716+0000","last_became_active":"2024-09-21T10:28:56.813103+0000","last_became_peered":"2024-09-21T10:28:56.813103+0000","last_unstale":"2024-09-21T10:33:37.005716+0000","last_undegraded":"2024-09-21T10:33:37.005716+0000","last_fullsized":"2024-09-21T10:33:37.005716+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:20.438161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1841,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023727+0000","last_change":"2024-09-21T10:29:04.650051+0000","last_active":"2024-09-21T10:33:38.023727+0000","last_peered":"2024-09-21T10:33:38.023727+0000","last_clean":"2024-09-21T10:33:38.023727+0000","last_became_active":"2024-09-21T10:29:04.649705+0000","last_became_peered":"2024-09-21T10:29:04.649705+0000","last_unstale":"2024-09-21T10:33:38.023727+0000","last_undegraded":"2024-09-21T10:33:38.023727+0000","last_fullsized":"2024-09-21T10:33:38.023727+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:45.854900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1228,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005784+0000","last_change":"2024-09-21T10:27:19.325859+0000","last_active":"2024-09-21T10:33:37.005784+0000","last_peered":"2024-09-21T10:33:37.005784+0000","last_clean":"2024-09-21T10:33:37.005784+0000","last_became_active":"2024-09-21T10:27:19.325540+0000","last_became_peered":"2024-09-21T10:27:19.325540+0000","last_unstale":"2024-09-21T10:33:37.005784+0000","last_undegraded":"2024-09-21T10:33:37.005784+0000","last_fullsized":"2024-09-21T10:33:37.005784+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:11.324592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1899,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021701+0000","last_change":"2024-09-21T10:28:37.333977+0000","last_active":"2024-09-21T10:33:38.021701+0000","last_peered":"2024-09-21T10:33:38.021701+0000","last_clean":"2024-09-21T10:33:38.021701+0000","last_became_active":"2024-09-21T10:28:37.333630+0000","last_became_peered":"2024-09-21T10:28:37.333630+0000","last_unstale":"2024-09-21T10:33:38.021701+0000","last_undegraded":"2024-09-21T10:33:38.021701+0000","last_fullsized":"2024-09-21T10:33:38.021701+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:07.637543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1998,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023554+0000","last_change":"2024-09-21T10:27:19.331712+0000","last_active":"2024-09-21T10:33:38.023554+0000","last_peered":"2024-09-21T10:33:38.023554+0000","last_clean":"2024-09-21T10:33:38.023554+0000","last_became_active":"2024-09-21T10:27:19.330959+0000","last_became_peered":"2024-09-21T10:27:19.330959+0000","last_unstale":"2024-09-21T10:33:38.023554+0000","last_undegraded":"2024-09-21T10:33:38.023554+0000","last_fullsized":"2024-09-21T10:33:38.023554+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:46.389484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2013,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.019455+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:33:38.019455+0000","last_peered":"2024-09-21T10:33:38.019455+0000","last_clean":"2024-09-21T10:33:38.019455+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:33:38.019455+0000","last_undegraded":"2024-09-21T10:33:38.019455+0000","last_fullsized":"2024-09-21T10:33:38.019455+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1925,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.020986+0000","last_change":"2024-09-21T10:28:56.819014+0000","last_active":"2024-09-21T10:33:38.020986+0000","last_peered":"2024-09-21T10:33:38.020986+0000","last_clean":"2024-09-21T10:33:38.020986+0000","last_became_active":"2024-09-21T10:28:56.818640+0000","last_became_peered":"2024-09-21T10:28:56.818640+0000","last_unstale":"2024-09-21T10:33:38.020986+0000","last_undegraded":"2024-09-21T10:33:38.020986+0000","last_fullsized":"2024-09-21T10:33:38.020986+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:49.759102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1248,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.009271+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:33:37.009271+0000","last_peered":"2024-09-21T10:33:37.009271+0000","last_clean":"2024-09-21T10:33:37.009271+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:33:37.009271+0000","last_undegraded":"2024-09-21T10:33:37.009271+0000","last_fullsized":"2024-09-21T10:33:37.009271+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1911,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021302+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:33:38.021302+0000","last_peered":"2024-09-21T10:33:38.021302+0000","last_clean":"2024-09-21T10:33:38.021302+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:33:38.021302+0000","last_undegraded":"2024-09-21T10:33:38.021302+0000","last_fullsized":"2024-09-21T10:33:38.021302+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2024,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.019064+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:33:38.019064+0000","last_peered":"2024-09-21T10:33:38.019064+0000","last_clean":"2024-09-21T10:33:38.019064+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:33:38.019064+0000","last_undegraded":"2024-09-21T10:33:38.019064+0000","last_fullsized":"2024-09-21T10:33:38.019064+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":26,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.015611+0000","last_change":"2024-09-21T10:33:20.897168+0000","last_active":"2024-09-21T10:33:38.015611+0000","last_peered":"2024-09-21T10:33:38.015611+0000","last_clean":"2024-09-21T10:33:38.015611+0000","last_became_active":"2024-09-21T10:33:20.896714+0000","last_became_peered":"2024-09-21T10:33:20.896714+0000","last_unstale":"2024-09-21T10:33:38.015611+0000","last_undegraded":"2024-09-21T10:33:38.015611+0000","last_fullsized":"2024-09-21T10:33:38.015611+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:34:21.487509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1894,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.022355+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:33:38.022355+0000","last_peered":"2024-09-21T10:33:38.022355+0000","last_clean":"2024-09-21T10:33:38.022355+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:33:38.022355+0000","last_undegraded":"2024-09-21T10:33:38.022355+0000","last_fullsized":"2024-09-21T10:33:38.022355+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":40,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008299+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:33:37.008299+0000","last_peered":"2024-09-21T10:33:37.008299+0000","last_clean":"2024-09-21T10:33:37.008299+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:33:37.008299+0000","last_undegraded":"2024-09-21T10:33:37.008299+0000","last_fullsized":"2024-09-21T10:33:37.008299+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1542,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023958+0000","last_change":"2024-09-21T10:27:19.327772+0000","last_active":"2024-09-21T10:33:38.023958+0000","last_peered":"2024-09-21T10:33:38.023958+0000","last_clean":"2024-09-21T10:33:38.023958+0000","last_became_active":"2024-09-21T10:27:19.327467+0000","last_became_peered":"2024-09-21T10:27:19.327467+0000","last_unstale":"2024-09-21T10:33:38.023958+0000","last_undegraded":"2024-09-21T10:33:38.023958+0000","last_fullsized":"2024-09-21T10:33:38.023958+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:28:38.639278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1997,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.024703+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:33:38.024703+0000","last_peered":"2024-09-21T10:33:38.024703+0000","last_clean":"2024-09-21T10:33:38.024703+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:33:38.024703+0000","last_undegraded":"2024-09-21T10:33:38.024703+0000","last_fullsized":"2024-09-21T10:33:38.024703+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1263,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005932+0000","last_change":"2024-09-21T10:27:19.330694+0000","last_active":"2024-09-21T10:33:37.005932+0000","last_peered":"2024-09-21T10:33:37.005932+0000","last_clean":"2024-09-21T10:33:37.005932+0000","last_became_active":"2024-09-21T10:27:19.329455+0000","last_became_peered":"2024-09-21T10:27:19.329455+0000","last_unstale":"2024-09-21T10:33:37.005932+0000","last_undegraded":"2024-09-21T10:33:37.005932+0000","last_fullsized":"2024-09-21T10:33:37.005932+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:27.883673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1257,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.006099+0000","last_change":"2024-09-21T10:27:19.371314+0000","last_active":"2024-09-21T10:33:37.006099+0000","last_peered":"2024-09-21T10:33:37.006099+0000","last_clean":"2024-09-21T10:33:37.006099+0000","last_became_active":"2024-09-21T10:27:19.370747+0000","last_became_peered":"2024-09-21T10:27:19.370747+0000","last_unstale":"2024-09-21T10:33:37.006099+0000","last_undegraded":"2024-09-21T10:33:37.006099+0000","last_fullsized":"2024-09-21T10:33:37.006099+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:00.522037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":152,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021355+0000","last_change":"2024-09-21T10:30:09.413781+0000","last_active":"2024-09-21T10:33:38.021355+0000","last_peered":"2024-09-21T10:33:38.021355+0000","last_clean":"2024-09-21T10:33:38.021355+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:33:38.021355+0000","last_undegraded":"2024-09-21T10:33:38.021355+0000","last_fullsized":"2024-09-21T10:33:38.021355+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:34:20.045835+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1248,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005605+0000","last_change":"2024-09-21T10:30:09.423426+0000","last_active":"2024-09-21T10:33:37.005605+0000","last_peered":"2024-09-21T10:33:37.005605+0000","last_clean":"2024-09-21T10:33:37.005605+0000","last_became_active":"2024-09-21T10:30:09.422262+0000","last_became_peered":"2024-09-21T10:30:09.422262+0000","last_unstale":"2024-09-21T10:33:37.005605+0000","last_undegraded":"2024-09-21T10:33:37.005605+0000","last_fullsized":"2024-09-21T10:33:37.005605+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:25.701866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1244,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.009231+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:33:37.009231+0000","last_peered":"2024-09-21T10:33:37.009231+0000","last_clean":"2024-09-21T10:33:37.009231+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:33:37.009231+0000","last_undegraded":"2024-09-21T10:33:37.009231+0000","last_fullsized":"2024-09-21T10:33:37.009231+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":133,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.025036+0000","last_change":"2024-09-21T10:29:04.342115+0000","last_active":"2024-09-21T10:33:38.025036+0000","last_peered":"2024-09-21T10:33:38.025036+0000","last_clean":"2024-09-21T10:33:38.025036+0000","last_became_active":"2024-09-21T10:29:04.340202+0000","last_became_peered":"2024-09-21T10:29:04.340202+0000","last_unstale":"2024-09-21T10:33:38.025036+0000","last_undegraded":"2024-09-21T10:33:38.025036+0000","last_fullsized":"2024-09-21T10:33:38.025036+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:52.282454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":95,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.006343+0000","last_change":"2024-09-21T10:33:25.422314+0000","last_active":"2024-09-21T10:33:37.006343+0000","last_peered":"2024-09-21T10:33:37.006343+0000","last_clean":"2024-09-21T10:33:37.006343+0000","last_became_active":"2024-09-21T10:29:56.991594+0000","last_became_peered":"2024-09-21T10:29:56.991594+0000","last_unstale":"2024-09-21T10:33:37.006343+0000","last_undegraded":"2024-09-21T10:33:37.006343+0000","last_fullsized":"2024-09-21T10:33:37.006343+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.422256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.422256+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:08.909618+0000","scrub_duration":0.0019284840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1909,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021097+0000","last_change":"2024-09-21T10:28:56.817623+0000","last_active":"2024-09-21T10:33:38.021097+0000","last_peered":"2024-09-21T10:33:38.021097+0000","last_clean":"2024-09-21T10:33:38.021097+0000","last_became_active":"2024-09-21T10:28:56.817337+0000","last_became_peered":"2024-09-21T10:28:56.817337+0000","last_unstale":"2024-09-21T10:33:38.021097+0000","last_undegraded":"2024-09-21T10:33:38.021097+0000","last_fullsized":"2024-09-21T10:33:38.021097+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:31.718364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2007,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.025361+0000","last_change":"2024-09-21T10:27:19.326897+0000","last_active":"2024-09-21T10:33:38.025361+0000","last_peered":"2024-09-21T10:33:38.025361+0000","last_clean":"2024-09-21T10:33:38.025361+0000","last_became_active":"2024-09-21T10:27:19.324476+0000","last_became_peered":"2024-09-21T10:27:19.324476+0000","last_unstale":"2024-09-21T10:33:38.025361+0000","last_undegraded":"2024-09-21T10:33:38.025361+0000","last_fullsized":"2024-09-21T10:33:38.025361+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:57.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1893,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.022087+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:33:38.022087+0000","last_peered":"2024-09-21T10:33:38.022087+0000","last_clean":"2024-09-21T10:33:38.022087+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:33:38.022087+0000","last_undegraded":"2024-09-21T10:33:38.022087+0000","last_fullsized":"2024-09-21T10:33:38.022087+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":415,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.009044+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:33:37.009044+0000","last_peered":"2024-09-21T10:33:37.009044+0000","last_clean":"2024-09-21T10:33:37.009044+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:33:37.009044+0000","last_undegraded":"2024-09-21T10:33:37.009044+0000","last_fullsized":"2024-09-21T10:33:37.009044+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2016,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.019064+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:33:38.019064+0000","last_peered":"2024-09-21T10:33:38.019064+0000","last_clean":"2024-09-21T10:33:38.019064+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:33:38.019064+0000","last_undegraded":"2024-09-21T10:33:38.019064+0000","last_fullsized":"2024-09-21T10:33:38.019064+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1287,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011614+0000","last_change":"2024-09-21T10:29:04.338405+0000","last_active":"2024-09-21T10:33:37.011614+0000","last_peered":"2024-09-21T10:33:37.011614+0000","last_clean":"2024-09-21T10:33:37.011614+0000","last_became_active":"2024-09-21T10:29:04.338028+0000","last_became_peered":"2024-09-21T10:29:04.338028+0000","last_unstale":"2024-09-21T10:33:37.011614+0000","last_undegraded":"2024-09-21T10:33:37.011614+0000","last_fullsized":"2024-09-21T10:33:37.011614+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:52.709092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2002,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023918+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:33:38.023918+0000","last_peered":"2024-09-21T10:33:38.023918+0000","last_clean":"2024-09-21T10:33:38.023918+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:33:38.023918+0000","last_undegraded":"2024-09-21T10:33:38.023918+0000","last_fullsized":"2024-09-21T10:33:38.023918+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":415,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.009162+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:33:37.009162+0000","last_peered":"2024-09-21T10:33:37.009162+0000","last_clean":"2024-09-21T10:33:37.009162+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:33:37.009162+0000","last_undegraded":"2024-09-21T10:33:37.009162+0000","last_fullsized":"2024-09-21T10:33:37.009162+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2000,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.024504+0000","last_change":"2024-09-21T10:30:09.408634+0000","last_active":"2024-09-21T10:33:38.024504+0000","last_peered":"2024-09-21T10:33:38.024504+0000","last_clean":"2024-09-21T10:33:38.024504+0000","last_became_active":"2024-09-21T10:30:09.408316+0000","last_became_peered":"2024-09-21T10:30:09.408316+0000","last_unstale":"2024-09-21T10:33:38.024504+0000","last_undegraded":"2024-09-21T10:33:38.024504+0000","last_fullsized":"2024-09-21T10:33:38.024504+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:36.075653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2002,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.024092+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:33:38.024092+0000","last_peered":"2024-09-21T10:33:38.024092+0000","last_clean":"2024-09-21T10:33:38.024092+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:33:38.024092+0000","last_undegraded":"2024-09-21T10:33:38.024092+0000","last_fullsized":"2024-09-21T10:33:38.024092+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":40,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008261+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:33:37.008261+0000","last_peered":"2024-09-21T10:33:37.008261+0000","last_clean":"2024-09-21T10:33:37.008261+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:33:37.008261+0000","last_undegraded":"2024-09-21T10:33:37.008261+0000","last_fullsized":"2024-09-21T10:33:37.008261+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":50,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.007935+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:33:37.007935+0000","last_peered":"2024-09-21T10:33:37.007935+0000","last_clean":"2024-09-21T10:33:37.007935+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:33:37.007935+0000","last_undegraded":"2024-09-21T10:33:37.007935+0000","last_fullsized":"2024-09-21T10:33:37.007935+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1999,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.024007+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:33:38.024007+0000","last_peered":"2024-09-21T10:33:38.024007+0000","last_clean":"2024-09-21T10:33:38.024007+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:33:38.024007+0000","last_undegraded":"2024-09-21T10:33:38.024007+0000","last_fullsized":"2024-09-21T10:33:38.024007+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1258,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005901+0000","last_change":"2024-09-21T10:27:19.333723+0000","last_active":"2024-09-21T10:33:37.005901+0000","last_peered":"2024-09-21T10:33:37.005901+0000","last_clean":"2024-09-21T10:33:37.005901+0000","last_became_active":"2024-09-21T10:27:19.333492+0000","last_became_peered":"2024-09-21T10:27:19.333492+0000","last_unstale":"2024-09-21T10:33:37.005901+0000","last_undegraded":"2024-09-21T10:33:37.005901+0000","last_fullsized":"2024-09-21T10:33:37.005901+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:03.860673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":44,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008021+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:33:37.008021+0000","last_peered":"2024-09-21T10:33:37.008021+0000","last_clean":"2024-09-21T10:33:37.008021+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:33:37.008021+0000","last_undegraded":"2024-09-21T10:33:37.008021+0000","last_fullsized":"2024-09-21T10:33:37.008021+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":1446,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011215+0000","last_change":"2024-09-21T10:33:26.562718+0000","last_active":"2024-09-21T10:33:37.011215+0000","last_peered":"2024-09-21T10:33:37.011215+0000","last_clean":"2024-09-21T10:33:37.011215+0000","last_became_active":"2024-09-21T10:29:57.002088+0000","last_became_peered":"2024-09-21T10:29:57.002088+0000","last_unstale":"2024-09-21T10:33:37.011215+0000","last_undegraded":"2024-09-21T10:33:37.011215+0000","last_fullsized":"2024-09-21T10:33:37.011215+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.562623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.562623+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:07.381791+0000","scrub_duration":0.001604517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2010,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.026585+0000","last_change":"2024-09-21T10:30:09.415060+0000","last_active":"2024-09-21T10:33:38.026585+0000","last_peered":"2024-09-21T10:33:38.026585+0000","last_clean":"2024-09-21T10:33:38.026585+0000","last_became_active":"2024-09-21T10:30:09.414412+0000","last_became_peered":"2024-09-21T10:30:09.414412+0000","last_unstale":"2024-09-21T10:33:38.026585+0000","last_undegraded":"2024-09-21T10:33:38.026585+0000","last_fullsized":"2024-09-21T10:33:38.026585+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:25:36.187303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1999,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023819+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:33:38.023819+0000","last_peered":"2024-09-21T10:33:38.023819+0000","last_clean":"2024-09-21T10:33:38.023819+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:33:38.023819+0000","last_undegraded":"2024-09-21T10:33:38.023819+0000","last_fullsized":"2024-09-21T10:33:38.023819+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1930,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.015493+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:33:38.015493+0000","last_peered":"2024-09-21T10:33:38.015493+0000","last_clean":"2024-09-21T10:33:38.015493+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:33:38.015493+0000","last_undegraded":"2024-09-21T10:33:38.015493+0000","last_fullsized":"2024-09-21T10:33:38.015493+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":155,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023955+0000","last_change":"2024-09-21T10:30:09.415904+0000","last_active":"2024-09-21T10:33:38.023955+0000","last_peered":"2024-09-21T10:33:38.023955+0000","last_clean":"2024-09-21T10:33:38.023955+0000","last_became_active":"2024-09-21T10:30:09.415524+0000","last_became_peered":"2024-09-21T10:30:09.415524+0000","last_unstale":"2024-09-21T10:33:38.023955+0000","last_undegraded":"2024-09-21T10:33:38.023955+0000","last_fullsized":"2024-09-21T10:33:38.023955+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:34:22.743516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":782,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.025518+0000","last_change":"2024-09-21T10:28:56.817541+0000","last_active":"2024-09-21T10:33:38.025518+0000","last_peered":"2024-09-21T10:33:38.025518+0000","last_clean":"2024-09-21T10:33:38.025518+0000","last_became_active":"2024-09-21T10:28:56.817221+0000","last_became_peered":"2024-09-21T10:28:56.817221+0000","last_unstale":"2024-09-21T10:33:38.025518+0000","last_undegraded":"2024-09-21T10:33:38.025518+0000","last_fullsized":"2024-09-21T10:33:38.025518+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:51.568259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2024,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.018483+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:33:38.018483+0000","last_peered":"2024-09-21T10:33:38.018483+0000","last_clean":"2024-09-21T10:33:38.018483+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:33:38.018483+0000","last_undegraded":"2024-09-21T10:33:38.018483+0000","last_fullsized":"2024-09-21T10:33:38.018483+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1277,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011580+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:33:37.011580+0000","last_peered":"2024-09-21T10:33:37.011580+0000","last_clean":"2024-09-21T10:33:37.011580+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:33:37.011580+0000","last_undegraded":"2024-09-21T10:33:37.011580+0000","last_fullsized":"2024-09-21T10:33:37.011580+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1240,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008979+0000","last_change":"2024-09-21T10:27:19.327425+0000","last_active":"2024-09-21T10:33:37.008979+0000","last_peered":"2024-09-21T10:33:37.008979+0000","last_clean":"2024-09-21T10:33:37.008979+0000","last_became_active":"2024-09-21T10:27:19.326641+0000","last_became_peered":"2024-09-21T10:27:19.326641+0000","last_unstale":"2024-09-21T10:33:37.008979+0000","last_undegraded":"2024-09-21T10:33:37.008979+0000","last_fullsized":"2024-09-21T10:33:37.008979+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:31:37.389673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1915,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.020941+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:33:38.020941+0000","last_peered":"2024-09-21T10:33:38.020941+0000","last_clean":"2024-09-21T10:33:38.020941+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:33:38.020941+0000","last_undegraded":"2024-09-21T10:33:38.020941+0000","last_fullsized":"2024-09-21T10:33:38.020941+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":44,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008167+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:33:37.008167+0000","last_peered":"2024-09-21T10:33:37.008167+0000","last_clean":"2024-09-21T10:33:37.008167+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:33:37.008167+0000","last_undegraded":"2024-09-21T10:33:37.008167+0000","last_fullsized":"2024-09-21T10:33:37.008167+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1450,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011447+0000","last_change":"2024-09-21T10:33:25.551397+0000","last_active":"2024-09-21T10:33:37.011447+0000","last_peered":"2024-09-21T10:33:37.011447+0000","last_clean":"2024-09-21T10:33:37.011447+0000","last_became_active":"2024-09-21T10:29:57.005035+0000","last_became_peered":"2024-09-21T10:29:57.005035+0000","last_unstale":"2024-09-21T10:33:37.011447+0000","last_undegraded":"2024-09-21T10:33:37.011447+0000","last_fullsized":"2024-09-21T10:33:37.011447+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:20.412919+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":40,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008079+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:33:37.008079+0000","last_peered":"2024-09-21T10:33:37.008079+0000","last_clean":"2024-09-21T10:33:37.008079+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:33:37.008079+0000","last_undegraded":"2024-09-21T10:33:37.008079+0000","last_fullsized":"2024-09-21T10:33:37.008079+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1998,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.026538+0000","last_change":"2024-09-21T10:27:19.327581+0000","last_active":"2024-09-21T10:33:38.026538+0000","last_peered":"2024-09-21T10:33:38.026538+0000","last_clean":"2024-09-21T10:33:38.026538+0000","last_became_active":"2024-09-21T10:27:19.327252+0000","last_became_peered":"2024-09-21T10:27:19.327252+0000","last_unstale":"2024-09-21T10:33:38.026538+0000","last_undegraded":"2024-09-21T10:33:38.026538+0000","last_fullsized":"2024-09-21T10:33:38.026538+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:46.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1902,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021206+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:33:38.021206+0000","last_peered":"2024-09-21T10:33:38.021206+0000","last_clean":"2024-09-21T10:33:38.021206+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:33:38.021206+0000","last_undegraded":"2024-09-21T10:33:38.021206+0000","last_fullsized":"2024-09-21T10:33:38.021206+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":168,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023776+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:33:38.023776+0000","last_peered":"2024-09-21T10:33:38.023776+0000","last_clean":"2024-09-21T10:33:38.023776+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:33:38.023776+0000","last_undegraded":"2024-09-21T10:33:38.023776+0000","last_fullsized":"2024-09-21T10:33:38.023776+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1257,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.006065+0000","last_change":"2024-09-21T10:27:19.329566+0000","last_active":"2024-09-21T10:33:37.006065+0000","last_peered":"2024-09-21T10:33:37.006065+0000","last_clean":"2024-09-21T10:33:37.006065+0000","last_became_active":"2024-09-21T10:27:19.329307+0000","last_became_peered":"2024-09-21T10:27:19.329307+0000","last_unstale":"2024-09-21T10:33:37.006065+0000","last_undegraded":"2024-09-21T10:33:37.006065+0000","last_fullsized":"2024-09-21T10:33:37.006065+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:59.305906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1210,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005609+0000","last_change":"2024-09-21T10:27:19.326055+0000","last_active":"2024-09-21T10:33:37.005609+0000","last_peered":"2024-09-21T10:33:37.005609+0000","last_clean":"2024-09-21T10:33:37.005609+0000","last_became_active":"2024-09-21T10:27:19.324778+0000","last_became_peered":"2024-09-21T10:27:19.324778+0000","last_unstale":"2024-09-21T10:33:37.005609+0000","last_undegraded":"2024-09-21T10:33:37.005609+0000","last_fullsized":"2024-09-21T10:33:37.005609+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:12.572967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":44,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.007905+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:33:37.007905+0000","last_peered":"2024-09-21T10:33:37.007905+0000","last_clean":"2024-09-21T10:33:37.007905+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:33:37.007905+0000","last_undegraded":"2024-09-21T10:33:37.007905+0000","last_fullsized":"2024-09-21T10:33:37.007905+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1217,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.006202+0000","last_change":"2024-09-21T10:27:20.323254+0000","last_active":"2024-09-21T10:33:37.006202+0000","last_peered":"2024-09-21T10:33:37.006202+0000","last_clean":"2024-09-21T10:33:37.006202+0000","last_became_active":"2024-09-21T10:27:20.322751+0000","last_became_peered":"2024-09-21T10:27:20.322751+0000","last_unstale":"2024-09-21T10:33:37.006202+0000","last_undegraded":"2024-09-21T10:33:37.006202+0000","last_fullsized":"2024-09-21T10:33:37.006202+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.909953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1246,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.009044+0000","last_change":"2024-09-21T10:27:19.326938+0000","last_active":"2024-09-21T10:33:37.009044+0000","last_peered":"2024-09-21T10:33:37.009044+0000","last_clean":"2024-09-21T10:33:37.009044+0000","last_became_active":"2024-09-21T10:27:19.326640+0000","last_became_peered":"2024-09-21T10:27:19.326640+0000","last_unstale":"2024-09-21T10:33:37.009044+0000","last_undegraded":"2024-09-21T10:33:37.009044+0000","last_fullsized":"2024-09-21T10:33:37.009044+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:24.413432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1411,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011137+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:33:37.011137+0000","last_peered":"2024-09-21T10:33:37.011137+0000","last_clean":"2024-09-21T10:33:37.011137+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:33:37.011137+0000","last_undegraded":"2024-09-21T10:33:37.011137+0000","last_fullsized":"2024-09-21T10:33:37.011137+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1257,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005914+0000","last_change":"2024-09-21T10:27:19.332150+0000","last_active":"2024-09-21T10:33:37.005914+0000","last_peered":"2024-09-21T10:33:37.005914+0000","last_clean":"2024-09-21T10:33:37.005914+0000","last_became_active":"2024-09-21T10:27:19.331901+0000","last_became_peered":"2024-09-21T10:27:19.331901+0000","last_unstale":"2024-09-21T10:33:37.005914+0000","last_undegraded":"2024-09-21T10:33:37.005914+0000","last_fullsized":"2024-09-21T10:33:37.005914+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:07.758296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":108,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021285+0000","last_change":"2024-09-21T10:28:56.834029+0000","last_active":"2024-09-21T10:33:38.021285+0000","last_peered":"2024-09-21T10:33:38.021285+0000","last_clean":"2024-09-21T10:33:38.021285+0000","last_became_active":"2024-09-21T10:28:56.817594+0000","last_became_peered":"2024-09-21T10:28:56.817594+0000","last_unstale":"2024-09-21T10:33:38.021285+0000","last_undegraded":"2024-09-21T10:33:38.021285+0000","last_fullsized":"2024-09-21T10:33:38.021285+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:44.035052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":27,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.006299+0000","last_change":"2024-09-21T10:33:27.479201+0000","last_active":"2024-09-21T10:33:37.006299+0000","last_peered":"2024-09-21T10:33:37.006299+0000","last_clean":"2024-09-21T10:33:37.006299+0000","last_became_active":"2024-09-21T10:33:20.899955+0000","last_became_peered":"2024-09-21T10:33:20.899955+0000","last_unstale":"2024-09-21T10:33:37.006299+0000","last_undegraded":"2024-09-21T10:33:37.006299+0000","last_fullsized":"2024-09-21T10:33:37.006299+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.479064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.479064+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:44.683933+0000","scrub_duration":0.0013296740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2023,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.018966+0000","last_change":"2024-09-21T10:27:19.370824+0000","last_active":"2024-09-21T10:33:38.018966+0000","last_peered":"2024-09-21T10:33:38.018966+0000","last_clean":"2024-09-21T10:33:38.018966+0000","last_became_active":"2024-09-21T10:27:19.370477+0000","last_became_peered":"2024-09-21T10:27:19.370477+0000","last_unstale":"2024-09-21T10:33:38.018966+0000","last_undegraded":"2024-09-21T10:33:38.018966+0000","last_fullsized":"2024-09-21T10:33:38.018966+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:17.020888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":40,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008270+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:33:37.008270+0000","last_peered":"2024-09-21T10:33:37.008270+0000","last_clean":"2024-09-21T10:33:37.008270+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:33:37.008270+0000","last_undegraded":"2024-09-21T10:33:37.008270+0000","last_fullsized":"2024-09-21T10:33:37.008270+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":424,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:39.741469+0000","last_change":"2024-09-21T10:33:39.741224+0000","last_active":"2024-09-21T10:33:39.741469+0000","last_peered":"2024-09-21T10:33:39.741469+0000","last_clean":"2024-09-21T10:33:39.741469+0000","last_became_active":"2024-09-21T10:33:20.898899+0000","last_became_peered":"2024-09-21T10:33:20.898899+0000","last_unstale":"2024-09-21T10:33:39.741469+0000","last_undegraded":"2024-09-21T10:33:39.741469+0000","last_fullsized":"2024-09-21T10:33:39.741469+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:14.593280+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1228,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008894+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:33:37.008894+0000","last_peered":"2024-09-21T10:33:37.008894+0000","last_clean":"2024-09-21T10:33:37.008894+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:33:37.008894+0000","last_undegraded":"2024-09-21T10:33:37.008894+0000","last_fullsized":"2024-09-21T10:33:37.008894+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1277,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011560+0000","last_change":"2024-09-21T10:30:09.415150+0000","last_active":"2024-09-21T10:33:37.011560+0000","last_peered":"2024-09-21T10:33:37.011560+0000","last_clean":"2024-09-21T10:33:37.011560+0000","last_became_active":"2024-09-21T10:30:09.402385+0000","last_became_peered":"2024-09-21T10:30:09.402385+0000","last_unstale":"2024-09-21T10:33:37.011560+0000","last_undegraded":"2024-09-21T10:33:37.011560+0000","last_fullsized":"2024-09-21T10:33:37.011560+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:53.975042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":167,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021218+0000","last_change":"2024-09-21T10:33:21.766789+0000","last_active":"2024-09-21T10:33:38.021218+0000","last_peered":"2024-09-21T10:33:38.021218+0000","last_clean":"2024-09-21T10:33:38.021218+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:33:38.021218+0000","last_undegraded":"2024-09-21T10:33:38.021218+0000","last_fullsized":"2024-09-21T10:33:38.021218+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:34:22.015436+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1253,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.005682+0000","last_change":"2024-09-21T10:27:19.327246+0000","last_active":"2024-09-21T10:33:37.005682+0000","last_peered":"2024-09-21T10:33:37.005682+0000","last_clean":"2024-09-21T10:33:37.005682+0000","last_became_active":"2024-09-21T10:27:19.326343+0000","last_became_peered":"2024-09-21T10:27:19.326343+0000","last_unstale":"2024-09-21T10:33:37.005682+0000","last_undegraded":"2024-09-21T10:33:37.005682+0000","last_fullsized":"2024-09-21T10:33:37.005682+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:37.654823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1267,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011932+0000","last_change":"2024-09-21T10:27:19.324762+0000","last_active":"2024-09-21T10:33:37.011932+0000","last_peered":"2024-09-21T10:33:37.011932+0000","last_clean":"2024-09-21T10:33:37.011932+0000","last_became_active":"2024-09-21T10:27:19.324341+0000","last_became_peered":"2024-09-21T10:27:19.324341+0000","last_unstale":"2024-09-21T10:33:37.011932+0000","last_undegraded":"2024-09-21T10:33:37.011932+0000","last_fullsized":"2024-09-21T10:33:37.011932+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:42.722645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":27,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011252+0000","last_change":"2024-09-21T10:33:27.552769+0000","last_active":"2024-09-21T10:33:37.011252+0000","last_peered":"2024-09-21T10:33:37.011252+0000","last_clean":"2024-09-21T10:33:37.011252+0000","last_became_active":"2024-09-21T10:33:20.902059+0000","last_became_peered":"2024-09-21T10:33:20.902059+0000","last_unstale":"2024-09-21T10:33:37.011252+0000","last_undegraded":"2024-09-21T10:33:37.011252+0000","last_fullsized":"2024-09-21T10:33:37.011252+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:41.326748+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":40,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.007955+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:33:37.007955+0000","last_peered":"2024-09-21T10:33:37.007955+0000","last_clean":"2024-09-21T10:33:37.007955+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:33:37.007955+0000","last_undegraded":"2024-09-21T10:33:37.007955+0000","last_fullsized":"2024-09-21T10:33:37.007955+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1280,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011764+0000","last_change":"2024-09-21T10:28:56.826222+0000","last_active":"2024-09-21T10:33:37.011764+0000","last_peered":"2024-09-21T10:33:37.011764+0000","last_clean":"2024-09-21T10:33:37.011764+0000","last_became_active":"2024-09-21T10:28:56.813187+0000","last_became_peered":"2024-09-21T10:28:56.813187+0000","last_unstale":"2024-09-21T10:33:37.011764+0000","last_undegraded":"2024-09-21T10:33:37.011764+0000","last_fullsized":"2024-09-21T10:33:37.011764+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:28.841806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":27,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.244625+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:33:38.244625+0000","last_peered":"2024-09-21T10:33:38.244625+0000","last_clean":"2024-09-21T10:33:38.244625+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:33:38.244625+0000","last_undegraded":"2024-09-21T10:33:38.244625+0000","last_fullsized":"2024-09-21T10:33:38.244625+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1305,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008838+0000","last_change":"2024-09-21T10:30:09.424136+0000","last_active":"2024-09-21T10:33:37.008838+0000","last_peered":"2024-09-21T10:33:37.008838+0000","last_clean":"2024-09-21T10:33:37.008838+0000","last_became_active":"2024-09-21T10:30:09.414027+0000","last_became_peered":"2024-09-21T10:30:09.414027+0000","last_unstale":"2024-09-21T10:33:37.008838+0000","last_undegraded":"2024-09-21T10:33:37.008838+0000","last_fullsized":"2024-09-21T10:33:37.008838+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:28.506487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1892,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021898+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:33:38.021898+0000","last_peered":"2024-09-21T10:33:38.021898+0000","last_clean":"2024-09-21T10:33:38.021898+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:33:38.021898+0000","last_undegraded":"2024-09-21T10:33:38.021898+0000","last_fullsized":"2024-09-21T10:33:38.021898+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":160,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.023673+0000","last_change":"2024-09-21T10:33:20.897520+0000","last_active":"2024-09-21T10:33:38.023673+0000","last_peered":"2024-09-21T10:33:38.023673+0000","last_clean":"2024-09-21T10:33:38.023673+0000","last_became_active":"2024-09-21T10:33:20.897071+0000","last_became_peered":"2024-09-21T10:33:20.897071+0000","last_unstale":"2024-09-21T10:33:38.023673+0000","last_undegraded":"2024-09-21T10:33:38.023673+0000","last_fullsized":"2024-09-21T10:33:38.023673+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:34:24.677314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1275,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011281+0000","last_change":"2024-09-21T10:26:04.187372+0000","last_active":"2024-09-21T10:33:37.011281+0000","last_peered":"2024-09-21T10:33:37.011281+0000","last_clean":"2024-09-21T10:33:37.011281+0000","last_became_active":"2024-09-21T10:26:04.186026+0000","last_became_peered":"2024-09-21T10:26:04.186026+0000","last_unstale":"2024-09-21T10:33:37.011281+0000","last_undegraded":"2024-09-21T10:33:37.011281+0000","last_fullsized":"2024-09-21T10:33:37.011281+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:55.130410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2011,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.019372+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:33:38.019372+0000","last_peered":"2024-09-21T10:33:38.019372+0000","last_clean":"2024-09-21T10:33:38.019372+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:33:38.019372+0000","last_undegraded":"2024-09-21T10:33:38.019372+0000","last_fullsized":"2024-09-21T10:33:38.019372+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":27,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.007794+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:33:37.007794+0000","last_peered":"2024-09-21T10:33:37.007794+0000","last_clean":"2024-09-21T10:33:37.007794+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:33:37.007794+0000","last_undegraded":"2024-09-21T10:33:37.007794+0000","last_fullsized":"2024-09-21T10:33:37.007794+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2010,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.018459+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:33:38.018459+0000","last_peered":"2024-09-21T10:33:38.018459+0000","last_clean":"2024-09-21T10:33:38.018459+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:33:38.018459+0000","last_undegraded":"2024-09-21T10:33:38.018459+0000","last_fullsized":"2024-09-21T10:33:38.018459+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":40,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.008295+0000","last_change":"2024-09-21T10:30:09.405230+0000","last_active":"2024-09-21T10:33:37.008295+0000","last_peered":"2024-09-21T10:33:37.008295+0000","last_clean":"2024-09-21T10:33:37.008295+0000","last_became_active":"2024-09-21T10:30:09.404929+0000","last_became_peered":"2024-09-21T10:30:09.404929+0000","last_unstale":"2024-09-21T10:33:37.008295+0000","last_undegraded":"2024-09-21T10:33:37.008295+0000","last_fullsized":"2024-09-21T10:33:37.008295+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:01.454535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":60,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.007990+0000","last_change":"2024-09-21T10:33:30.245355+0000","last_active":"2024-09-21T10:33:37.007990+0000","last_peered":"2024-09-21T10:33:37.007990+0000","last_clean":"2024-09-21T10:33:37.007990+0000","last_became_active":"2024-09-21T10:33:20.902088+0000","last_became_peered":"2024-09-21T10:33:20.902088+0000","last_unstale":"2024-09-21T10:33:37.007990+0000","last_undegraded":"2024-09-21T10:33:37.007990+0000","last_fullsized":"2024-09-21T10:33:37.007990+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:30.245301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:30.245301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:43.434664+0000","scrub_duration":0.0018729759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1280,"reported_epoch":1289,"state":"active+clean","last_fresh":"2024-09-21T10:33:37.011468+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:33:37.011468+0000","last_peered":"2024-09-21T10:33:37.011468+0000","last_clean":"2024-09-21T10:33:37.011468+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:33:37.011468+0000","last_undegraded":"2024-09-21T10:33:37.011468+0000","last_fullsized":"2024-09-21T10:33:37.011468+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1900,"reported_epoch":1290,"state":"active+clean","last_fresh":"2024-09-21T10:33:38.021456+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:33:38.021456+0000","last_peered":"2024-09-21T10:33:38.021456+0000","last_clean":"2024-09-21T10:33:38.021456+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:33:38.021456+0000","last_undegraded":"2024-09-21T10:33:38.021456+0000","last_fullsized":"2024-09-21T10:33:38.021456+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":47,"num_pg":16,"stat_sum":{"num_bytes":1660944384,"num_objects":25344,"num_object_clones":0,"num_object_copies":76032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25344,"num_write_kb":1622016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5093064704,"data_stored":5093064704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25344,"ondisk_log_size":25344,"up":48,"acting":48,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":1426,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397970,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732940,"kb_used_data":435588,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93012212,"statfs":{"total":95995035648,"available":95244505088,"internally_reserved":0,"allocated":446042112,"data_stored":440715393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049630,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1448268,"kb_used_data":1099716,"kb_used_omap":242,"kb_used_meta":348237,"kb_avail":92296884,"statfs":{"total":95995035648,"available":94512009216,"internally_reserved":0,"allocated":1126109184,"data_stored":1120783006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":356594911},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790428,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":590084,"kb_used_data":284412,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93155068,"statfs":{"total":95995035648,"available":95390789632,"internally_reserved":0,"allocated":291237888,"data_stored":285915694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395534,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1162568,"kb_used_data":839040,"kb_used_omap":224,"kb_used_meta":323295,"kb_avail":92582584,"statfs":{"total":95995035648,"available":94804566016,"internally_reserved":0,"allocated":859176960,"data_stored":853854323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":331054985},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644588,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1158076,"kb_used_data":746932,"kb_used_omap":67,"kb_used_meta":411068,"kb_avail":92587076,"statfs":{"total":95995035648,"available":94809165824,"internally_reserved":0,"allocated":764858368,"data_stored":759543817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":420934493},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527606,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":968896,"kb_used_data":516600,"kb_used_omap":75,"kb_used_meta":452212,"kb_avail":92776256,"statfs":{"total":95995035648,"available":95002886144,"internally_reserved":0,"allocated":528998400,"data_stored":523680291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":463065242},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274759,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374148,"kb_used_data":969852,"kb_used_omap":51,"kb_used_meta":404236,"kb_avail":92371004,"statfs":{"total":95995035648,"available":94587908096,"internally_reserved":0,"allocated":993128448,"data_stored":987805244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":413938005},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138353997,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":724852,"kb_used_data":392940,"kb_used_omap":66,"kb_used_meta":331837,"kb_avail":93020300,"statfs":{"total":95995035648,"available":95252787200,"internally_reserved":0,"allocated":402370560,"data_stored":397063583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":339801412},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":387579904,"data_stored":387579904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1031471104,"data_stored":1031471104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":544997376,"data_stored":544997376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":750190592,"data_stored":750190592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":548208640,"data_stored":548208640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":280166400,"data_stored":280166400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1092091904,"data_stored":1092091904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":458358784,"data_stored":458358784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:33:49.692 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:33:49.692 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_2 pg_num 26 2024-09-21T10:33:49.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:49.798 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:49.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:49.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:49.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:49.953 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 14 16 35484 35468 158.319 132.438 0.0188777 0.00631079 2024-09-21T10:33:50.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.076 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:50.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:50.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:50.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:50.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:50.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.348 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:50.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:50.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:50.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:33:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:33:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:33:50.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:50.633 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:50.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:50.772 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:50.864 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:50 smithi119 ceph-mon[101432]: pgmap v2008: 109 pgs: 109 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-21T10:33:50.864 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:50 smithi119 ceph-mon[105288]: pgmap v2008: 109 pgs: 109 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-21T10:33:50.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:50.910 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:50.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:50.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:50.954 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 15 16 37551 37535 156.376 129.188 0.00917805 0.00639 2024-09-21T10:33:51.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:50 smithi145 ceph-mon[89359]: pgmap v2008: 109 pgs: 109 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 152 MiB/s wr, 2.43k op/s 2024-09-21T10:33:51.048 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:51.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:51.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:51.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:51.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:51.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:51.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:51.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:51.451 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:51.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:51.553 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:51.578 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:51.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:51.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:51.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:51.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:51.850 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:51.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:51.954 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 16 16 39152 39136 152.856 100.062 0.00292735 0.0065267 2024-09-21T10:33:51.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:51.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:51.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:52.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:52.112 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:52.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:52.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:52.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.367 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:52.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:52.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.494 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:52.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:52.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:52.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:52.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:52.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:52.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:52.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:52.915 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:52.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:52.928 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:52.932 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:33:52.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 17 16 40969 40953 150.536 113.562 0.0219461 0.00663644 2024-09-21T10:33:53.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:52 smithi145 ceph-mon[89359]: pgmap v2009: 109 pgs: 109 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-21T10:33:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:53.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:53.074 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:52 smithi119 ceph-mon[101432]: pgmap v2009: 109 pgs: 109 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-21T10:33:53.075 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:52 smithi119 ceph-mon[105288]: pgmap v2009: 109 pgs: 109 active+clean; 2.1 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 149 MiB/s wr, 2.38k op/s 2024-09-21T10:33:53.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:53.196 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:53.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:53.340 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:53.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:53.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:53.574 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:53.603 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:53.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:53.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:53.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:53.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:53.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:53.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 18 16 42868 42852 148.766 118.688 0.00711117 0.00671714 2024-09-21T10:33:53.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.007 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:54.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:54.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:54.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.141 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:54.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:54.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:54.263 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:54.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:54.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:54.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:54.542 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:54.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:33:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:33:54] "GET /metrics HTTP/1.1" 200 36876 "" "Prometheus/2.51.0" 2024-09-21T10:33:54.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:54 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:54.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:54 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:54.644 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:54.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:54.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:54.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:54.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:54 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:33:54.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:54.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:54.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:54.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:54.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:54.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 19 16 44351 44335 145.814 92.6875 0.0636319 0.00685142 2024-09-21T10:33:55.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:55.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:55.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:55.193 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:55.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:55.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:55.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:55.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:55.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:55.459 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:55.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:55.591 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:55 smithi119 ceph-mon[101432]: pgmap v2010: 109 pgs: 109 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-21T10:33:55.594 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:55 smithi119 ceph-mon[105288]: pgmap v2010: 109 pgs: 109 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-21T10:33:55.594 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:33:55.594 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:33:55.606 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:55.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:55.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:55.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:55 smithi145 ceph-mon[89359]: pgmap v2010: 109 pgs: 109 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 143 MiB/s wr, 2.29k op/s 2024-09-21T10:33:55.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:55.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:55.866 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:55.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:55.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:33:55.954227+0000 min lat: 0.00109908 max lat: 0.259852 avg lat: 0.00696604 2024-09-21T10:33:55.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:33:55.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 20 16 45929 45913 143.455 98.625 0.010527 0.00696604 2024-09-21T10:33:55.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:56.001 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:56.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:56.133 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:56.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:56.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:56.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:56.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:56.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:56.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:33:56.543 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:56.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:33:56.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:56.685 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:56.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:56.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:56.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:56.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:56.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:56.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 21 16 47565 47549 141.492 102.25 0.0046711 0.00706253 2024-09-21T10:33:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:56.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:57.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:33:57.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:33:57.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:57.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:57.355 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:33:57.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:57.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:57.477 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:57.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:57 smithi119 ceph-mon[105288]: pgmap v2011: 109 pgs: 109 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:33:57.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3593887693' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:33:57.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:57 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:33:57.505 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:57 smithi119 ceph-mon[101432]: pgmap v2011: 109 pgs: 109 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:33:57.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3593887693' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:33:57.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:57 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:33:57.509 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:57.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:57.647 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:57.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:57.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:57.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:57 smithi145 ceph-mon[89359]: pgmap v2011: 109 pgs: 109 active+clean; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 118 MiB/s wr, 1.89k op/s 2024-09-21T10:33:57.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3593887693' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:33:57.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:57 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:33:57.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:57.938 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:57.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:57.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 22 16 49211 49195 139.736 102.875 0.00749619 0.00715163 2024-09-21T10:33:58.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:58.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:58.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:58.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:58.214 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:58.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:33:58.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:33:58.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:58.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:58.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:58 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:58 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:33:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:58 smithi119 ceph-mon[105288]: osdmap e1291: 8 total, 8 up, 8 in 2024-09-21T10:33:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:58 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:58 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:33:58.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:58 smithi119 ceph-mon[101432]: osdmap e1291: 8 total, 8 up, 8 in 2024-09-21T10:33:58.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:58.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:33:58.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:33:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:58 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:33:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:58 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:33:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:58 smithi145 ceph-mon[89359]: osdmap e1291: 8 total, 8 up, 8 in 2024-09-21T10:33:58.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:58.902 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:58.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:33:58.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 23 16 50956 50940 138.402 109.062 0.00670332 0.00721996 2024-09-21T10:33:59.004 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:59.030 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:59.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:59.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:59.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:59.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:59.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:33:59.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:33:59.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:33:59 smithi119 ceph-mon[101432]: pgmap v2013: 109 pgs: 109 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:33:59.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:33:59 smithi119 ceph-mon[105288]: pgmap v2013: 109 pgs: 109 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:33:59.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:33:59.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:59.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:33:59.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.722 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:59.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:33:59 smithi145 ceph-mon[89359]: pgmap v2013: 109 pgs: 109 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:33:59.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:33:59.860 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:33:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:33:59.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:33:59.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 24 16 52642 52626 137.026 105.375 0.0091837 0.00729286 2024-09-21T10:33:59.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:33:59.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:33:59.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:33:59.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:00.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:00.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:00.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:00.248 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:00.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:00.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:00.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:34:00.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:00.479 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:00.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:34:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:34:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:00.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:00.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:00.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:00.737 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:00.771 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:00.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:00.858 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:34:00.859 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:34:00.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:00.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:00.893 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:00.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:00.935 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:34:00.936 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:34:00.955 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 25 16 53981 53965 134.892 83.6875 0.0162995 0.00740792 2024-09-21T10:34:00.978 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:34:00.978 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:34:00.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:01.011 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:34:01.011 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:34:01.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:01.047 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:34:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:34:01.076 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:34:01.077 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:34:01.114 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:34:01.114 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:34:01.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:01.149 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:34:01.150 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:34:01.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:01.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:01.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:01.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:01.416 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:01.435 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:01 smithi145 ceph-mon[89359]: pgmap v2014: 109 pgs: 109 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-21T10:34:01.518 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:01.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[101432]: pgmap v2014: 109 pgs: 109 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-21T10:34:01.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/523897870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:01.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3717013923' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:34:01.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:34:01.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[105288]: pgmap v2014: 109 pgs: 109 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-21T10:34:01.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/523897870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:01.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3717013923' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:34:01.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:01 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:34:01.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:01.668 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:01.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:01.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/523897870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3717013923' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:34:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:01 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-21T10:34:01.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:01.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:01.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:01.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 26 16 54900 54884 131.913 57.4375 0.00679576 0.00757561 2024-09-21T10:34:02.016 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:02.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:02.146 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 47 pg_num to 26 2024-09-21T10:34:02.154 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:02.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:02.289 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:02.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:02.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:02.488 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:02.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:02.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:02 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:34:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:02 smithi119 ceph-mon[105288]: osdmap e1292: 8 total, 8 up, 8 in 2024-09-21T10:34:02.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:34:02.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:02 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:34:02.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:02 smithi119 ceph-mon[101432]: osdmap e1292: 8 total, 8 up, 8 in 2024-09-21T10:34:02.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:34:02.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.739 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:02.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:02 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-21T10:34:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:02 smithi145 ceph-mon[89359]: osdmap e1292: 8 total, 8 up, 8 in 2024-09-21T10:34:02.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-21T10:34:02.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:02.861 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:02.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 27 16 55581 55565 128.604 42.5625 0.00401516 0.00761772 2024-09-21T10:34:02.963 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:02.984 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:02.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:02.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:03.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:03.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:03.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:03.232 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:03.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:03.358 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:03.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:03.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:03.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:03.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:03.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:03.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:03.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:03 smithi145 ceph-mon[89359]: pgmap v2016: 109 pgs: 109 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:34:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:34:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:03 smithi145 ceph-mon[89359]: osdmap e1293: 8 total, 8 up, 8 in 2024-09-21T10:34:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[105288]: pgmap v2016: 109 pgs: 109 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:34:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:34:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[105288]: osdmap e1293: 8 total, 8 up, 8 in 2024-09-21T10:34:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[101432]: pgmap v2016: 109 pgs: 109 active+clean; 3.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-21T10:34:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T10:34:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[101432]: osdmap e1293: 8 total, 8 up, 8 in 2024-09-21T10:34:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:03.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:03.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:03.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:03.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:03.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 28 16 56629 56613 126.35 65.5 0.0132653 0.00790996 2024-09-21T10:34:04.007 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:04.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:04.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:04.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:04.177 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:04.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:04.314 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:04.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:04.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:04.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:04.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:04.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:04.725 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:04.758 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:34:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:34:04] "GET /metrics HTTP/1.1" 200 36885 "" "Prometheus/2.51.0" 2024-09-21T10:34:04.758 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:04 smithi119 ceph-mon[105288]: osdmap e1294: 8 total, 8 up, 8 in 2024-09-21T10:34:04.759 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:04 smithi119 ceph-mon[101432]: osdmap e1294: 8 total, 8 up, 8 in 2024-09-21T10:34:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:04 smithi145 ceph-mon[89359]: osdmap e1294: 8 total, 8 up, 8 in 2024-09-21T10:34:04.827 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:04.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:04.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:04.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 29 15 58734 58719 126.531 131.625 0.00677515 0.0078986 2024-09-21T10:34:04.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.001 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:05.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:05.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:05.134 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:05.272 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:05.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:05.407 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:05.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:05.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:05.612 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:05.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:05.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:05.748 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:05 smithi119 ceph-mon[101432]: pgmap v2019: 119 pgs: 1 peering, 8 unknown, 110 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-09-21T10:34:05.749 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:05 smithi119 ceph-mon[105288]: pgmap v2019: 119 pgs: 1 peering, 8 unknown, 110 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-09-21T10:34:05.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:05 smithi145 ceph-mon[89359]: pgmap v2019: 119 pgs: 1 peering, 8 unknown, 110 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 88 MiB/s wr, 1.42k op/s 2024-09-21T10:34:05.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:05.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:05.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 30 16 60775 60759 126.563 127.5 0.00203899 0.00789606 2024-09-21T10:34:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:05.972 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:05.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:06.073 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:06.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:06.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:06.248 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:06.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:06.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:06.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:06.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:06.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:06.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:06.767 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:06.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:06.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:06.939 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:06.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:06.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 31 16 62766 62750 126.494 124.438 0.00898542 0.00790062 2024-09-21T10:34:07.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:07.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:07.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:07.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:07.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:07.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:07.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:07.468 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:07.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:07.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:34:07.609 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:07.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:07.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:07 smithi119 ceph-mon[101432]: pgmap v2020: 119 pgs: 1 peering, 4 unknown, 114 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s; 21 KiB/s, 0 objects/s recovering 2024-09-21T10:34:07.647 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:07 smithi119 ceph-mon[105288]: pgmap v2020: 119 pgs: 1 peering, 4 unknown, 114 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s; 21 KiB/s, 0 objects/s recovering 2024-09-21T10:34:07.711 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:07.743 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:07 smithi145 ceph-mon[89359]: pgmap v2020: 119 pgs: 1 peering, 4 unknown, 114 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 60 MiB/s wr, 967 op/s; 21 KiB/s, 0 objects/s recovering 2024-09-21T10:34:07.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:07.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:07.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:07.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 32 16 64520 64504 125.966 109.625 0.0126147 0.0079337 2024-09-21T10:34:07.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:08.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:08.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:08.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:08.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:08.288 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:08.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:08.425 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:08.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:08.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:08.663 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:08.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:08.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:08.837 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:08.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:08.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:08.956 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 33 16 66414 66398 125.736 118.375 0.00531198 0.0079482 2024-09-21T10:34:08.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:08.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:09.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:09.112 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:09.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:09.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:09.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:09.253 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:09.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:09.354 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:09.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:09.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:09.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:09.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:09.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:09.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:09 smithi119 ceph-mon[101432]: pgmap v2021: 119 pgs: 1 peering, 1 unknown, 117 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 55 MiB/s wr, 878 op/s; 30 KiB/s, 0 objects/s recovering 2024-09-21T10:34:09.555 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:09 smithi119 ceph-mon[105288]: pgmap v2021: 119 pgs: 1 peering, 1 unknown, 117 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 55 MiB/s wr, 878 op/s; 30 KiB/s, 0 objects/s recovering 2024-09-21T10:34:09.609 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:230922 45.1 45:8fcd486c:::benchmark_data_smithi119_168706_object100752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1211)", 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:18.969891+0000", 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 410.63318745999999, 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.486679368, 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.610 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 230922 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969891+0000", 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969891+0000", 2024-09-21T10:34:09.611 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969893+0000", 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.717e-06 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969898+0000", 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2449999999999998e-06 2024-09-21T10:34:09.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969915+0000", 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7119e-05 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969923+0000", 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7300000000000005e-06 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969983+0000", 2024-09-21T10:34:09.613 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0186999999999998e-05 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.969992+0000", 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.123e-06 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.436349+0000", 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46635789100000002 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.614 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.436483+0000", 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000133375 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.436576+0000", 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2867000000000004e-05 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.449155+0000", 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01257904 2024-09-21T10:34:09.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.449203+0000", 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8158999999999997e-05 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.453013+0000", 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038103199999999999 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.616 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.456233+0000", 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032201840000000001 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.456342+0000", 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010872199999999999 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.456571+0000", 2024-09-21T10:34:09.617 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00022868899999999999 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:230936 45.1 45:891b61cf:::benchmark_data_smithi119_168706_object100766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1211)", 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:18.989991+0000", 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 410.61308814699998, 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46707375800000001, 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.618 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 230936 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.989991+0000", 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.619 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.989991+0000", 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.989992+0000", 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6309999999999999e-06 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.989997+0000", 2024-09-21T10:34:09.620 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5600000000000004e-06 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.990012+0000", 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5588000000000001e-05 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.990020+0000", 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5669999999999997e-06 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.621 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.990087+0000", 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6610000000000001e-05 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:18.990096+0000", 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5219999999999996e-06 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.440979+0000", 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.45088234500000002 2024-09-21T10:34:09.622 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.441105+0000", 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000126363 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.441249+0000", 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000143817 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.623 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.449261+0000", 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080127589999999995 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.453081+0000", 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038190960000000001 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.453152+0000", 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1186000000000006e-05 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.624 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.456630+0000", 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034780890000000002 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.456793+0000", 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016351399999999999 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.457064+0000", 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00027111099999999999 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.625 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.35976.0:230978 45.1 45:8550b32b:::benchmark_data_smithi119_168706_object100808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1211)", 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:27:19.052339+0000", 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 410.55073942000001, 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.404998354, 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.626 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.35976", 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 230978 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052339+0000", 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.627 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052339+0000", 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052341+0000", 2024-09-21T10:34:09.628 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5319999999999999e-06 2024-09-21T10:34:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052345+0000", 2024-09-21T10:34:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4630000000000003e-06 2024-09-21T10:34:09.629 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052357+0000", 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1338000000000001e-05 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052362+0000", 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2390000000000002e-06 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.630 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052417+0000", 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4827000000000002e-05 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.052425+0000", 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1340000000000004e-06 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.631 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.444735+0000", 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.39230958700000002 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.444889+0000", 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015433499999999999 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5,6", 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.445034+0000", 2024-09-21T10:34:09.632 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014527499999999999 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.453138+0000", 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0081034339999999996 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.453159+0000", 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1358000000000001e-05 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.633 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.454697+0000", 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015376210000000001 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.457118+0000", 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024214380000000001 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.457203+0000", 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5261999999999998e-05 2024-09-21T10:34:09.634 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:27:19.457338+0000", 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000134511 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:152158 46.9 46:990781f7:::benchmark_data_smithi119_206743_object47057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:17.375445+0000", 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 172.22763355199999, 2024-09-21T10:34:09.635 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.28534780700000001, 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:34:09.636 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 152158 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375445+0000", 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375445+0000", 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.637 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375447+0000", 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.375e-06 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375452+0000", 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3680000000000001e-06 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375463+0000", 2024-09-21T10:34:09.638 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0579e-05 2024-09-21T10:34:09.639 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.639 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.639 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.639 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375469+0000", 2024-09-21T10:34:09.639 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8349999999999998e-06 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375521+0000", 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1202000000000002e-05 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375726+0000", 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020537999999999999 2024-09-21T10:34:09.640 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.375839+0000", 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000113238 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.380038+0000", 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041982790000000001 2024-09-21T10:34:09.641 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.642 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.642 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.642 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.381213+0000", 2024-09-21T10:34:09.642 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011752240000000001 2024-09-21T10:34:09.642 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.642 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:09.643 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.643 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.660689+0000", 2024-09-21T10:34:09.643 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27947671499999999 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.660746+0000", 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6597e-05 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.660793+0000", 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7015000000000003e-05 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.644 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:152177 46.9 46:9b1239d2:::benchmark_data_smithi119_206743_object47076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:17.405262+0000", 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 172.19781640100001, 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26371115699999997, 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:34:09.645 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 152177 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405262+0000", 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405262+0000", 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.646 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405263+0000", 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.035e-06 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405268+0000", 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0980000000000004e-06 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405276+0000", 2024-09-21T10:34:09.647 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7169999999999997e-06 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405281+0000", 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4939999999999997e-06 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405334+0000", 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3158000000000002e-05 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.648 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405461+0000", 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012740900000000001 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.405575+0000", 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000113322 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.409521+0000", 2024-09-21T10:34:09.649 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039460179999999999 2024-09-21T10:34:09.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.650 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.650 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.650 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.410176+0000", 2024-09-21T10:34:09.650 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00065547499999999998 2024-09-21T10:34:09.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.668870+0000", 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25869356599999999 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.651 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.652 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.668928+0000", 2024-09-21T10:34:09.652 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8085999999999999e-05 2024-09-21T10:34:09.652 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.652 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.652 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.652 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:17.668974+0000", 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5779000000000003e-05 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.653 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:155870 46.b 46:d900c5cf:::benchmark_data_smithi119_206743_object50769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1267)", 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:22.306240+0000", 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 167.29683902599999, 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23280841799999999, 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.654 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 155870 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.306240+0000", 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.655 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.306240+0000", 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.306241+0000", 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.296e-06 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.656 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.306244+0000", 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5790000000000002e-06 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.306252+0000", 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8520000000000004e-06 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.306257+0000", 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.242e-06 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.657 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.441821+0000", 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.13556433100000001 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.441934+0000", 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011289 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.658 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.478740+0000", 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.036806278999999997 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.478840+0000", 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9977999999999997e-05 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.659 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.478985+0000", 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014517999999999999 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.486957+0000", 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079713900000000001 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.660 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.489817+0000", 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028600800000000001 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.535443+0000", 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.045626433000000001 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.535502+0000", 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8675999999999999e-05 2024-09-21T10:34:09.661 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:22.539048+0000", 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003546212 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:163978 46.b 46:dffc492e:::benchmark_data_smithi119_206743_object58877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1268)", 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:33.262711+0000", 2024-09-21T10:34:09.662 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 156.34036818600001, 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.241971462, 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 163978 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.663 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262711+0000", 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262711+0000", 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262713+0000", 2024-09-21T10:34:09.664 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8819999999999999e-06 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262718+0000", 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8139999999999997e-06 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262730+0000", 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1805e-05 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.665 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262738+0000", 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5179999999999998e-06 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262804+0000", 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6328000000000006e-05 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.666 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262937+0000", 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013334800000000001 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.268645+0000", 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057077730000000002 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.667 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.268703+0000", 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7797000000000002e-05 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.268867+0000", 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000164442 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.668 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281516+0000", 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012648313 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281555+0000", 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9184999999999998e-05 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.501427+0000", 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.219872346 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.669 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.501523+0000", 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6273999999999993e-05 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.504682+0000", 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031586370000000002 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.670 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:163991 46.b 46:d214d743:::benchmark_data_smithi119_206743_object58890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1268)", 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:33.277884+0000", 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 156.32519513, 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.240157019, 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:34:09.671 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 163991 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277884+0000", 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.672 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277884+0000", 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277885+0000", 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5009999999999999e-06 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.673 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277890+0000", 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5480000000000003e-06 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277901+0000", 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1134999999999999e-05 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.674 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277908+0000", 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2099999999999996e-06 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281580+0000", 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036715089999999999 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281645+0000", 2024-09-21T10:34:09.675 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5312000000000001e-05 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.288724+0000", 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007079381 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.288793+0000", 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8491999999999994e-05 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.676 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.289471+0000", 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00067867700000000003 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.299520+0000", 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010048443000000001 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.299536+0000", 2024-09-21T10:34:09.677 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6291999999999999e-05 2024-09-21T10:34:09.678 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.678 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.678 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.678 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.515106+0000", 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21557004900000001 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.515164+0000", 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7834999999999998e-05 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.679 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.518041+0000", 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028766350000000002 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:192766 46.9 46:9d2e794c:::benchmark_data_smithi119_206743_object87665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1269)", 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:32:12.232572+0000", 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 117.37050720400001, 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.248148233, 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.680 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192766 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232572+0000", 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.681 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232572+0000", 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232573+0000", 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4839999999999999e-06 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232577+0000", 2024-09-21T10:34:09.682 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.33e-06 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232588+0000", 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0647e-05 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232594+0000", 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6149999999999996e-06 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.683 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232641+0000", 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6866999999999997e-05 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232724+0000", 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3366000000000001e-05 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232801+0000", 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6679000000000002e-05 2024-09-21T10:34:09.684 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.236637+0000", 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038365209999999999 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.236882+0000", 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024469100000000001 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.480552+0000", 2024-09-21T10:34:09.685 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24366981200000001 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.480684+0000", 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000132667 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.480720+0000", 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5553999999999999e-05 2024-09-21T10:34:09.686 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54809 47.6 47:62b03f81:::benchmark_data_smithi119_238957_object54808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.645023+0000", 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9580557469999995, 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.2786305, 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.687 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54809 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645023+0000", 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645023+0000", 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.688 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645025+0000", 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9719999999999999e-06 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645112+0000", 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6680000000000004e-05 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645129+0000", 2024-09-21T10:34:09.689 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6762e-05 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645137+0000", 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0989999999999999e-06 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645217+0000", 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0133e-05 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645358+0000", 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000141018 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645527+0000", 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016966999999999999 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647754+0000", 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022260779999999998 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.649544+0000", 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017901670000000001 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923550+0000", 2024-09-21T10:34:09.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27400582499999998 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923610+0000", 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0229000000000002e-05 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923654+0000", 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3866999999999999e-05 2024-09-21T10:34:09.693 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54810 47.3 47:c9045a95:::benchmark_data_smithi119_238957_object54809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.645164+0000", 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9579144790000003, 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27704334800000002, 2024-09-21T10:34:09.694 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54810 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645164+0000", 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645164+0000", 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645165+0000", 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6899999999999997e-07 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645213+0000", 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7490000000000001e-05 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645221+0000", 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.585e-06 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645227+0000", 2024-09-21T10:34:09.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0090000000000004e-06 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645279+0000", 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379000000000001e-05 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645404+0000", 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012490799999999999 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645532+0000", 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012801699999999999 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.649757+0000", 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042247789999999997 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.650723+0000", 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000966258 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.922047+0000", 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27132334600000002 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.922103+0000", 2024-09-21T10:34:09.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5900999999999999e-05 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.922208+0000", 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010500700000000001 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.701 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54814 47.3 47:c491dd3a:::benchmark_data_smithi119_238957_object54813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.647488+0000", 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9555904670000004, 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27654822899999998, 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54814 2024-09-21T10:34:09.702 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647488+0000", 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647488+0000", 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.703 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647490+0000", 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9470000000000002e-06 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647521+0000", 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0806999999999998e-05 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647536+0000", 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5344000000000001e-05 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.704 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647545+0000", 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1680000000000004e-06 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647598+0000", 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3816000000000001e-05 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.705 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647764+0000", 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016596800000000001 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647898+0000", 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013345100000000001 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.649906+0000", 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020083200000000001 2024-09-21T10:34:09.706 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651295+0000", 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013887680000000001 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923936+0000", 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27264077599999997 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.707 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923984+0000", 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8647999999999999e-05 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924037+0000", 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2216000000000003e-05 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54822 47.6 47:676ee123:::benchmark_data_smithi119_238957_object54821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.651507+0000", 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9515714979999998, 2024-09-21T10:34:09.708 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27229856499999999, 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54822 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651507+0000", 2024-09-21T10:34:09.709 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651507+0000", 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651509+0000", 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.395e-06 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.710 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651530+0000", 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1491000000000001e-05 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651543+0000", 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3192e-05 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651550+0000", 2024-09-21T10:34:09.711 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2400000000000004e-06 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651614+0000", 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4203999999999997e-05 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651763+0000", 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000149603 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:34:09.712 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651859+0000", 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5863e-05 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653689+0000", 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00182942 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654168+0000", 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047968099999999998 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.713 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923704+0000", 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26953568999999999 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923768+0000", 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4109999999999994e-05 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.714 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923806+0000", 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7676000000000003e-05 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54824 47.3 47:c6adad3d:::benchmark_data_smithi119_238957_object54823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.653983+0000", 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.949095797, 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27022529699999998, 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.715 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54824 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653983+0000", 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.716 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653983+0000", 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653985+0000", 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7230000000000001e-06 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654035+0000", 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0497000000000001e-05 2024-09-21T10:34:09.717 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654048+0000", 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2682e-05 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654056+0000", 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9999999999999996e-06 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.718 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654115+0000", 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9545e-05 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654238+0000", 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012221700000000001 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654361+0000", 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123657 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.719 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656784+0000", 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024229429999999999 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656899+0000", 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114821 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924076+0000", 2024-09-21T10:34:09.720 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.267176529 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924151+0000", 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5177000000000002e-05 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924208+0000", 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7506000000000001e-05 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.721 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54826 47.6 47:6297b7c8:::benchmark_data_smithi119_238957_object54825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.654495+0000", 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9485841380000002, 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26946603200000002, 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.722 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54826 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654495+0000", 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.723 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654495+0000", 2024-09-21T10:34:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.724 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654496+0000", 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0410000000000001e-06 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654549+0000", 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3205999999999998e-05 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.725 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654563+0000", 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3793e-05 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654570+0000", 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5820000000000004e-06 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654626+0000", 2024-09-21T10:34:09.726 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5566000000000003e-05 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654759+0000", 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013260999999999999 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654910+0000", 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015190000000000001 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.727 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656822+0000", 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001912003 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656868+0000", 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6035999999999998e-05 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.728 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923863+0000", 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26699409200000002 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923922+0000", 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9947000000000001e-05 2024-09-21T10:34:09.729 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923961+0000", 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8256000000000001e-05 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54829 47.6 47:6dbd0281:::benchmark_data_smithi119_238957_object54828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.658574+0000", 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9445048289999995, 2024-09-21T10:34:09.730 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26858875599999998, 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54829 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658574+0000", 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.731 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658574+0000", 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658576+0000", 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7460000000000001e-06 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.732 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658629+0000", 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3380000000000001e-05 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658646+0000", 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7034999999999999e-05 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658654+0000", 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2379999999999997e-06 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.733 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658714+0000", 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0053000000000003e-05 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658841+0000", 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012661899999999999 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:34:09.734 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658968+0000", 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012709200000000001 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.660925+0000", 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019565519999999999 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.661023+0000", 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8535000000000004e-05 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.735 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927013+0000", 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26598982599999998 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927115+0000", 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102339 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927163+0000", 2024-09-21T10:34:09.736 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7341e-05 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54830 47.3 47:ccba9b51:::benchmark_data_smithi119_238957_object54829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.659013+0000", 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9440653799999996, 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26840792899999999, 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.737 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54830 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659013+0000", 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.738 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659013+0000", 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659014+0000", 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0200000000000001e-07 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659051+0000", 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7007e-05 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659059+0000", 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1319999999999994e-06 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659064+0000", 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4240000000000003e-06 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659120+0000", 2024-09-21T10:34:09.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6322e-05 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659247+0000", 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000127179 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659374+0000", 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012619199999999999 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.661230+0000", 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018564409999999999 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.661469+0000", 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00023875200000000001 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927285+0000", 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26581638400000002 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927381+0000", 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6274999999999996e-05 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927421+0000", 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0018999999999997e-05 2024-09-21T10:34:09.743 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54833 47.3 47:c8abb190:::benchmark_data_smithi119_238957_object54832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.664934+0000", 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.9381447569999999, 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26395025999999999, 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.744 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54833 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664934+0000", 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664934+0000", 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664935+0000", 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2839999999999999e-06 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664974+0000", 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8825999999999997e-05 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664985+0000", 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0944999999999999e-05 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664990+0000", 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8690000000000003e-06 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.665030+0000", 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0249e-05 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.665120+0000", 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9876999999999993e-05 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.665210+0000", 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9479000000000002e-05 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.666610+0000", 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014004950000000001 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.667132+0000", 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052222300000000004 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.928787+0000", 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26165429600000001 2024-09-21T10:34:09.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.928836+0000", 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9849999999999999e-05 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.928884+0000", 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7867000000000001e-05 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55562 47.13 47:ce06e73b:::benchmark_data_smithi119_238957_object55561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406134+0000", 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.1969450840000002, 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.028150865, 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55562 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.751 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406134+0000", 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406134+0000", 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406136+0000", 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2859999999999998e-06 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406165+0000", 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9339000000000001e-05 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406183+0000", 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7816e-05 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406193+0000", 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.842e-06 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.482248+0000", 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.076055306000000003 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.482258+0000", 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9490000000000008e-06 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.434220+0000", 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.95196215299999998 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.434285+0000", 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4173999999999999e-05 2024-09-21T10:34:09.755 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55574 47.16 47:68107290:::benchmark_data_smithi119_238957_object55573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.411282+0000", 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 7.1917969560000001, 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0768573640000001, 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:34:09.756 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55574 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411282+0000", 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411282+0000", 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411285+0000", 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8949999999999998e-06 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411316+0000", 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1191e-05 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411341+0000", 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.472e-05 2024-09-21T10:34:09.758 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411352+0000", 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.131e-05 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.500144+0000", 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.088791963000000002 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:34:09.759 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.500155+0000", 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0655999999999999e-05 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.479289+0000", 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.97913472499999998 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.479451+0000", 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016205599999999999 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:34:09.760 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.479692+0000", 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000240583 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481857+0000", 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021645470000000002 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.482008+0000", 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000151068 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.761 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.488040+0000", 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060320979999999996 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.488095+0000", 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5127000000000003e-05 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.488139+0000", 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4425000000000002e-05 2024-09-21T10:34:09.762 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.763 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.763 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.763 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:34:09.763 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:34:09.763 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:34:09.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:09.764 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:09.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:09 smithi145 ceph-mon[89359]: pgmap v2021: 119 pgs: 1 peering, 1 unknown, 117 active+clean; 3.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 55 MiB/s wr, 878 op/s; 30 KiB/s, 0 objects/s recovering 2024-09-21T10:34:09.806 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:34:09.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:34:09.807 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:34:09.807 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.821854 2024-09-21T10:34:09.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 0.8218539163183195 2024-09-21T10:34:09.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:09.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:09.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:09.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 34 16 68137 68121 125.205 107.688 0.00899984 0.00798213 2024-09-21T10:34:10.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.048 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:10.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:10.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:10.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:10.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:10.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:10.427 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:10.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:10.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:10.468 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:34:10.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:10 smithi145 ceph-mon[89359]: pgmap v2022: 119 pgs: 119 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s; 24 KiB/s, 0 objects/s recovering 2024-09-21T10:34:10.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:34:10.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/944843895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:10.547 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:10.547 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:34:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:34:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:10.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:10.579 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[101432]: pgmap v2022: 119 pgs: 119 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s; 24 KiB/s, 0 objects/s recovering 2024-09-21T10:34:10.579 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:34:10.580 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/944843895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:10.580 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:10.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[105288]: pgmap v2022: 119 pgs: 119 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 95 MiB/s wr, 1.51k op/s; 24 KiB/s, 0 objects/s recovering 2024-09-21T10:34:10.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"} : dispatch 2024-09-21T10:34:10.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/944843895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:10.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:10.593 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:10.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:10.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.840 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:10.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:10.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:10.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 35 16 69331 69315 123.76 74.625 0.00198178 0.00798648 2024-09-21T10:34:10.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:10.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:11.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:11.086 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:11.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:11.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.334 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:11.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:11.435 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:11.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:11.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:34:11.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:11.581 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:11.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:11.608 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:11.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:34:11.683 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:11.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:11.730 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:34:11.730 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:34:11.731 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[101432]: osdmap e1295: 8 total, 8 up, 8 in 2024-09-21T10:34:11.731 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:34:11.731 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:34:11.731 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:34:11.732 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:11 smithi119 ceph-mon[105288]: osdmap e1295: 8 total, 8 up, 8 in 2024-09-21T10:34:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:11 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:34:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "17"}]': finished 2024-09-21T10:34:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:34:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:11 smithi145 ceph-mon[89359]: osdmap e1295: 8 total, 8 up, 8 in 2024-09-21T10:34:11.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:11.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:11.932 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:11.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:11.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:11.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 36 16 69331 69315 120.322 0 - 0.00798648 2024-09-21T10:34:12.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:12.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:12.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:12.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:12.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:12.328 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:12.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:12.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:12.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:12.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:12.580 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:12.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:12 smithi119 ceph-mon[101432]: osdmap e1296: 8 total, 8 up, 8 in 2024-09-21T10:34:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:12 smithi119 ceph-mon[101432]: pgmap v2025: 119 pgs: 1 unknown, 118 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 24 KiB/s, 0 objects/s recovering 2024-09-21T10:34:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:12 smithi119 ceph-mon[105288]: osdmap e1296: 8 total, 8 up, 8 in 2024-09-21T10:34:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:12 smithi119 ceph-mon[105288]: pgmap v2025: 119 pgs: 1 unknown, 118 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 24 KiB/s, 0 objects/s recovering 2024-09-21T10:34:12.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:12.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:12.751 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:12 smithi145 ceph-mon[89359]: osdmap e1296: 8 total, 8 up, 8 in 2024-09-21T10:34:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:12 smithi145 ceph-mon[89359]: pgmap v2025: 119 pgs: 1 unknown, 118 active+clean; 3.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 94 MiB/s wr, 1.50k op/s; 24 KiB/s, 0 objects/s recovering 2024-09-21T10:34:12.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:12.831 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:12.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:12.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 37 16 69463 69447 117.293 4.125 0.00695382 0.00852059 2024-09-21T10:34:12.973 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:12.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:13.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:13.108 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:13.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:13.248 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:13.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:13.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:13.371 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:13.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:13.506 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:13.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:13.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:13.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:13.768 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:13 smithi119 ceph-mon[105288]: osdmap e1297: 8 total, 8 up, 8 in 2024-09-21T10:34:13.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:13 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:13.769 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:13 smithi119 ceph-mon[101432]: osdmap e1297: 8 total, 8 up, 8 in 2024-09-21T10:34:13.769 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:13 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:13.789 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:13 smithi145 ceph-mon[89359]: osdmap e1297: 8 total, 8 up, 8 in 2024-09-21T10:34:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:13 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:13.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:13.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:13.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:13.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 38 16 71187 71171 117.042 107.75 0.00926977 0.00853956 2024-09-21T10:34:14.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:14.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:14.136 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:34:14.136 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:34:14.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:14.171 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:34:14.172 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:34:14.198 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:14.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:14.213 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:34:14.213 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:34:14.254 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:34:14.254 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:34:14.285 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:34:14.285 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:34:14.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:14.314 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:34:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:34:14.335 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:14.346 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:34:14.346 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:34:14.381 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:34:14.381 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:34:14.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:14.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:14.574 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:14.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:34:14] "GET /metrics HTTP/1.1" 200 36885 "" "Prometheus/2.51.0" 2024-09-21T10:34:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-mon[105288]: pgmap v2027: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s; 2171/207135 objects misplaced (1.048%); 53 KiB/s, 0 objects/s recovering 2024-09-21T10:34:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:34:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1101404111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-mon[101432]: pgmap v2027: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s; 2171/207135 objects misplaced (1.048%); 53 KiB/s, 0 objects/s recovering 2024-09-21T10:34:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:34:14.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1101404111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:14.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:14.743 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:14 smithi145 ceph-mon[89359]: pgmap v2027: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 128 MiB/s wr, 2.05k op/s; 2171/207135 objects misplaced (1.048%); 53 KiB/s, 0 objects/s recovering 2024-09-21T10:34:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"} : dispatch 2024-09-21T10:34:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1101404111' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:14.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:14.870 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:14.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 39 16 72723 72707 116.502 96 0.00318372 0.00853024 2024-09-21T10:34:14.972 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:14.996 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:14.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:15.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:15.121 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:15.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:15.246 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:15.348 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:15.369 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:15.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:15.492 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:15.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:15.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:34:15.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:15.617 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:15.719 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:15.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:34:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:15 smithi145 ceph-mon[89359]: osdmap e1298: 8 total, 8 up, 8 in 2024-09-21T10:34:15.802 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:34:15.802 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:15 smithi119 ceph-mon[105288]: osdmap e1298: 8 total, 8 up, 8 in 2024-09-21T10:34:15.802 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "18"}]': finished 2024-09-21T10:34:15.802 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:15 smithi119 ceph-mon[101432]: osdmap e1298: 8 total, 8 up, 8 in 2024-09-21T10:34:15.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:15.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:15.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:34:15.956251+0000 min lat: 0.00109908 max lat: 2.40251 avg lat: 0.00852841 2024-09-21T10:34:15.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:34:15.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 40 16 72747 72731 113.627 1.5 0.00217794 0.00852841 2024-09-21T10:34:15.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:15.986 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:15.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:16.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:16.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:16.207 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:16.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:16.232 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:16.334 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:16.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:16.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:16.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:16.478 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:16.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:16.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:16.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:16.739 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:16 smithi119 ceph-mon[105288]: osdmap e1299: 8 total, 8 up, 8 in 2024-09-21T10:34:16.740 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:16 smithi119 ceph-mon[105288]: pgmap v2030: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 2171/210651 objects misplaced (1.031%); 76 KiB/s, 1 objects/s recovering 2024-09-21T10:34:16.740 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:34:16.741 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:16 smithi119 ceph-mon[101432]: osdmap e1299: 8 total, 8 up, 8 in 2024-09-21T10:34:16.741 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:16 smithi119 ceph-mon[101432]: pgmap v2030: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 2171/210651 objects misplaced (1.031%); 76 KiB/s, 1 objects/s recovering 2024-09-21T10:34:16.741 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:34:16.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:16 smithi145 ceph-mon[89359]: osdmap e1299: 8 total, 8 up, 8 in 2024-09-21T10:34:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:16 smithi145 ceph-mon[89359]: pgmap v2030: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 2171/210651 objects misplaced (1.031%); 76 KiB/s, 1 objects/s recovering 2024-09-21T10:34:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"} : dispatch 2024-09-21T10:34:16.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:16.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:16.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:16.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 41 16 73286 73270 111.677 33.6875 0.00404911 0.00892834 2024-09-21T10:34:16.989 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:17.120 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:17.146 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:17.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:17.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:17.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:17.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:17.394 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:17.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:17.510 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.7 primary-affinity to 0.821854 (8538612) 2024-09-21T10:34:17.523 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:34:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[105288]: osdmap e1300: 8 total, 8 up, 8 in 2024-09-21T10:34:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/276925193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195} : dispatch 2024-09-21T10:34:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195} : dispatch 2024-09-21T10:34:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:34:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[101432]: osdmap e1300: 8 total, 8 up, 8 in 2024-09-21T10:34:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/276925193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195} : dispatch 2024-09-21T10:34:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:17 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195} : dispatch 2024-09-21T10:34:17.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:17.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:17.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:17.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "19"}]': finished 2024-09-21T10:34:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:17 smithi145 ceph-mon[89359]: osdmap e1300: 8 total, 8 up, 8 in 2024-09-21T10:34:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/276925193' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195} : dispatch 2024-09-21T10:34:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:17 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195} : dispatch 2024-09-21T10:34:17.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:34:17.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:17.893 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:17.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:17.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 42 16 73514 73498 109.358 14.25 0.00546873 0.00891266 2024-09-21T10:34:17.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:18.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:18.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:18.142 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:18.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:18.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:18.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:18.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:18.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:18.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:18.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:18.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:18.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195}]': finished 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: osdmap e1301: 8 total, 8 up, 8 in 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: pgmap v2033: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 39 MiB/s wr, 629 op/s; 2171/214689 objects misplaced (1.011%) 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:34:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:18 smithi145 ceph-mon[89359]: osdmap e1302: 8 total, 8 up, 8 in 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195}]': finished 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: osdmap e1301: 8 total, 8 up, 8 in 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: pgmap v2033: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 39 MiB/s wr, 629 op/s; 2171/214689 objects misplaced (1.011%) 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:34:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[101432]: osdmap e1302: 8 total, 8 up, 8 in 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.8218539163183195}]': finished 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: osdmap e1301: 8 total, 8 up, 8 in 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: pgmap v2033: 119 pgs: 1 active+recovering+remapped, 118 active+clean; 4.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 39 MiB/s wr, 629 op/s; 2171/214689 objects misplaced (1.011%) 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"} : dispatch 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "20"}]': finished 2024-09-21T10:34:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:18 smithi119 ceph-mon[105288]: osdmap e1302: 8 total, 8 up, 8 in 2024-09-21T10:34:18.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:18.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:18.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:18.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:18.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 43 16 73920 73904 107.404 25.375 0.0015395 0.00926225 2024-09-21T10:34:19.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:19.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:19.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.158 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:19.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:19.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:19.279 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:19.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:19.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:19.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:19.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:19.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:19.528 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:34:19.528 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:34:19.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:19.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:19.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.771 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:19.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:19.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:19.893 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:19.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:19.957 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 44 16 73920 73904 104.963 0 - 0.00926225 2024-09-21T10:34:19.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:20.016 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:20.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:20.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:20.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:20.263 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:20.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:20.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:20.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:20.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:34:20.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:20 smithi145 ceph-mon[89359]: osdmap e1303: 8 total, 8 up, 8 in 2024-09-21T10:34:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:20 smithi145 ceph-mon[89359]: pgmap v2036: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 116 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.5 KiB/s rd, 47 MiB/s wr, 757 op/s; 2246/219678 objects degraded (1.022%); 3667/219678 objects misplaced (1.669%); 12 MiB/s, 187 objects/s recovering 2024-09-21T10:34:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:34:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:34:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:20.611 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:20.632 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:20.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:20.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:20.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:20 smithi119 ceph-mon[105288]: osdmap e1303: 8 total, 8 up, 8 in 2024-09-21T10:34:20.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:20 smithi119 ceph-mon[105288]: pgmap v2036: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 116 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.5 KiB/s rd, 47 MiB/s wr, 757 op/s; 2246/219678 objects degraded (1.022%); 3667/219678 objects misplaced (1.669%); 12 MiB/s, 187 objects/s recovering 2024-09-21T10:34:20.796 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:20 smithi119 ceph-mon[101432]: osdmap e1303: 8 total, 8 up, 8 in 2024-09-21T10:34:20.796 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:20 smithi119 ceph-mon[101432]: pgmap v2036: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 116 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.5 KiB/s rd, 47 MiB/s wr, 757 op/s; 2246/219678 objects degraded (1.022%); 3667/219678 objects misplaced (1.669%); 12 MiB/s, 187 objects/s recovering 2024-09-21T10:34:20.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:20.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:20.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:20.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 45 16 74309 74293 103.171 12.1562 0.0320802 0.00968833 2024-09-21T10:34:21.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:21.048 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:21.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:21.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:21.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:21.273 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:21.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:21.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:21.426 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:21.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.608 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:21.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:21.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:21.710 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:21 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:34:21.710 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:21 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 2246/219678 objects degraded (1.022%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:21.710 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:21 smithi119 ceph-mon[105288]: osdmap e1304: 8 total, 8 up, 8 in 2024-09-21T10:34:21.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:21 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:34:21.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:21 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 2246/219678 objects degraded (1.022%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:21.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:21 smithi119 ceph-mon[101432]: osdmap e1304: 8 total, 8 up, 8 in 2024-09-21T10:34:21.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:21.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:21 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:34:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:21 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 2246/219678 objects degraded (1.022%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:21 smithi145 ceph-mon[89359]: osdmap e1304: 8 total, 8 up, 8 in 2024-09-21T10:34:21.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:21.880 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:21.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:21.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 46 16 75848 75832 103.019 96.1875 0.0083757 0.00970253 2024-09-21T10:34:21.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:22.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:22.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:22.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:22.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:22.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:22.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:22.433 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:22.500 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:34:22.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1721355125' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:34:22.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:22 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:34:22.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:22 smithi119 ceph-mon[101432]: pgmap v2038: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 115 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.2 KiB/s rd, 34 MiB/s wr, 541 op/s; 4510/221385 objects degraded (2.037%); 3667/221385 objects misplaced (1.656%); 11 MiB/s, 180 objects/s recovering 2024-09-21T10:34:22.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1721355125' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:34:22.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:22 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:34:22.519 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:22 smithi119 ceph-mon[105288]: pgmap v2038: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 115 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.2 KiB/s rd, 34 MiB/s wr, 541 op/s; 4510/221385 objects degraded (2.037%); 3667/221385 objects misplaced (1.656%); 11 MiB/s, 180 objects/s recovering 2024-09-21T10:34:22.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:22.566 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:22.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:22.704 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1721355125' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:34:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:22 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:34:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:22 smithi145 ceph-mon[89359]: pgmap v2038: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 1 peering, 115 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.2 KiB/s rd, 34 MiB/s wr, 541 op/s; 4510/221385 objects degraded (2.037%); 3667/221385 objects misplaced (1.656%); 11 MiB/s, 180 objects/s recovering 2024-09-21T10:34:22.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:22.840 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:22.942 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:22.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 47 16 77479 77463 102.996 101.938 0.0202586 0.00970412 2024-09-21T10:34:22.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:22.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:23.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:23.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:23.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:23.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:23.249 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:23.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:23.371 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:23.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:23.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:23.505 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:23.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:23 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:34:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:34:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:23 smithi119 ceph-mon[101432]: osdmap e1305: 8 total, 8 up, 8 in 2024-09-21T10:34:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:23 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:34:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:34:23.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:23 smithi119 ceph-mon[105288]: osdmap e1305: 8 total, 8 up, 8 in 2024-09-21T10:34:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.649 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:23.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:23.750 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:23.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:23 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:34:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:34:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:23 smithi145 ceph-mon[89359]: osdmap e1305: 8 total, 8 up, 8 in 2024-09-21T10:34:23.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:23.923 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:23.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:23.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 48 15 79023 79008 102.862 96.5625 0.00739681 0.00971769 2024-09-21T10:34:24.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:24.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:24.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:24.160 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:24.192 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:24.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:24.253 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:34:24.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:34:24.253 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-09-21T10:34:24.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:34:24.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:24.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:24.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:24.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:24.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:24.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:24 smithi119 ceph-mon[101432]: pgmap v2040: 119 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 6.6 KiB/s rd, 37 MiB/s wr, 594 op/s; 6808/227799 objects degraded (2.989%); 2665/227799 objects misplaced (1.170%); 12 MiB/s, 194 objects/s recovering 2024-09-21T10:34:24.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:24 smithi119 ceph-mon[105288]: pgmap v2040: 119 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 6.6 KiB/s rd, 37 MiB/s wr, 594 op/s; 6808/227799 objects degraded (2.989%); 2665/227799 objects misplaced (1.170%); 12 MiB/s, 194 objects/s recovering 2024-09-21T10:34:24.544 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:34:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:34:24] "GET /metrics HTTP/1.1" 200 36885 "" "Prometheus/2.51.0" 2024-09-21T10:34:24.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:24.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:24.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:24.750 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:24.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:24 smithi145 ceph-mon[89359]: pgmap v2040: 119 pgs: 1 active+recovering+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 115 active+clean; 4.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 6.6 KiB/s rd, 37 MiB/s wr, 594 op/s; 6808/227799 objects degraded (2.989%); 2665/227799 objects misplaced (1.170%); 12 MiB/s, 194 objects/s recovering 2024-09-21T10:34:24.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:24.893 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:24.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:24.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 49 16 80571 80555 102.736 96.6875 0.00595644 0.00972814 2024-09-21T10:34:24.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:25.021 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:25.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:25.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:25.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:25.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:25.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:25.429 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:25.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:25.566 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:25.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.693 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:25.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:25.768 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:25 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:34:25.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:25 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:34:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:25 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:34:25.794 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:25.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:25.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:25.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 50 16 82119 82103 102.616 96.75 0.0105361 0.00974029 2024-09-21T10:34:25.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:25.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:26.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:26.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:26.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:26.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:26.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:26.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:26.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:26.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:26.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:26 smithi119 ceph-mon[101432]: pgmap v2041: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 42 MiB/s wr, 665 op/s; 6808/232080 objects degraded (2.933%); 2665/232080 objects misplaced (1.148%); 10 MiB/s, 161 objects/s recovering 2024-09-21T10:34:26.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:26 smithi119 ceph-mon[105288]: pgmap v2041: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 42 MiB/s wr, 665 op/s; 6808/232080 objects degraded (2.933%); 2665/232080 objects misplaced (1.148%); 10 MiB/s, 161 objects/s recovering 2024-09-21T10:34:26.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:26.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:26.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:26 smithi145 ceph-mon[89359]: pgmap v2041: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 42 MiB/s wr, 665 op/s; 6808/232080 objects degraded (2.933%); 2665/232080 objects misplaced (1.148%); 10 MiB/s, 161 objects/s recovering 2024-09-21T10:34:26.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:26.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:26.956 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:26.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:26.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 51 16 83593 83577 102.41 92.125 0.0186978 0.00975959 2024-09-21T10:34:27.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:27.090 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:27.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:27.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:27.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:27.367 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:27.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:27.500 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:27.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:27.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:27 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 6808/232080 objects degraded (2.933%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:34:27.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:27 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 6808/232080 objects degraded (2.933%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:34:27.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:27.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:27.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:27.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:27 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 6808/232080 objects degraded (2.933%), 3 pgs degraded (PG_DEGRADED) 2024-09-21T10:34:27.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:27.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:27.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 52 16 84868 84852 101.973 79.6875 0.0121418 0.00976935 2024-09-21T10:34:27.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:28.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:28.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:28.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:28.272 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:28.304 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:28.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:28.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:34:28.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:28.427 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:28.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:28.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:28.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:28.681 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:28.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:28.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:28 smithi145 ceph-mon[89359]: osdmap e1306: 8 total, 8 up, 8 in 2024-09-21T10:34:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:28 smithi145 ceph-mon[89359]: pgmap v2043: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s; 6808/240690 objects degraded (2.829%); 2665/240690 objects misplaced (1.107%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:34:28.823 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:28 smithi119 ceph-mon[105288]: osdmap e1306: 8 total, 8 up, 8 in 2024-09-21T10:34:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:28 smithi119 ceph-mon[105288]: pgmap v2043: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s; 6808/240690 objects degraded (2.829%); 2665/240690 objects misplaced (1.107%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:34:28.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:28 smithi119 ceph-mon[101432]: osdmap e1306: 8 total, 8 up, 8 in 2024-09-21T10:34:28.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:28 smithi119 ceph-mon[101432]: pgmap v2043: 119 pgs: 1 active+recovering+remapped, 1 active+clean+scrubbing, 3 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 60 MiB/s wr, 954 op/s; 6808/240690 objects degraded (2.829%); 2665/240690 objects misplaced (1.107%); 8.7 MiB/s, 139 objects/s recovering 2024-09-21T10:34:28.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:28.946 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:28.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:28.958 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 53 16 85471 85455 100.76 37.6875 0.00767452 0.00991743 2024-09-21T10:34:29.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:29.080 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:29.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:29.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:29.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:29.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:29.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:29.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:29.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:29.584 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:29.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:29.721 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:29 smithi145 ceph-mon[89359]: osdmap e1307: 8 total, 8 up, 8 in 2024-09-21T10:34:29.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:29.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:29.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:29 smithi119 ceph-mon[101432]: osdmap e1307: 8 total, 8 up, 8 in 2024-09-21T10:34:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:29 smithi119 ceph-mon[105288]: osdmap e1307: 8 total, 8 up, 8 in 2024-09-21T10:34:29.959 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 54 16 86271 86255 99.8198 50 0.00542059 0.0100054 2024-09-21T10:34:29.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:29.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:29.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:30.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:30.125 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:30.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:30.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:30.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:30.387 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:30.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:30.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:30.523 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:30.537 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:34:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:34:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:30.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:30.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:30.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:30.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:30 smithi145 ceph-mon[89359]: pgmap v2045: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 4562/250500 objects degraded (1.821%); 4703/250500 objects misplaced (1.877%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:34:30.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:30.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:30.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:30 smithi119 ceph-mon[101432]: pgmap v2045: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 4562/250500 objects degraded (1.821%); 4703/250500 objects misplaced (1.877%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:34:30.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:30 smithi119 ceph-mon[105288]: pgmap v2045: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 114 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 81 MiB/s wr, 1.29k op/s; 4562/250500 objects degraded (1.821%); 4703/250500 objects misplaced (1.877%); 9.6 MiB/s, 154 objects/s recovering 2024-09-21T10:34:30.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:30.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:30.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:30.959 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 55 16 87422 87406 99.3127 71.9375 0.0206569 0.0100636 2024-09-21T10:34:31.038 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:31.066 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:31.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:31.191 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:31.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:31.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:31.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:31.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:31.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:31.479 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:31.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:31.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:31.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:31 smithi145 ceph-mon[89359]: 47.15 scrub starts 2024-09-21T10:34:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:31 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:34:31.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:31 smithi119 ceph-mon[101432]: 47.15 scrub starts 2024-09-21T10:34:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:31 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:34:31.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:31 smithi119 ceph-mon[105288]: 47.15 scrub starts 2024-09-21T10:34:31.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:31 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:34:31.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:31.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:31.959 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 56 16 88557 88541 98.8059 70.9375 0.0065448 0.0101164 2024-09-21T10:34:31.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:32.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:32.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:32.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:32.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:32.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:32.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:32.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:32.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:32.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:32.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:32.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:32.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:32.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:32.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:32 smithi145 ceph-mon[89359]: 47.15 scrub ok 2024-09-21T10:34:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:32 smithi145 ceph-mon[89359]: pgmap v2046: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s; 2298/257568 objects degraded (0.892%); 6302/257568 objects misplaced (2.447%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:34:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:32.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:32 smithi119 ceph-mon[105288]: 47.15 scrub ok 2024-09-21T10:34:32.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:32 smithi119 ceph-mon[105288]: pgmap v2046: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s; 2298/257568 objects degraded (0.892%); 6302/257568 objects misplaced (2.447%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:34:32.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:32 smithi119 ceph-mon[101432]: 47.15 scrub ok 2024-09-21T10:34:32.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:32 smithi119 ceph-mon[101432]: pgmap v2046: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 78 MiB/s wr, 1.24k op/s; 2298/257568 objects degraded (0.892%); 6302/257568 objects misplaced (2.447%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:34:32.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:32.949 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:32.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:32.959 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 57 16 89363 89347 97.9562 50.375 0.00663486 0.0102028 2024-09-21T10:34:33.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:33.082 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:33.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:33.205 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:33.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:33.347 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:33.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:33.479 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:33.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:33 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2298/257568 objects degraded (0.892%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:33 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2298/257568 objects degraded (0.892%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:33.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:33.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:33.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:33.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:33.860 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:33 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2298/257568 objects degraded (0.892%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:33.860 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:33.860 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:33.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:33.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:33.959 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 58 16 90613 90597 97.6142 78.125 0.0206737 0.0102396 2024-09-21T10:34:33.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:34.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:34.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:34.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:34.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:34.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:34.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:34.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:34.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:34.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:34.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:34.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:34.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:34.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:34.803 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:34 smithi119 ceph-mon[101432]: pgmap v2047: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 2298/264330 objects degraded (0.869%); 6302/264330 objects misplaced (2.384%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:34:34.803 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/100078461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:34.803 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:34:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:34:34] "GET /metrics HTTP/1.1" 200 36908 "" "Prometheus/2.51.0" 2024-09-21T10:34:34.804 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:34 smithi119 ceph-mon[105288]: pgmap v2047: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 2298/264330 objects degraded (0.869%); 6302/264330 objects misplaced (2.384%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:34:34.804 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/100078461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:34.821 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:34.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:34.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:34.959 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 59 16 91701 91685 97.1109 68 0.0168233 0.010292 2024-09-21T10:34:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:34.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:34 smithi145 ceph-mon[89359]: pgmap v2047: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 84 MiB/s wr, 1.34k op/s; 2298/264330 objects degraded (0.869%); 6302/264330 objects misplaced (2.384%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:34:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/100078461' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:34:35.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:35.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:35.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:35.252 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:35.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:35.354 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.387 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:35.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:35.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:34:35.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:34:35.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:34:35.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:34:35.397 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:34:35.397 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:34:35.397 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:34:35.397 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:34:35.397 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:34:35.397 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:34:35.398 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:34:35.398 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:34:35.398 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:34:35.398 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:34:35.398 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:34:35.398 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:34:35.488 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:35.509 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:35.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:35.631 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:35.733 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:35.754 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:35.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:35 smithi119 ceph-mon[101432]: 47.5 scrub starts 2024-09-21T10:34:35.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:35 smithi119 ceph-mon[105288]: 47.5 scrub starts 2024-09-21T10:34:35.890 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:35.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:35.960 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:34:35.958947+0000 min lat: 0.00109908 max lat: 2.40251 avg lat: 0.010356 2024-09-21T10:34:35.960 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:34:35.960 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 60 16 92683 92667 96.5153 61.375 0.0120564 0.010356 2024-09-21T10:34:35.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:36.022 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:35 smithi145 ceph-mon[89359]: 47.5 scrub starts 2024-09-21T10:34:36.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:36.168 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:36.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:36.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:36.438 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:36.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:36.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:36.665 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:36.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:36.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:36.840 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:36 smithi119 ceph-mon[101432]: 47.5 scrub ok 2024-09-21T10:34:36.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:36 smithi119 ceph-mon[101432]: pgmap v2048: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s; 2298/268119 objects degraded (0.857%); 5760/268119 objects misplaced (2.148%); 8.0 MiB/s, 127 objects/s recovering 2024-09-21T10:34:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:36 smithi119 ceph-mon[105288]: 47.5 scrub ok 2024-09-21T10:34:36.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:36 smithi119 ceph-mon[105288]: pgmap v2048: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s; 2298/268119 objects degraded (0.857%); 5760/268119 objects misplaced (2.148%); 8.0 MiB/s, 127 objects/s recovering 2024-09-21T10:34:36.942 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:36.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:36 smithi145 ceph-mon[89359]: 47.5 scrub ok 2024-09-21T10:34:36.943 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:36 smithi145 ceph-mon[89359]: pgmap v2048: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 49 MiB/s wr, 789 op/s; 2298/268119 objects degraded (0.857%); 5760/268119 objects misplaced (2.148%); 8.0 MiB/s, 127 objects/s recovering 2024-09-21T10:34:36.960 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 61 16 93613 93597 95.8857 58.125 0.00426293 0.010406 2024-09-21T10:34:36.966 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:36.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:37.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:37.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:37.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:37.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:37.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.382 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:37.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:37.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:37.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.531 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:37.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:37.632 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:37.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:37.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:37.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:37.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:37.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:37.960 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:37.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:37.962 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 62 16 94817 94801 95.5528 75.25 0.0158901 0.0104609 2024-09-21T10:34:38.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:38.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:38.189 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:38.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:38.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:38.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:38.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:38.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:38.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.639 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:38.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:38.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:38.767 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[105288]: pgmap v2049: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 54 MiB/s wr, 859 op/s; 2298/274623 objects degraded (0.837%); 5476/274623 objects misplaced (1.994%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[101432]: pgmap v2049: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 54 MiB/s wr, 859 op/s; 2298/274623 objects degraded (0.837%); 5476/274623 objects misplaced (1.994%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:34:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:38.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:38.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:38.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:38.960 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 63 16 95956 95940 95.166 71.1875 0.0145246 0.0105028 2024-09-21T10:34:38.997 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:39.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:38 smithi145 ceph-mon[89359]: pgmap v2049: 119 pgs: 1 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 54 MiB/s wr, 859 op/s; 2298/274623 objects degraded (0.837%); 5476/274623 objects misplaced (1.994%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:34:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:39.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:34:39.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:39.172 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:39.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:39.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:39.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:39.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:39.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:39.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:39.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:39.704 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:39.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:39.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2298/274623 objects degraded (0.837%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2298/274623 objects degraded (0.837%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:39.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:39.961 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 64 16 96920 96904 94.6197 60.25 0.00622247 0.0105637 2024-09-21T10:34:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:39.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:39.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2298/274623 objects degraded (0.837%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:40.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:40.096 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:40.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:40.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:40.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:40.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:40.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:40.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:34:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:34:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:40.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:40.635 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:40.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:40.781 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:40.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:40.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:40 smithi119 ceph-mon[101432]: pgmap v2050: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 115 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 48 MiB/s wr, 775 op/s; 2298/282195 objects degraded (0.814%); 4936/282195 objects misplaced (1.749%); 8.9 MiB/s, 143 objects/s recovering 2024-09-21T10:34:40.869 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:40 smithi119 ceph-mon[105288]: pgmap v2050: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 115 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 48 MiB/s wr, 775 op/s; 2298/282195 objects degraded (0.814%); 4936/282195 objects misplaced (1.749%); 8.9 MiB/s, 143 objects/s recovering 2024-09-21T10:34:40.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:40.920 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:40.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:40.962 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 65 16 98221 98205 94.4126 81.3125 0.00877297 0.0105872 2024-09-21T10:34:41.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:41.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:40 smithi145 ceph-mon[89359]: pgmap v2050: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+remapped, 115 active+clean; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 48 MiB/s wr, 775 op/s; 2298/282195 objects degraded (0.814%); 4936/282195 objects misplaced (1.749%); 8.9 MiB/s, 143 objects/s recovering 2024-09-21T10:34:41.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:41.166 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:41.166 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:41.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:41.192 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:41.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:41.211 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:34:41.212 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:34:41.212 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:34:41.212 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:34:41.212 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:34:41.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:41.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:41.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:41.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:41.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:41.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.752 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:41.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:41.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:41.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:41.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:41.962 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 66 16 99488 99472 94.1818 79.1875 0.00423489 0.0106132 2024-09-21T10:34:41.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.038 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:42.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:42.139 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:42.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:42.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:42.307 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:42.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:42.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:42.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:34:42.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:42.582 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:42.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:42.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:42.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:42.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:42.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:42 smithi119 ceph-mon[105288]: pgmap v2051: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 MiB/s wr, 602 op/s; 2298/289152 objects degraded (0.795%); 4698/289152 objects misplaced (1.625%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:34:42.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:42 smithi119 ceph-mon[101432]: pgmap v2051: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 MiB/s wr, 602 op/s; 2298/289152 objects degraded (0.795%); 4698/289152 objects misplaced (1.625%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:34:42.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:42.962 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 67 16 100711 100695 93.9168 76.4375 0.0102303 0.0106422 2024-09-21T10:34:42.995 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:42.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:43.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:42 smithi145 ceph-mon[89359]: pgmap v2051: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 38 MiB/s wr, 602 op/s; 2298/289152 objects degraded (0.795%); 4698/289152 objects misplaced (1.625%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:34:43.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:43.132 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:43.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:43.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:43.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:43.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:43.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:43.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:43.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:43.676 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:43.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:43.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:43.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:43.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:43.946 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:43.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:43.963 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 68 16 101864 101848 93.5954 72.0625 0.0132031 0.01068 2024-09-21T10:34:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:43 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2298/297582 objects degraded (0.772%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:44.047 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:44.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:44.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:43 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2298/297582 objects degraded (0.772%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:43 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2298/297582 objects degraded (0.772%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:44.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:44.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:44.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:44.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:44.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:44.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:44.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:44.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:44.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:34:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:34:44] "GET /metrics HTTP/1.1" 200 36908 "" "Prometheus/2.51.0" 2024-09-21T10:34:44.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:44.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.783 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:44.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:44.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:44.916 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:44.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:44.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:44.963 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 69 16 103201 103185 93.4499 83.5625 0.00968565 0.010696 2024-09-21T10:34:45.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:44 smithi145 ceph-mon[89359]: pgmap v2052: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 40 MiB/s wr, 639 op/s; 2298/297582 objects degraded (0.772%); 4698/297582 objects misplaced (1.579%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:34:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:45.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:45.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:44 smithi119 ceph-mon[101432]: pgmap v2052: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 40 MiB/s wr, 639 op/s; 2298/297582 objects degraded (0.772%); 4698/297582 objects misplaced (1.579%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:34:45.127 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:44 smithi119 ceph-mon[105288]: pgmap v2052: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 40 MiB/s wr, 639 op/s; 2298/297582 objects degraded (0.772%); 4698/297582 objects misplaced (1.579%); 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:34:45.159 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:45.193 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:45.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:45.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:45.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:45.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:45.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:45.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:45.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:45.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:45.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:45.890 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:45.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:45.963 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 70 16 104475 104459 93.2523 79.625 0.00808997 0.0107188 2024-09-21T10:34:45.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:46.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.026 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:46.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:46.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:46.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:46.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:46.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:46.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:46.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:46.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.581 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:46.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:46.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:46.718 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:46.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:46.861 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:46.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:46.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:46 smithi119 ceph-mon[105288]: pgmap v2053: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 36 MiB/s wr, 571 op/s; 2298/301893 objects degraded (0.761%); 4060/301893 objects misplaced (1.345%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T10:34:46.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:46 smithi119 ceph-mon[101432]: pgmap v2053: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 36 MiB/s wr, 571 op/s; 2298/301893 objects degraded (0.761%); 4060/301893 objects misplaced (1.345%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T10:34:46.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:46.963 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 71 16 105736 105720 93.0479 78.8125 0.00989357 0.0107424 2024-09-21T10:34:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:46.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:46.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:46 smithi145 ceph-mon[89359]: pgmap v2053: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 36 MiB/s wr, 571 op/s; 2298/301893 objects degraded (0.761%); 4060/301893 objects misplaced (1.345%); 7.2 MiB/s, 115 objects/s recovering 2024-09-21T10:34:47.100 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:47.132 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:47.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:47.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:47.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:47.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:47.404 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:47.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:47.542 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:47.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:47.676 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:47.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:47.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:47.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:47.950 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:47.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:47.963 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 72 16 107018 107002 92.8684 80.125 0.024437 0.0107626 2024-09-21T10:34:48.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:48.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:48.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:48.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:48.185 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:34:48.324 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:48.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:48.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:48.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:48.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:48.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:48.642 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:48.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:48.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:48.884 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:48.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:48.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:48.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 73 16 108045 108029 92.4755 64.1875 0.0248399 0.0108092 2024-09-21T10:34:49.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.060 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:49.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:48 smithi119 ceph-mon[101432]: pgmap v2054: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 906 op/s; 2298/309597 objects degraded (0.742%); 3799/309597 objects misplaced (1.227%); 7.1 MiB/s, 114 objects/s recovering 2024-09-21T10:34:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:48 smithi119 ceph-mon[105288]: pgmap v2054: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 906 op/s; 2298/309597 objects degraded (0.742%); 3799/309597 objects misplaced (1.227%); 7.1 MiB/s, 114 objects/s recovering 2024-09-21T10:34:49.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:49.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:48 smithi145 ceph-mon[89359]: pgmap v2054: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 57 MiB/s wr, 906 op/s; 2298/309597 objects degraded (0.742%); 3799/309597 objects misplaced (1.227%); 7.1 MiB/s, 114 objects/s recovering 2024-09-21T10:34:49.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:49.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:49.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:49.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:49.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:49.599 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:49.702 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.745 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:49.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:49.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:49.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:49.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:49.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:34:49.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 74 16 109242 109226 92.2365 74.8125 0.00579159 0.0108367 2024-09-21T10:34:49.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:50.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:50.110 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:49 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2298/309597 objects degraded (0.742%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:49 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2298/309597 objects degraded (0.742%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:50.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:50.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:49 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2298/309597 objects degraded (0.742%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:50.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:50.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:50.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:50.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:50.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:50.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:34:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:34:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:34:50.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:50.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:50.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:50.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:50.844 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:50 smithi119 ceph-mon[101432]: pgmap v2055: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 965 op/s; 2298/318222 objects degraded (0.722%); 3121/318222 objects misplaced (0.981%); 7.4 MiB/s, 118 objects/s recovering 2024-09-21T10:34:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:50 smithi119 ceph-mon[105288]: pgmap v2055: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 965 op/s; 2298/318222 objects degraded (0.722%); 3121/318222 objects misplaced (0.981%); 7.4 MiB/s, 118 objects/s recovering 2024-09-21T10:34:50.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:50.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 75 16 110438 110422 92.0033 74.75 0.0301982 0.0108642 2024-09-21T10:34:50.981 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:50.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:51.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:51.110 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:51.119 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:50 smithi145 ceph-mon[89359]: pgmap v2055: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 60 MiB/s wr, 965 op/s; 2298/318222 objects degraded (0.722%); 3121/318222 objects misplaced (0.981%); 7.4 MiB/s, 118 objects/s recovering 2024-09-21T10:34:51.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:51.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:51.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:51.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:51.421 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:51.427 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:34:51.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:51.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:51.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:51.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:51.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:51.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:51.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:51.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:51.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:51.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:51.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 76 16 111548 111532 91.7055 69.375 0.0105882 0.0108996 2024-09-21T10:34:52.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:52.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:52.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.244 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:52.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:52.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.366 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:52.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:52.467 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:52.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:52.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:52.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:52.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:52.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:52.840 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:34:52.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:52.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:52.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:52.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:52.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 77 16 112722 112706 91.4673 73.375 0.00724013 0.0109279 2024-09-21T10:34:53.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.049 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:53.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:52 smithi119 ceph-mon[105288]: pgmap v2056: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s; 2298/325599 objects degraded (0.706%); 2839/325599 objects misplaced (0.872%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T10:34:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1959924163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:53.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:52 smithi119 ceph-mon[101432]: pgmap v2056: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s; 2298/325599 objects degraded (0.706%); 2839/325599 objects misplaced (0.872%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T10:34:53.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1959924163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:53.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:53.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:53.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:53.283 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:52 smithi145 ceph-mon[89359]: pgmap v2056: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 60 MiB/s wr, 959 op/s; 2298/325599 objects degraded (0.706%); 2839/325599 objects misplaced (0.872%); 7.5 MiB/s, 119 objects/s recovering 2024-09-21T10:34:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1959924163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:34:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:53.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:53.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:53.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:53.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:53.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:53.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:53.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:53.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:53.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:53.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:53.868 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:53.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:53.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 78 16 114013 113997 91.329 80.6875 0.0140986 0.0109449 2024-09-21T10:34:53.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:54.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:54.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:53 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:34:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1959924163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:34:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:53 smithi119 ceph-mon[105288]: osdmap e1308: 8 total, 8 up, 8 in 2024-09-21T10:34:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:53 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:34:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1959924163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:34:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:53 smithi119 ceph-mon[101432]: osdmap e1308: 8 total, 8 up, 8 in 2024-09-21T10:34:54.133 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:54.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.275 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:54.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:53 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:34:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1959924163' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:34:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:53 smithi145 ceph-mon[89359]: osdmap e1308: 8 total, 8 up, 8 in 2024-09-21T10:34:54.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:54.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:54.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:54.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:54.657 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:54.688 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:54.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:54.717 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:34:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:34:54] "GET /metrics HTTP/1.1" 200 36912 "" "Prometheus/2.51.0" 2024-09-21T10:34:54.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:54.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.832 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:54.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:54.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:54.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 79 16 115276 115260 91.1721 78.9375 0.0158164 0.0109636 2024-09-21T10:34:54.967 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:54.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:55.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:55.110 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:54 smithi119 ceph-mon[105288]: pgmap v2058: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 2298/333735 objects degraded (0.689%); 2839/333735 objects misplaced (0.851%); 7.5 MiB/s, 120 objects/s recovering 2024-09-21T10:34:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:54 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2298/333735 objects degraded (0.689%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:54 smithi119 ceph-mon[101432]: pgmap v2058: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 2298/333735 objects degraded (0.689%); 2839/333735 objects misplaced (0.851%); 7.5 MiB/s, 120 objects/s recovering 2024-09-21T10:34:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:54 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2298/333735 objects degraded (0.689%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:55.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:55.249 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:54 smithi145 ceph-mon[89359]: pgmap v2058: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 2298/333735 objects degraded (0.689%); 2839/333735 objects misplaced (0.851%); 7.5 MiB/s, 120 objects/s recovering 2024-09-21T10:34:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:54 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2298/333735 objects degraded (0.689%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:55.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:55.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:55.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:55.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:55.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:55.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:55.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:55.767 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:55.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:55.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:55.936 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:55.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:55.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:34:55.963669+0000 min lat: 0.00109908 max lat: 2.40251 avg lat: 0.0109917 2024-09-21T10:34:55.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:34:55.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 80 15 116439 116424 90.9418 72.75 0.0126651 0.0109917 2024-09-21T10:34:56.038 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:56.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:56.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:56.198 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:56.300 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:56.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:56.423 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:56.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:56.556 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:56.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:56.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:56.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:56.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:56.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:56.964 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 81 16 117426 117410 90.5798 61.625 0.0116332 0.0110354 2024-09-21T10:34:56.982 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:56.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:56.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:57.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:57.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:56 smithi119 ceph-mon[105288]: pgmap v2059: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 2298/338241 objects degraded (0.679%); 2141/338241 objects misplaced (0.633%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:34:57.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:56 smithi119 ceph-mon[101432]: pgmap v2059: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 2298/338241 objects degraded (0.679%); 2141/338241 objects misplaced (0.633%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:34:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:57.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:57.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:57.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:56 smithi145 ceph-mon[89359]: pgmap v2059: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 6.9 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 76 MiB/s wr, 1.21k op/s; 2298/338241 objects degraded (0.679%); 2141/338241 objects misplaced (0.633%); 7.7 MiB/s, 123 objects/s recovering 2024-09-21T10:34:57.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:34:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:34:57.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:34:57.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:57.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:34:57.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:57.688 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:57.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:57.790 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:57.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:57.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:57.964 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:57.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:57.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 82 16 118597 118581 90.3676 73.1875 0.0131074 0.011061 2024-09-21T10:34:58.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:58.103 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:58.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:58.225 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:58.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:58.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:58.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:58.504 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:58.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:58.632 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:58.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:34:58.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:58.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:34:58.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:58.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:58.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:58.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:58.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 83 16 119828 119812 90.2057 76.9375 0.0140798 0.0110813 2024-09-21T10:34:59.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:58 smithi119 ceph-mon[105288]: pgmap v2060: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 2298/345501 objects degraded (0.665%); 1787/345501 objects misplaced (0.517%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:34:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:58 smithi119 ceph-mon[101432]: pgmap v2060: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 2298/345501 objects degraded (0.665%); 1787/345501 objects misplaced (0.517%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:34:59.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:34:59.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:34:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:58 smithi145 ceph-mon[89359]: pgmap v2060: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 75 MiB/s wr, 1.20k op/s; 2298/345501 objects degraded (0.665%); 1787/345501 objects misplaced (0.517%); 8.3 MiB/s, 132 objects/s recovering 2024-09-21T10:34:59.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:59.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:59.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:59.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:34:59.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:59.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:34:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.593 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:59.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:34:59.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:34:59.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:34:59.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:34:59.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:34:59.866 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:59.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:34:59.961 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:34:59.962 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:34:59.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 84 16 120974 120958 89.9845 71.625 0.00314358 0.0111063 2024-09-21T10:34:59.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:34:59.973 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:34:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:34:59.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:34:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:34:59.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:34:59.995 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:34:59 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2298/345501 objects degraded (0.665%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:34:59.995 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:34:59.996 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:34:59.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:34:59 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2298/345501 objects degraded (0.665%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:35:00.043 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:35:00.043 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:35:00.088 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:35:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:35:00.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:00.116 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:35:00.116 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:35:00.128 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:00.147 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:35:00.147 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:35:00.156 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:35:00.158 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:35:00.175 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:35:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:35:00.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:34:59 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2298/345501 objects degraded (0.665%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:35:00.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:00.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:00.360 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:00.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:00.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:00.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:35:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:35:00.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:00.642 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:00.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:00.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:00.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:00.896 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:00.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:00.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 85 16 121288 121272 89.1568 19.625 0.0068817 0.0112115 2024-09-21T10:35:00.998 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:01.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:01.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:01.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:01.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:01.182 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[105288]: pgmap v2061: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 2298/353610 objects degraded (0.650%); 805/353610 objects misplaced (0.228%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:35:01.182 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[105288]: osdmap e1309: 8 total, 8 up, 8 in 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/677888247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[105288]: 47.0 deep-scrub starts 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[105288]: from='client.55687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[105288]: 47.16 scrub starts 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[101432]: pgmap v2061: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 2298/353610 objects degraded (0.650%); 805/353610 objects misplaced (0.228%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[101432]: osdmap e1309: 8 total, 8 up, 8 in 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/677888247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[101432]: 47.0 deep-scrub starts 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[101432]: from='client.55687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:35:01.183 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:00 smithi119 ceph-mon[101432]: 47.16 scrub starts 2024-09-21T10:35:01.265 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-mon[89359]: pgmap v2061: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.2 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 74 MiB/s wr, 1.18k op/s; 2298/353610 objects degraded (0.650%); 805/353610 objects misplaced (0.228%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:35:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-mon[89359]: osdmap e1309: 8 total, 8 up, 8 in 2024-09-21T10:35:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/677888247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-mon[89359]: 47.0 deep-scrub starts 2024-09-21T10:35:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-mon[89359]: from='client.55687 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:35:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:00 smithi145 ceph-mon[89359]: 47.16 scrub starts 2024-09-21T10:35:01.301 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:01.403 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:01.441 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:01.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:01.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.581 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:01.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:01.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:01.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:01.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:01.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:01.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:01.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:01.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 86 16 122508 122492 89.0063 76.25 0.0119545 0.0112285 2024-09-21T10:35:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:02.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:02.132 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:02.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:02.234 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:02.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:02.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:01 smithi145 ceph-mon[89359]: 47.16 scrub ok 2024-09-21T10:35:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:01 smithi145 ceph-mon[89359]: osdmap e1310: 8 total, 8 up, 8 in 2024-09-21T10:35:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:01 smithi145 ceph-mon[89359]: 47.1 scrub starts 2024-09-21T10:35:02.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:01 smithi119 ceph-mon[101432]: 47.16 scrub ok 2024-09-21T10:35:02.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:01 smithi119 ceph-mon[101432]: osdmap e1310: 8 total, 8 up, 8 in 2024-09-21T10:35:02.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:01 smithi119 ceph-mon[101432]: 47.1 scrub starts 2024-09-21T10:35:02.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:01 smithi119 ceph-mon[105288]: 47.16 scrub ok 2024-09-21T10:35:02.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:01 smithi119 ceph-mon[105288]: osdmap e1310: 8 total, 8 up, 8 in 2024-09-21T10:35:02.354 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:01 smithi119 ceph-mon[105288]: 47.1 scrub starts 2024-09-21T10:35:02.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:02.390 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:02.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:02.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:02.617 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:02.640 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:02.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:02.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:02.804 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:35:02.804 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:35:02.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:02.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:02.965 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 87 16 122958 122942 88.3065 28.125 0.0164315 0.0113197 2024-09-21T10:35:02.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:03.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:03.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:03.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[101432]: pgmap v2064: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 2298/360000 objects degraded (0.638%); 269/360000 objects misplaced (0.075%); 14 MiB/s, 216 objects/s recovering 2024-09-21T10:35:03.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[101432]: osdmap e1311: 8 total, 8 up, 8 in 2024-09-21T10:35:03.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:03.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:03.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[105288]: pgmap v2064: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 2298/360000 objects degraded (0.638%); 269/360000 objects misplaced (0.075%); 14 MiB/s, 216 objects/s recovering 2024-09-21T10:35:03.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[105288]: osdmap e1311: 8 total, 8 up, 8 in 2024-09-21T10:35:03.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:03.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:03.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:03.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:03.287 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:03.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:02 smithi145 ceph-mon[89359]: pgmap v2064: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 116 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 2298/360000 objects degraded (0.638%); 269/360000 objects misplaced (0.075%); 14 MiB/s, 216 objects/s recovering 2024-09-21T10:35:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:02 smithi145 ceph-mon[89359]: osdmap e1311: 8 total, 8 up, 8 in 2024-09-21T10:35:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:03.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:03.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:03.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:03.559 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:03.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:03.661 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:03.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:03.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:03.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:03.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:03.948 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2062,"stamp":"2024-09-21T10:34:59.868143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7723647136,"num_objects":117870,"num_object_clones":0,"num_object_copies":353610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":117870,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":112260,"num_write_kb":7180265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6021,"num_bytes_recovered":393310290,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145585,"ondisk_log_size":145585,"up":357,"acting":354,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":363,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26886872,"kb_used_data":23360792,"kb_used_omap":731,"kb_used_meta":3525220,"kb_avail":723074344,"statfs":{"total":767960285184,"available":740428128256,"internally_reserved":0,"allocated":23921451008,"data_stored":23877861663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":749144,"internal_metadata":3609825704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,5,8,7,4,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":611909632,"num_objects":9337,"num_object_clones":0,"num_object_copies":28011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2034,"num_objects_unfound":0,"num_objects_dirty":9337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9337,"num_write_kb":597568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1194,"num_bytes_recovered":78249984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.192494"},"pg_stats":[{"pgid":"47.19","version":"1308'5369","reported_seq":5410,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:57.330935+0000","last_change":"2024-09-21T10:34:03.481259+0000","last_active":"2024-09-21T10:34:57.330935+0000","last_peered":"2024-09-21T10:34:57.330935+0000","last_clean":"2024-09-21T10:34:57.330935+0000","last_became_active":"2024-09-21T10:34:03.479509+0000","last_became_peered":"2024-09-21T10:34:03.479509+0000","last_unstale":"2024-09-21T10:34:57.330935+0000","last_undegraded":"2024-09-21T10:34:57.330935+0000","last_fullsized":"2024-09-21T10:34:57.330935+0000","mapping_epoch":1293,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5369,"log_dups_size":0,"ondisk_log_size":5369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238878720,"num_objects":3645,"num_object_clones":0,"num_object_copies":10935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3645,"num_write_kb":233280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.18","version":"1308'5541","reported_seq":5580,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:57.974237+0000","last_change":"2024-09-21T10:34:03.464722+0000","last_active":"2024-09-21T10:34:57.974237+0000","last_peered":"2024-09-21T10:34:57.974237+0000","last_clean":"2024-09-21T10:34:57.974237+0000","last_became_active":"2024-09-21T10:34:03.463031+0000","last_became_peered":"2024-09-21T10:34:03.463031+0000","last_unstale":"2024-09-21T10:34:57.974237+0000","last_undegraded":"2024-09-21T10:34:57.974237+0000","last_fullsized":"2024-09-21T10:34:57.974237+0000","mapping_epoch":1293,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247857152,"num_objects":3782,"num_object_clones":0,"num_object_copies":11346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3782,"num_write_kb":242048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"47.17","version":"1308'5411","reported_seq":5484,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:56.197498+0000","last_change":"2024-09-21T10:34:03.488597+0000","last_active":"2024-09-21T10:34:56.197498+0000","last_peered":"2024-09-21T10:34:56.197498+0000","last_clean":"2024-09-21T10:34:56.197498+0000","last_became_active":"2024-09-21T10:34:03.474882+0000","last_became_peered":"2024-09-21T10:34:03.474882+0000","last_unstale":"2024-09-21T10:34:56.197498+0000","last_undegraded":"2024-09-21T10:34:56.197498+0000","last_fullsized":"2024-09-21T10:34:56.197498+0000","mapping_epoch":1293,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237961216,"num_objects":3631,"num_object_clones":0,"num_object_copies":10893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3631,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":232384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.16","version":"1308'5470","reported_seq":5540,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.689122+0000","last_change":"2024-09-21T10:34:03.481058+0000","last_active":"2024-09-21T10:34:58.689122+0000","last_peered":"2024-09-21T10:34:58.689122+0000","last_clean":"2024-09-21T10:34:58.689122+0000","last_became_active":"2024-09-21T10:34:03.477622+0000","last_became_peered":"2024-09-21T10:34:03.477622+0000","last_unstale":"2024-09-21T10:34:58.689122+0000","last_undegraded":"2024-09-21T10:34:58.689122+0000","last_fullsized":"2024-09-21T10:34:58.689122+0000","mapping_epoch":1293,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5470,"log_dups_size":0,"ondisk_log_size":5470,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243302400,"num_objects":3713,"num_object_clones":0,"num_object_copies":11139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3713,"num_write_kb":237600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.15","version":"1308'5397","reported_seq":5502,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:55.128665+0000","last_change":"2024-09-21T10:34:31.058853+0000","last_active":"2024-09-21T10:34:55.128665+0000","last_peered":"2024-09-21T10:34:55.128665+0000","last_clean":"2024-09-21T10:34:55.128665+0000","last_became_active":"2024-09-21T10:34:03.479448+0000","last_became_peered":"2024-09-21T10:34:03.479448+0000","last_unstale":"2024-09-21T10:34:55.128665+0000","last_undegraded":"2024-09-21T10:34:55.128665+0000","last_fullsized":"2024-09-21T10:34:55.128665+0000","mapping_epoch":1293,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"1307'4541","last_scrub_stamp":"2024-09-21T10:34:31.058760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:34:31.058760+0000","objects_scrubbed":2710,"log_size":5397,"log_dups_size":0,"ondisk_log_size":5397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:18:59.183676+0000","scrub_duration":881,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":234618880,"num_objects":3580,"num_object_clones":0,"num_object_copies":10740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":870,"num_write_kb":55680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.14","version":"1308'5396","reported_seq":5469,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:55.157560+0000","last_change":"2024-09-21T10:34:32.183111+0000","last_active":"2024-09-21T10:34:55.157560+0000","last_peered":"2024-09-21T10:34:55.157560+0000","last_clean":"2024-09-21T10:34:55.157560+0000","last_became_active":"2024-09-21T10:34:03.475925+0000","last_became_peered":"2024-09-21T10:34:03.475925+0000","last_unstale":"2024-09-21T10:34:55.157560+0000","last_undegraded":"2024-09-21T10:34:55.157560+0000","last_fullsized":"2024-09-21T10:34:55.157560+0000","mapping_epoch":1293,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5396,"log_dups_size":0,"ondisk_log_size":5396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T13:10:51.082930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238977024,"num_objects":3647,"num_object_clones":0,"num_object_copies":10941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3647,"num_write_kb":233376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.13","version":"1308'5453","reported_seq":5521,"reported_epoch":1308,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:34:58.438082+0000","last_change":"2024-09-21T10:34:20.701370+0000","last_active":"2024-09-21T10:34:58.438082+0000","last_peered":"2024-09-21T10:34:58.438082+0000","last_clean":"2024-09-21T10:34:17.198135+0000","last_became_active":"2024-09-21T10:34:20.685100+0000","last_became_peered":"2024-09-21T10:34:20.685100+0000","last_unstale":"2024-09-21T10:34:58.438082+0000","last_undegraded":"2024-09-21T10:34:20.469516+0000","last_fullsized":"2024-09-21T10:34:20.461133+0000","mapping_epoch":1303,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5453,"log_dups_size":0,"ondisk_log_size":5453,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242057216,"num_objects":3694,"num_object_clones":0,"num_object_copies":11082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3694,"num_write_kb":236384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":2298},{"shards":"2,7","objects":1396}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.12","version":"1308'5331","reported_seq":8731,"reported_epoch":1308,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:34:56.216972+0000","last_change":"2024-09-21T10:34:27.039901+0000","last_active":"2024-09-21T10:34:56.216972+0000","last_peered":"2024-09-21T10:34:56.216972+0000","last_clean":"2024-09-21T10:34:15.500943+0000","last_became_active":"2024-09-21T10:34:18.577403+0000","last_became_peered":"2024-09-21T10:34:18.577403+0000","last_unstale":"2024-09-21T10:34:56.216972+0000","last_undegraded":"2024-09-21T10:34:56.216972+0000","last_fullsized":"2024-09-21T10:34:18.511469+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5331,"log_dups_size":0,"ondisk_log_size":5331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237436928,"num_objects":3623,"num_object_clones":0,"num_object_copies":10869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":599,"num_objects_unfound":0,"num_objects_dirty":3623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3623,"num_write_kb":231872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1665,"num_bytes_recovered":109117440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[1,6],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6","objects":3024},{"shards":"1,6,7","objects":599}],"blocked_by":[],"up_primary":2,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.11","version":"1308'5428","reported_seq":11912,"reported_epoch":1308,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:34:58.783349+0000","last_change":"2024-09-21T10:34:27.044385+0000","last_active":"2024-09-21T10:34:58.783349+0000","last_peered":"2024-09-21T10:34:58.783349+0000","last_clean":"2024-09-21T10:34:14.448652+0000","last_became_active":"2024-09-21T10:34:16.582086+0000","last_became_peered":"2024-09-21T10:34:16.582086+0000","last_unstale":"2024-09-21T10:34:58.783349+0000","last_undegraded":"2024-09-21T10:34:58.783349+0000","last_fullsized":"2024-09-21T10:34:16.510737+0000","mapping_epoch":1299,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5428,"log_dups_size":0,"ondisk_log_size":5428,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241074176,"num_objects":3679,"num_object_clones":0,"num_object_copies":11037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":206,"num_objects_unfound":0,"num_objects_dirty":3679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3679,"num_write_kb":235424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2143,"num_bytes_recovered":140443648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[0,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":103},{"shards":"0,5","objects":3576}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.10","version":"1308'5443","reported_seq":9855,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:56.202392+0000","last_change":"2024-09-21T10:34:28.559775+0000","last_active":"2024-09-21T10:34:56.202392+0000","last_peered":"2024-09-21T10:34:56.202392+0000","last_clean":"2024-09-21T10:34:56.202392+0000","last_became_active":"2024-09-21T10:34:28.554866+0000","last_became_peered":"2024-09-21T10:34:28.554866+0000","last_unstale":"2024-09-21T10:34:56.202392+0000","last_undegraded":"2024-09-21T10:34:56.202392+0000","last_fullsized":"2024-09-21T10:34:56.202392+0000","mapping_epoch":1306,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1307,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241598464,"num_objects":3687,"num_object_clones":0,"num_object_copies":11061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3687,"num_write_kb":235936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2176,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.f","version":"1308'7289","reported_seq":7320,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:57.398916+0000","last_change":"2024-09-21T10:33:28.930280+0000","last_active":"2024-09-21T10:34:57.398916+0000","last_peered":"2024-09-21T10:34:57.398916+0000","last_clean":"2024-09-21T10:34:57.398916+0000","last_became_active":"2024-09-21T10:33:28.928885+0000","last_became_peered":"2024-09-21T10:33:28.928885+0000","last_unstale":"2024-09-21T10:34:57.398916+0000","last_undegraded":"2024-09-21T10:34:57.398916+0000","last_fullsized":"2024-09-21T10:34:57.398916+0000","mapping_epoch":1285,"log_start":"1290'1100","ondisk_log_start":"1290'1100","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":6189,"log_dups_size":0,"ondisk_log_size":6189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:20.937916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477691904,"num_objects":7289,"num_object_clones":0,"num_object_copies":21867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7289,"num_write_kb":466496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.e","version":"1308'7467","reported_seq":7498,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.428253+0000","last_change":"2024-09-21T10:33:28.932881+0000","last_active":"2024-09-21T10:34:58.428253+0000","last_peered":"2024-09-21T10:34:58.428253+0000","last_clean":"2024-09-21T10:34:58.428253+0000","last_became_active":"2024-09-21T10:33:28.932543+0000","last_became_peered":"2024-09-21T10:33:28.932543+0000","last_unstale":"2024-09-21T10:34:58.428253+0000","last_undegraded":"2024-09-21T10:34:58.428253+0000","last_fullsized":"2024-09-21T10:34:58.428253+0000","mapping_epoch":1285,"log_start":"1290'1200","ondisk_log_start":"1290'1200","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":6267,"log_dups_size":0,"ondisk_log_size":6267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:30:10.930926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":489357312,"num_objects":7467,"num_object_clones":0,"num_object_copies":22401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7467,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7467,"num_write_kb":477888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.d","version":"1308'7362","reported_seq":7393,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:57.988289+0000","last_change":"2024-09-21T10:33:29.381012+0000","last_active":"2024-09-21T10:34:57.988289+0000","last_peered":"2024-09-21T10:34:57.988289+0000","last_clean":"2024-09-21T10:34:57.988289+0000","last_became_active":"2024-09-21T10:33:29.380701+0000","last_became_peered":"2024-09-21T10:33:29.380701+0000","last_unstale":"2024-09-21T10:34:57.988289+0000","last_undegraded":"2024-09-21T10:34:57.988289+0000","last_fullsized":"2024-09-21T10:34:57.988289+0000","mapping_epoch":1285,"log_start":"1290'1300","ondisk_log_start":"1290'1300","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":6062,"log_dups_size":0,"ondisk_log_size":6062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:17:26.203462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482476032,"num_objects":7362,"num_object_clones":0,"num_object_copies":22086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":471168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"47.c","version":"1308'7246","reported_seq":7309,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:56.203046+0000","last_change":"2024-09-21T10:33:28.929528+0000","last_active":"2024-09-21T10:34:56.203046+0000","last_peered":"2024-09-21T10:34:56.203046+0000","last_clean":"2024-09-21T10:34:56.203046+0000","last_became_active":"2024-09-21T10:33:28.929202+0000","last_became_peered":"2024-09-21T10:33:28.929202+0000","last_unstale":"2024-09-21T10:34:56.203046+0000","last_undegraded":"2024-09-21T10:34:56.203046+0000","last_fullsized":"2024-09-21T10:34:56.203046+0000","mapping_epoch":1285,"log_start":"1290'2400","ondisk_log_start":"1290'2400","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4846,"log_dups_size":0,"ondisk_log_size":4846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:32:17.174993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474873856,"num_objects":7246,"num_object_clones":0,"num_object_copies":21738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7246,"num_write_kb":463744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.b","version":"1308'7421","reported_seq":7452,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.424087+0000","last_change":"2024-09-21T10:33:28.930894+0000","last_active":"2024-09-21T10:34:58.424087+0000","last_peered":"2024-09-21T10:34:58.424087+0000","last_clean":"2024-09-21T10:34:58.424087+0000","last_became_active":"2024-09-21T10:33:28.930610+0000","last_became_peered":"2024-09-21T10:33:28.930610+0000","last_unstale":"2024-09-21T10:34:58.424087+0000","last_undegraded":"2024-09-21T10:34:58.424087+0000","last_fullsized":"2024-09-21T10:34:58.424087+0000","mapping_epoch":1285,"log_start":"1290'1100","ondisk_log_start":"1290'1100","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":6321,"log_dups_size":0,"ondisk_log_size":6321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:52:14.319122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":486342656,"num_objects":7421,"num_object_clones":0,"num_object_copies":22263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7421,"num_write_kb":474944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.a","version":"1308'7241","reported_seq":7304,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:56.219862+0000","last_change":"2024-09-21T10:33:29.379333+0000","last_active":"2024-09-21T10:34:56.219862+0000","last_peered":"2024-09-21T10:34:56.219862+0000","last_clean":"2024-09-21T10:34:56.219862+0000","last_became_active":"2024-09-21T10:33:29.379043+0000","last_became_peered":"2024-09-21T10:33:29.379043+0000","last_unstale":"2024-09-21T10:34:56.219862+0000","last_undegraded":"2024-09-21T10:34:56.219862+0000","last_fullsized":"2024-09-21T10:34:56.219862+0000","mapping_epoch":1285,"log_start":"1290'2400","ondisk_log_start":"1290'2400","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:28.440544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474546176,"num_objects":7241,"num_object_clones":0,"num_object_copies":21723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7241,"num_write_kb":463424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.9","version":"1308'5475","reported_seq":5517,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:57.358440+0000","last_change":"2024-09-21T10:34:03.466288+0000","last_active":"2024-09-21T10:34:57.358440+0000","last_peered":"2024-09-21T10:34:57.358440+0000","last_clean":"2024-09-21T10:34:57.358440+0000","last_became_active":"2024-09-21T10:34:03.464143+0000","last_became_peered":"2024-09-21T10:34:03.464143+0000","last_unstale":"2024-09-21T10:34:57.358440+0000","last_undegraded":"2024-09-21T10:34:57.358440+0000","last_fullsized":"2024-09-21T10:34:57.358440+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5475,"log_dups_size":0,"ondisk_log_size":5475,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245825536,"num_objects":3751,"num_object_clones":0,"num_object_copies":11253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3751,"num_write_kb":240064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.8","version":"1308'5566","reported_seq":5606,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:57.964451+0000","last_change":"2024-09-21T10:34:03.451987+0000","last_active":"2024-09-21T10:34:57.964451+0000","last_peered":"2024-09-21T10:34:57.964451+0000","last_clean":"2024-09-21T10:34:57.964451+0000","last_became_active":"2024-09-21T10:34:03.450591+0000","last_became_peered":"2024-09-21T10:34:03.450591+0000","last_unstale":"2024-09-21T10:34:57.964451+0000","last_undegraded":"2024-09-21T10:34:57.964451+0000","last_fullsized":"2024-09-21T10:34:57.964451+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5566,"log_dups_size":0,"ondisk_log_size":5566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249495552,"num_objects":3807,"num_object_clones":0,"num_object_copies":11421,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":243648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"47.7","version":"1308'5497","reported_seq":5566,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:56.176504+0000","last_change":"2024-09-21T10:34:03.472512+0000","last_active":"2024-09-21T10:34:56.176504+0000","last_peered":"2024-09-21T10:34:56.176504+0000","last_clean":"2024-09-21T10:34:56.176504+0000","last_became_active":"2024-09-21T10:34:03.464562+0000","last_became_peered":"2024-09-21T10:34:03.464562+0000","last_unstale":"2024-09-21T10:34:56.176504+0000","last_undegraded":"2024-09-21T10:34:56.176504+0000","last_fullsized":"2024-09-21T10:34:56.176504+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243597312,"num_objects":3717,"num_object_clones":0,"num_object_copies":11151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3717,"num_write_kb":237888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.6","version":"1308'5488","reported_seq":5558,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.713267+0000","last_change":"2024-09-21T10:34:03.428791+0000","last_active":"2024-09-21T10:34:58.713267+0000","last_peered":"2024-09-21T10:34:58.713267+0000","last_clean":"2024-09-21T10:34:58.713267+0000","last_became_active":"2024-09-21T10:34:03.425142+0000","last_became_peered":"2024-09-21T10:34:03.425142+0000","last_unstale":"2024-09-21T10:34:58.713267+0000","last_undegraded":"2024-09-21T10:34:58.713267+0000","last_fullsized":"2024-09-21T10:34:58.713267+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5488,"log_dups_size":0,"ondisk_log_size":5488,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244482048,"num_objects":3730,"num_object_clones":0,"num_object_copies":11190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3730,"num_write_kb":238752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.5","version":"1308'5477","reported_seq":5586,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:55.149743+0000","last_change":"2024-09-21T10:34:35.228534+0000","last_active":"2024-09-21T10:34:55.149743+0000","last_peered":"2024-09-21T10:34:55.149743+0000","last_clean":"2024-09-21T10:34:55.149743+0000","last_became_active":"2024-09-21T10:34:03.470400+0000","last_became_peered":"2024-09-21T10:34:03.470400+0000","last_unstale":"2024-09-21T10:34:55.149743+0000","last_undegraded":"2024-09-21T10:34:55.149743+0000","last_fullsized":"2024-09-21T10:34:55.149743+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1307'4740","last_scrub_stamp":"2024-09-21T10:34:35.228281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:34:35.228281+0000","objects_scrubbed":2944,"log_size":5477,"log_dups_size":0,"ondisk_log_size":5477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:21:21.021192+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242352128,"num_objects":3698,"num_object_clones":0,"num_object_copies":11094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":754,"num_write_kb":48256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.4","version":"1308'5351","reported_seq":5423,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:55.130563+0000","last_change":"2024-09-21T10:34:03.482381+0000","last_active":"2024-09-21T10:34:55.130563+0000","last_peered":"2024-09-21T10:34:55.130563+0000","last_clean":"2024-09-21T10:34:55.130563+0000","last_became_active":"2024-09-21T10:34:03.453546+0000","last_became_peered":"2024-09-21T10:34:03.453546+0000","last_unstale":"2024-09-21T10:34:55.130563+0000","last_undegraded":"2024-09-21T10:34:55.130563+0000","last_fullsized":"2024-09-21T10:34:55.130563+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5351,"log_dups_size":0,"ondisk_log_size":5351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:35:20.034883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236027904,"num_objects":3601,"num_object_clones":0,"num_object_copies":10803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3601,"num_write_kb":230496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.3","version":"1308'5483","reported_seq":5552,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.749007+0000","last_change":"2024-09-21T10:34:03.399282+0000","last_active":"2024-09-21T10:34:58.749007+0000","last_peered":"2024-09-21T10:34:58.749007+0000","last_clean":"2024-09-21T10:34:58.749007+0000","last_became_active":"2024-09-21T10:34:03.397807+0000","last_became_peered":"2024-09-21T10:34:03.397807+0000","last_unstale":"2024-09-21T10:34:58.749007+0000","last_undegraded":"2024-09-21T10:34:58.749007+0000","last_fullsized":"2024-09-21T10:34:58.749007+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5483,"log_dups_size":0,"ondisk_log_size":5483,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244023296,"num_objects":3723,"num_object_clones":0,"num_object_copies":11169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":238304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.2","version":"1308'5352","reported_seq":5421,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:56.195116+0000","last_change":"2024-09-21T10:34:03.443755+0000","last_active":"2024-09-21T10:34:56.195116+0000","last_peered":"2024-09-21T10:34:56.195116+0000","last_clean":"2024-09-21T10:34:56.195116+0000","last_became_active":"2024-09-21T10:34:03.441889+0000","last_became_peered":"2024-09-21T10:34:03.441889+0000","last_unstale":"2024-09-21T10:34:56.195116+0000","last_undegraded":"2024-09-21T10:34:56.195116+0000","last_fullsized":"2024-09-21T10:34:56.195116+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5352,"log_dups_size":0,"ondisk_log_size":5352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238813184,"num_objects":3644,"num_object_clones":0,"num_object_copies":10932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3644,"num_write_kb":233216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.1","version":"1308'5493","reported_seq":5535,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.482657+0000","last_change":"2024-09-21T10:34:03.465110+0000","last_active":"2024-09-21T10:34:58.482657+0000","last_peered":"2024-09-21T10:34:58.482657+0000","last_clean":"2024-09-21T10:34:58.482657+0000","last_became_active":"2024-09-21T10:34:03.461587+0000","last_became_peered":"2024-09-21T10:34:03.461587+0000","last_unstale":"2024-09-21T10:34:58.482657+0000","last_undegraded":"2024-09-21T10:34:58.482657+0000","last_fullsized":"2024-09-21T10:34:58.482657+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5493,"log_dups_size":0,"ondisk_log_size":5493,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245334016,"num_objects":3743,"num_object_clones":0,"num_object_copies":11229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3743,"num_write_kb":239584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"47.0","version":"1308'5480","reported_seq":5522,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:58.440234+0000","last_change":"2024-09-21T10:34:03.407618+0000","last_active":"2024-09-21T10:34:58.440234+0000","last_peered":"2024-09-21T10:34:58.440234+0000","last_clean":"2024-09-21T10:34:58.440234+0000","last_became_active":"2024-09-21T10:34:03.405996+0000","last_became_peered":"2024-09-21T10:34:03.405996+0000","last_unstale":"2024-09-21T10:34:58.440234+0000","last_undegraded":"2024-09-21T10:34:58.440234+0000","last_fullsized":"2024-09-21T10:34:58.440234+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244023296,"num_objects":3723,"num_object_clones":0,"num_object_copies":11169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3723,"num_write_kb":238304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":64,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530609+0000","last_change":"2024-09-21T10:30:09.420642+0000","last_active":"2024-09-21T10:34:28.530609+0000","last_peered":"2024-09-21T10:34:28.530609+0000","last_clean":"2024-09-21T10:34:28.530609+0000","last_became_active":"2024-09-21T10:30:09.407144+0000","last_became_peered":"2024-09-21T10:30:09.407144+0000","last_unstale":"2024-09-21T10:34:28.530609+0000","last_undegraded":"2024-09-21T10:34:28.530609+0000","last_fullsized":"2024-09-21T10:34:28.530609+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:49.017939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1285,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556417+0000","last_change":"2024-09-21T10:28:37.587114+0000","last_active":"2024-09-21T10:34:28.556417+0000","last_peered":"2024-09-21T10:34:28.556417+0000","last_clean":"2024-09-21T10:34:28.556417+0000","last_became_active":"2024-09-21T10:28:37.586686+0000","last_became_peered":"2024-09-21T10:28:37.586686+0000","last_unstale":"2024-09-21T10:34:28.556417+0000","last_undegraded":"2024-09-21T10:34:28.556417+0000","last_fullsized":"2024-09-21T10:34:28.556417+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:41:45.424696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1934,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856197+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:34:52.856197+0000","last_peered":"2024-09-21T10:34:52.856197+0000","last_clean":"2024-09-21T10:34:52.856197+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:34:52.856197+0000","last_undegraded":"2024-09-21T10:34:52.856197+0000","last_fullsized":"2024-09-21T10:34:52.856197+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1953,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.853059+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:34:52.853059+0000","last_peered":"2024-09-21T10:34:52.853059+0000","last_clean":"2024-09-21T10:34:52.853059+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:34:52.853059+0000","last_undegraded":"2024-09-21T10:34:52.853059+0000","last_fullsized":"2024-09-21T10:34:52.853059+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":62,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530080+0000","last_change":"2024-09-21T10:30:09.427552+0000","last_active":"2024-09-21T10:34:28.530080+0000","last_peered":"2024-09-21T10:34:28.530080+0000","last_clean":"2024-09-21T10:34:28.530080+0000","last_became_active":"2024-09-21T10:30:09.412897+0000","last_became_peered":"2024-09-21T10:30:09.412897+0000","last_unstale":"2024-09-21T10:34:28.530080+0000","last_undegraded":"2024-09-21T10:34:28.530080+0000","last_fullsized":"2024-09-21T10:34:28.530080+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:49.300103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1271,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556386+0000","last_change":"2024-09-21T10:27:19.333245+0000","last_active":"2024-09-21T10:34:28.556386+0000","last_peered":"2024-09-21T10:34:28.556386+0000","last_clean":"2024-09-21T10:34:28.556386+0000","last_became_active":"2024-09-21T10:27:19.332965+0000","last_became_peered":"2024-09-21T10:27:19.332965+0000","last_unstale":"2024-09-21T10:34:28.556386+0000","last_undegraded":"2024-09-21T10:34:28.556386+0000","last_fullsized":"2024-09-21T10:34:28.556386+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:04.828777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":62,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529738+0000","last_change":"2024-09-21T10:30:09.425518+0000","last_active":"2024-09-21T10:34:28.529738+0000","last_peered":"2024-09-21T10:34:28.529738+0000","last_clean":"2024-09-21T10:34:28.529738+0000","last_became_active":"2024-09-21T10:30:09.412102+0000","last_became_peered":"2024-09-21T10:30:09.412102+0000","last_unstale":"2024-09-21T10:34:28.529738+0000","last_undegraded":"2024-09-21T10:34:28.529738+0000","last_fullsized":"2024-09-21T10:34:28.529738+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:39:44.848838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1281,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556702+0000","last_change":"2024-09-21T10:33:38.024141+0000","last_active":"2024-09-21T10:34:28.556702+0000","last_peered":"2024-09-21T10:34:28.556702+0000","last_clean":"2024-09-21T10:34:28.556702+0000","last_became_active":"2024-09-21T10:33:38.023837+0000","last_became_peered":"2024-09-21T10:33:38.023837+0000","last_unstale":"2024-09-21T10:34:28.556702+0000","last_undegraded":"2024-09-21T10:34:28.556702+0000","last_fullsized":"2024-09-21T10:34:28.556702+0000","mapping_epoch":1289,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1290,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:11.353781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":73,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556525+0000","last_change":"2024-09-21T10:28:56.813482+0000","last_active":"2024-09-21T10:34:28.556525+0000","last_peered":"2024-09-21T10:34:28.556525+0000","last_clean":"2024-09-21T10:34:28.556525+0000","last_became_active":"2024-09-21T10:28:56.813103+0000","last_became_peered":"2024-09-21T10:28:56.813103+0000","last_unstale":"2024-09-21T10:34:28.556525+0000","last_undegraded":"2024-09-21T10:34:28.556525+0000","last_fullsized":"2024-09-21T10:34:28.556525+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:31:20.438161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1878,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859027+0000","last_change":"2024-09-21T10:29:04.650051+0000","last_active":"2024-09-21T10:34:52.859027+0000","last_peered":"2024-09-21T10:34:52.859027+0000","last_clean":"2024-09-21T10:34:52.859027+0000","last_became_active":"2024-09-21T10:29:04.649705+0000","last_became_peered":"2024-09-21T10:29:04.649705+0000","last_unstale":"2024-09-21T10:34:52.859027+0000","last_undegraded":"2024-09-21T10:34:52.859027+0000","last_fullsized":"2024-09-21T10:34:52.859027+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:45.854900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":1246,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556448+0000","last_change":"2024-09-21T10:27:19.325859+0000","last_active":"2024-09-21T10:34:28.556448+0000","last_peered":"2024-09-21T10:34:28.556448+0000","last_clean":"2024-09-21T10:34:28.556448+0000","last_became_active":"2024-09-21T10:27:19.325540+0000","last_became_peered":"2024-09-21T10:27:19.325540+0000","last_unstale":"2024-09-21T10:34:28.556448+0000","last_undegraded":"2024-09-21T10:34:28.556448+0000","last_fullsized":"2024-09-21T10:34:28.556448+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:11.324592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1936,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.853987+0000","last_change":"2024-09-21T10:28:37.333977+0000","last_active":"2024-09-21T10:34:52.853987+0000","last_peered":"2024-09-21T10:34:52.853987+0000","last_clean":"2024-09-21T10:34:52.853987+0000","last_became_active":"2024-09-21T10:28:37.333630+0000","last_became_peered":"2024-09-21T10:28:37.333630+0000","last_unstale":"2024-09-21T10:34:52.853987+0000","last_undegraded":"2024-09-21T10:34:52.853987+0000","last_fullsized":"2024-09-21T10:34:52.853987+0000","mapping_epoch":1241,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:56:07.637543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2034,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859565+0000","last_change":"2024-09-21T10:27:19.331712+0000","last_active":"2024-09-21T10:34:52.859565+0000","last_peered":"2024-09-21T10:34:52.859565+0000","last_clean":"2024-09-21T10:34:52.859565+0000","last_became_active":"2024-09-21T10:27:19.330959+0000","last_became_peered":"2024-09-21T10:27:19.330959+0000","last_unstale":"2024-09-21T10:34:52.859565+0000","last_undegraded":"2024-09-21T10:34:52.859565+0000","last_fullsized":"2024-09-21T10:34:52.859565+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:10:46.389484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2050,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.861671+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:34:52.861671+0000","last_peered":"2024-09-21T10:34:52.861671+0000","last_clean":"2024-09-21T10:34:52.861671+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:34:52.861671+0000","last_undegraded":"2024-09-21T10:34:52.861671+0000","last_fullsized":"2024-09-21T10:34:52.861671+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1961,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852379+0000","last_change":"2024-09-21T10:28:56.819014+0000","last_active":"2024-09-21T10:34:52.852379+0000","last_peered":"2024-09-21T10:34:52.852379+0000","last_clean":"2024-09-21T10:34:52.852379+0000","last_became_active":"2024-09-21T10:28:56.818640+0000","last_became_peered":"2024-09-21T10:28:56.818640+0000","last_unstale":"2024-09-21T10:34:52.852379+0000","last_undegraded":"2024-09-21T10:34:52.852379+0000","last_fullsized":"2024-09-21T10:34:52.852379+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:49.759102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1266,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531252+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:34:28.531252+0000","last_peered":"2024-09-21T10:34:28.531252+0000","last_clean":"2024-09-21T10:34:28.531252+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:34:28.531252+0000","last_undegraded":"2024-09-21T10:34:28.531252+0000","last_fullsized":"2024-09-21T10:34:28.531252+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1948,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852572+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:34:52.852572+0000","last_peered":"2024-09-21T10:34:52.852572+0000","last_clean":"2024-09-21T10:34:52.852572+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:34:52.852572+0000","last_undegraded":"2024-09-21T10:34:52.852572+0000","last_fullsized":"2024-09-21T10:34:52.852572+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2060,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.861327+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:34:52.861327+0000","last_peered":"2024-09-21T10:34:52.861327+0000","last_clean":"2024-09-21T10:34:52.861327+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:34:52.861327+0000","last_undegraded":"2024-09-21T10:34:52.861327+0000","last_fullsized":"2024-09-21T10:34:52.861327+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":62,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.857017+0000","last_change":"2024-09-21T10:33:20.897168+0000","last_active":"2024-09-21T10:34:52.857017+0000","last_peered":"2024-09-21T10:34:52.857017+0000","last_clean":"2024-09-21T10:34:52.857017+0000","last_became_active":"2024-09-21T10:33:20.896714+0000","last_became_peered":"2024-09-21T10:33:20.896714+0000","last_unstale":"2024-09-21T10:34:52.857017+0000","last_undegraded":"2024-09-21T10:34:52.857017+0000","last_fullsized":"2024-09-21T10:34:52.857017+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1931,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.853297+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:34:52.853297+0000","last_peered":"2024-09-21T10:34:52.853297+0000","last_clean":"2024-09-21T10:34:52.853297+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:34:52.853297+0000","last_undegraded":"2024-09-21T10:34:52.853297+0000","last_fullsized":"2024-09-21T10:34:52.853297+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":58,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530230+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:34:28.530230+0000","last_peered":"2024-09-21T10:34:28.530230+0000","last_clean":"2024-09-21T10:34:28.530230+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:34:28.530230+0000","last_undegraded":"2024-09-21T10:34:28.530230+0000","last_fullsized":"2024-09-21T10:34:28.530230+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1578,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856126+0000","last_change":"2024-09-21T10:27:19.327772+0000","last_active":"2024-09-21T10:34:52.856126+0000","last_peered":"2024-09-21T10:34:52.856126+0000","last_clean":"2024-09-21T10:34:52.856126+0000","last_became_active":"2024-09-21T10:27:19.327467+0000","last_became_peered":"2024-09-21T10:27:19.327467+0000","last_unstale":"2024-09-21T10:34:52.856126+0000","last_undegraded":"2024-09-21T10:34:52.856126+0000","last_fullsized":"2024-09-21T10:34:52.856126+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:28:38.639278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2034,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.863745+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:34:52.863745+0000","last_peered":"2024-09-21T10:34:52.863745+0000","last_clean":"2024-09-21T10:34:52.863745+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:34:52.863745+0000","last_undegraded":"2024-09-21T10:34:52.863745+0000","last_fullsized":"2024-09-21T10:34:52.863745+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":1281,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556663+0000","last_change":"2024-09-21T10:27:19.330694+0000","last_active":"2024-09-21T10:34:28.556663+0000","last_peered":"2024-09-21T10:34:28.556663+0000","last_clean":"2024-09-21T10:34:28.556663+0000","last_became_active":"2024-09-21T10:27:19.329455+0000","last_became_peered":"2024-09-21T10:27:19.329455+0000","last_unstale":"2024-09-21T10:34:28.556663+0000","last_undegraded":"2024-09-21T10:34:28.556663+0000","last_fullsized":"2024-09-21T10:34:28.556663+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:11:27.883673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1275,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556833+0000","last_change":"2024-09-21T10:27:19.371314+0000","last_active":"2024-09-21T10:34:28.556833+0000","last_peered":"2024-09-21T10:34:28.556833+0000","last_clean":"2024-09-21T10:34:28.556833+0000","last_became_active":"2024-09-21T10:27:19.370747+0000","last_became_peered":"2024-09-21T10:27:19.370747+0000","last_unstale":"2024-09-21T10:34:28.556833+0000","last_undegraded":"2024-09-21T10:34:28.556833+0000","last_fullsized":"2024-09-21T10:34:28.556833+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:45:00.522037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":190,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852655+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:34:52.852655+0000","last_peered":"2024-09-21T10:34:52.852655+0000","last_clean":"2024-09-21T10:34:52.852655+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:34:52.852655+0000","last_undegraded":"2024-09-21T10:34:52.852655+0000","last_fullsized":"2024-09-21T10:34:52.852655+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1266,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556251+0000","last_change":"2024-09-21T10:30:09.423426+0000","last_active":"2024-09-21T10:34:28.556251+0000","last_peered":"2024-09-21T10:34:28.556251+0000","last_clean":"2024-09-21T10:34:28.556251+0000","last_became_active":"2024-09-21T10:30:09.422262+0000","last_became_peered":"2024-09-21T10:30:09.422262+0000","last_unstale":"2024-09-21T10:34:28.556251+0000","last_undegraded":"2024-09-21T10:34:28.556251+0000","last_fullsized":"2024-09-21T10:34:28.556251+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:35:25.701866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1262,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531103+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:34:28.531103+0000","last_peered":"2024-09-21T10:34:28.531103+0000","last_clean":"2024-09-21T10:34:28.531103+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:34:28.531103+0000","last_undegraded":"2024-09-21T10:34:28.531103+0000","last_fullsized":"2024-09-21T10:34:28.531103+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":169,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856279+0000","last_change":"2024-09-21T10:29:04.342115+0000","last_active":"2024-09-21T10:34:52.856279+0000","last_peered":"2024-09-21T10:34:52.856279+0000","last_clean":"2024-09-21T10:34:52.856279+0000","last_became_active":"2024-09-21T10:29:04.340202+0000","last_became_peered":"2024-09-21T10:29:04.340202+0000","last_unstale":"2024-09-21T10:34:52.856279+0000","last_undegraded":"2024-09-21T10:34:52.856279+0000","last_fullsized":"2024-09-21T10:34:52.856279+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:52.282454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":113,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556889+0000","last_change":"2024-09-21T10:33:25.422314+0000","last_active":"2024-09-21T10:34:28.556889+0000","last_peered":"2024-09-21T10:34:28.556889+0000","last_clean":"2024-09-21T10:34:28.556889+0000","last_became_active":"2024-09-21T10:29:56.991594+0000","last_became_peered":"2024-09-21T10:29:56.991594+0000","last_unstale":"2024-09-21T10:34:28.556889+0000","last_undegraded":"2024-09-21T10:34:28.556889+0000","last_fullsized":"2024-09-21T10:34:28.556889+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.422256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.422256+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:11:08.909618+0000","scrub_duration":0.0019284840000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1945,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852450+0000","last_change":"2024-09-21T10:28:56.817623+0000","last_active":"2024-09-21T10:34:52.852450+0000","last_peered":"2024-09-21T10:34:52.852450+0000","last_clean":"2024-09-21T10:34:52.852450+0000","last_became_active":"2024-09-21T10:28:56.817337+0000","last_became_peered":"2024-09-21T10:28:56.817337+0000","last_unstale":"2024-09-21T10:34:52.852450+0000","last_undegraded":"2024-09-21T10:34:52.852450+0000","last_fullsized":"2024-09-21T10:34:52.852450+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:31.718364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2044,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859868+0000","last_change":"2024-09-21T10:27:19.326897+0000","last_active":"2024-09-21T10:34:52.859868+0000","last_peered":"2024-09-21T10:34:52.859868+0000","last_clean":"2024-09-21T10:34:52.859868+0000","last_became_active":"2024-09-21T10:27:19.324476+0000","last_became_peered":"2024-09-21T10:27:19.324476+0000","last_unstale":"2024-09-21T10:34:52.859868+0000","last_undegraded":"2024-09-21T10:34:52.859868+0000","last_fullsized":"2024-09-21T10:34:52.859868+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:57.091307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1929,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.853789+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:34:52.853789+0000","last_peered":"2024-09-21T10:34:52.853789+0000","last_clean":"2024-09-21T10:34:52.853789+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:34:52.853789+0000","last_undegraded":"2024-09-21T10:34:52.853789+0000","last_fullsized":"2024-09-21T10:34:52.853789+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":433,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531732+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:34:28.531732+0000","last_peered":"2024-09-21T10:34:28.531732+0000","last_clean":"2024-09-21T10:34:28.531732+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:34:28.531732+0000","last_undegraded":"2024-09-21T10:34:28.531732+0000","last_fullsized":"2024-09-21T10:34:28.531732+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2052,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.861532+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:34:52.861532+0000","last_peered":"2024-09-21T10:34:52.861532+0000","last_clean":"2024-09-21T10:34:52.861532+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:34:52.861532+0000","last_undegraded":"2024-09-21T10:34:52.861532+0000","last_fullsized":"2024-09-21T10:34:52.861532+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1305,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529576+0000","last_change":"2024-09-21T10:29:04.338405+0000","last_active":"2024-09-21T10:34:28.529576+0000","last_peered":"2024-09-21T10:34:28.529576+0000","last_clean":"2024-09-21T10:34:28.529576+0000","last_became_active":"2024-09-21T10:29:04.338028+0000","last_became_peered":"2024-09-21T10:29:04.338028+0000","last_unstale":"2024-09-21T10:34:28.529576+0000","last_undegraded":"2024-09-21T10:34:28.529576+0000","last_fullsized":"2024-09-21T10:34:28.529576+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:53:52.709092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2038,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859408+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:34:52.859408+0000","last_peered":"2024-09-21T10:34:52.859408+0000","last_clean":"2024-09-21T10:34:52.859408+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:34:52.859408+0000","last_undegraded":"2024-09-21T10:34:52.859408+0000","last_fullsized":"2024-09-21T10:34:52.859408+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":433,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530293+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:34:28.530293+0000","last_peered":"2024-09-21T10:34:28.530293+0000","last_clean":"2024-09-21T10:34:28.530293+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:34:28.530293+0000","last_undegraded":"2024-09-21T10:34:28.530293+0000","last_fullsized":"2024-09-21T10:34:28.530293+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2036,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.865327+0000","last_change":"2024-09-21T10:30:09.408634+0000","last_active":"2024-09-21T10:34:52.865327+0000","last_peered":"2024-09-21T10:34:52.865327+0000","last_clean":"2024-09-21T10:34:52.865327+0000","last_became_active":"2024-09-21T10:30:09.408316+0000","last_became_peered":"2024-09-21T10:30:09.408316+0000","last_unstale":"2024-09-21T10:34:52.865327+0000","last_undegraded":"2024-09-21T10:34:52.865327+0000","last_fullsized":"2024-09-21T10:34:52.865327+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:10:36.075653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2038,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859577+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:34:52.859577+0000","last_peered":"2024-09-21T10:34:52.859577+0000","last_clean":"2024-09-21T10:34:52.859577+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:34:52.859577+0000","last_undegraded":"2024-09-21T10:34:52.859577+0000","last_fullsized":"2024-09-21T10:34:52.859577+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":58,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530001+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:34:28.530001+0000","last_peered":"2024-09-21T10:34:28.530001+0000","last_clean":"2024-09-21T10:34:28.530001+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:34:28.530001+0000","last_undegraded":"2024-09-21T10:34:28.530001+0000","last_fullsized":"2024-09-21T10:34:28.530001+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":68,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528921+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:34:28.528921+0000","last_peered":"2024-09-21T10:34:28.528921+0000","last_clean":"2024-09-21T10:34:28.528921+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:34:28.528921+0000","last_undegraded":"2024-09-21T10:34:28.528921+0000","last_fullsized":"2024-09-21T10:34:28.528921+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2035,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859496+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:34:52.859496+0000","last_peered":"2024-09-21T10:34:52.859496+0000","last_clean":"2024-09-21T10:34:52.859496+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:34:52.859496+0000","last_undegraded":"2024-09-21T10:34:52.859496+0000","last_fullsized":"2024-09-21T10:34:52.859496+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1276,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556624+0000","last_change":"2024-09-21T10:27:19.333723+0000","last_active":"2024-09-21T10:34:28.556624+0000","last_peered":"2024-09-21T10:34:28.556624+0000","last_clean":"2024-09-21T10:34:28.556624+0000","last_became_active":"2024-09-21T10:27:19.333492+0000","last_became_peered":"2024-09-21T10:27:19.333492+0000","last_unstale":"2024-09-21T10:34:28.556624+0000","last_undegraded":"2024-09-21T10:34:28.556624+0000","last_fullsized":"2024-09-21T10:34:28.556624+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:03.860673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":62,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530536+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:34:28.530536+0000","last_peered":"2024-09-21T10:34:28.530536+0000","last_clean":"2024-09-21T10:34:28.530536+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:34:28.530536+0000","last_undegraded":"2024-09-21T10:34:28.530536+0000","last_fullsized":"2024-09-21T10:34:28.530536+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":342,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.860750+0000","last_change":"2024-09-21T10:34:18.587659+0000","last_active":"2024-09-21T10:34:52.860750+0000","last_peered":"2024-09-21T10:34:52.860750+0000","last_clean":"2024-09-21T10:34:52.860750+0000","last_became_active":"2024-09-21T10:34:18.587278+0000","last_became_peered":"2024-09-21T10:34:18.587278+0000","last_unstale":"2024-09-21T10:34:52.860750+0000","last_undegraded":"2024-09-21T10:34:52.860750+0000","last_fullsized":"2024-09-21T10:34:52.860750+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.562623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.562623+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2047,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.864496+0000","last_change":"2024-09-21T10:30:09.415060+0000","last_active":"2024-09-21T10:34:52.864496+0000","last_peered":"2024-09-21T10:34:52.864496+0000","last_clean":"2024-09-21T10:34:52.864496+0000","last_became_active":"2024-09-21T10:30:09.414412+0000","last_became_peered":"2024-09-21T10:30:09.414412+0000","last_unstale":"2024-09-21T10:34:52.864496+0000","last_undegraded":"2024-09-21T10:34:52.864496+0000","last_fullsized":"2024-09-21T10:34:52.864496+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:25:36.187303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2036,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856178+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:34:52.856178+0000","last_peered":"2024-09-21T10:34:52.856178+0000","last_clean":"2024-09-21T10:34:52.856178+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:34:52.856178+0000","last_undegraded":"2024-09-21T10:34:52.856178+0000","last_fullsized":"2024-09-21T10:34:52.856178+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1966,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.855262+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:34:52.855262+0000","last_peered":"2024-09-21T10:34:52.855262+0000","last_clean":"2024-09-21T10:34:52.855262+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:34:52.855262+0000","last_undegraded":"2024-09-21T10:34:52.855262+0000","last_fullsized":"2024-09-21T10:34:52.855262+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":192,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856965+0000","last_change":"2024-09-21T10:30:09.415904+0000","last_active":"2024-09-21T10:34:52.856965+0000","last_peered":"2024-09-21T10:34:52.856965+0000","last_clean":"2024-09-21T10:34:52.856965+0000","last_became_active":"2024-09-21T10:30:09.415524+0000","last_became_peered":"2024-09-21T10:30:09.415524+0000","last_unstale":"2024-09-21T10:34:52.856965+0000","last_undegraded":"2024-09-21T10:34:52.856965+0000","last_fullsized":"2024-09-21T10:34:52.856965+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":818,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.859961+0000","last_change":"2024-09-21T10:28:56.817541+0000","last_active":"2024-09-21T10:34:52.859961+0000","last_peered":"2024-09-21T10:34:52.859961+0000","last_clean":"2024-09-21T10:34:52.859961+0000","last_became_active":"2024-09-21T10:28:56.817221+0000","last_became_peered":"2024-09-21T10:28:56.817221+0000","last_unstale":"2024-09-21T10:34:52.859961+0000","last_undegraded":"2024-09-21T10:34:52.859961+0000","last_fullsized":"2024-09-21T10:34:52.859961+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:51.568259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2060,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.860741+0000","last_change":"2024-09-21T10:08:55.085920+0000","last_active":"2024-09-21T10:34:52.860741+0000","last_peered":"2024-09-21T10:34:52.860741+0000","last_clean":"2024-09-21T10:34:52.860741+0000","last_became_active":"2024-09-21T10:08:55.085636+0000","last_became_peered":"2024-09-21T10:08:55.085636+0000","last_unstale":"2024-09-21T10:34:52.860741+0000","last_undegraded":"2024-09-21T10:34:52.860741+0000","last_fullsized":"2024-09-21T10:34:52.860741+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:47:01.923309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1295,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529107+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:34:28.529107+0000","last_peered":"2024-09-21T10:34:28.529107+0000","last_clean":"2024-09-21T10:34:28.529107+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:34:28.529107+0000","last_undegraded":"2024-09-21T10:34:28.529107+0000","last_fullsized":"2024-09-21T10:34:28.529107+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1258,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531727+0000","last_change":"2024-09-21T10:27:19.327425+0000","last_active":"2024-09-21T10:34:28.531727+0000","last_peered":"2024-09-21T10:34:28.531727+0000","last_clean":"2024-09-21T10:34:28.531727+0000","last_became_active":"2024-09-21T10:27:19.326641+0000","last_became_peered":"2024-09-21T10:27:19.326641+0000","last_unstale":"2024-09-21T10:34:28.531727+0000","last_undegraded":"2024-09-21T10:34:28.531727+0000","last_fullsized":"2024-09-21T10:34:28.531727+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:31:37.389673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1952,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.853277+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:34:52.853277+0000","last_peered":"2024-09-21T10:34:52.853277+0000","last_clean":"2024-09-21T10:34:52.853277+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:34:52.853277+0000","last_undegraded":"2024-09-21T10:34:52.853277+0000","last_fullsized":"2024-09-21T10:34:52.853277+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":62,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529396+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:34:28.529396+0000","last_peered":"2024-09-21T10:34:28.529396+0000","last_clean":"2024-09-21T10:34:28.529396+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:34:28.529396+0000","last_undegraded":"2024-09-21T10:34:28.529396+0000","last_fullsized":"2024-09-21T10:34:28.529396+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1468,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528937+0000","last_change":"2024-09-21T10:33:25.551397+0000","last_active":"2024-09-21T10:34:28.528937+0000","last_peered":"2024-09-21T10:34:28.528937+0000","last_clean":"2024-09-21T10:34:28.528937+0000","last_became_active":"2024-09-21T10:29:57.005035+0000","last_became_peered":"2024-09-21T10:29:57.005035+0000","last_unstale":"2024-09-21T10:34:28.528937+0000","last_undegraded":"2024-09-21T10:34:28.528937+0000","last_fullsized":"2024-09-21T10:34:28.528937+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:20.412919+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":58,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529164+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:34:28.529164+0000","last_peered":"2024-09-21T10:34:28.529164+0000","last_clean":"2024-09-21T10:34:28.529164+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:34:28.529164+0000","last_undegraded":"2024-09-21T10:34:28.529164+0000","last_fullsized":"2024-09-21T10:34:28.529164+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2034,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.864359+0000","last_change":"2024-09-21T10:27:19.327581+0000","last_active":"2024-09-21T10:34:52.864359+0000","last_peered":"2024-09-21T10:34:52.864359+0000","last_clean":"2024-09-21T10:34:52.864359+0000","last_became_active":"2024-09-21T10:27:19.327252+0000","last_became_peered":"2024-09-21T10:27:19.327252+0000","last_unstale":"2024-09-21T10:34:52.864359+0000","last_undegraded":"2024-09-21T10:34:52.864359+0000","last_fullsized":"2024-09-21T10:34:52.864359+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:11:46.428607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1938,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852480+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:34:52.852480+0000","last_peered":"2024-09-21T10:34:52.852480+0000","last_clean":"2024-09-21T10:34:52.852480+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:34:52.852480+0000","last_undegraded":"2024-09-21T10:34:52.852480+0000","last_fullsized":"2024-09-21T10:34:52.852480+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":205,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856872+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:34:52.856872+0000","last_peered":"2024-09-21T10:34:52.856872+0000","last_clean":"2024-09-21T10:34:52.856872+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:34:52.856872+0000","last_undegraded":"2024-09-21T10:34:52.856872+0000","last_fullsized":"2024-09-21T10:34:52.856872+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1275,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556768+0000","last_change":"2024-09-21T10:27:19.329566+0000","last_active":"2024-09-21T10:34:28.556768+0000","last_peered":"2024-09-21T10:34:28.556768+0000","last_clean":"2024-09-21T10:34:28.556768+0000","last_became_active":"2024-09-21T10:27:19.329307+0000","last_became_peered":"2024-09-21T10:27:19.329307+0000","last_unstale":"2024-09-21T10:34:28.556768+0000","last_undegraded":"2024-09-21T10:34:28.556768+0000","last_fullsized":"2024-09-21T10:34:28.556768+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:35:59.305906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1228,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556363+0000","last_change":"2024-09-21T10:27:19.326055+0000","last_active":"2024-09-21T10:34:28.556363+0000","last_peered":"2024-09-21T10:34:28.556363+0000","last_clean":"2024-09-21T10:34:28.556363+0000","last_became_active":"2024-09-21T10:27:19.324778+0000","last_became_peered":"2024-09-21T10:27:19.324778+0000","last_unstale":"2024-09-21T10:34:28.556363+0000","last_undegraded":"2024-09-21T10:34:28.556363+0000","last_fullsized":"2024-09-21T10:34:28.556363+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:12:12.572967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":62,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528779+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:34:28.528779+0000","last_peered":"2024-09-21T10:34:28.528779+0000","last_clean":"2024-09-21T10:34:28.528779+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:34:28.528779+0000","last_undegraded":"2024-09-21T10:34:28.528779+0000","last_fullsized":"2024-09-21T10:34:28.528779+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1235,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556810+0000","last_change":"2024-09-21T10:27:20.323254+0000","last_active":"2024-09-21T10:34:28.556810+0000","last_peered":"2024-09-21T10:34:28.556810+0000","last_clean":"2024-09-21T10:34:28.556810+0000","last_became_active":"2024-09-21T10:27:20.322751+0000","last_became_peered":"2024-09-21T10:27:20.322751+0000","last_unstale":"2024-09-21T10:34:28.556810+0000","last_undegraded":"2024-09-21T10:34:28.556810+0000","last_fullsized":"2024-09-21T10:34:28.556810+0000","mapping_epoch":1212,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:04:06.909953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1264,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531414+0000","last_change":"2024-09-21T10:27:19.326938+0000","last_active":"2024-09-21T10:34:28.531414+0000","last_peered":"2024-09-21T10:34:28.531414+0000","last_clean":"2024-09-21T10:34:28.531414+0000","last_became_active":"2024-09-21T10:27:19.326640+0000","last_became_peered":"2024-09-21T10:27:19.326640+0000","last_unstale":"2024-09-21T10:34:28.531414+0000","last_undegraded":"2024-09-21T10:34:28.531414+0000","last_fullsized":"2024-09-21T10:34:28.531414+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:24.413432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1429,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528956+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:34:28.528956+0000","last_peered":"2024-09-21T10:34:28.528956+0000","last_clean":"2024-09-21T10:34:28.528956+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:34:28.528956+0000","last_undegraded":"2024-09-21T10:34:28.528956+0000","last_fullsized":"2024-09-21T10:34:28.528956+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1275,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556571+0000","last_change":"2024-09-21T10:27:19.332150+0000","last_active":"2024-09-21T10:34:28.556571+0000","last_peered":"2024-09-21T10:34:28.556571+0000","last_clean":"2024-09-21T10:34:28.556571+0000","last_became_active":"2024-09-21T10:27:19.331901+0000","last_became_peered":"2024-09-21T10:27:19.331901+0000","last_unstale":"2024-09-21T10:34:28.556571+0000","last_undegraded":"2024-09-21T10:34:28.556571+0000","last_fullsized":"2024-09-21T10:34:28.556571+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:20:07.758296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":144,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852350+0000","last_change":"2024-09-21T10:28:56.834029+0000","last_active":"2024-09-21T10:34:52.852350+0000","last_peered":"2024-09-21T10:34:52.852350+0000","last_clean":"2024-09-21T10:34:52.852350+0000","last_became_active":"2024-09-21T10:28:56.817594+0000","last_became_peered":"2024-09-21T10:28:56.817594+0000","last_unstale":"2024-09-21T10:34:52.852350+0000","last_undegraded":"2024-09-21T10:34:52.852350+0000","last_fullsized":"2024-09-21T10:34:52.852350+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:44.035052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":45,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556630+0000","last_change":"2024-09-21T10:33:27.479201+0000","last_active":"2024-09-21T10:34:28.556630+0000","last_peered":"2024-09-21T10:34:28.556630+0000","last_clean":"2024-09-21T10:34:28.556630+0000","last_became_active":"2024-09-21T10:33:20.899955+0000","last_became_peered":"2024-09-21T10:33:20.899955+0000","last_unstale":"2024-09-21T10:34:28.556630+0000","last_undegraded":"2024-09-21T10:34:28.556630+0000","last_fullsized":"2024-09-21T10:34:28.556630+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.479064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.479064+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:38:44.683933+0000","scrub_duration":0.0013296740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2059,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.860721+0000","last_change":"2024-09-21T10:27:19.370824+0000","last_active":"2024-09-21T10:34:52.860721+0000","last_peered":"2024-09-21T10:34:52.860721+0000","last_clean":"2024-09-21T10:34:52.860721+0000","last_became_active":"2024-09-21T10:27:19.370477+0000","last_became_peered":"2024-09-21T10:27:19.370477+0000","last_unstale":"2024-09-21T10:34:52.860721+0000","last_undegraded":"2024-09-21T10:34:52.860721+0000","last_fullsized":"2024-09-21T10:34:52.860721+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:42:17.020888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":58,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529709+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:34:28.529709+0000","last_peered":"2024-09-21T10:34:28.529709+0000","last_clean":"2024-09-21T10:34:28.529709+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:34:28.529709+0000","last_undegraded":"2024-09-21T10:34:28.529709+0000","last_fullsized":"2024-09-21T10:34:28.529709+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":442,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531179+0000","last_change":"2024-09-21T10:33:39.741224+0000","last_active":"2024-09-21T10:34:28.531179+0000","last_peered":"2024-09-21T10:34:28.531179+0000","last_clean":"2024-09-21T10:34:28.531179+0000","last_became_active":"2024-09-21T10:33:20.898899+0000","last_became_peered":"2024-09-21T10:33:20.898899+0000","last_unstale":"2024-09-21T10:34:28.531179+0000","last_undegraded":"2024-09-21T10:34:28.531179+0000","last_fullsized":"2024-09-21T10:34:28.531179+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:14.593280+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1246,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.531184+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:34:28.531184+0000","last_peered":"2024-09-21T10:34:28.531184+0000","last_clean":"2024-09-21T10:34:28.531184+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:34:28.531184+0000","last_undegraded":"2024-09-21T10:34:28.531184+0000","last_fullsized":"2024-09-21T10:34:28.531184+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1253,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529908+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:34:28.529908+0000","last_peered":"2024-09-21T10:34:28.529908+0000","last_clean":"2024-09-21T10:34:28.529908+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:34:28.529908+0000","last_undegraded":"2024-09-21T10:34:28.529908+0000","last_fullsized":"2024-09-21T10:34:28.529908+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":205,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852559+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:34:52.852559+0000","last_peered":"2024-09-21T10:34:52.852559+0000","last_clean":"2024-09-21T10:34:52.852559+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:34:52.852559+0000","last_undegraded":"2024-09-21T10:34:52.852559+0000","last_fullsized":"2024-09-21T10:34:52.852559+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1271,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.556355+0000","last_change":"2024-09-21T10:27:19.327246+0000","last_active":"2024-09-21T10:34:28.556355+0000","last_peered":"2024-09-21T10:34:28.556355+0000","last_clean":"2024-09-21T10:34:28.556355+0000","last_became_active":"2024-09-21T10:27:19.326343+0000","last_became_peered":"2024-09-21T10:27:19.326343+0000","last_unstale":"2024-09-21T10:34:28.556355+0000","last_undegraded":"2024-09-21T10:34:28.556355+0000","last_fullsized":"2024-09-21T10:34:28.556355+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:37:37.654823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1285,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529935+0000","last_change":"2024-09-21T10:27:19.324762+0000","last_active":"2024-09-21T10:34:28.529935+0000","last_peered":"2024-09-21T10:34:28.529935+0000","last_clean":"2024-09-21T10:34:28.529935+0000","last_became_active":"2024-09-21T10:27:19.324341+0000","last_became_peered":"2024-09-21T10:27:19.324341+0000","last_unstale":"2024-09-21T10:34:28.529935+0000","last_undegraded":"2024-09-21T10:34:28.529935+0000","last_fullsized":"2024-09-21T10:34:28.529935+0000","mapping_epoch":1211,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1212,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:08:42.722645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":45,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529432+0000","last_change":"2024-09-21T10:33:27.552769+0000","last_active":"2024-09-21T10:34:28.529432+0000","last_peered":"2024-09-21T10:34:28.529432+0000","last_clean":"2024-09-21T10:34:28.529432+0000","last_became_active":"2024-09-21T10:33:20.902059+0000","last_became_peered":"2024-09-21T10:33:20.902059+0000","last_unstale":"2024-09-21T10:34:28.529432+0000","last_undegraded":"2024-09-21T10:34:28.529432+0000","last_fullsized":"2024-09-21T10:34:28.529432+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:53:41.326748+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":58,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528873+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:34:28.528873+0000","last_peered":"2024-09-21T10:34:28.528873+0000","last_clean":"2024-09-21T10:34:28.528873+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:34:28.528873+0000","last_undegraded":"2024-09-21T10:34:28.528873+0000","last_fullsized":"2024-09-21T10:34:28.528873+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1298,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530804+0000","last_change":"2024-09-21T10:28:56.826222+0000","last_active":"2024-09-21T10:34:28.530804+0000","last_peered":"2024-09-21T10:34:28.530804+0000","last_clean":"2024-09-21T10:34:28.530804+0000","last_became_active":"2024-09-21T10:28:56.813187+0000","last_became_peered":"2024-09-21T10:28:56.813187+0000","last_unstale":"2024-09-21T10:34:28.530804+0000","last_undegraded":"2024-09-21T10:34:28.530804+0000","last_fullsized":"2024-09-21T10:34:28.530804+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:59:28.841806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":45,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528803+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:34:28.528803+0000","last_peered":"2024-09-21T10:34:28.528803+0000","last_clean":"2024-09-21T10:34:28.528803+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:34:28.528803+0000","last_undegraded":"2024-09-21T10:34:28.528803+0000","last_fullsized":"2024-09-21T10:34:28.528803+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1323,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530925+0000","last_change":"2024-09-21T10:30:09.424136+0000","last_active":"2024-09-21T10:34:28.530925+0000","last_peered":"2024-09-21T10:34:28.530925+0000","last_clean":"2024-09-21T10:34:28.530925+0000","last_became_active":"2024-09-21T10:30:09.414027+0000","last_became_peered":"2024-09-21T10:30:09.414027+0000","last_unstale":"2024-09-21T10:34:28.530925+0000","last_undegraded":"2024-09-21T10:34:28.530925+0000","last_fullsized":"2024-09-21T10:34:28.530925+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:28.506487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1928,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.853591+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:34:52.853591+0000","last_peered":"2024-09-21T10:34:52.853591+0000","last_clean":"2024-09-21T10:34:52.853591+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:34:52.853591+0000","last_undegraded":"2024-09-21T10:34:52.853591+0000","last_fullsized":"2024-09-21T10:34:52.853591+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":197,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.856773+0000","last_change":"2024-09-21T10:33:20.897520+0000","last_active":"2024-09-21T10:34:52.856773+0000","last_peered":"2024-09-21T10:34:52.856773+0000","last_clean":"2024-09-21T10:34:52.856773+0000","last_became_active":"2024-09-21T10:33:20.897071+0000","last_became_peered":"2024-09-21T10:33:20.897071+0000","last_unstale":"2024-09-21T10:34:52.856773+0000","last_undegraded":"2024-09-21T10:34:52.856773+0000","last_fullsized":"2024-09-21T10:34:52.856773+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1193,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.855422+0000","last_change":"2024-09-21T10:34:18.518899+0000","last_active":"2024-09-21T10:34:52.855422+0000","last_peered":"2024-09-21T10:34:52.855422+0000","last_clean":"2024-09-21T10:34:52.855422+0000","last_became_active":"2024-09-21T10:34:18.518573+0000","last_became_peered":"2024-09-21T10:34:18.518573+0000","last_unstale":"2024-09-21T10:34:52.855422+0000","last_undegraded":"2024-09-21T10:34:52.855422+0000","last_fullsized":"2024-09-21T10:34:52.855422+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:10.155306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2047,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.861598+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:34:52.861598+0000","last_peered":"2024-09-21T10:34:52.861598+0000","last_clean":"2024-09-21T10:34:52.861598+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:34:52.861598+0000","last_undegraded":"2024-09-21T10:34:52.861598+0000","last_fullsized":"2024-09-21T10:34:52.861598+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":45,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.528694+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:34:28.528694+0000","last_peered":"2024-09-21T10:34:28.528694+0000","last_clean":"2024-09-21T10:34:28.528694+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:34:28.528694+0000","last_undegraded":"2024-09-21T10:34:28.528694+0000","last_fullsized":"2024-09-21T10:34:28.528694+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2047,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.860620+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:34:52.860620+0000","last_peered":"2024-09-21T10:34:52.860620+0000","last_clean":"2024-09-21T10:34:52.860620+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:34:52.860620+0000","last_undegraded":"2024-09-21T10:34:52.860620+0000","last_fullsized":"2024-09-21T10:34:52.860620+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":58,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.530000+0000","last_change":"2024-09-21T10:30:09.405230+0000","last_active":"2024-09-21T10:34:28.530000+0000","last_peered":"2024-09-21T10:34:28.530000+0000","last_clean":"2024-09-21T10:34:28.530000+0000","last_became_active":"2024-09-21T10:30:09.404929+0000","last_became_peered":"2024-09-21T10:30:09.404929+0000","last_unstale":"2024-09-21T10:34:28.530000+0000","last_undegraded":"2024-09-21T10:34:28.530000+0000","last_fullsized":"2024-09-21T10:34:28.530000+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:01.454535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":78,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529003+0000","last_change":"2024-09-21T10:33:30.245355+0000","last_active":"2024-09-21T10:34:28.529003+0000","last_peered":"2024-09-21T10:34:28.529003+0000","last_clean":"2024-09-21T10:34:28.529003+0000","last_became_active":"2024-09-21T10:33:20.902088+0000","last_became_peered":"2024-09-21T10:33:20.902088+0000","last_unstale":"2024-09-21T10:34:28.529003+0000","last_undegraded":"2024-09-21T10:34:28.529003+0000","last_fullsized":"2024-09-21T10:34:28.529003+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:30.245301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:30.245301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:43.434664+0000","scrub_duration":0.0018729759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1298,"reported_epoch":1307,"state":"active+clean","last_fresh":"2024-09-21T10:34:28.529485+0000","last_change":"2024-09-21T10:08:06.541759+0000","last_active":"2024-09-21T10:34:28.529485+0000","last_peered":"2024-09-21T10:34:28.529485+0000","last_clean":"2024-09-21T10:34:28.529485+0000","last_became_active":"2024-09-21T10:08:06.541245+0000","last_became_peered":"2024-09-21T10:08:06.541245+0000","last_unstale":"2024-09-21T10:34:28.529485+0000","last_undegraded":"2024-09-21T10:34:28.529485+0000","last_fullsized":"2024-09-21T10:34:28.529485+0000","mapping_epoch":626,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":627,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:45:34.949988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1937,"reported_epoch":1308,"state":"active+clean","last_fresh":"2024-09-21T10:34:52.852744+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:34:52.852744+0000","last_peered":"2024-09-21T10:34:52.852744+0000","last_clean":"2024-09-21T10:34:52.852744+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:34:52.852744+0000","last_undegraded":"2024-09-21T10:34:52.852744+0000","last_fullsized":"2024-09-21T10:34:52.852744+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":47,"num_pg":26,"stat_sum":{"num_bytes":7723024384,"num_objects":117844,"num_object_clones":0,"num_object_copies":353532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2298,"num_objects_misplaced":805,"num_objects_unfound":0,"num_objects_dirty":117844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":112190,"num_write_kb":7180160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5987,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23700439040,"data_stored":23700439040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143427,"ondisk_log_size":143427,"up":78,"acting":75,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":617203,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":944883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1356,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453397985,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3099948,"kb_used_data":2505892,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":90645204,"statfs":{"total":95995035648,"available":92820688896,"internally_reserved":0,"allocated":2566033408,"data_stored":2560578092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049644,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4578348,"kb_used_data":4238180,"kb_used_omap":242,"kb_used_meta":339917,"kb_avail":89166804,"statfs":{"total":95995035648,"available":91306807296,"internally_reserved":0,"allocated":4339896320,"data_stored":4334441545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":348075231},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790443,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1745636,"kb_used_data":1435804,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91999516,"statfs":{"total":95995035648,"available":94207504384,"internally_reserved":0,"allocated":1470263296,"data_stored":1464812505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395549,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3448936,"kb_used_data":3129568,"kb_used_omap":224,"kb_used_meta":319135,"kb_avail":90296216,"statfs":{"total":95995035648,"available":92463325184,"internally_reserved":0,"allocated":3204677632,"data_stored":3199226398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":326795145},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644602,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3664540,"kb_used_data":3177556,"kb_used_omap":67,"kb_used_meta":486908,"kb_avail":90080612,"statfs":{"total":95995035648,"available":92242546688,"internally_reserved":0,"allocated":3253817344,"data_stored":3248374196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":498594653},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527620,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3050400,"kb_used_data":2507864,"kb_used_omap":75,"kb_used_meta":542452,"kb_avail":90694752,"statfs":{"total":95995035648,"available":92871426048,"internally_reserved":0,"allocated":2568052736,"data_stored":2562606030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":555471002},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274773,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5086692,"kb_used_data":4539612,"kb_used_omap":51,"kb_used_meta":546956,"kb_avail":88658460,"statfs":{"total":95995035648,"available":90786263040,"internally_reserved":0,"allocated":4648562688,"data_stored":4643110887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":560083285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,0,4,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354012,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2212372,"kb_used_data":1826316,"kb_used_omap":66,"kb_used_meta":385981,"kb_avail":91532780,"statfs":{"total":95995035648,"available":93729566720,"internally_reserved":0,"allocated":1870147584,"data_stored":1864712010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":395244868},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1865220096,"data_stored":1865220096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4634771456,"data_stored":4634771456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2564947968,"data_stored":2564947968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3240230912,"data_stored":3240230912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3026780160,"data_stored":3026780160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1463025664,"data_stored":1463025664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4339597312,"data_stored":4339597312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2565865472,"data_stored":2565865472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:35:03.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_2 pgp_num 26 2024-09-21T10:35:03.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 88 16 124248 124232 88.2189 80.625 0.00499268 0.0113304 2024-09-21T10:35:03.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:03.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:04.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:04.110 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:04.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:04.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:03 smithi145 ceph-mon[89359]: 47.1 scrub ok 2024-09-21T10:35:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:03 smithi145 ceph-mon[89359]: 47.8 scrub starts 2024-09-21T10:35:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:03 smithi145 ceph-mon[89359]: 47.8 scrub ok 2024-09-21T10:35:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:03 smithi145 ceph-mon[89359]: osdmap e1312: 8 total, 8 up, 8 in 2024-09-21T10:35:04.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:03 smithi145 ceph-mon[89359]: 47.18 scrub starts 2024-09-21T10:35:04.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:35:04.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[101432]: 47.1 scrub ok 2024-09-21T10:35:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[101432]: 47.8 scrub starts 2024-09-21T10:35:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[101432]: 47.8 scrub ok 2024-09-21T10:35:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[101432]: osdmap e1312: 8 total, 8 up, 8 in 2024-09-21T10:35:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[101432]: 47.18 scrub starts 2024-09-21T10:35:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:35:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[105288]: 47.1 scrub ok 2024-09-21T10:35:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[105288]: 47.8 scrub starts 2024-09-21T10:35:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[105288]: 47.8 scrub ok 2024-09-21T10:35:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[105288]: osdmap e1312: 8 total, 8 up, 8 in 2024-09-21T10:35:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[105288]: 47.18 scrub starts 2024-09-21T10:35:04.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-21T10:35:04.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:04.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:04.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:04.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:35:04.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:04.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:04.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:04.787 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:35:04] "GET /metrics HTTP/1.1" 200 36917 "" "Prometheus/2.51.0" 2024-09-21T10:35:04.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:04.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:04.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:04.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 89 16 124639 124623 87.5023 24.4375 0.004133 0.0113271 2024-09-21T10:35:05.012 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:05.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:05.044 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:05.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:05.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:05.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:05.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:05.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: pgmap v2067: 119 pgs: 2 active+recovering+undersized+remapped, 117 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2214/366819 objects misplaced (0.604%); 12 MiB/s, 196 objects/s recovering 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2298/360000 objects degraded (0.638%), 1 pg degraded) 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: osdmap e1313: 8 total, 8 up, 8 in 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: 47.0 deep-scrub starts 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: 47.4 scrub starts 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: osdmap e1314: 8 total, 8 up, 8 in 2024-09-21T10:35:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: 47.18 scrub ok 2024-09-21T10:35:05.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:04 smithi145 ceph-mon[89359]: 47.4 scrub ok 2024-09-21T10:35:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: pgmap v2067: 119 pgs: 2 active+recovering+undersized+remapped, 117 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2214/366819 objects misplaced (0.604%); 12 MiB/s, 196 objects/s recovering 2024-09-21T10:35:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2298/360000 objects degraded (0.638%), 1 pg degraded) 2024-09-21T10:35:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: osdmap e1313: 8 total, 8 up, 8 in 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: 47.0 deep-scrub starts 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: 47.4 scrub starts 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: osdmap e1314: 8 total, 8 up, 8 in 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: 47.18 scrub ok 2024-09-21T10:35:05.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[101432]: 47.4 scrub ok 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: pgmap v2067: 119 pgs: 2 active+recovering+undersized+remapped, 117 active+clean; 7.5 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2214/366819 objects misplaced (0.604%); 12 MiB/s, 196 objects/s recovering 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2298/360000 objects degraded (0.638%), 1 pg degraded) 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: osdmap e1313: 8 total, 8 up, 8 in 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: 47.0 deep-scrub starts 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: 47.4 scrub starts 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: osdmap e1314: 8 total, 8 up, 8 in 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: 47.18 scrub ok 2024-09-21T10:35:05.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:04 smithi119 ceph-mon[105288]: 47.4 scrub ok 2024-09-21T10:35:05.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:35:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:05.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:05.417 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:05.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:05.519 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:05.547 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:05.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:05.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:05.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:05.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:05.851 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:05.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total time run: 90.0091 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total writes made: 124826 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Write size: 65536 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object size: 65536 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Bandwidth (MB/sec): 86.6759 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Bandwidth: 45.9287 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max bandwidth (MB/sec): 248.5 2024-09-21T10:35:05.966 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average IOPS: 1386 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev IOPS: 734.868 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max IOPS: 3976 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min IOPS: 0 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average Latency(s): 0.011533 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Latency(s): 0.0529009 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max latency(s): 2.40251 2024-09-21T10:35:05.967 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min latency(s): 0.00109908 2024-09-21T10:35:05.972 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:35:05.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:05.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:06.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:06.116 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:06.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:05 smithi119 ceph-mon[105288]: 47.11 deep-scrub starts 2024-09-21T10:35:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:05 smithi119 ceph-mon[105288]: 47.6 scrub starts 2024-09-21T10:35:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:05 smithi119 ceph-mon[105288]: osdmap e1315: 8 total, 8 up, 8 in 2024-09-21T10:35:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:05 smithi119 ceph-mon[101432]: 47.11 deep-scrub starts 2024-09-21T10:35:06.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:05 smithi119 ceph-mon[101432]: 47.6 scrub starts 2024-09-21T10:35:06.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:05 smithi119 ceph-mon[101432]: osdmap e1315: 8 total, 8 up, 8 in 2024-09-21T10:35:06.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:06.249 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:06.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:05 smithi145 ceph-mon[89359]: 47.11 deep-scrub starts 2024-09-21T10:35:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:05 smithi145 ceph-mon[89359]: 47.6 scrub starts 2024-09-21T10:35:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:05 smithi145 ceph-mon[89359]: osdmap e1315: 8 total, 8 up, 8 in 2024-09-21T10:35:06.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:06.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:06.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:06.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:06.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:06.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:06.794 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:06.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:06.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:06.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:06.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:07.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:07.061 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:07.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:07.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:07.242 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:06 smithi119 ceph-mon[101432]: pgmap v2071: 119 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 117 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 2151/369732 objects misplaced (0.582%); 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:35:07.242 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:06 smithi119 ceph-mon[101432]: 47.6 scrub ok 2024-09-21T10:35:07.243 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:06 smithi119 ceph-mon[105288]: pgmap v2071: 119 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 117 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 2151/369732 objects misplaced (0.582%); 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:35:07.243 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:06 smithi119 ceph-mon[105288]: 47.6 scrub ok 2024-09-21T10:35:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:06 smithi145 ceph-mon[89359]: pgmap v2071: 119 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 117 active+clean; 7.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 2151/369732 objects misplaced (0.582%); 5.1 MiB/s, 82 objects/s recovering 2024-09-21T10:35:07.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:06 smithi145 ceph-mon[89359]: 47.6 scrub ok 2024-09-21T10:35:07.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:07.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:07.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:07.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:07.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:07.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:07.622 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:07.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:07.752 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:07.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:07.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:07.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:07.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:08.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:08.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:08.258 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:07 smithi119 ceph-mon[101432]: 47.9 scrub starts 2024-09-21T10:35:08.258 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:07 smithi119 ceph-mon[105288]: 47.9 scrub starts 2024-09-21T10:35:08.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:08.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:07 smithi145 ceph-mon[89359]: 47.9 scrub starts 2024-09-21T10:35:08.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:08.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:08.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:08.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:08.566 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:08.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:08.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:08.701 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:08.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:08.842 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:08.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:08.975 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:08.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:09.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:09.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:08 smithi119 ceph-mon[101432]: pgmap v2072: 119 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+remapped, 116 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2151/371973 objects misplaced (0.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:35:09.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:08 smithi119 ceph-mon[101432]: 47.19 scrub starts 2024-09-21T10:35:09.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:08 smithi119 ceph-mon[101432]: 47.9 scrub ok 2024-09-21T10:35:09.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:08 smithi119 ceph-mon[105288]: pgmap v2072: 119 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+remapped, 116 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2151/371973 objects misplaced (0.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:35:09.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:08 smithi119 ceph-mon[105288]: 47.19 scrub starts 2024-09-21T10:35:09.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:08 smithi119 ceph-mon[105288]: 47.9 scrub ok 2024-09-21T10:35:09.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:09.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:09.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:08 smithi145 ceph-mon[89359]: pgmap v2072: 119 pgs: 1 active+clean+scrubbing, 1 remapped+peering, 1 active+recovering+undersized+remapped, 116 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 2151/371973 objects misplaced (0.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-21T10:35:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:08 smithi145 ceph-mon[89359]: 47.19 scrub starts 2024-09-21T10:35:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:08 smithi145 ceph-mon[89359]: 47.9 scrub ok 2024-09-21T10:35:09.355 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:09.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:09.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:09.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:09.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:09.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:09.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:09.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:09.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:09.938 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:09.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:10.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:10.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:10.211 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:09 smithi145 ceph-mon[89359]: 47.19 scrub ok 2024-09-21T10:35:10.217 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:09 smithi119 ceph-mon[101432]: 47.19 scrub ok 2024-09-21T10:35:10.217 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:09 smithi119 ceph-mon[105288]: 47.19 scrub ok 2024-09-21T10:35:10.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:10.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.350 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:10.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:10.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:10.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:35:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:35:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:35:10.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:10.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:10.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:10.762 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:10.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:10.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:10.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:11.001 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:11.037 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:11.104 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:10 smithi119 ceph-mon[105288]: pgmap v2073: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 remapped+peering, 115 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 1763/370281 objects misplaced (0.476%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:35:11.105 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:10 smithi119 ceph-mon[101432]: pgmap v2073: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 remapped+peering, 115 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 1763/370281 objects misplaced (0.476%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:35:11.139 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:11.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:11.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:10 smithi145 ceph-mon[89359]: pgmap v2073: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 1 remapped+peering, 115 active+clean; 7.5 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 1763/370281 objects misplaced (0.476%); 4.7 MiB/s, 74 objects/s recovering 2024-09-21T10:35:11.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:11.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:11.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:11.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:11.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:11.627 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:11.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:11.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:11.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:11.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:11.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:11.965 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:12.001 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:12.048 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:35:12.048 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:35:12.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:12.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:12.219 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:35:12.219 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:35:12.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:12.273 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:12.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"} : dispatch 2024-09-21T10:35:12.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:12.402 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:12.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:12.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:12.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:12.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:12.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:12.781 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:12.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:12.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:12.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:12.960 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:35:13.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:13.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:13.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:13.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:13.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:13.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:12 smithi145 ceph-mon[89359]: pgmap v2074: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 115 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 131 B/s rd, 0 op/s; 5465/364389 objects misplaced (1.500%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:35:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:35:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:12 smithi145 ceph-mon[89359]: osdmap e1316: 8 total, 8 up, 8 in 2024-09-21T10:35:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4176798544' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:35:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:12 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[101432]: pgmap v2074: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 115 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 131 B/s rd, 0 op/s; 5465/364389 objects misplaced (1.500%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[101432]: osdmap e1316: 8 total, 8 up, 8 in 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4176798544' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[105288]: pgmap v2074: 119 pgs: 2 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 115 active+clean; 7.4 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 131 B/s rd, 0 op/s; 5465/364389 objects misplaced (1.500%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:35:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "22"}]': finished 2024-09-21T10:35:13.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[105288]: osdmap e1316: 8 total, 8 up, 8 in 2024-09-21T10:35:13.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4176798544' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:35:13.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:12 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:35:13.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:13.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:13.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:13.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:13.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:13.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:13.665 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:13.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:13.788 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:13.890 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:13.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:13.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:14.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:14.045 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:14.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:14.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:14.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:13 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:13 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:35:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:13 smithi145 ceph-mon[89359]: osdmap e1317: 8 total, 8 up, 8 in 2024-09-21T10:35:14.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:13 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:35:14.302 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:14.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:35:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[105288]: osdmap e1317: 8 total, 8 up, 8 in 2024-09-21T10:35:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:35:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:35:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[101432]: osdmap e1317: 8 total, 8 up, 8 in 2024-09-21T10:35:14.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:13 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"} : dispatch 2024-09-21T10:35:14.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:14.436 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:14.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:14.568 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:14.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:14.703 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:14.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:14.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:35:14] "GET /metrics HTTP/1.1" 200 36917 "" "Prometheus/2.51.0" 2024-09-21T10:35:14.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:14.958 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:14.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:15.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:15.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:15.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:15.211 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:15.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:14 smithi145 ceph-mon[89359]: pgmap v2077: 119 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 116 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 5066/357111 objects misplaced (1.419%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:35:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:35:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:14 smithi145 ceph-mon[89359]: osdmap e1318: 8 total, 8 up, 8 in 2024-09-21T10:35:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:14 smithi145 ceph-mon[89359]: osdmap e1319: 8 total, 8 up, 8 in 2024-09-21T10:35:15.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:15.334 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:15.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[101432]: pgmap v2077: 119 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 116 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 5066/357111 objects misplaced (1.419%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[101432]: osdmap e1318: 8 total, 8 up, 8 in 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[101432]: osdmap e1319: 8 total, 8 up, 8 in 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[105288]: pgmap v2077: 119 pgs: 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 116 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 5066/357111 objects misplaced (1.419%); 7.6 MiB/s, 121 objects/s recovering 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "23"}]': finished 2024-09-21T10:35:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[105288]: osdmap e1318: 8 total, 8 up, 8 in 2024-09-21T10:35:15.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:14 smithi119 ceph-mon[105288]: osdmap e1319: 8 total, 8 up, 8 in 2024-09-21T10:35:15.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:15.457 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:15.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:15.582 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:15.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:15.709 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:15.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:15.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:15.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:15.988 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:15.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:16.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:16.123 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:16.224 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:35:16.225 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:16.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:16.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:16.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:16.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:16 smithi119 ceph-mon[101432]: osdmap e1320: 8 total, 8 up, 8 in 2024-09-21T10:35:16.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:16 smithi119 ceph-mon[101432]: pgmap v2081: 119 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 115 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 4716/354618 objects misplaced (1.330%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:35:16.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:16.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:16.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:16.679 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:16.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:16 smithi145 ceph-mon[89359]: osdmap e1320: 8 total, 8 up, 8 in 2024-09-21T10:35:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:16 smithi145 ceph-mon[89359]: pgmap v2081: 119 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 115 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 4716/354618 objects misplaced (1.330%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:35:16.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:16 smithi119 ceph-mon[105288]: osdmap e1320: 8 total, 8 up, 8 in 2024-09-21T10:35:16.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:16 smithi119 ceph-mon[105288]: pgmap v2081: 119 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 115 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 4716/354618 objects misplaced (1.330%); 13 MiB/s, 200 objects/s recovering 2024-09-21T10:35:16.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:16.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:16.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:16.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:17.053 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:17.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:17.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:17.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:17.332 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.370 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:17.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:17.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:17.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:17.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:17.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:17.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:17.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:17.856 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:17.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:17.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:17.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:18.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:18.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:18.161 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:18.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:18.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:18.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:18.421 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:18.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:18.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:18.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:18.692 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:18.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:18.832 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:18.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:18.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:18.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:18 smithi145 ceph-mon[89359]: 47.0 deep-scrub ok 2024-09-21T10:35:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:18 smithi145 ceph-mon[89359]: pgmap v2082: 119 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 115 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4716/351951 objects misplaced (1.340%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:35:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:19.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:19.104 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[101432]: 47.0 deep-scrub ok 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[101432]: pgmap v2082: 119 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 115 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4716/351951 objects misplaced (1.340%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[105288]: 47.0 deep-scrub ok 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[105288]: pgmap v2082: 119 pgs: 1 remapped+peering, 1 active+clean+scrubbing+deep, 2 active+recovering+undersized+remapped, 115 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 0 B/s wr, 735 op/s; 4716/351951 objects misplaced (1.340%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:19.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:19.244 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:19.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:19.382 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:19.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:19.506 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:19.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:19.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:19.641 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:19.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:19.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:19.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:19.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:19.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:19.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:19.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:20.035 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:19 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 11381/347430 objects degraded (3.276%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:19 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 11381/347430 objects degraded (3.276%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:20.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:20.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:20.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:19 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 11381/347430 objects degraded (3.276%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:20.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:20.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:20.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:20.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:20.444 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:35:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:35:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:35:20.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:20.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:20.686 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:20.728 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:20.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:20.867 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:20.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:20.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:20.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:21.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:21.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:20 smithi119 ceph-mon[101432]: pgmap v2083: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 11381/347430 objects degraded (3.276%); 8038/347430 objects misplaced (2.314%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:35:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:20 smithi119 ceph-mon[105288]: pgmap v2083: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 11381/347430 objects degraded (3.276%); 8038/347430 objects misplaced (2.314%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:35:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.144 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:21.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:21.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:21.272 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:20 smithi145 ceph-mon[89359]: pgmap v2083: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 7.1 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 11381/347430 objects degraded (3.276%); 8038/347430 objects misplaced (2.314%); 11 MiB/s, 173 objects/s recovering 2024-09-21T10:35:21.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:21.406 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:21.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:21.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:21.649 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:21.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:21.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.826 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:21.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:21.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:21.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:21.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:22.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:22.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:22.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:22.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.237 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:22.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:22.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:22.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:22.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:22.516 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:22.618 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:22.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:22.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:22.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:22.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:22.932 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:22.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:22.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:23.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:23.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.069 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:23.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:22 smithi119 ceph-mon[101432]: pgmap v2084: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 78 op/s; 11381/341769 objects degraded (3.330%); 7773/341769 objects misplaced (2.274%); 11 MiB/s, 169 objects/s recovering 2024-09-21T10:35:23.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:22 smithi119 ceph-mon[105288]: pgmap v2084: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 78 op/s; 11381/341769 objects degraded (3.330%); 7773/341769 objects misplaced (2.274%); 11 MiB/s, 169 objects/s recovering 2024-09-21T10:35:23.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:23.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:22 smithi145 ceph-mon[89359]: pgmap v2084: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 78 op/s; 11381/341769 objects degraded (3.330%); 7773/341769 objects misplaced (2.274%); 11 MiB/s, 169 objects/s recovering 2024-09-21T10:35:23.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:23.358 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:23.460 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:23.493 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:23.595 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:23.626 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:23.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:23.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:23.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:23.852 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 47 pgp_num to 26 2024-09-21T10:35:23.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:23.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:23.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:23.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:23.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:24.041 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:24.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1074194061' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:35:24.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1074194061' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:35:24.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:24.179 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:24.280 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1074194061' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"} : dispatch 2024-09-21T10:35:24.309 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:24.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:24.458 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:24.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:24.599 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:24.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:24.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:24.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:24.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:24.851 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:35:24] "GET /metrics HTTP/1.1" 200 36913 "" "Prometheus/2.51.0" 2024-09-21T10:35:24.852 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-mon[101432]: pgmap v2085: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 11381/333291 objects degraded (3.415%); 7401/333291 objects misplaced (2.221%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:35:24.852 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1074194061' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:35:24.852 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-mon[101432]: osdmap e1321: 8 total, 8 up, 8 in 2024-09-21T10:35:24.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:35:24.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:24.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:24.981 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:25.010 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:25.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-mon[105288]: pgmap v2085: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 11381/333291 objects degraded (3.415%); 7401/333291 objects misplaced (2.221%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:35:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1074194061' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:35:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:24 smithi119 ceph-mon[105288]: osdmap e1321: 8 total, 8 up, 8 in 2024-09-21T10:35:25.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:25.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:25.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:24 smithi145 ceph-mon[89359]: pgmap v2085: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 11381/333291 objects degraded (3.415%); 7401/333291 objects misplaced (2.221%); 11 MiB/s, 182 objects/s recovering 2024-09-21T10:35:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1074194061' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "26"}]': finished 2024-09-21T10:35:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:24 smithi145 ceph-mon[89359]: osdmap e1321: 8 total, 8 up, 8 in 2024-09-21T10:35:25.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:25.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:25.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:25.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:25.611 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:35:25.612 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:35:25.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:25.667 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:35:25.668 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:35:25.670 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:25.703 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:35:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:35:25.736 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:35:25.736 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:35:25.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:25.775 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:35:25.775 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:35:25.805 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:35:25.805 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:35:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:25.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:25.840 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:35:25.840 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:35:25.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:35:25.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:35:25.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:25.945 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:25.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:26.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:26.079 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:26.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:26.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:26.293 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.294 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.294 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2052013321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:26.294 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:35:26.294 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:35:26.294 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2052013321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:26.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:35:26.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:35:26.295 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2052013321' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:26.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:35:26.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:35:26.297 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:26.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:26.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:26.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:26.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:26.532 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:26.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:26.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:26.718 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:26.755 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:26.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:26.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:26.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:26.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:27.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:27.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:27.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:27.142 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:27.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:27.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:27.293 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:27 smithi145 ceph-mon[89359]: pgmap v2087: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 441 op/s; 11039/329220 objects degraded (3.353%); 7181/329220 objects misplaced (2.181%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:35:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:27 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 11039/329220 objects degraded (3.353%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:35:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:27.364 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:27 smithi119 ceph-mon[105288]: pgmap v2087: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 441 op/s; 11039/329220 objects degraded (3.353%); 7181/329220 objects misplaced (2.181%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:35:27.365 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:27 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 11039/329220 objects degraded (3.353%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:35:27.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:27 smithi119 ceph-mon[101432]: pgmap v2087: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 441 op/s; 11039/329220 objects degraded (3.353%); 7181/329220 objects misplaced (2.181%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:35:27.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:27 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 11039/329220 objects degraded (3.353%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:35:27.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:27.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:27.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:27.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:27.673 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.712 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:27.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:27.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.849 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:27.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:27.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:27.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:27.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:27.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:28.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:28.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:28.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:28.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:28.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:28.401 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:28.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:28.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:28.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:28.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:28.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:28.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:28.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:28.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:28.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:28.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:29.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:29.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:35:29.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:29.244 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:29.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:29 smithi119 ceph-mon[105288]: pgmap v2088: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 11039/323466 objects degraded (3.413%); 7181/323466 objects misplaced (2.220%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:35:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:29 smithi119 ceph-mon[101432]: pgmap v2088: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 11039/323466 objects degraded (3.413%); 7181/323466 objects misplaced (2.220%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:35:29.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:29.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:29.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:29 smithi145 ceph-mon[89359]: pgmap v2088: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 544 op/s; 11039/323466 objects degraded (3.413%); 7181/323466 objects misplaced (2.220%); 11 MiB/s, 172 objects/s recovering 2024-09-21T10:35:29.600 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:29.624 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:29.726 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:29.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:29.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:29.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:29.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:29.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:29.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:29.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:30.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:30.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:30.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:30.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:30.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:30 smithi119 ceph-mon[101432]: osdmap e1322: 8 total, 8 up, 8 in 2024-09-21T10:35:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:30 smithi119 ceph-mon[105288]: osdmap e1322: 8 total, 8 up, 8 in 2024-09-21T10:35:30.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:30.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:30.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:30.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:35:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:35:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:35:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:30 smithi145 ceph-mon[89359]: osdmap e1322: 8 total, 8 up, 8 in 2024-09-21T10:35:30.620 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:30.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:30.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:30.783 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:30.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:30.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:30.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:31.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:31.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:31.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.177 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:31.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:31.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:31.313 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:31.341 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:35:31.341 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:35:31.341 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:35:31.341 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:35:31.341 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:35:31.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:35:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:31 smithi119 ceph-mon[105288]: pgmap v2090: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 10643/315645 objects degraded (3.372%); 6778/315645 objects misplaced (2.147%); 11 MiB/s, 170 objects/s recovering 2024-09-21T10:35:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:31 smithi119 ceph-mon[105288]: osdmap e1323: 8 total, 8 up, 8 in 2024-09-21T10:35:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:31 smithi119 ceph-mon[101432]: pgmap v2090: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 10643/315645 objects degraded (3.372%); 6778/315645 objects misplaced (2.147%); 11 MiB/s, 170 objects/s recovering 2024-09-21T10:35:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:31 smithi119 ceph-mon[101432]: osdmap e1323: 8 total, 8 up, 8 in 2024-09-21T10:35:31.416 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:31.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:31.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:31 smithi145 ceph-mon[89359]: pgmap v2090: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.4 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 10643/315645 objects degraded (3.372%); 6778/315645 objects misplaced (2.147%); 11 MiB/s, 170 objects/s recovering 2024-09-21T10:35:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:31 smithi145 ceph-mon[89359]: osdmap e1323: 8 total, 8 up, 8 in 2024-09-21T10:35:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:31.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:31.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:31.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:31.837 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:31.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:31.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:31.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:32.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:32.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:32.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:32.244 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:32.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:32 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10643/310491 objects degraded (3.428%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:35:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:32 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10643/310491 objects degraded (3.428%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:35:32.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:32.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:32.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:32.512 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:32 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10643/310491 objects degraded (3.428%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:35:32.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:32.634 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:32.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:32.773 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:32.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:32.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:32.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:33.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:33.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:33.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:33.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:33.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:33.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:33.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:33.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:33.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:33 smithi119 ceph-mon[101432]: pgmap v2092: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 10643/310491 objects degraded (3.428%); 6499/310491 objects misplaced (2.093%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:35:33.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:33.507 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:33.507 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:33 smithi119 ceph-mon[105288]: pgmap v2092: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 10643/310491 objects degraded (3.428%); 6499/310491 objects misplaced (2.093%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:35:33.507 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:33.507 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:33 smithi145 ceph-mon[89359]: pgmap v2092: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 115 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 881 op/s; 10643/310491 objects degraded (3.428%); 6499/310491 objects misplaced (2.093%); 8.8 MiB/s, 140 objects/s recovering 2024-09-21T10:35:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:35:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:33.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.603 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:33.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:33.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:33.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:33.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:33.877 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:33.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:33.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:34.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:34.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:34.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:34.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:34.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:34.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:34.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.436 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:34.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:34.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.575 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:34.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:34.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:34.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:34.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:34.826 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:35:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:35:34] "GET /metrics HTTP/1.1" 200 36913 "" "Prometheus/2.51.0" 2024-09-21T10:35:34.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:34.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:34.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:34.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:34.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:35.077 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:35.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.142 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:35.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:35.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:35.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:35 smithi119 ceph-mon[105288]: pgmap v2093: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 10643/303777 objects degraded (3.504%); 6481/303777 objects misplaced (2.133%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:35:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:35 smithi119 ceph-mon[101432]: pgmap v2093: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 10643/303777 objects degraded (3.504%); 6481/303777 objects misplaced (2.133%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:35:35.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:35.400 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:35.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:35.535 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:35 smithi145 ceph-mon[89359]: pgmap v2093: 119 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 116 active+clean; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 10643/303777 objects degraded (3.504%); 6481/303777 objects misplaced (2.133%); 7.0 MiB/s, 112 objects/s recovering 2024-09-21T10:35:35.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:35.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:35.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:35.803 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:35.812 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:35:35.812 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:35:35.812 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:35:35.812 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:35:35.812 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:35:35.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:35.934 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:35.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:36.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:36.072 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:36.173 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.214 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:36.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:36.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:36.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:36.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:36.493 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:36.595 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:36.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:36.711 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:36.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:36.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:36.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:36.938 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:36.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:37.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:37.069 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:37.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:37.136 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:35:37.171 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:37.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:37.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:37.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:37.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:37.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:37 smithi119 ceph-mon[101432]: pgmap v2094: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 10359/300093 objects degraded (3.452%); 6229/300093 objects misplaced (2.076%); 10 MiB/s, 161 objects/s recovering 2024-09-21T10:35:37.391 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3260698325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:35:37.392 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:37 smithi119 ceph-mon[105288]: pgmap v2094: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 10359/300093 objects degraded (3.452%); 6229/300093 objects misplaced (2.076%); 10 MiB/s, 161 objects/s recovering 2024-09-21T10:35:37.392 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3260698325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:35:37.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.478 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:37.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:37 smithi145 ceph-mon[89359]: pgmap v2094: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 6.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 0 B/s wr, 973 op/s; 10359/300093 objects degraded (3.452%); 6229/300093 objects misplaced (2.076%); 10 MiB/s, 161 objects/s recovering 2024-09-21T10:35:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3260698325' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:35:37.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:37.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:37.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:37.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:37.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:37.892 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:37.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:37.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:38.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:38.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:35:38.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:38.179 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:38.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:38.315 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:38 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:38.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3260698325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:35:38.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:38 smithi119 ceph-mon[101432]: osdmap e1324: 8 total, 8 up, 8 in 2024-09-21T10:35:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:38 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3260698325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:35:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:38 smithi119 ceph-mon[105288]: osdmap e1324: 8 total, 8 up, 8 in 2024-09-21T10:35:38.329 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:38.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:38.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:38 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3260698325' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:35:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:38 smithi145 ceph-mon[89359]: osdmap e1324: 8 total, 8 up, 8 in 2024-09-21T10:35:38.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:38.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:38.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:38.737 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:38.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:38.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:38.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:38.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:38.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.027 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:39.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:39.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:39.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:39.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:39.304 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:39.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:39.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:39.530 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:39 smithi119 ceph-mon[105288]: pgmap v2096: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 10359/294282 objects degraded (3.520%); 6229/294282 objects misplaced (2.117%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:35:39.530 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:39 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 10359/294282 objects degraded (3.520%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:39.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:39 smithi119 ceph-mon[101432]: pgmap v2096: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 10359/294282 objects degraded (3.520%); 6229/294282 objects misplaced (2.117%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:35:39.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:39 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 10359/294282 objects degraded (3.520%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:39.537 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:39.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:39 smithi145 ceph-mon[89359]: pgmap v2096: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 889 op/s; 10359/294282 objects degraded (3.520%); 6229/294282 objects misplaced (2.117%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:35:39.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:39 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 10359/294282 objects degraded (3.520%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.575 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:39.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:39.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:39.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:39.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:39.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:39.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:39.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:39.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:39.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:40.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:40.114 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:40.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:40.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:40.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:40.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:40.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:40.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:35:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:35:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:35:40.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:40.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:40.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:40.796 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:40.875 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:35:40.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:40.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:41.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:41.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:41.133 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:41 smithi119 ceph-mon[101432]: pgmap v2097: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9995/286263 objects degraded (3.492%); 6229/286263 objects misplaced (2.176%); 5.7 MiB/s, 90 objects/s recovering 2024-09-21T10:35:41.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:41.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:41.279 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:41.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:41.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:41.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:41.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.471 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:41.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:41 smithi145 ceph-mon[89359]: pgmap v2097: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9995/286263 objects degraded (3.492%); 6229/286263 objects misplaced (2.176%); 5.7 MiB/s, 90 objects/s recovering 2024-09-21T10:35:41.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:41.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:41 smithi119 ceph-mon[105288]: pgmap v2097: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9995/286263 objects degraded (3.492%); 6229/286263 objects misplaced (2.176%); 5.7 MiB/s, 90 objects/s recovering 2024-09-21T10:35:41.714 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:41.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:41.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:41.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:41.885 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:41.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:41.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:42.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:42.039 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:35:42.039 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:35:42.039 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:35:42.039 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:35:42.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:35:42.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:35:42.102 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:35:42.102 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:35:42.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:42.132 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:35:42.132 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:35:42.161 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:42.163 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:35:42.163 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:35:42.210 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:35:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:35:42.246 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:35:42.246 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:35:42.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:42.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:42.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:42 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3075140036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:42.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:42 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3075140036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:42.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:42.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:42.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:42 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3075140036' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:42.577 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:42.660 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:35:42.660 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:35:42.661 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:35:42.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:42.712 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:42.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:42.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:42.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:42.990 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:42.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:43.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:43.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:43.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:43.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:43.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:43.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:43 smithi119 ceph-mon[105288]: pgmap v2098: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9995/279849 objects degraded (3.572%); 5985/279849 objects misplaced (2.139%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:35:43.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:43 smithi119 ceph-mon[101432]: pgmap v2098: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9995/279849 objects degraded (3.572%); 5985/279849 objects misplaced (2.139%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:35:43.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:43.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:43 smithi145 ceph-mon[89359]: pgmap v2098: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.02k op/s; 9995/279849 objects degraded (3.572%); 5985/279849 objects misplaced (2.139%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:35:43.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:43.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:43.685 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:43.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:43.823 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:43.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:43.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:43.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:44.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:44.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:44.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:44.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:44.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:44.370 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:44.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:44.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:44.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:44.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9995/271986 objects degraded (3.675%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:44.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9995/271986 objects degraded (3.675%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9995/271986 objects degraded (3.675%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:44.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:44.644 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:44.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:44.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:35:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:35:44] "GET /metrics HTTP/1.1" 200 36913 "" "Prometheus/2.51.0" 2024-09-21T10:35:44.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:44.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:44.905 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:44.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:45.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:45.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:45.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:45.168 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:45.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:45.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:45.328 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:45 smithi119 ceph-mon[105288]: pgmap v2099: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 9995/271986 objects degraded (3.675%); 5985/271986 objects misplaced (2.200%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:35:45.328 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:45 smithi119 ceph-mon[101432]: pgmap v2099: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 9995/271986 objects degraded (3.675%); 5985/271986 objects misplaced (2.200%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:35:45.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:45.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:45 smithi145 ceph-mon[89359]: pgmap v2099: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 9995/271986 objects degraded (3.675%); 5985/271986 objects misplaced (2.200%); 5.1 MiB/s, 81 objects/s recovering 2024-09-21T10:35:45.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:45.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:45.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:45.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:45.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:45.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:35:45.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:45.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:45.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:46.003 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:46.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:46.125 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:46.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:46.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:46.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:46.406 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:46.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:46.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:46.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:46.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:46.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:46.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:46.807 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:46.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:46.943 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:46.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:47.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:47.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:47.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:47.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:47.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:47.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:47.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:47.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:47.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:47.497 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:47 smithi119 ceph-mon[101432]: pgmap v2100: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 9625/267753 objects degraded (3.595%); 5718/267753 objects misplaced (2.136%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T10:35:47.497 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:47 smithi119 ceph-mon[105288]: pgmap v2100: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 9625/267753 objects degraded (3.595%); 5718/267753 objects misplaced (2.136%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T10:35:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:47 smithi145 ceph-mon[89359]: pgmap v2100: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.08k op/s; 9625/267753 objects degraded (3.595%); 5718/267753 objects misplaced (2.136%); 5.5 MiB/s, 87 objects/s recovering 2024-09-21T10:35:47.598 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.621 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:47.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:47.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:47.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:47.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:47.878 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:47.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:47.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:48.018 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:48.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:48.120 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:48.162 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:48.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:48.286 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:48.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.421 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:48.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:48.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:48.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:48.564 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:48.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:35:48.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:48.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:48.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:48.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:48.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:48.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:48.966 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:48.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:49.067 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:49.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:49.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:49.225 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:49.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:49.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:49.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:49.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:49.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:49 smithi145 ceph-mon[89359]: pgmap v2101: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 9625/261774 objects degraded (3.677%); 5718/261774 objects misplaced (2.184%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:35:49.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:49 smithi119 ceph-mon[101432]: pgmap v2101: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 9625/261774 objects degraded (3.677%); 5718/261774 objects misplaced (2.184%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:35:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:49 smithi119 ceph-mon[105288]: pgmap v2101: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.09k op/s; 9625/261774 objects degraded (3.677%); 5718/261774 objects misplaced (2.184%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:35:49.642 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:49.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:49.782 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:49.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:49.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:49.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:50.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.042 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:50.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:50.102 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:35:50.102 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:35:50.103 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T10:35:50.103 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:35:50.103 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:35:50.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T10:35:50.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:50.180 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:50.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:50.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:50.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:50.419 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:50.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:35:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:35:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:35:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:50 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9625/261774 objects degraded (3.677%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:50.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.589 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:50.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:50.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:50 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9625/261774 objects degraded (3.677%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:50 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9625/261774 objects degraded (3.677%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:50.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:50.728 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:50.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:50.866 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:50.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:50.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:51.009 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:51.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:51.147 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:51.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:51.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:51.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:51.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:51.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:51.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:51.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:51.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:51 smithi145 ceph-mon[89359]: pgmap v2102: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 9305/253695 objects degraded (3.668%); 5718/253695 objects misplaced (2.254%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:35:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:51 smithi119 ceph-mon[101432]: pgmap v2102: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 9305/253695 objects degraded (3.668%); 5718/253695 objects misplaced (2.254%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:35:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:51 smithi119 ceph-mon[105288]: pgmap v2102: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.13k op/s; 9305/253695 objects degraded (3.668%); 5718/253695 objects misplaced (2.254%); 5.4 MiB/s, 86 objects/s recovering 2024-09-21T10:35:51.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:51.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:51.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:51.908 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:51.945 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:51.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:52.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:52.088 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:52.183 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:35:52.183 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:35:52.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:52.224 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:52.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:52.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:52.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:52.493 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:52.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:52.632 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:52.734 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:52.775 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:52.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:52.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:52.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:52.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:52.986 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:52.987 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:53.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:53.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:53.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:53.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:53.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.348 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:53.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:53.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:53.478 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:53 smithi145 ceph-mon[89359]: pgmap v2103: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 9305/247947 objects degraded (3.753%); 5418/247947 objects misplaced (2.185%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:35:53.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:53.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:53.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:53 smithi119 ceph-mon[105288]: pgmap v2103: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 9305/247947 objects degraded (3.753%); 5418/247947 objects misplaced (2.185%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:35:53.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:53 smithi119 ceph-mon[101432]: pgmap v2103: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.06k op/s; 9305/247947 objects degraded (3.753%); 5418/247947 objects misplaced (2.185%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:35:53.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:53.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:53.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:53.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:53.885 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:53.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:53.986 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:54.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:54.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:54.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:54.241 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:35:54.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:54.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:35:54.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:54.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:54.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:54.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:54.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3143609453' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:35:54.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3143609453' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:35:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3143609453' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:35:54.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:54.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:54.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:54.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:54.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:54.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:35:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:35:54] "GET /metrics HTTP/1.1" 200 36912 "" "Prometheus/2.51.0" 2024-09-21T10:35:54.870 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:54.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:54.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.019 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:55.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:55.120 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:55.161 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:55.263 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:55.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:55.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:55.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[101432]: pgmap v2104: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 9305/240147 objects degraded (3.875%); 5418/240147 objects misplaced (2.256%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3143609453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[101432]: osdmap e1325: 8 total, 8 up, 8 in 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 9305/240147 objects degraded (3.875%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[105288]: pgmap v2104: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 9305/240147 objects degraded (3.875%); 5418/240147 objects misplaced (2.256%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3143609453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[105288]: osdmap e1325: 8 total, 8 up, 8 in 2024-09-21T10:35:55.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:55 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 9305/240147 objects degraded (3.875%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:55.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:55 smithi145 ceph-mon[89359]: pgmap v2104: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.10k op/s; 9305/240147 objects degraded (3.875%); 5418/240147 objects misplaced (2.256%); 4.7 MiB/s, 75 objects/s recovering 2024-09-21T10:35:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:55 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:35:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3143609453' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:35:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:55 smithi145 ceph-mon[89359]: osdmap e1325: 8 total, 8 up, 8 in 2024-09-21T10:35:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:55 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 9305/240147 objects degraded (3.875%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:35:55.562 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:55.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:35:55.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:35:55.799 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:55.831 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:55.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:55.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:55.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:56.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:56.108 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:56.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:56.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:56.248 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:56.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:56.350 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:56.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:56.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:56.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:56.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:56.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:56.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:56.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:56.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:56.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:56.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:56.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:56.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:35:57.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:57.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:57.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:57.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:57.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:57.279 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:57.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:57.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:57.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:57 smithi119 ceph-mon[105288]: pgmap v2106: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8999/236337 objects degraded (3.808%); 5107/236337 objects misplaced (2.161%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:35:57.358 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:57 smithi119 ceph-mon[101432]: pgmap v2106: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8999/236337 objects degraded (3.808%); 5107/236337 objects misplaced (2.161%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:35:57.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:57.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:57.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:57.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:57.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:57 smithi145 ceph-mon[89359]: pgmap v2106: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8999/236337 objects degraded (3.808%); 5107/236337 objects misplaced (2.161%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:35:57.560 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:57.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:57.662 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:35:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:57.690 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:57.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:35:57.791 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:57.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:57.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:57.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:35:57.966 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:35:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:58.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:35:58.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:58.148 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:58.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:58.283 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:58.385 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:58.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:58.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:58.562 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:58.663 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:58.701 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:58.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:58.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:58.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:58.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:35:58.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:58.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:35:59.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:59.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:59.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:35:59.232 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:35:59.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:59.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:59 smithi119 ceph-mon[105288]: pgmap v2107: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8999/230283 objects degraded (3.908%); 5107/230283 objects misplaced (2.218%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:35:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:35:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3362975046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:59 smithi119 ceph-mon[101432]: pgmap v2107: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8999/230283 objects degraded (3.908%); 5107/230283 objects misplaced (2.218%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:35:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:35:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3362975046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:59.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:35:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.493 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:59.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:35:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:59 smithi145 ceph-mon[89359]: pgmap v2107: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8999/230283 objects degraded (3.908%); 5107/230283 objects misplaced (2.218%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:35:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:35:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3362975046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:35:59.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:59.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:59.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:35:59.753 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:35:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:35:59.855 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:59.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:35:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:35:59.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:35:59.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:35:59.998 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:35:59.999 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:36:00.000 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:36:00.032 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:00.134 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:00.177 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:00.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:00.407 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:00.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:00.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8999/230283 objects degraded (3.908%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:36:00.545 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:36:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:36:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:36:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.575 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8999/230283 objects degraded (3.908%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:36:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8999/230283 objects degraded (3.908%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:36:00.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:00.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:00.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:00.856 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:00.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:00.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:00.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:01.099 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:01.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:01.237 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:01.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:01.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.397 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:01.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:01.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.522 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:01.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:01 smithi145 ceph-mon[89359]: 47.13 scrub starts 2024-09-21T10:36:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:01 smithi145 ceph-mon[89359]: pgmap v2108: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8677/222222 objects degraded (3.905%); 5107/222222 objects misplaced (2.298%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:36:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:01 smithi145 ceph-mon[89359]: 47.13 scrub ok 2024-09-21T10:36:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:01 smithi119 ceph-mon[105288]: 47.13 scrub starts 2024-09-21T10:36:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:01 smithi119 ceph-mon[105288]: pgmap v2108: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8677/222222 objects degraded (3.905%); 5107/222222 objects misplaced (2.298%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:36:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:01 smithi119 ceph-mon[105288]: 47.13 scrub ok 2024-09-21T10:36:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:01 smithi119 ceph-mon[101432]: 47.13 scrub starts 2024-09-21T10:36:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:01 smithi119 ceph-mon[101432]: pgmap v2108: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.05k op/s; 8677/222222 objects degraded (3.905%); 5107/222222 objects misplaced (2.298%); 5.8 MiB/s, 92 objects/s recovering 2024-09-21T10:36:01.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:01 smithi119 ceph-mon[101432]: 47.13 scrub ok 2024-09-21T10:36:01.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:01.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:01.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:01.808 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:01.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:01.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:01.940 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:01.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:01.949 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:36:01.949 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "" 2024-09-21T10:36:01.949 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:36:02.042 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:02.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:02.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:02.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:02.217 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:02.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:02.358 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:02.460 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:02.499 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:02.601 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:02.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:02.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:02.780 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:02.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:02.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:02.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:03.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:03.066 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:03.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:03.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:03.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:03.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:03.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:03 smithi119 ceph-mon[101432]: pgmap v2109: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 8677/216660 objects degraded (4.005%); 4766/216660 objects misplaced (2.200%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:36:03.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:03 smithi119 ceph-mon[101432]: 47.2 scrub starts 2024-09-21T10:36:03.304 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:03.304 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:03 smithi119 ceph-mon[105288]: pgmap v2109: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 8677/216660 objects degraded (4.005%); 4766/216660 objects misplaced (2.200%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:36:03.304 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:03 smithi119 ceph-mon[105288]: 47.2 scrub starts 2024-09-21T10:36:03.304 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:03.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:03.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:03.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.498 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:03.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:03.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:03.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:03.781 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:03 smithi145 ceph-mon[89359]: pgmap v2109: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 8677/216660 objects degraded (4.005%); 4766/216660 objects misplaced (2.200%); 6.0 MiB/s, 96 objects/s recovering 2024-09-21T10:36:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:03 smithi145 ceph-mon[89359]: 47.2 scrub starts 2024-09-21T10:36:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:03.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:03.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:03.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:03.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:04.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:04.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:04.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:04.156 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:04.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:04.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:04.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:04.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:04 smithi119 ceph-mon[105288]: 47.2 scrub ok 2024-09-21T10:36:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:04 smithi119 ceph-mon[101432]: 47.2 scrub ok 2024-09-21T10:36:04.441 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:04.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:04.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:04.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:04.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:04.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:04.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:04.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:04 smithi145 ceph-mon[89359]: 47.2 scrub ok 2024-09-21T10:36:04.849 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:36:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:36:04] "GET /metrics HTTP/1.1" 200 36916 "" "Prometheus/2.51.0" 2024-09-21T10:36:04.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:04.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:04.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:05.026 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:05.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:05.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:36:05.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:05.276 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:05.307 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:05.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:05.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:05.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:05.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:05 smithi119 ceph-mon[101432]: pgmap v2110: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 470 op/s; 8677/209307 objects degraded (4.146%); 4766/209307 objects misplaced (2.277%) 2024-09-21T10:36:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:05 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8677/209307 objects degraded (4.146%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:36:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:05 smithi119 ceph-mon[105288]: pgmap v2110: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 470 op/s; 8677/209307 objects degraded (4.146%); 4766/209307 objects misplaced (2.277%) 2024-09-21T10:36:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:05 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8677/209307 objects degraded (4.146%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:36:05.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:05.722 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:05 smithi145 ceph-mon[89359]: pgmap v2110: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 470 op/s; 8677/209307 objects degraded (4.146%); 4766/209307 objects misplaced (2.277%) 2024-09-21T10:36:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:05 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8677/209307 objects degraded (4.146%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:36:05.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:05.858 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:05.927 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:36:05.927 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:36:05.927 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.927 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:184332 47.14 47:2dbd6d6a:::benchmark_data_smithi119_238957_object59503:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:05.900050+0000", 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012736041, 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012766992, 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 184332 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:36:05.928 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.900050+0000", 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.900050+0000", 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.900051+0000", 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6530000000000001e-06 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.929 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.900056+0000", 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5909999999999998e-06 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.900069+0000", 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2699e-05 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.900077+0000", 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0420000000000003e-06 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.901327+0000", 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012505019999999999 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.901445+0000", 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118304 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.901936+0000", 2024-09-21T10:36:05.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049028799999999997 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.905052+0000", 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003116527 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.905423+0000", 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037039499999999998 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46418.0:184343 47.1 e1325/1293 47:84ba9183:::benchmark_data_smithi119_238957_object59514:head v 1325'7536, mlcod=1325'7534)", 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:05.911284+0000", 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00150175, 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015940939999999999, 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911284+0000", 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:36:05.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911284+0000", 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911285+0000", 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5900000000000005e-07 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911290+0000", 2024-09-21T10:36:05.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.49e-06 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911306+0000", 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5588999999999999e-05 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911314+0000", 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9570000000000002e-06 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911367+0000", 2024-09-21T10:36:05.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2948999999999997e-05 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:05.911386+0000", 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8861000000000001e-05 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 2 2024-09-21T10:36:05.936 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:36:05.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:05.982 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:05.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:05.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:06.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:06.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:06.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:06.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:06.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:06.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:06.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:06.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:06.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:06.770 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:06.803 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:06.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:06.938 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:06.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:07.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:07.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:07.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:07.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:07.320 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:07.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:07.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:07.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:07.599 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:07 smithi119 ceph-mon[105288]: pgmap v2111: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 469 op/s; 8353/205383 objects degraded (4.067%); 4344/205383 objects misplaced (2.115%) 2024-09-21T10:36:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:07 smithi119 ceph-mon[101432]: pgmap v2111: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 469 op/s; 8353/205383 objects degraded (4.067%); 4344/205383 objects misplaced (2.115%) 2024-09-21T10:36:07.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:07.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:07.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:07.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:07 smithi145 ceph-mon[89359]: pgmap v2111: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 469 op/s; 8353/205383 objects degraded (4.067%); 4344/205383 objects misplaced (2.115%) 2024-09-21T10:36:07.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:07.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:07.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:07.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:07.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:08.027 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:08.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:08.153 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:08.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:08.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:08.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:08.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:08.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:08.554 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:08.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:08.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:36:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:08.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:08.794 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:08.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:08.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:08.972 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:08.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:09.073 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:09.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:09.138 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:36:09.138 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:36:09.138 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 6, 7, 5] 2024-09-21T10:36:09.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T10:36:09.141 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:09.199 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:09.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:09.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:09.327 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:09 smithi119 ceph-mon[101432]: pgmap v2112: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 131 op/s; 8165/198012 objects degraded (4.123%); 4344/198012 objects misplaced (2.194%) 2024-09-21T10:36:09.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.371 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:09.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:09.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:09.533 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:09.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:09 smithi119 ceph-mon[105288]: pgmap v2112: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 131 op/s; 8165/198012 objects degraded (4.123%); 4344/198012 objects misplaced (2.194%) 2024-09-21T10:36:09.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:09.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:09.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:09 smithi145 ceph-mon[89359]: pgmap v2112: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 131 op/s; 8165/198012 objects degraded (4.123%); 4344/198012 objects misplaced (2.194%) 2024-09-21T10:36:09.807 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:09.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:09.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:09.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:10.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:10.088 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:10.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:10.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:10.225 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:10.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:10.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:10.347 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:36:10.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:10.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:10.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:36:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:36:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:36:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:10 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 8165/198012 objects degraded (4.123%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3330085719' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:10.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:10 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 8165/198012 objects degraded (4.123%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3330085719' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:10.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:10 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 8165/198012 objects degraded (4.123%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3330085719' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:10.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:10.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:10.750 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:10.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:10.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:10.873 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:10.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:10.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:10.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:11.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:11.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:11.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.244 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:11.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:11.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:11.369 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:11.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:11.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.506 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:11.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:11.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[105288]: pgmap v2113: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 131 op/s; 8165/191934 objects degraded (4.254%); 4344/191934 objects misplaced (2.263%) 2024-09-21T10:36:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:36:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:36:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[105288]: osdmap e1326: 8 total, 8 up, 8 in 2024-09-21T10:36:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[101432]: pgmap v2113: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 131 op/s; 8165/191934 objects degraded (4.254%); 4344/191934 objects misplaced (2.263%) 2024-09-21T10:36:11.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:36:11.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:36:11.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:11 smithi119 ceph-mon[101432]: osdmap e1326: 8 total, 8 up, 8 in 2024-09-21T10:36:11.637 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:11.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:11.739 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:11.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:11 smithi145 ceph-mon[89359]: pgmap v2113: 119 pgs: 1 active+recovering+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 3.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 131 op/s; 8165/191934 objects degraded (4.254%); 4344/191934 objects misplaced (2.263%) 2024-09-21T10:36:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:11 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:36:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:36:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:11 smithi145 ceph-mon[89359]: osdmap e1326: 8 total, 8 up, 8 in 2024-09-21T10:36:11.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:11.914 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:11.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:12.016 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:12.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:12.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:12.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:12.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:12.324 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:12.426 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.467 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:12.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:12.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:12.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:12 smithi119 ceph-mon[101432]: osdmap e1327: 8 total, 8 up, 8 in 2024-09-21T10:36:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:12 smithi119 ceph-mon[105288]: osdmap e1327: 8 total, 8 up, 8 in 2024-09-21T10:36:12.709 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:12.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:12 smithi145 ceph-mon[89359]: osdmap e1327: 8 total, 8 up, 8 in 2024-09-21T10:36:12.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:12.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:12.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:12.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:13.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:13.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:13.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:13.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:13.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:13.430 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:13.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:13.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:13 smithi119 ceph-mon[105288]: pgmap v2116: 119 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 242 op/s; 8165/186600 objects degraded (4.376%); 3893/186600 objects misplaced (2.086%); 8.8 MiB/s, 141 objects/s recovering 2024-09-21T10:36:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:13 smithi119 ceph-mon[101432]: pgmap v2116: 119 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 242 op/s; 8165/186600 objects degraded (4.376%); 3893/186600 objects misplaced (2.086%); 8.8 MiB/s, 141 objects/s recovering 2024-09-21T10:36:13.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:13.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:13 smithi145 ceph-mon[89359]: pgmap v2116: 119 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 242 op/s; 8165/186600 objects degraded (4.376%); 3893/186600 objects misplaced (2.086%); 8.8 MiB/s, 141 objects/s recovering 2024-09-21T10:36:13.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:13.840 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:13.941 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:13.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:13.979 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:13.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:14.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:14.110 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:14.147 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:14.166 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:36:14.166 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:36:14.166 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:36:14.166 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:36:14.166 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:36:14.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:14.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:14.344 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:14.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:14.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:14.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:14.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:14.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:14.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:14.781 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:36:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:36:14] "GET /metrics HTTP/1.1" 200 36916 "" "Prometheus/2.51.0" 2024-09-21T10:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:14.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:14.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:14.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:15.065 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:15.100 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:15.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:15.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:15.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:15.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:15.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.528 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:15.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:15.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:36:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:15 smithi119 ceph-mon[101432]: pgmap v2117: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 116 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 442 op/s; 3893/177870 objects degraded (2.189%); 7329/177870 objects misplaced (4.120%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:36:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:15 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3893/177870 objects degraded (2.189%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:15 smithi119 ceph-mon[105288]: pgmap v2117: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 116 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 442 op/s; 3893/177870 objects degraded (2.189%); 7329/177870 objects misplaced (4.120%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:36:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:15 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3893/177870 objects degraded (2.189%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:15.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:15.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:15.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:15.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:15 smithi145 ceph-mon[89359]: pgmap v2117: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 116 active+clean; 3.6 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 442 op/s; 3893/177870 objects degraded (2.189%); 7329/177870 objects misplaced (4.120%); 5.9 MiB/s, 94 objects/s recovering 2024-09-21T10:36:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:15 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3893/177870 objects degraded (2.189%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:15.806 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:15.908 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:15.940 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:15.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:16.042 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:16.080 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:16.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:16.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:16.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:16.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:16.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:16.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:16.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:16.495 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:16.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.639 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:16.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:16.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:16.771 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:16.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:16.905 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:17.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:17.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:17.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:17.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:17.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:17.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:17.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:17.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:17.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:17.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:17.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:17.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:17.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.580 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:17.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:17.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:17.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:17 smithi119 ceph-mon[101432]: pgmap v2118: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 116 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 3893/176004 objects degraded (2.212%); 7329/176004 objects misplaced (4.164%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:36:17.689 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:17 smithi119 ceph-mon[105288]: pgmap v2118: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 116 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 3893/176004 objects degraded (2.212%); 7329/176004 objects misplaced (4.164%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:36:17.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:17.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:17 smithi145 ceph-mon[89359]: pgmap v2118: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 116 active+clean; 3.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 916 op/s; 3893/176004 objects degraded (2.212%); 7329/176004 objects misplaced (4.164%); 5.2 MiB/s, 83 objects/s recovering 2024-09-21T10:36:17.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:17.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:17.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:17.994 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:17.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:18.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:36:18.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:18.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:18.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:18.267 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:18.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:18.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:18.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:18.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:18.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:18.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:18.821 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:18.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:18.910 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:18 smithi119 ceph-mon[105288]: pgmap v2119: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 116 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3893/167553 objects degraded (2.323%); 6955/167553 objects misplaced (4.151%); 6.6 MiB/s, 106 objects/s recovering 2024-09-21T10:36:18.911 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:18.911 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:18.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:18 smithi119 ceph-mon[101432]: pgmap v2119: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 116 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3893/167553 objects degraded (2.323%); 6955/167553 objects misplaced (4.151%); 6.6 MiB/s, 106 objects/s recovering 2024-09-21T10:36:18.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:18.911 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:18.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:18.945 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:18.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:18 smithi145 ceph-mon[89359]: pgmap v2119: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 116 active+clean; 3.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 3893/167553 objects degraded (2.323%); 6955/167553 objects misplaced (4.151%); 6.6 MiB/s, 106 objects/s recovering 2024-09-21T10:36:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:19.046 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:19.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:19.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:19.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:19.211 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:19.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:19.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:19.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.481 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:19.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:19.501 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:36:19.501 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:36:19.539 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:36:19.539 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:36:19.563 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:19.580 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:36:19.580 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:36:19.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:19.623 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:36:19.624 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:36:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:19.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:19.660 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:36:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:36:19.697 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:36:19.697 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:36:19.714 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:19.737 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:36:19.738 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:36:19.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:19.773 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:36:19.773 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:36:19.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:19.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:19.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:19.954 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1325573002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:36:19.954 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3893/167553 objects degraded (2.323%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:36:19.954 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1325573002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:36:19.954 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3893/167553 objects degraded (2.323%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:36:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:19.956 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:19.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:20.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:20.092 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:20.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:20.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1325573002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:36:20.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3893/167553 objects degraded (2.323%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:36:20.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:20.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:20.228 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:36:20.087+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1327 definitely_dead 0 2024-09-21T10:36:20.327 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:20.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:20.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:20.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:36:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:36:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:36:20.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:20.640 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:20.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:20.775 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:20.840 INFO:teuthology.orchestra.run.smithi119.stderr:marked out osd.4. 2024-09-21T10:36:20.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:20.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:20.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:21.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:21.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[101432]: pgmap v2120: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 116 active+clean; 3.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3893/161352 objects degraded (2.413%); 6955/161352 objects misplaced (4.310%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1939846497' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[101432]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[105288]: pgmap v2120: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 116 active+clean; 3.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3893/161352 objects degraded (2.413%); 6955/161352 objects misplaced (4.310%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1939846497' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:36:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:36:21.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:20 smithi119 ceph-mon[105288]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:36:21.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:21.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:21.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:21.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:21.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:20 smithi145 ceph-mon[89359]: pgmap v2120: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovery_wait, 116 active+clean; 3.3 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 3893/161352 objects degraded (2.413%); 6955/161352 objects misplaced (4.310%); 2.7 MiB/s, 42 objects/s recovering 2024-09-21T10:36:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1939846497' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:36:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:20 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-21T10:36:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:20 smithi145 ceph-mon[89359]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T10:36:21.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:21.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:21.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:21.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:21.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:21.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:21.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:21.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:21.891 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:21.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:21.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:22.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:22.034 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:21 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:36:22.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:21 smithi119 ceph-mon[101432]: osdmap e1328: 8 total, 8 up, 7 in 2024-09-21T10:36:22.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:21 smithi119 ceph-mon[101432]: osdmap e1329: 8 total, 8 up, 7 in 2024-09-21T10:36:22.035 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:21 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:36:22.035 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:21 smithi119 ceph-mon[105288]: osdmap e1328: 8 total, 8 up, 7 in 2024-09-21T10:36:22.035 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:21 smithi119 ceph-mon[105288]: osdmap e1329: 8 total, 8 up, 7 in 2024-09-21T10:36:22.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:22.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:22.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:22.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:21 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T10:36:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:21 smithi145 ceph-mon[89359]: osdmap e1328: 8 total, 8 up, 7 in 2024-09-21T10:36:22.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:21 smithi145 ceph-mon[89359]: osdmap e1329: 8 total, 8 up, 7 in 2024-09-21T10:36:22.362 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:22.383 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:22.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:22.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:22.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.645 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:22.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:22.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:22.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:22.891 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:22.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:22.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:23.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:23.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:22 smithi119 ceph-mon[101432]: pgmap v2122: 119 pgs: 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 101 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 3893/155394 objects degraded (2.505%); 6955/155394 objects misplaced (4.476%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:36:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:22 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:36:23.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:22 smithi119 ceph-mon[101432]: osdmap e1330: 8 total, 8 up, 7 in 2024-09-21T10:36:23.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:22 smithi119 ceph-mon[105288]: pgmap v2122: 119 pgs: 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 101 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 3893/155394 objects degraded (2.505%); 6955/155394 objects misplaced (4.476%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:36:23.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:22 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:36:23.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:22 smithi119 ceph-mon[105288]: osdmap e1330: 8 total, 8 up, 7 in 2024-09-21T10:36:23.145 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:23.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:22 smithi145 ceph-mon[89359]: pgmap v2122: 119 pgs: 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+remapped, 101 active+clean; 3.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 1.04k op/s; 3893/155394 objects degraded (2.505%); 6955/155394 objects misplaced (4.476%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:36:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:22 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:36:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:22 smithi145 ceph-mon[89359]: osdmap e1330: 8 total, 8 up, 7 in 2024-09-21T10:36:23.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:23.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:23.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:23.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:23.572 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:23.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:23.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:23.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:23.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:23.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:23.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:23.995 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:23.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:24.005 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:36:24.005 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.11 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'7917 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.0 starting backfill to osd.1 from (0'0,0'0] MAX to 1327'10887 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.14 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8020 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.d starting backfill to osd.0 from (0'0,0'0] MAX to 1327'12382 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.4 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8044 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.b starting backfill to osd.2 from (0'0,0'0] MAX to 1327'12454 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.8 starting backfill to osd.2 from (0'0,0'0] MAX to 1327'8154 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.17 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8081 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.18 starting backfill to osd.2 from (0'0,0'0] MAX to 1327'8089 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[101432]: 47.7 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8091 2024-09-21T10:36:24.006 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.11 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'7917 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.0 starting backfill to osd.1 from (0'0,0'0] MAX to 1327'10887 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.14 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8020 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.d starting backfill to osd.0 from (0'0,0'0] MAX to 1327'12382 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.4 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8044 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.b starting backfill to osd.2 from (0'0,0'0] MAX to 1327'12454 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.8 starting backfill to osd.2 from (0'0,0'0] MAX to 1327'8154 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.17 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8081 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.18 starting backfill to osd.2 from (0'0,0'0] MAX to 1327'8089 2024-09-21T10:36:24.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:23 smithi119 ceph-mon[105288]: 47.7 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8091 2024-09-21T10:36:24.097 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:24.126 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:24.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:24.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:24.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:24.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.11 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'7917 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.0 starting backfill to osd.1 from (0'0,0'0] MAX to 1327'10887 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.14 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8020 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.d starting backfill to osd.0 from (0'0,0'0] MAX to 1327'12382 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.4 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8044 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.b starting backfill to osd.2 from (0'0,0'0] MAX to 1327'12454 2024-09-21T10:36:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.8 starting backfill to osd.2 from (0'0,0'0] MAX to 1327'8154 2024-09-21T10:36:24.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.17 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8081 2024-09-21T10:36:24.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.18 starting backfill to osd.2 from (0'0,0'0] MAX to 1327'8089 2024-09-21T10:36:24.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:23 smithi145 ceph-mon[89359]: 47.7 starting backfill to osd.0 from (0'0,0'0] MAX to 1327'8091 2024-09-21T10:36:24.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:24.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:24.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:24.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:24.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:24.561 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:24.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:24.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:24.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:24.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:24.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:24.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:24.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:24.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:36:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:36:24] "GET /metrics HTTP/1.1" 200 36913 "" "Prometheus/2.51.0" 2024-09-21T10:36:24.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:25.002 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:25.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:25.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:25.240 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:25.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:25.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:24 smithi145 ceph-mon[89359]: pgmap v2125: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 remapped+peering, 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 97 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 942 op/s; 3893/150645 objects degraded (2.584%); 10041/150645 objects misplaced (6.665%); 2.3 MiB/s, 1 keys/s, 38 objects/s recovering 2024-09-21T10:36:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:24 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3893/150645 objects degraded (2.584%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:25.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:24 smithi119 ceph-mon[101432]: pgmap v2125: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 remapped+peering, 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 97 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 942 op/s; 3893/150645 objects degraded (2.584%); 10041/150645 objects misplaced (6.665%); 2.3 MiB/s, 1 keys/s, 38 objects/s recovering 2024-09-21T10:36:25.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:24 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3893/150645 objects degraded (2.584%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:25.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:24 smithi119 ceph-mon[105288]: pgmap v2125: 119 pgs: 2 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 remapped+peering, 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 97 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 942 op/s; 3893/150645 objects degraded (2.584%); 10041/150645 objects misplaced (6.665%); 2.3 MiB/s, 1 keys/s, 38 objects/s recovering 2024-09-21T10:36:25.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:24 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3893/150645 objects degraded (2.584%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:36:25.363 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:36:25.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.420 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:25.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:25.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:25.559 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:25.661 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:25.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:25.757 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:36:25.794 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:25.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:25.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:25.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:25.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:25.988 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:25.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:26.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:26.164 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:25 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:36:26.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:26.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:26.265 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:26.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:26.299 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:26.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:25 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:36:26.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:25 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:36:26.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:26.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:26.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:26.577 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:26.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:26.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:26.810 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:26.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:26.924 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:36:26.961 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:26.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:26.993 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:26.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:27.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:36:27.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:36:27.059 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-09-21T10:36:27.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:36:27.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:27.103 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:26 smithi119 ceph-mon[101432]: pgmap v2126: 119 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 2 remapped+peering, 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 658 op/s; 3893/149619 objects degraded (2.602%); 12849/149619 objects misplaced (8.588%); 4.7 MiB/s, 1 keys/s, 77 objects/s recovering 2024-09-21T10:36:27.103 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1574505530' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:36:27.103 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:26 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:36:27.103 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:26 smithi119 ceph-mon[105288]: pgmap v2126: 119 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 2 remapped+peering, 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 658 op/s; 3893/149619 objects degraded (2.602%); 12849/149619 objects misplaced (8.588%); 4.7 MiB/s, 1 keys/s, 77 objects/s recovering 2024-09-21T10:36:27.103 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1574505530' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:36:27.104 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:26 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:36:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:27.192 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:27.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:27.281 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:26 smithi145 ceph-mon[89359]: pgmap v2126: 119 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 2 remapped+peering, 2 unknown, 13 peering, 1 active+recovery_wait+undersized+degraded+remapped, 96 active+clean; 3.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 658 op/s; 3893/149619 objects degraded (2.602%); 12849/149619 objects misplaced (8.588%); 4.7 MiB/s, 1 keys/s, 77 objects/s recovering 2024-09-21T10:36:27.282 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1574505530' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:36:27.282 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:26 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:36:27.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:27.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:27.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:27.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:27.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:27.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:27.619 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:27.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:27.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:27.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:27.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:27.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:27.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:27.923 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:27.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:28.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:28.056 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:28.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:28.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:28.239 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:27 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:36:28.240 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:27 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:36:28.240 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:27 smithi145 ceph-mon[89359]: osdmap e1331: 8 total, 8 up, 7 in 2024-09-21T10:36:28.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:28.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:27 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:36:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:27 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:36:28.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:27 smithi119 ceph-mon[101432]: osdmap e1331: 8 total, 8 up, 7 in 2024-09-21T10:36:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:27 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:36:28.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:27 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:36:28.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:27 smithi119 ceph-mon[105288]: osdmap e1331: 8 total, 8 up, 7 in 2024-09-21T10:36:28.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:28.485 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:28.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:28.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:28.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:28.783 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:28.857 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:28.878 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:28.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:28.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:29.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:29.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:28 smithi119 ceph-mon[101432]: pgmap v2128: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 687 op/s; 5465/143142 objects degraded (3.818%); 19122/143142 objects misplaced (13.359%); 5.8 MiB/s, 1 keys/s, 94 objects/s recovering 2024-09-21T10:36:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:28 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:36:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:28 smithi119 ceph-mon[105288]: pgmap v2128: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 687 op/s; 5465/143142 objects degraded (3.818%); 19122/143142 objects misplaced (13.359%); 5.8 MiB/s, 1 keys/s, 94 objects/s recovering 2024-09-21T10:36:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:28 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:36:29.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:29.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:29.273 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:36:29.273 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:36:29.273 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:28 smithi145 ceph-mon[89359]: pgmap v2128: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 11 KiB/s rd, 25 KiB/s wr, 687 op/s; 5465/143142 objects degraded (3.818%); 19122/143142 objects misplaced (13.359%); 5.8 MiB/s, 1 keys/s, 94 objects/s recovering 2024-09-21T10:36:29.273 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:28 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-21T10:36:29.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:29.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:29.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:29.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.531 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:29.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:29.632 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:29.667 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:29.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:29.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:29.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:29.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:29.943 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:29.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:29.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:36:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5465/143142 objects degraded (3.818%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:30.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:30.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:30.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:30.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:30.214 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:30.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:30.349 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:36:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:36:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:36:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.365 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:30.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5465/143142 objects degraded (3.818%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5465/143142 objects degraded (3.818%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:30.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:36:30.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:30.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:30.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:30.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:30.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:30.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:30.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:30.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:30.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:31.007 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:30 smithi145 ceph-mon[89359]: pgmap v2129: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 8.6 KiB/s rd, 19 KiB/s wr, 673 op/s; 5465/139593 objects degraded (3.915%); 18160/139593 objects misplaced (13.009%); 7.7 MiB/s, 1 keys/s, 124 objects/s recovering 2024-09-21T10:36:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:31.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:31.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:31.234 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:31.283 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:31.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:30 smithi119 ceph-mon[101432]: pgmap v2129: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 8.6 KiB/s rd, 19 KiB/s wr, 673 op/s; 5465/139593 objects degraded (3.915%); 18160/139593 objects misplaced (13.009%); 7.7 MiB/s, 1 keys/s, 124 objects/s recovering 2024-09-21T10:36:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:30 smithi119 ceph-mon[105288]: pgmap v2129: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 8.6 KiB/s rd, 19 KiB/s wr, 673 op/s; 5465/139593 objects degraded (3.915%); 18160/139593 objects misplaced (13.009%); 7.7 MiB/s, 1 keys/s, 124 objects/s recovering 2024-09-21T10:36:31.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:31.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:31.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:31.552 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:31.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:31.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:31.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:31.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:31.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:31.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:31.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:31.973 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:31.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:31.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:32.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:32.106 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:32.208 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:32.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:32.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:32.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:32.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:32.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:32.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:32.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:32.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:32.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:32.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:32.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:32.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:32.998 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:32.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:33.060 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:32 smithi145 ceph-mon[89359]: pgmap v2130: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 7.7 KiB/s rd, 17 KiB/s wr, 702 op/s; 5465/134040 objects degraded (4.077%); 16698/134040 objects misplaced (12.457%); 11 MiB/s, 1 keys/s, 181 objects/s recovering 2024-09-21T10:36:33.060 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:33.060 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:33.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:33.206 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:36:33.206 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:36:33.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:33.241 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:32 smithi119 ceph-mon[101432]: pgmap v2130: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 7.7 KiB/s rd, 17 KiB/s wr, 702 op/s; 5465/134040 objects degraded (4.077%); 16698/134040 objects misplaced (12.457%); 11 MiB/s, 1 keys/s, 181 objects/s recovering 2024-09-21T10:36:33.241 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:33.241 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:33.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:32 smithi119 ceph-mon[105288]: pgmap v2130: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 7.7 KiB/s rd, 17 KiB/s wr, 702 op/s; 5465/134040 objects degraded (4.077%); 16698/134040 objects misplaced (12.457%); 11 MiB/s, 1 keys/s, 181 objects/s recovering 2024-09-21T10:36:33.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:33.241 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:33.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:33.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:33.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:33.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:33.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:33.686 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:33.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:33.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:33.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:33.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:33.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:36:33.945 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:33.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:33.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:34.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:34.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:34.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.223 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:34.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:34.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:34.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:34.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:34.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:34.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:34.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:34.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:34.616 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:34.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:34.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:34.855 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:36:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:36:34] "GET /metrics HTTP/1.1" 200 36934 "" "Prometheus/2.51.0" 2024-09-21T10:36:34.863 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:34.884 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:34.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:34.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:34 smithi119 ceph-mon[101432]: pgmap v2131: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 3.4 KiB/s rd, 7.4 KiB/s wr, 751 op/s; 5291/128148 objects degraded (4.129%); 15654/128148 objects misplaced (12.216%); 12 MiB/s, 200 objects/s recovering 2024-09-21T10:36:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:34 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5291/128148 objects degraded (4.129%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:34 smithi119 ceph-mon[105288]: pgmap v2131: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 3.4 KiB/s rd, 7.4 KiB/s wr, 751 op/s; 5291/128148 objects degraded (4.129%); 15654/128148 objects misplaced (12.216%); 12 MiB/s, 200 objects/s recovering 2024-09-21T10:36:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:34 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5291/128148 objects degraded (4.129%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:35.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:35.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:35.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:35.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:34 smithi145 ceph-mon[89359]: pgmap v2131: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 8 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 3.4 KiB/s rd, 7.4 KiB/s wr, 751 op/s; 5291/128148 objects degraded (4.129%); 15654/128148 objects misplaced (12.216%); 12 MiB/s, 200 objects/s recovering 2024-09-21T10:36:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:34 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5291/128148 objects degraded (4.129%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:35.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:35.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:35.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:35.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:35.560 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:35.661 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:35.691 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:35.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:35.815 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:35.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:35.938 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:35.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:36.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:36.068 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:36.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:36.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:36.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:35 smithi145 ceph-mon[89359]: osdmap e1332: 8 total, 8 up, 7 in 2024-09-21T10:36:36.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:36.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:36.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:35 smithi119 ceph-mon[101432]: osdmap e1332: 8 total, 8 up, 7 in 2024-09-21T10:36:36.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:35 smithi119 ceph-mon[105288]: osdmap e1332: 8 total, 8 up, 7 in 2024-09-21T10:36:36.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:36.467 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:36.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:36.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:36.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:36.711 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:36.744 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:36.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:36.850 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:36.893 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:36.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:36.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:37.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:37.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:37.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:37.174 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:37.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:37.276 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:36 smithi145 ceph-mon[89359]: pgmap v2133: 119 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 5291/126555 objects degraded (4.181%); 14812/126555 objects misplaced (11.704%); 16 MiB/s, 263 objects/s recovering 2024-09-21T10:36:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:36 smithi145 ceph-mon[89359]: osdmap e1333: 8 total, 8 up, 7 in 2024-09-21T10:36:37.301 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:36 smithi119 ceph-mon[101432]: pgmap v2133: 119 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 5291/126555 objects degraded (4.181%); 14812/126555 objects misplaced (11.704%); 16 MiB/s, 263 objects/s recovering 2024-09-21T10:36:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:36 smithi119 ceph-mon[101432]: osdmap e1333: 8 total, 8 up, 7 in 2024-09-21T10:36:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:36 smithi119 ceph-mon[105288]: pgmap v2133: 119 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 7 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 748 op/s; 5291/126555 objects degraded (4.181%); 14812/126555 objects misplaced (11.704%); 16 MiB/s, 263 objects/s recovering 2024-09-21T10:36:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:36 smithi119 ceph-mon[105288]: osdmap e1333: 8 total, 8 up, 7 in 2024-09-21T10:36:37.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:37.430 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:37.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:37.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:37.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:37.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:37.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:37.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:37.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:37.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:37.899 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:37.920 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:37.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:38.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:38.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:38.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:37 smithi119 ceph-mon[101432]: osdmap e1334: 8 total, 8 up, 7 in 2024-09-21T10:36:38.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:37 smithi119 ceph-mon[105288]: osdmap e1334: 8 total, 8 up, 7 in 2024-09-21T10:36:38.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:38.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:38.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:38.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:38.290 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:37 smithi145 ceph-mon[89359]: osdmap e1334: 8 total, 8 up, 7 in 2024-09-21T10:36:38.330 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:38.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:38.463 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:38.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:38.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:38.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:38.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:38.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:38.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:38.866 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:38.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:38.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:39.007 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:39.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:39.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:39.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:39.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:39.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:39.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:38 smithi145 ceph-mon[89359]: pgmap v2136: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 7 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 806 op/s; 5047/119523 objects degraded (4.223%); 13966/119523 objects misplaced (11.685%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:36:39.296 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:38 smithi145 ceph-mon[89359]: osdmap e1335: 8 total, 8 up, 7 in 2024-09-21T10:36:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:38 smithi119 ceph-mon[101432]: pgmap v2136: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 7 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 806 op/s; 5047/119523 objects degraded (4.223%); 13966/119523 objects misplaced (11.685%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:36:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:39 smithi119 ceph-mon[101432]: osdmap e1335: 8 total, 8 up, 7 in 2024-09-21T10:36:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:38 smithi119 ceph-mon[105288]: pgmap v2136: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 7 active+remapped+backfill_wait, 2 active+remapped+backfilling, 107 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 806 op/s; 5047/119523 objects degraded (4.223%); 13966/119523 objects misplaced (11.685%); 15 MiB/s, 238 objects/s recovering 2024-09-21T10:36:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:39 smithi119 ceph-mon[105288]: osdmap e1335: 8 total, 8 up, 7 in 2024-09-21T10:36:39.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:39.433 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:39.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:39.562 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:39.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:39.691 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:39.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:39.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:39.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:39.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:39.966 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:39.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:40.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:40.090 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:40.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:40.127 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:40 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5047/119523 objects degraded (4.223%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:40.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2124962612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:36:40.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:40.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:40.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:40 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5047/119523 objects degraded (4.223%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:40.229 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2124962612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:36:40.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:40 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5047/119523 objects degraded (4.223%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:40.230 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2124962612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:36:40.230 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:36:40.230 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:36:40.230 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:36:40.230 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:36:40.230 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:36:40.230 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:36:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:36:40.257 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:36:40.257 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:36:40.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:206271 45.1 45:82987e0b:::benchmark_data_smithi119_168706_object76101:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:40.841044+0000", 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 599.27397290099998, 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.050415605, 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 206271 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.258 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841044+0000", 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841044+0000", 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841042+0000", 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:40.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841048+0000", 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1990000000000003e-06 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841064+0000", 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6398000000000001e-05 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841071+0000", 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1360000000000002e-06 2024-09-21T10:36:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841131+0000", 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0254000000000003e-05 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.841138+0000", 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6650000000000003e-06 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.872528+0000", 2024-09-21T10:36:40.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0313896119999999 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.876002+0000", 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003474361 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0", 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.876098+0000", 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5988000000000003e-05 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.887217+0000", 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011118846999999999 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.891320+0000", 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041023659999999997 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.891370+0000", 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.99e-05 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.891459+0000", 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9635000000000005e-05 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:206339 45.1 45:8cf2172c:::benchmark_data_smithi119_168706_object76169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:40.927717+0000", 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 599.187299435, 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96891153200000002, 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:40.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 206339 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927717+0000", 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927717+0000", 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927716+0000", 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927722+0000", 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5490000000000003e-06 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927734+0000", 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2136999999999999e-05 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927741+0000", 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1620000000000002e-06 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927793+0000", 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1411e-05 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.927802+0000", 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0489999999999993e-06 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.879226+0000", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.95142409400000005 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.879342+0000", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116457 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.879455+0000", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011266 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.887223+0000", 2024-09-21T10:36:40.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077681649999999996 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896538+0000", 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.009315149 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896573+0000", 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4255000000000003e-05 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896629+0000", 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5831000000000001e-05 2024-09-21T10:36:40.269 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:206363 45.1 45:81c5df67:::benchmark_data_smithi119_168706_object76193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1200)", 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:26:40.966827+0000", 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 599.14818992699998, 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92998550800000002, 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 206363 2024-09-21T10:36:40.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966827+0000", 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966827+0000", 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966825+0000", 2024-09-21T10:36:40.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966834+0000", 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7299999999999994e-06 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966851+0000", 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7292999999999999e-05 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966858+0000", 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6830000000000002e-06 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966906+0000", 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8470999999999997e-05 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:40.966916+0000", 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.005e-05 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.883112+0000", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.91619621500000004 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.883215+0000", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010313499999999999 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.883290+0000", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4449000000000005e-05 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.887230+0000", 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039403370000000004 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896672+0000", 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0094420649999999995 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896721+0000", 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8977999999999998e-05 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:26:41.896812+0000", 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0816999999999995e-05 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:54700 47.1 47:8d0349cd:::benchmark_data_smithi119_238957_object54699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:01.490653+0000", 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 158.62436366899999, 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10272278999999999, 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 54700 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490653+0000", 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490653+0000", 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490651+0000", 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490685+0000", 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3769999999999997e-05 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490703+0000", 2024-09-21T10:36:40.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8076000000000001e-05 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490710+0000", 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4509999999999997e-06 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490768+0000", 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8638000000000003e-05 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.490902+0000", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013367699999999999 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.491052+0000", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000150383 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.493080+0000", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020275760000000001 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.493368+0000", 2024-09-21T10:36:40.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000287769 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.593277+0000", 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.099909716999999995 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.593341+0000", 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3244999999999995e-05 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:01.593376+0000", 2024-09-21T10:36:40.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4934999999999997e-05 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55565 47.11 47:8d2e24a3:::benchmark_data_smithi119_238957_object55564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.408320+0000", 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 157.70669662899999, 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0753061399999999, 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.282 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55565 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408320+0000", 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408320+0000", 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408316+0000", 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408370+0000", 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3699000000000002e-05 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408390+0000", 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9998999999999999e-05 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408399+0000", 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6969999999999999e-06 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.463739+0000", 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.055340093999999999 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.463760+0000", 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1586e-05 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.473493+0000", 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0097327549999999 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.473569+0000", 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6061000000000002e-05 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.473678+0000", 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108972 2024-09-21T10:36:40.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.474612+0000", 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00093417999999999995 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.475720+0000", 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011081470000000001 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.483517+0000", 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077964180000000003 2024-09-21T10:36:40.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.483585+0000", 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8565000000000005e-05 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.483626+0000", 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0572999999999999e-05 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.288 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55573 47.10 47:0ef07b0d:::benchmark_data_smithi119_238957_object55572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.408412+0000", 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 157.70660458, 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0028219249999999, 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55573 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408412+0000", 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408412+0000", 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408411+0000", 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408448+0000", 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7005999999999998e-05 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408455+0000", 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.092e-06 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408460+0000", 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1609999999999996e-06 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.478472+0000", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070012106000000005 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.478482+0000", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696000000000001e-05 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.408993+0000", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93051024100000002 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.409084+0000", 2024-09-21T10:36:40.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1686000000000003e-05 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.409181+0000", 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6817999999999994e-05 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.410864+0000", 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016828469999999999 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411115+0000", 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000251108 2024-09-21T10:36:40.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411145+0000", 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9635999999999999e-05 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411186+0000", 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1270000000000003e-05 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411234+0000", 2024-09-21T10:36:40.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7766000000000003e-05 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55581 47.10 47:0cc25ec3:::benchmark_data_smithi119_238957_object55580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.568526+0000", 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 157.54649017, 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.84286496399999999, 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.295 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55581 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568526+0000", 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568526+0000", 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.296 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568525+0000", 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568586+0000", 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1499000000000002e-05 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568596+0000", 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6930000000000002e-06 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568602+0000", 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5069999999999997e-06 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568643+0000", 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0488999999999998e-05 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568650+0000", 2024-09-21T10:36:40.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3509999999999996e-06 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.409915+0000", 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.84126443799999995 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.410032+0000", 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117043 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:36:40.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.410172+0000", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000140075 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411246+0000", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001074336 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411278+0000", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2325e-05 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411341+0000", 2024-09-21T10:36:40.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2892999999999996e-05 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411371+0000", 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9804000000000001e-05 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411391+0000", 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0091000000000001e-05 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:146542 47.0 47:01b22221:::benchmark_data_smithi119_238957_object21713:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1322)", 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:29.304845+0000", 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.810171526999994, 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.086941936999999997, 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 146542 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304845+0000", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304845+0000", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304843+0000", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304851+0000", 2024-09-21T10:36:40.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8399999999999995e-06 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304867+0000", 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6191000000000001e-05 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304873+0000", 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4010000000000002e-06 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.304937+0000", 2024-09-21T10:36:40.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4181000000000001e-05 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.305065+0000", 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000127661 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.305705+0000", 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00064020099999999996 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.309511+0000", 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038058369999999998 2024-09-21T10:36:40.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.310134+0000", 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062267500000000005 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.391631+0000", 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.081497595000000006 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.391743+0000", 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000111603 2024-09-21T10:36:40.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:29.391787+0000", 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3872000000000003e-05 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:146845 47.1 47:841683d9:::benchmark_data_smithi119_238957_object22016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:30.107251+0000", 2024-09-21T10:36:40.307 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 70.007765301000006, 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10138256499999999, 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 146845 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107251+0000", 2024-09-21T10:36:40.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107251+0000", 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107249+0000", 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107257+0000", 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.123e-06 2024-09-21T10:36:40.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107270+0000", 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2516e-05 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107275+0000", 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4709999999999998e-06 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107340+0000", 2024-09-21T10:36:40.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4400999999999995e-05 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107464+0000", 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012471800000000001 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.107575+0000", 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110099 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.110405+0000", 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028302140000000002 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.110939+0000", 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00053407900000000002 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.208538+0000", 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.097599134000000004 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.208592+0000", 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3681999999999999e-05 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:30.208634+0000", 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2104999999999998e-05 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:148053 47.1 47:85150950:::benchmark_data_smithi119_238957_object23224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:31.373557+0000", 2024-09-21T10:36:40.313 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 68.741459395000007, 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.106037285, 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 148053 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373557+0000", 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373557+0000", 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373556+0000", 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373563+0000", 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2339999999999997e-06 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373580+0000", 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6849000000000001e-05 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373587+0000", 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5190000000000003e-06 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373636+0000", 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8939e-05 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373763+0000", 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012636599999999999 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.373889+0000", 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012616799999999999 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.376182+0000", 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022933049999999998 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.479509+0000", 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.103327066 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.479532+0000", 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2586e-05 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.479568+0000", 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5815999999999997e-05 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:31.479594+0000", 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6684000000000001e-05 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46418.0:152063 47.11 e1323/1309 47:8b3cf2fb:::benchmark_data_smithi119_238957_object27234:head v 1323'6438, mlcod=1323'6435)", 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:35.238870+0000", 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 64.876146865999999, 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15509666699999999, 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238870+0000", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238870+0000", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238869+0000", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238875+0000", 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9159999999999998e-06 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238885+0000", 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0387000000000001e-05 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238890+0000", 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9910000000000002e-06 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238939+0000", 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8686000000000001e-05 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.238956+0000", 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6742e-05 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.393916+0000", 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15496044 2024-09-21T10:36:40.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.393966+0000", 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0129000000000001e-05 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46418.0:152066 47.10 e1323/1306 47:0edc5cc9:::benchmark_data_smithi119_238957_object27237:head v 1323'6542, mlcod=1323'6540)", 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:35.239322+0000", 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 64.875694809999999, 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.16206368099999999, 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.323 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239322+0000", 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239322+0000", 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239321+0000", 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239330+0000", 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3759999999999997e-06 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239342+0000", 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1997999999999999e-05 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239348+0000", 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4299999999999997e-06 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239398+0000", 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0657e-05 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.239416+0000", 2024-09-21T10:36:40.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7329e-05 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.401354+0000", 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.16193807800000001 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.401385+0000", 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1572000000000002e-05 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "MOSDPGPush(47.14 1323/1314 [PushOp(47:2babe832:::benchmark_data_smithi119_238957_object50847:head, version: 1291'3184, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(47:2babe832:::benchmark_data_smithi119_238957_object50847:head@1291'3184, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:35.245835+0000", 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 64.869181190000006, 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.147726208, 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.245835+0000", 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.245835+0000", 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.245834+0000", 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.245899+0000", 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5085000000000005e-05 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391180+0000", 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14528121499999999 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391228+0000", 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7321999999999999e-05 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391333+0000", 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010528500000000001 2024-09-21T10:36:40.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391341+0000", 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6980000000000004e-06 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.393561+0000", 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022199870000000001 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.331 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46418.0:152074 47.11 e1323/1309 47:899b88c3:::benchmark_data_smithi119_238957_object27245:head v 1323'6439, mlcod=1323'6436)", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:35.249514+0000", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 64.865502738999993, 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14895560799999999, 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.249514+0000", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.249514+0000", 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.249512+0000", 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.249523+0000", 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0353000000000001e-05 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391334+0000", 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14181152299999999 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391340+0000", 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4589999999999997e-06 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391705+0000", 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036589899999999999 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391726+0000", 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0815999999999999e-05 2024-09-21T10:36:40.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.398353+0000", 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0066267790000000002 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.398469+0000", 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116282 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.335 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:152086 47.1 47:84834a50:::benchmark_data_smithi119_238957_object27257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1323)", 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:35.259305+0000", 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 64.855711036000002, 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14399554000000001, 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 152086 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.259305+0000", 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.259305+0000", 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.259303+0000", 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.259314+0000", 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1004e-05 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391260+0000", 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.13194518299999999 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391288+0000", 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8459999999999999e-05 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391536+0000", 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00024767099999999998 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391648+0000", 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112173 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.391751+0000", 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010275799999999999 2024-09-21T10:36:40.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.396162+0000", 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044110410000000001 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.398245+0000", 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002083035 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.403114+0000", 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048693679999999998 2024-09-21T10:36:40.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.403191+0000", 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7177999999999999e-05 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:35.403301+0000", 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109716 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.341 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:158630 47.1 47:83d5697f:::benchmark_data_smithi119_238957_object33801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:41.193631+0000", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 58.921385839999999, 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.111438669, 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 158630 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193631+0000", 2024-09-21T10:36:40.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193631+0000", 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193629+0000", 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193636+0000", 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2330000000000003e-06 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193651+0000", 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5789000000000001e-05 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193658+0000", 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4439999999999996e-06 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193706+0000", 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.791e-05 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.193831+0000", 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012545199999999999 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.194086+0000", 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025498699999999998 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.197801+0000", 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037147759999999999 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.199218+0000", 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001417015 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.304979+0000", 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10576081499999999 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305031+0000", 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1882999999999998e-05 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305069+0000", 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8537000000000001e-05 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.347 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:158639 47.1 47:83e9a097:::benchmark_data_smithi119_238957_object33810:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:41.204108+0000", 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 58.910908395, 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.101079975, 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 158639 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.348 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204108+0000", 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204108+0000", 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204106+0000", 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.349 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204114+0000", 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3420000000000002e-06 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204127+0000", 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2948e-05 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.350 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204132+0000", 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6869999999999999e-06 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204473+0000", 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00034059500000000001 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.204773+0000", 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000299458 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.205072+0000", 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029989099999999999 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.209311+0000", 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0042383719999999998 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.209332+0000", 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.073e-05 2024-09-21T10:36:40.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305105+0000", 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.095773213999999995 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305147+0000", 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2709000000000001e-05 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305188+0000", 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0605999999999997e-05 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:158641 47.1 47:80d17782:::benchmark_data_smithi119_238957_object33812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1324)", 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:35:41.206092+0000", 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 58.908924442, 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.099196147999999998, 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.354 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 158641 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.206092+0000", 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.206092+0000", 2024-09-21T10:36:40.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.206091+0000", 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.206097+0000", 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1920000000000002e-06 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.206111+0000", 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4516e-05 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.206116+0000", 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4399999999999998e-06 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.207820+0000", 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001704491 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.207906+0000", 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5740000000000002e-05 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.208030+0000", 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012344400000000001 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.209414+0000", 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013839670000000001 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.210787+0000", 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00137346 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305223+0000", 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.094436366999999993 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305261+0000", 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7796000000000002e-05 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:35:41.305288+0000", 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6973000000000001e-05 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:197709 47.0 47:06842883:::benchmark_data_smithi119_238957_object72880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.919516+0000", 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 21.195500236000001, 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.159543614, 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.360 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 197709 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919516+0000", 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919516+0000", 2024-09-21T10:36:40.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919515+0000", 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919522+0000", 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.357e-06 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919535+0000", 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3016999999999999e-05 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919541+0000", 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7969999999999997e-06 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919811+0000", 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027059399999999998 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.919924+0000", 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113093 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.920581+0000", 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00065646699999999999 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.923456+0000", 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002874616 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.076302+0000", 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.152846698 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.078864+0000", 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025616229999999999 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.079017+0000", 2024-09-21T10:36:40.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000153182 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.079060+0000", 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2790000000000002e-05 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:197718 47.0 47:05ec008f:::benchmark_data_smithi119_238957_object72889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.931384+0000", 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 21.183632515999999, 2024-09-21T10:36:40.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15167708999999999, 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 197718 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931384+0000", 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931384+0000", 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931382+0000", 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931391+0000", 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5429999999999994e-06 2024-09-21T10:36:40.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931404+0000", 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3655000000000001e-05 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931410+0000", 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1210000000000003e-06 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:40.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931459+0000", 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8318000000000001e-05 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.931545+0000", 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5847000000000007e-05 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.932176+0000", 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00063193400000000001 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934985+0000", 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028084220000000001 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.078008+0000", 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14302282999999999 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.082627+0000", 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046189910000000002 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.082941+0000", 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031438400000000002 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.083061+0000", 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000119961 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:40.372 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:36:40.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:40.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:40.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:40.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:36:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:36:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:36:40.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:40.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:40.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:40.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:40.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:40.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:41.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:41.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:41.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:41.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:36:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:41.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:41.291 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:41 smithi145 ceph-mon[89359]: pgmap v2138: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 7 active+remapped+backfill_wait, 1 active+remapped+backfilling, 108 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 5047/116946 objects degraded (4.316%); 13134/116946 objects misplaced (11.231%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:36:41.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:41.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:41 smithi119 ceph-mon[101432]: pgmap v2138: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 7 active+remapped+backfill_wait, 1 active+remapped+backfilling, 108 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 5047/116946 objects degraded (4.316%); 13134/116946 objects misplaced (11.231%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:36:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:41 smithi119 ceph-mon[105288]: pgmap v2138: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 7 active+remapped+backfill_wait, 1 active+remapped+backfilling, 108 active+clean; 2.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 5047/116946 objects degraded (4.316%); 13134/116946 objects misplaced (11.231%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:36:41.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.494 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:41.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:41.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:41.607 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:41.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:41.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:41.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:41.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:41.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:41.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:42.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:42.035 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:42.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:42.174 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:42.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:42.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:42.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:42.397 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.419 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:42.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:42.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:42.565 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:42.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:42.719 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:42.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:42.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:42.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:42.969 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:42.969 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:42 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:42.969 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:42 smithi145 ceph-mon[89359]: pgmap v2139: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 109 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 5047/112923 objects degraded (4.469%); 11760/112923 objects misplaced (10.414%); 19 MiB/s, 300 objects/s recovering 2024-09-21T10:36:42.982 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:42.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:43.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:43.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:43.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:42 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:42 smithi119 ceph-mon[101432]: pgmap v2139: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 109 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 5047/112923 objects degraded (4.469%); 11760/112923 objects misplaced (10.414%); 19 MiB/s, 300 objects/s recovering 2024-09-21T10:36:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:42 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:42 smithi119 ceph-mon[105288]: pgmap v2139: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 7 active+remapped+backfill_wait, 109 active+clean; 2.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 757 op/s; 5047/112923 objects degraded (4.469%); 11760/112923 objects misplaced (10.414%); 19 MiB/s, 300 objects/s recovering 2024-09-21T10:36:43.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:43.230 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:43.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:43.356 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:43.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:43.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:43.480 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:43.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:43.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:43.603 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:43.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:43.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:43.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:43.773 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:43 smithi145 ceph-mon[89359]: osdmap e1336: 8 total, 8 up, 7 in 2024-09-21T10:36:43.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:43.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:43.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:43.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:43.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:44.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:44.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:43 smithi119 ceph-mon[101432]: osdmap e1336: 8 total, 8 up, 7 in 2024-09-21T10:36:44.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:43 smithi119 ceph-mon[105288]: osdmap e1336: 8 total, 8 up, 7 in 2024-09-21T10:36:44.120 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:44.221 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:44.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:44.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:44.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.385 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:44.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:44.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:44.511 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:44.612 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:44.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:44.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:44.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:36:44] "GET /metrics HTTP/1.1" 200 36934 "" "Prometheus/2.51.0" 2024-09-21T10:36:44.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-mon[101432]: osdmap e1337: 8 total, 8 up, 7 in 2024-09-21T10:36:44.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-mon[101432]: pgmap v2142: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 109 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 4825/106800 objects degraded (4.518%); 11081/106800 objects misplaced (10.375%); 18 MiB/s, 285 objects/s recovering 2024-09-21T10:36:44.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 5047/112923 objects degraded (4.469%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:44.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-mon[105288]: osdmap e1337: 8 total, 8 up, 7 in 2024-09-21T10:36:44.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-mon[105288]: pgmap v2142: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 109 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 4825/106800 objects degraded (4.518%); 11081/106800 objects misplaced (10.375%); 18 MiB/s, 285 objects/s recovering 2024-09-21T10:36:44.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:44 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 5047/112923 objects degraded (4.469%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:44.885 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:44 smithi145 ceph-mon[89359]: osdmap e1337: 8 total, 8 up, 7 in 2024-09-21T10:36:44.885 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:44 smithi145 ceph-mon[89359]: pgmap v2142: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_wait, 109 active+clean; 2.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 706 op/s; 4825/106800 objects degraded (4.518%); 11081/106800 objects misplaced (10.375%); 18 MiB/s, 285 objects/s recovering 2024-09-21T10:36:44.885 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:44 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 5047/112923 objects degraded (4.469%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:44.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:44.932 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:44.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:45.033 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:45.065 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:45.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:45.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:45.207 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:45.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:45.332 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:216916 47.e 47:7df1dec6:::benchmark_data_smithi119_238957_object92087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:45.380319+0000", 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.011368425999999999, 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011399431, 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:45.394 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 216916 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.380319+0000", 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.380319+0000", 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:45.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.380318+0000", 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.380323+0000", 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8029999999999996e-06 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:45.396 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.380338+0000", 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5209e-05 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.380344+0000", 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9259999999999997e-06 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.382805+0000", 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002461263 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:45.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.382976+0000", 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170977 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.383104+0000", 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012778400000000001 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.386061+0000", 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029568509999999999 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.386077+0000", 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6269e-05 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:216922 47.b 47:d72466af:::benchmark_data_smithi119_238957_object92093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:45.388589+0000", 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0030984749999999998, 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031847690000000001, 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:45.399 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 216922 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388589+0000", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388589+0000", 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:45.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388588+0000", 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388592+0000", 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6609999999999999e-06 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388603+0000", 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0307e-05 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388609+0000", 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9240000000000004e-06 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388643+0000", 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4400000000000003e-05 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388729+0000", 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6174999999999994e-05 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,4,6", 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.388804+0000", 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4892000000000001e-05 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:45.391720+0000", 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029154210000000001 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 2 2024-09-21T10:36:45.403 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:36:45.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:45.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:45.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:45.597 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:45.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:45.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:45.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:45.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:45.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:45.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.043 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:46.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:46.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:36:46.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:46.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:46.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:46.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:46.450 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:46.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:46.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:46.549 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:46.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:46.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:46.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:46.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:46.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:46.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:46.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:47.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:47.034 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:46 smithi145 ceph-mon[89359]: pgmap v2143: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 4825/101382 objects degraded (4.759%); 10138/101382 objects misplaced (10.000%); 17 MiB/s, 278 objects/s recovering 2024-09-21T10:36:47.050 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:47.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:46 smithi119 ceph-mon[101432]: pgmap v2143: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 4825/101382 objects degraded (4.759%); 10138/101382 objects misplaced (10.000%); 17 MiB/s, 278 objects/s recovering 2024-09-21T10:36:47.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:46 smithi119 ceph-mon[105288]: pgmap v2143: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 2.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 781 op/s; 4825/101382 objects degraded (4.759%); 10138/101382 objects misplaced (10.000%); 17 MiB/s, 278 objects/s recovering 2024-09-21T10:36:47.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:47.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:47.289 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:47.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:47.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:47.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:47.468 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:47.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:47.613 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:47.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:47.743 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:47.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:47.871 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:47.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:47.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:47.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:36:47.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:48.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.023 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:48.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:48.106 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:48.106 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:48.124 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:48.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:48.256 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:36:48.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:48.310 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:48.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:48.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:48.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:48.597 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:48.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:48.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:48.767 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:36:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:36:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:36:48.767 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:36:48.767 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:36:48.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:48.860 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:36:48.866 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:36:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:48.871 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:48.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:48.926 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:48 smithi145 ceph-mon[89359]: pgmap v2144: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 4593/98061 objects degraded (4.684%); 9120/98061 objects misplaced (9.300%); 20 MiB/s, 326 objects/s recovering 2024-09-21T10:36:48.972 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:49.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:49.107 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:48 smithi119 ceph-mon[105288]: pgmap v2144: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 4593/98061 objects degraded (4.684%); 9120/98061 objects misplaced (9.300%); 20 MiB/s, 326 objects/s recovering 2024-09-21T10:36:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:48 smithi119 ceph-mon[101432]: pgmap v2144: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 2.0 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 786 op/s; 4593/98061 objects degraded (4.684%); 9120/98061 objects misplaced (9.300%); 20 MiB/s, 326 objects/s recovering 2024-09-21T10:36:49.147 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:49.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:49.248 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.277 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:49.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:49.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:49.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:49.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:49.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:49.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:49.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:49.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:49.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:49.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:49.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:49.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:49.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:36:50.027 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:50.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:50.172 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:50.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:50.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:50.343 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:50.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:50.359 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:50.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:50.477 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:50.501 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:36:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:36:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:36:50.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:50 smithi145 ceph-mon[89359]: from='client.46538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:36:50.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:50.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:50.501 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:50 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 4593/98061 objects degraded (4.684%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:50.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[105288]: from='client.46538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:36:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:50.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 4593/98061 objects degraded (4.684%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:50.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[101432]: from='client.46538 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:36:50.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:50.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:36:50.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:50 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 4593/98061 objects degraded (4.684%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:50.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:50.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:50.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:50.900 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:50.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:51.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:51.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:51.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:51.180 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:51.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:51.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:51.314 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:51.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:51.456 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:51.491 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2144,"stamp":"2024-09-21T10:36:47.718698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2141016670,"num_objects":32687,"num_object_clones":0,"num_object_copies":98061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9120,"num_objects_unfound":0,"num_objects_dirty":32687,"num_whiteouts":0,"num_read":41,"num_read_kb":103,"num_write":164993,"num_write_kb":4889342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19037,"num_bytes_recovered":1245336850,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155322,"ondisk_log_size":155322,"up":357,"acting":355,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":362,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10383908,"kb_used_data":6642084,"kb_used_omap":507,"kb_used_meta":3741124,"kb_avail":645832156,"statfs":{"total":671965249536,"available":661332127744,"internally_reserved":0,"allocated":6801494016,"data_stored":6761909067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":3830911519},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,9,8,4,7,1,2,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-412549120,"num_objects":-6295,"num_object_clones":0,"num_object_copies":-18885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-454,"num_objects_misplaced":-4014,"num_objects_unfound":0,"num_objects_dirty":-6295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6295,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2611,"num_bytes_recovered":171114496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003226"},"pg_stats":[{"pgid":"47.19","version":"1337'8481","reported_seq":8711,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:47.460087+0000","last_change":"2024-09-21T10:35:09.220454+0000","last_active":"2024-09-21T10:36:47.460087+0000","last_peered":"2024-09-21T10:36:47.460087+0000","last_clean":"2024-09-21T10:36:47.460087+0000","last_became_active":"2024-09-21T10:34:03.479509+0000","last_became_peered":"2024-09-21T10:34:03.479509+0000","last_unstale":"2024-09-21T10:36:47.460087+0000","last_undegraded":"2024-09-21T10:36:47.460087+0000","last_fullsized":"2024-09-21T10:36:47.460087+0000","mapping_epoch":1293,"log_start":"1291'2980","ondisk_log_start":"1291'2980","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"1315'5650","last_scrub_stamp":"2024-09-21T10:35:09.220363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:09.220363+0000","objects_scrubbed":3738,"log_size":5501,"log_dups_size":0,"ondisk_log_size":5501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:01.520610+0000","scrub_duration":1.249326693,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58327040,"num_objects":890,"num_object_clones":0,"num_object_copies":2670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.18","version":"1337'8636","reported_seq":8885,"reported_epoch":1337,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:36:45.145085+0000","last_change":"2024-09-21T10:36:23.245098+0000","last_active":"2024-09-21T10:36:45.145085+0000","last_peered":"2024-09-21T10:36:45.145085+0000","last_clean":"2024-09-21T10:36:20.704439+0000","last_became_active":"2024-09-21T10:36:23.198970+0000","last_became_peered":"2024-09-21T10:36:23.198970+0000","last_unstale":"2024-09-21T10:36:45.145085+0000","last_undegraded":"2024-09-21T10:36:45.145085+0000","last_fullsized":"2024-09-21T10:36:45.145085+0000","mapping_epoch":1329,"log_start":"1304'4112","ondisk_log_start":"1304'4112","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"1313'5712","last_scrub_stamp":"2024-09-21T10:35:04.507669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:04.507669+0000","objects_scrubbed":3932,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1.3460112470000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1040,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2944,"num_write_kb":1664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":1040}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.17","version":"1337'8571","reported_seq":8711,"reported_epoch":1337,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:36:45.168293+0000","last_change":"2024-09-21T10:36:23.248873+0000","last_active":"2024-09-21T10:36:45.168293+0000","last_peered":"2024-09-21T10:36:45.168293+0000","last_clean":"2024-09-21T10:36:20.795223+0000","last_became_active":"2024-09-21T10:36:23.190491+0000","last_became_peered":"2024-09-21T10:36:23.190491+0000","last_unstale":"2024-09-21T10:36:45.168293+0000","last_undegraded":"2024-09-21T10:36:45.168293+0000","last_fullsized":"2024-09-21T10:36:45.168293+0000","mapping_epoch":1329,"log_start":"1297'4004","ondisk_log_start":"1297'4004","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65208320,"num_objects":995,"num_object_clones":0,"num_object_copies":2985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":995,"num_objects_unfound":0,"num_objects_dirty":995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6791,"num_write_kb":249152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,4],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":995}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.16","version":"1337'8549","reported_seq":10237,"reported_epoch":1337,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-21T10:36:45.164297+0000","last_change":"2024-09-21T10:36:23.371465+0000","last_active":"2024-09-21T10:36:45.164297+0000","last_peered":"2024-09-21T10:36:45.164297+0000","last_clean":"2024-09-21T10:35:13.205767+0000","last_became_active":"2024-09-21T10:36:23.118097+0000","last_became_peered":"2024-09-21T10:36:23.118097+0000","last_unstale":"2024-09-21T10:36:45.164297+0000","last_undegraded":"2024-09-21T10:36:45.164297+0000","last_fullsized":"2024-09-21T10:36:22.841544+0000","mapping_epoch":1329,"log_start":"1297'4046","ondisk_log_start":"1297'4046","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"1309'5526","last_scrub_stamp":"2024-09-21T10:35:00.654596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:00.654596+0000","objects_scrubbed":3799,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176704,"num_objects":964,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":737,"num_objects_unfound":0,"num_objects_dirty":964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3023,"num_write_kb":6016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":49348608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1,2","objects":737},{"shards":"1,6","objects":227}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.15","version":"1337'8591","reported_seq":16877,"reported_epoch":1337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-21T10:36:47.214775+0000","last_change":"2024-09-21T10:36:23.341048+0000","last_active":"2024-09-21T10:36:47.214775+0000","last_peered":"2024-09-21T10:36:47.214775+0000","last_clean":"2024-09-21T10:35:11.904906+0000","last_became_active":"2024-09-21T10:36:23.312543+0000","last_became_peered":"2024-09-21T10:36:23.312543+0000","last_unstale":"2024-09-21T10:36:47.214775+0000","last_undegraded":"2024-09-21T10:36:23.127845+0000","last_fullsized":"2024-09-21T10:36:22.836215+0000","mapping_epoch":1329,"log_start":"0'0","ondisk_log_start":"0'0","created":1293,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":5,"last_scrub":"1307'4541","last_scrub_stamp":"2024-09-21T10:34:31.058760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:34:31.058760+0000","objects_scrubbed":2710,"log_size":8591,"log_dups_size":0,"ondisk_log_size":8591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":881,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":960,"num_object_clones":0,"num_object_copies":2880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2705,"num_bytes_recovered":177274880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[0,7],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1,3","objects":957},{"shards":"0,7","objects":-118}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.14","version":"1337'8551","reported_seq":19344,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.314485+0000","last_change":"2024-09-21T10:36:36.051439+0000","last_active":"2024-09-21T10:36:45.314485+0000","last_peered":"2024-09-21T10:36:45.314485+0000","last_clean":"2024-09-21T10:36:45.314485+0000","last_became_active":"2024-09-21T10:36:36.031883+0000","last_became_peered":"2024-09-21T10:36:36.031883+0000","last_unstale":"2024-09-21T10:36:45.314485+0000","last_undegraded":"2024-09-21T10:36:45.314485+0000","last_fullsized":"2024-09-21T10:36:45.314485+0000","mapping_epoch":1332,"log_start":"1294'3526","ondisk_log_start":"1294'3526","created":1293,"last_epoch_clean":1333,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-21T10:37:42.719549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71139328,"num_objects":1086,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6802,"num_write_kb":252384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5319,"num_bytes_recovered":348520448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.13","version":"1336'8464","reported_seq":13266,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.907239+0000","last_change":"2024-09-21T10:36:00.667252+0000","last_active":"2024-09-21T10:36:42.907239+0000","last_peered":"2024-09-21T10:36:42.907239+0000","last_clean":"2024-09-21T10:36:42.907239+0000","last_became_active":"2024-09-21T10:35:30.069934+0000","last_became_peered":"2024-09-21T10:35:30.069934+0000","last_unstale":"2024-09-21T10:36:42.907239+0000","last_undegraded":"2024-09-21T10:36:42.907239+0000","last_fullsized":"2024-09-21T10:36:42.907239+0000","mapping_epoch":1322,"log_start":"1290'1652","ondisk_log_start":"1290'1652","created":1293,"last_epoch_clean":1323,"parent":"0.0","parent_split_bits":5,"last_scrub":"1325'7325","last_scrub_stamp":"2024-09-21T10:36:00.667182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:36:00.667182+0000","objects_scrubbed":2182,"log_size":6812,"log_dups_size":0,"ondisk_log_size":6812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:41:58.689806+0000","scrub_duration":1.0274110379999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67371008,"num_objects":1028,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1154,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.12","version":"1337'8447","reported_seq":13117,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.607816+0000","last_change":"2024-09-21T10:35:02.897838+0000","last_active":"2024-09-21T10:36:45.607816+0000","last_peered":"2024-09-21T10:36:45.607816+0000","last_clean":"2024-09-21T10:36:45.607816+0000","last_became_active":"2024-09-21T10:35:02.894960+0000","last_became_peered":"2024-09-21T10:35:02.894960+0000","last_unstale":"2024-09-21T10:36:45.607816+0000","last_undegraded":"2024-09-21T10:36:45.607816+0000","last_fullsized":"2024-09-21T10:36:45.607816+0000","mapping_epoch":1311,"log_start":"1288'141","ondisk_log_start":"1288'141","created":1293,"last_epoch_clean":1312,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":8306,"log_dups_size":0,"ondisk_log_size":8306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64290816,"num_objects":981,"num_object_clones":0,"num_object_copies":2943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6739,"num_write_kb":247040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2264,"num_bytes_recovered":148373504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"47.11","version":"1336'8372","reported_seq":15236,"reported_epoch":1336,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:36:42.979205+0000","last_change":"2024-09-21T10:36:23.141115+0000","last_active":"2024-09-21T10:36:42.979205+0000","last_peered":"2024-09-21T10:36:42.979205+0000","last_clean":"2024-09-21T10:36:20.813628+0000","last_became_active":"2024-09-21T10:36:23.049235+0000","last_became_peered":"2024-09-21T10:36:23.049235+0000","last_unstale":"2024-09-21T10:36:42.979205+0000","last_undegraded":"2024-09-21T10:36:42.979205+0000","last_fullsized":"2024-09-21T10:36:42.979205+0000","mapping_epoch":1329,"log_start":"1290'557","ondisk_log_start":"1290'557","created":1293,"last_epoch_clean":1310,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":7815,"log_dups_size":0,"ondisk_log_size":7815,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71073792,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1080,"num_objects_unfound":0,"num_objects_dirty":1085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6623,"num_write_kb":246624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2253,"num_bytes_recovered":147652608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[{"shards":"4,5,7","objects":1085}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"47.10","version":"1337'8619","reported_seq":13096,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.135508+0000","last_change":"2024-09-21T10:36:04.369020+0000","last_active":"2024-09-21T10:36:45.135508+0000","last_peered":"2024-09-21T10:36:45.135508+0000","last_clean":"2024-09-21T10:36:45.135508+0000","last_became_active":"2024-09-21T10:34:28.554866+0000","last_became_peered":"2024-09-21T10:34:28.554866+0000","last_unstale":"2024-09-21T10:36:45.135508+0000","last_undegraded":"2024-09-21T10:36:45.135508+0000","last_fullsized":"2024-09-21T10:36:45.135508+0000","mapping_epoch":1306,"log_start":"1302'4107","ondisk_log_start":"1302'4107","created":1293,"last_epoch_clean":1307,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4512,"log_dups_size":0,"ondisk_log_size":4512,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T23:04:00.742693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66093056,"num_objects":1009,"num_object_clones":0,"num_object_copies":3027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":251872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2176,"num_bytes_recovered":142606336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.f","version":"1337'13547","reported_seq":13607,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:47.482714+0000","last_change":"2024-09-21T10:33:28.930280+0000","last_active":"2024-09-21T10:36:47.482714+0000","last_peered":"2024-09-21T10:36:47.482714+0000","last_clean":"2024-09-21T10:36:47.482714+0000","last_became_active":"2024-09-21T10:33:28.928885+0000","last_became_peered":"2024-09-21T10:33:28.928885+0000","last_unstale":"2024-09-21T10:36:47.482714+0000","last_undegraded":"2024-09-21T10:36:47.482714+0000","last_fullsized":"2024-09-21T10:36:47.482714+0000","mapping_epoch":1285,"log_start":"1318'8191","ondisk_log_start":"1318'8191","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5356,"log_dups_size":0,"ondisk_log_size":5356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:49:20.937916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126681088,"num_objects":1933,"num_object_clones":0,"num_object_copies":5799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13547,"num_write_kb":495360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.e","version":"1336'13516","reported_seq":13575,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.044858+0000","last_change":"2024-09-21T10:33:28.932881+0000","last_active":"2024-09-21T10:36:43.044858+0000","last_peered":"2024-09-21T10:36:43.044858+0000","last_clean":"2024-09-21T10:36:43.044858+0000","last_became_active":"2024-09-21T10:33:28.932543+0000","last_became_peered":"2024-09-21T10:33:28.932543+0000","last_unstale":"2024-09-21T10:36:43.044858+0000","last_undegraded":"2024-09-21T10:36:43.044858+0000","last_fullsized":"2024-09-21T10:36:43.044858+0000","mapping_epoch":1285,"log_start":"1310'7628","ondisk_log_start":"1310'7628","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:30:10.930926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138936320,"num_objects":2120,"num_object_clones":0,"num_object_copies":6360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13516,"num_write_kb":500352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.d","version":"1337'13557","reported_seq":15516,"reported_epoch":1337,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T10:36:47.528884+0000","last_change":"2024-09-21T10:36:41.942447+0000","last_active":"2024-09-21T10:36:47.528884+0000","last_peered":"2024-09-21T10:36:47.528884+0000","last_clean":"2024-09-21T10:36:20.775904+0000","last_became_active":"2024-09-21T10:36:23.319001+0000","last_became_peered":"2024-09-21T10:36:23.319001+0000","last_unstale":"2024-09-21T10:36:47.528884+0000","last_undegraded":"2024-09-21T10:36:47.528884+0000","last_fullsized":"2024-09-21T10:36:47.528884+0000","mapping_epoch":1329,"log_start":"1320'8219","ondisk_log_start":"1320'8219","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5338,"log_dups_size":0,"ondisk_log_size":5338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:17:26.361367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123469848,"num_objects":1885,"num_object_clones":0,"num_object_copies":5655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1025,"num_objects_unfound":0,"num_objects_dirty":1885,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13557,"num_write_kb":494081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":943,"num_bytes_recovered":61800448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,4],"avail_no_missing":["6","2","4"],"object_location_counts":[{"shards":"2,4,6","objects":1885}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.c","version":"1337'13411","reported_seq":13537,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.150312+0000","last_change":"2024-09-21T10:33:28.929528+0000","last_active":"2024-09-21T10:36:45.150312+0000","last_peered":"2024-09-21T10:36:45.150312+0000","last_clean":"2024-09-21T10:36:45.150312+0000","last_became_active":"2024-09-21T10:33:28.929202+0000","last_became_peered":"2024-09-21T10:33:28.929202+0000","last_unstale":"2024-09-21T10:36:45.150312+0000","last_undegraded":"2024-09-21T10:36:45.150312+0000","last_fullsized":"2024-09-21T10:36:45.150312+0000","mapping_epoch":1285,"log_start":"1321'8869","ondisk_log_start":"1321'8869","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4542,"log_dups_size":0,"ondisk_log_size":4542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:15:33.256338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13411,"num_write_kb":496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.b","version":"1336'13460","reported_seq":13611,"reported_epoch":1336,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:36:42.887347+0000","last_change":"2024-09-21T10:36:23.336170+0000","last_active":"2024-09-21T10:36:42.887347+0000","last_peered":"2024-09-21T10:36:42.887347+0000","last_clean":"2024-09-21T10:36:20.840395+0000","last_became_active":"2024-09-21T10:36:23.047302+0000","last_became_peered":"2024-09-21T10:36:23.047302+0000","last_unstale":"2024-09-21T10:36:42.887347+0000","last_undegraded":"2024-09-21T10:36:42.887347+0000","last_fullsized":"2024-09-21T10:36:42.887347+0000","mapping_epoch":1329,"log_start":"1310'7572","ondisk_log_start":"1310'7572","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":5888,"log_dups_size":0,"ondisk_log_size":5888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:37:25.915801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2125,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13460,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2555904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":2150}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"47.a","version":"1337'13433","reported_seq":13559,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.177879+0000","last_change":"2024-09-21T10:33:29.379333+0000","last_active":"2024-09-21T10:36:45.177879+0000","last_peered":"2024-09-21T10:36:45.177879+0000","last_clean":"2024-09-21T10:36:45.177879+0000","last_became_active":"2024-09-21T10:33:29.379043+0000","last_became_peered":"2024-09-21T10:33:29.379043+0000","last_unstale":"2024-09-21T10:36:45.177879+0000","last_undegraded":"2024-09-21T10:36:45.177879+0000","last_fullsized":"2024-09-21T10:36:45.177879+0000","mapping_epoch":1285,"log_start":"1321'8944","ondisk_log_start":"1321'8944","created":1285,"last_epoch_clean":1286,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4489,"log_dups_size":0,"ondisk_log_size":4489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:39:21.731457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137166848,"num_objects":2093,"num_object_clones":0,"num_object_copies":6279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13433,"num_write_kb":496832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.9","version":"1337'8690","reported_seq":8928,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:47.503038+0000","last_change":"2024-09-21T10:35:08.354844+0000","last_active":"2024-09-21T10:36:47.503038+0000","last_peered":"2024-09-21T10:36:47.503038+0000","last_clean":"2024-09-21T10:36:47.503038+0000","last_became_active":"2024-09-21T10:34:03.464143+0000","last_became_peered":"2024-09-21T10:34:03.464143+0000","last_unstale":"2024-09-21T10:36:47.503038+0000","last_undegraded":"2024-09-21T10:36:47.503038+0000","last_fullsized":"2024-09-21T10:36:47.503038+0000","mapping_epoch":1285,"log_start":"1291'3356","ondisk_log_start":"1291'3356","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1315'5798","last_scrub_stamp":"2024-09-21T10:35:08.354760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:08.354760+0000","objects_scrubbed":3947,"log_size":5334,"log_dups_size":0,"ondisk_log_size":5334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:53.769179+0000","scrub_duration":1.3559361990000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68091904,"num_objects":1039,"num_object_clones":0,"num_object_copies":3117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"47.8","version":"1337'8738","reported_seq":8988,"reported_epoch":1337,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:36:45.143016+0000","last_change":"2024-09-21T10:36:23.245216+0000","last_active":"2024-09-21T10:36:45.143016+0000","last_peered":"2024-09-21T10:36:45.143016+0000","last_clean":"2024-09-21T10:36:20.799210+0000","last_became_active":"2024-09-21T10:36:23.196184+0000","last_became_peered":"2024-09-21T10:36:23.196184+0000","last_unstale":"2024-09-21T10:36:45.143016+0000","last_undegraded":"2024-09-21T10:36:45.143016+0000","last_fullsized":"2024-09-21T10:36:45.143016+0000","mapping_epoch":1329,"log_start":"1304'4177","ondisk_log_start":"1304'4177","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'5709","last_scrub_stamp":"2024-09-21T10:35:02.687257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:02.687257+0000","objects_scrubbed":3950,"log_size":4561,"log_dups_size":0,"ondisk_log_size":4561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.47758299700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69468160,"num_objects":1060,"num_object_clones":0,"num_object_copies":3180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1060,"num_objects_unfound":0,"num_objects_dirty":1060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3030,"num_write_kb":4480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":1060}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.7","version":"1337'8638","reported_seq":8774,"reported_epoch":1337,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:36:45.146680+0000","last_change":"2024-09-21T10:36:23.248735+0000","last_active":"2024-09-21T10:36:45.146680+0000","last_peered":"2024-09-21T10:36:45.146680+0000","last_clean":"2024-09-21T10:36:20.814956+0000","last_became_active":"2024-09-21T10:36:23.197172+0000","last_became_peered":"2024-09-21T10:36:23.197172+0000","last_unstale":"2024-09-21T10:36:45.146680+0000","last_undegraded":"2024-09-21T10:36:45.146680+0000","last_fullsized":"2024-09-21T10:36:45.146680+0000","mapping_epoch":1329,"log_start":"1300'4114","ondisk_log_start":"1300'4114","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":4524,"log_dups_size":0,"ondisk_log_size":4524,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69337088,"num_objects":1058,"num_object_clones":0,"num_object_copies":3174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1058,"num_objects_unfound":0,"num_objects_dirty":1058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6858,"num_write_kb":253312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,4],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":1058}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.6","version":"1337'8515","reported_seq":8694,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:47.158649+0000","last_change":"2024-09-21T10:35:06.025889+0000","last_active":"2024-09-21T10:36:47.158649+0000","last_peered":"2024-09-21T10:36:47.158649+0000","last_clean":"2024-09-21T10:36:47.158649+0000","last_became_active":"2024-09-21T10:34:03.425142+0000","last_became_peered":"2024-09-21T10:34:03.425142+0000","last_unstale":"2024-09-21T10:36:47.158649+0000","last_undegraded":"2024-09-21T10:36:47.158649+0000","last_fullsized":"2024-09-21T10:36:47.158649+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1315'5645","last_scrub_stamp":"2024-09-21T10:35:06.025698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:06.025698+0000","objects_scrubbed":3853,"log_size":8515,"log_dups_size":0,"ondisk_log_size":8515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:17:01.725274+0000","scrub_duration":680,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64552960,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2874,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.5","version":"1337'8724","reported_seq":8896,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.270161+0000","last_change":"2024-09-21T10:34:35.228534+0000","last_active":"2024-09-21T10:36:45.270161+0000","last_peered":"2024-09-21T10:36:45.270161+0000","last_clean":"2024-09-21T10:36:45.270161+0000","last_became_active":"2024-09-21T10:34:03.470400+0000","last_became_peered":"2024-09-21T10:34:03.470400+0000","last_unstale":"2024-09-21T10:36:45.270161+0000","last_undegraded":"2024-09-21T10:36:45.270161+0000","last_fullsized":"2024-09-21T10:36:45.270161+0000","mapping_epoch":1285,"log_start":"1294'3705","ondisk_log_start":"1294'3705","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1307'4740","last_scrub_stamp":"2024-09-21T10:34:35.228281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:34:35.228281+0000","objects_scrubbed":2944,"log_size":5019,"log_dups_size":0,"ondisk_log_size":5019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:06:08.319646+0000","scrub_duration":973,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67043328,"num_objects":1023,"num_object_clones":0,"num_object_copies":3069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4001,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.4","version":"1337'8576","reported_seq":11135,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.254674+0000","last_change":"2024-09-21T10:36:43.685096+0000","last_active":"2024-09-21T10:36:45.254674+0000","last_peered":"2024-09-21T10:36:45.254674+0000","last_clean":"2024-09-21T10:36:45.254674+0000","last_became_active":"2024-09-21T10:36:43.668971+0000","last_became_peered":"2024-09-21T10:36:43.668971+0000","last_unstale":"2024-09-21T10:36:45.254674+0000","last_undegraded":"2024-09-21T10:36:45.254674+0000","last_fullsized":"2024-09-21T10:36:45.254674+0000","mapping_epoch":1336,"log_start":"1294'3520","ondisk_log_start":"1294'3520","created":1285,"last_epoch_clean":1337,"parent":"0.0","parent_split_bits":0,"last_scrub":"1314'5648","last_scrub_stamp":"2024-09-21T10:35:04.669392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:04.669392+0000","objects_scrubbed":3935,"log_size":5056,"log_dups_size":0,"ondisk_log_size":5056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:13:13.791916+0000","scrub_duration":537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67371008,"num_objects":1028,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2931,"num_write_kb":768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1179,"num_bytes_recovered":77201408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"47.3","version":"1337'8585","reported_seq":8719,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:47.124056+0000","last_change":"2024-09-21T10:35:07.475275+0000","last_active":"2024-09-21T10:36:47.124056+0000","last_peered":"2024-09-21T10:36:47.124056+0000","last_clean":"2024-09-21T10:36:47.124056+0000","last_became_active":"2024-09-21T10:34:03.397807+0000","last_became_peered":"2024-09-21T10:34:03.397807+0000","last_unstale":"2024-09-21T10:36:47.124056+0000","last_undegraded":"2024-09-21T10:36:47.124056+0000","last_fullsized":"2024-09-21T10:36:47.124056+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.909401+0000","objects_scrubbed":0,"log_size":8585,"log_dups_size":0,"ondisk_log_size":8585,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60784640,"num_objects":927,"num_object_clones":0,"num_object_copies":2781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6825,"num_write_kb":248096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"47.2","version":"1337'8433","reported_seq":8595,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:45.164619+0000","last_change":"2024-09-21T10:36:03.011233+0000","last_active":"2024-09-21T10:36:45.164619+0000","last_peered":"2024-09-21T10:36:45.164619+0000","last_clean":"2024-09-21T10:36:45.164619+0000","last_became_active":"2024-09-21T10:34:03.441889+0000","last_became_peered":"2024-09-21T10:34:03.441889+0000","last_unstale":"2024-09-21T10:36:45.164619+0000","last_undegraded":"2024-09-21T10:36:45.164619+0000","last_fullsized":"2024-09-21T10:36:45.164619+0000","mapping_epoch":1285,"log_start":"1298'3933","ondisk_log_start":"1298'3933","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1325'7342","last_scrub_stamp":"2024-09-21T10:36:03.011114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:36:03.011114+0000","objects_scrubbed":2177,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:17:40.409847+0000","scrub_duration":663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70844416,"num_objects":1081,"num_object_clones":0,"num_object_copies":3243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1096,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"47.1","version":"1336'8511","reported_seq":8745,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.821925+0000","last_change":"2024-09-21T10:35:02.117172+0000","last_active":"2024-09-21T10:36:42.821925+0000","last_peered":"2024-09-21T10:36:42.821925+0000","last_clean":"2024-09-21T10:36:42.821925+0000","last_became_active":"2024-09-21T10:34:03.461587+0000","last_became_peered":"2024-09-21T10:34:03.461587+0000","last_unstale":"2024-09-21T10:36:42.821925+0000","last_undegraded":"2024-09-21T10:36:42.821925+0000","last_fullsized":"2024-09-21T10:36:42.821925+0000","mapping_epoch":1285,"log_start":"0'0","ondisk_log_start":"0'0","created":1285,"last_epoch_clean":1294,"parent":"0.0","parent_split_bits":0,"last_scrub":"1311'5602","last_scrub_stamp":"2024-09-21T10:35:02.117080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:33:27.909401+0000","last_clean_scrub_stamp":"2024-09-21T10:35:02.117080+0000","objects_scrubbed":3848,"log_size":8511,"log_dups_size":0,"ondisk_log_size":8511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:05.189011+0000","scrub_duration":1.1106482449999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69599232,"num_objects":1062,"num_object_clones":0,"num_object_copies":3186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"47.0","version":"1336'11371","reported_seq":14345,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.956095+0000","last_change":"2024-09-21T10:36:38.042331+0000","last_active":"2024-09-21T10:36:42.956095+0000","last_peered":"2024-09-21T10:36:42.956095+0000","last_clean":"2024-09-21T10:36:42.956095+0000","last_became_active":"2024-09-21T10:36:37.991637+0000","last_became_peered":"2024-09-21T10:36:37.991637+0000","last_unstale":"2024-09-21T10:36:42.956095+0000","last_undegraded":"2024-09-21T10:36:42.956095+0000","last_fullsized":"2024-09-21T10:36:42.956095+0000","mapping_epoch":1334,"log_start":"1307'4469","ondisk_log_start":"1307'4469","created":1285,"last_epoch_clean":1335,"parent":"0.0","parent_split_bits":0,"last_scrub":"1320'8845","last_scrub_stamp":"2024-09-21T10:35:17.272330+0000","last_deep_scrub":"1320'8845","last_deep_scrub_stamp":"2024-09-21T10:35:17.272330+0000","last_clean_scrub_stamp":"2024-09-21T10:35:17.272330+0000","objects_scrubbed":3767,"log_size":6902,"log_dups_size":0,"ondisk_log_size":6902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":14,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:59:45.504551+0000","scrub_duration":13.31810347,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2687,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1357,"num_bytes_recovered":88932352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":92,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987354+0000","last_change":"2024-09-21T10:30:09.420642+0000","last_active":"2024-09-21T10:36:37.987354+0000","last_peered":"2024-09-21T10:36:37.987354+0000","last_clean":"2024-09-21T10:36:37.987354+0000","last_became_active":"2024-09-21T10:30:09.407144+0000","last_became_peered":"2024-09-21T10:30:09.407144+0000","last_unstale":"2024-09-21T10:36:37.987354+0000","last_undegraded":"2024-09-21T10:36:37.987354+0000","last_fullsized":"2024-09-21T10:36:37.987354+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:04:49.017939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1231,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.664564+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:36:42.664564+0000","last_peered":"2024-09-21T10:36:42.664564+0000","last_clean":"2024-09-21T10:36:42.664564+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:36:42.664564+0000","last_undegraded":"2024-09-21T10:36:42.664564+0000","last_fullsized":"2024-09-21T10:36:42.664564+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":1993,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666533+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:36:43.666533+0000","last_peered":"2024-09-21T10:36:43.666533+0000","last_clean":"2024-09-21T10:36:43.666533+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:36:43.666533+0000","last_undegraded":"2024-09-21T10:36:43.666533+0000","last_fullsized":"2024-09-21T10:36:43.666533+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2012,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666146+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:36:43.666146+0000","last_peered":"2024-09-21T10:36:43.666146+0000","last_clean":"2024-09-21T10:36:43.666146+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:36:43.666146+0000","last_undegraded":"2024-09-21T10:36:43.666146+0000","last_fullsized":"2024-09-21T10:36:43.666146+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":104,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987136+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:36:37.987136+0000","last_peered":"2024-09-21T10:36:37.987136+0000","last_clean":"2024-09-21T10:36:37.987136+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:36:37.987136+0000","last_undegraded":"2024-09-21T10:36:37.987136+0000","last_fullsized":"2024-09-21T10:36:37.987136+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1196,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665919+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:36:43.665919+0000","last_peered":"2024-09-21T10:36:43.665919+0000","last_clean":"2024-09-21T10:36:43.665919+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:36:43.665919+0000","last_undegraded":"2024-09-21T10:36:43.665919+0000","last_fullsized":"2024-09-21T10:36:43.665919+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":104,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987522+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:36:37.987522+0000","last_peered":"2024-09-21T10:36:37.987522+0000","last_clean":"2024-09-21T10:36:37.987522+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:36:37.987522+0000","last_undegraded":"2024-09-21T10:36:37.987522+0000","last_fullsized":"2024-09-21T10:36:37.987522+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1283,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.664904+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:36:43.664904+0000","last_peered":"2024-09-21T10:36:43.664904+0000","last_clean":"2024-09-21T10:36:43.664904+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:36:43.664904+0000","last_undegraded":"2024-09-21T10:36:43.664904+0000","last_fullsized":"2024-09-21T10:36:43.664904+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":20,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.985888+0000","last_change":"2024-09-21T10:36:21.860297+0000","last_active":"2024-09-21T10:36:37.985888+0000","last_peered":"2024-09-21T10:36:37.985888+0000","last_clean":"2024-09-21T10:36:37.985888+0000","last_became_active":"2024-09-21T10:36:21.859377+0000","last_became_peered":"2024-09-21T10:36:21.859377+0000","last_unstale":"2024-09-21T10:36:37.985888+0000","last_undegraded":"2024-09-21T10:36:37.985888+0000","last_fullsized":"2024-09-21T10:36:37.985888+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:33:28.856920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1944,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667868+0000","last_change":"2024-09-21T10:36:21.849410+0000","last_active":"2024-09-21T10:36:43.667868+0000","last_peered":"2024-09-21T10:36:43.667868+0000","last_clean":"2024-09-21T10:36:43.667868+0000","last_became_active":"2024-09-21T10:36:21.849098+0000","last_became_peered":"2024-09-21T10:36:21.849098+0000","last_unstale":"2024-09-21T10:36:43.667868+0000","last_undegraded":"2024-09-21T10:36:43.667868+0000","last_fullsized":"2024-09-21T10:36:43.667868+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:46:33.009279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":56,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666302+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:36:43.666302+0000","last_peered":"2024-09-21T10:36:43.666302+0000","last_clean":"2024-09-21T10:36:43.666302+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:36:43.666302+0000","last_undegraded":"2024-09-21T10:36:43.666302+0000","last_fullsized":"2024-09-21T10:36:43.666302+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2002,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665681+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:36:43.665681+0000","last_peered":"2024-09-21T10:36:43.665681+0000","last_clean":"2024-09-21T10:36:43.665681+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:36:43.665681+0000","last_undegraded":"2024-09-21T10:36:43.665681+0000","last_fullsized":"2024-09-21T10:36:43.665681+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2100,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665061+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:36:43.665061+0000","last_peered":"2024-09-21T10:36:43.665061+0000","last_clean":"2024-09-21T10:36:43.665061+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:36:43.665061+0000","last_undegraded":"2024-09-21T10:36:43.665061+0000","last_fullsized":"2024-09-21T10:36:43.665061+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2109,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668498+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:36:43.668498+0000","last_peered":"2024-09-21T10:36:43.668498+0000","last_clean":"2024-09-21T10:36:43.668498+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:36:43.668498+0000","last_undegraded":"2024-09-21T10:36:43.668498+0000","last_fullsized":"2024-09-21T10:36:43.668498+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2020,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665801+0000","last_change":"2024-09-21T10:28:56.819014+0000","last_active":"2024-09-21T10:36:43.665801+0000","last_peered":"2024-09-21T10:36:43.665801+0000","last_clean":"2024-09-21T10:36:43.665801+0000","last_became_active":"2024-09-21T10:28:56.818640+0000","last_became_peered":"2024-09-21T10:28:56.818640+0000","last_unstale":"2024-09-21T10:36:43.665801+0000","last_undegraded":"2024-09-21T10:36:43.665801+0000","last_fullsized":"2024-09-21T10:36:43.665801+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:49.759102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1295,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.669292+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:36:42.669292+0000","last_peered":"2024-09-21T10:36:42.669292+0000","last_clean":"2024-09-21T10:36:42.669292+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:36:42.669292+0000","last_undegraded":"2024-09-21T10:36:42.669292+0000","last_fullsized":"2024-09-21T10:36:42.669292+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2006,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665989+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:36:43.665989+0000","last_peered":"2024-09-21T10:36:43.665989+0000","last_clean":"2024-09-21T10:36:43.665989+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:36:43.665989+0000","last_undegraded":"2024-09-21T10:36:43.665989+0000","last_fullsized":"2024-09-21T10:36:43.665989+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2119,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668543+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:36:43.668543+0000","last_peered":"2024-09-21T10:36:43.668543+0000","last_clean":"2024-09-21T10:36:43.668543+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:36:43.668543+0000","last_undegraded":"2024-09-21T10:36:43.668543+0000","last_fullsized":"2024-09-21T10:36:43.668543+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":128,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.687063+0000","last_change":"2024-09-21T10:36:21.852315+0000","last_active":"2024-09-21T10:36:43.687063+0000","last_peered":"2024-09-21T10:36:43.687063+0000","last_clean":"2024-09-21T10:36:43.687063+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:36:43.687063+0000","last_undegraded":"2024-09-21T10:36:43.687063+0000","last_fullsized":"2024-09-21T10:36:43.687063+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1989,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666251+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:36:43.666251+0000","last_peered":"2024-09-21T10:36:43.666251+0000","last_clean":"2024-09-21T10:36:43.666251+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:36:43.666251+0000","last_undegraded":"2024-09-21T10:36:43.666251+0000","last_fullsized":"2024-09-21T10:36:43.666251+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":86,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987157+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:36:37.987157+0000","last_peered":"2024-09-21T10:36:37.987157+0000","last_clean":"2024-09-21T10:36:37.987157+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:36:37.987157+0000","last_undegraded":"2024-09-21T10:36:37.987157+0000","last_fullsized":"2024-09-21T10:36:37.987157+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1650,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.669891+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:36:43.669891+0000","last_peered":"2024-09-21T10:36:43.669891+0000","last_clean":"2024-09-21T10:36:43.669891+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:36:43.669891+0000","last_undegraded":"2024-09-21T10:36:43.669891+0000","last_fullsized":"2024-09-21T10:36:43.669891+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2092,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667183+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:36:43.667183+0000","last_peered":"2024-09-21T10:36:43.667183+0000","last_clean":"2024-09-21T10:36:43.667183+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:36:43.667183+0000","last_undegraded":"2024-09-21T10:36:43.667183+0000","last_fullsized":"2024-09-21T10:36:43.667183+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":351,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986034+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:36:37.986034+0000","last_peered":"2024-09-21T10:36:37.986034+0000","last_clean":"2024-09-21T10:36:37.986034+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:36:37.986034+0000","last_undegraded":"2024-09-21T10:36:37.986034+0000","last_fullsized":"2024-09-21T10:36:37.986034+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1196,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.664897+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:36:42.664897+0000","last_peered":"2024-09-21T10:36:42.664897+0000","last_clean":"2024-09-21T10:36:42.664897+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:36:42.664897+0000","last_undegraded":"2024-09-21T10:36:42.664897+0000","last_fullsized":"2024-09-21T10:36:42.664897+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":249,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.664957+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:36:43.664957+0000","last_peered":"2024-09-21T10:36:43.664957+0000","last_clean":"2024-09-21T10:36:43.664957+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:36:43.664957+0000","last_undegraded":"2024-09-21T10:36:43.664957+0000","last_fullsized":"2024-09-21T10:36:43.664957+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1229,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986842+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:36:37.986842+0000","last_peered":"2024-09-21T10:36:37.986842+0000","last_clean":"2024-09-21T10:36:37.986842+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:36:37.986842+0000","last_undegraded":"2024-09-21T10:36:37.986842+0000","last_fullsized":"2024-09-21T10:36:37.986842+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1291,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.669144+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:36:42.669144+0000","last_peered":"2024-09-21T10:36:42.669144+0000","last_clean":"2024-09-21T10:36:42.669144+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:36:42.669144+0000","last_undegraded":"2024-09-21T10:36:42.669144+0000","last_fullsized":"2024-09-21T10:36:42.669144+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":239,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.676307+0000","last_change":"2024-09-21T10:36:21.864693+0000","last_active":"2024-09-21T10:36:43.676307+0000","last_peered":"2024-09-21T10:36:43.676307+0000","last_clean":"2024-09-21T10:36:43.676307+0000","last_became_active":"2024-09-21T10:36:21.854766+0000","last_became_peered":"2024-09-21T10:36:21.854766+0000","last_unstale":"2024-09-21T10:36:43.676307+0000","last_undegraded":"2024-09-21T10:36:43.676307+0000","last_fullsized":"2024-09-21T10:36:43.676307+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:24.603630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":30,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.688586+0000","last_change":"2024-09-21T10:36:21.851075+0000","last_active":"2024-09-21T10:36:43.688586+0000","last_peered":"2024-09-21T10:36:43.688586+0000","last_clean":"2024-09-21T10:36:43.688586+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:36:43.688586+0000","last_undegraded":"2024-09-21T10:36:43.688586+0000","last_fullsized":"2024-09-21T10:36:43.688586+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.422256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.422256+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":2004,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665891+0000","last_change":"2024-09-21T10:28:56.817623+0000","last_active":"2024-09-21T10:36:43.665891+0000","last_peered":"2024-09-21T10:36:43.665891+0000","last_clean":"2024-09-21T10:36:43.665891+0000","last_became_active":"2024-09-21T10:28:56.817337+0000","last_became_peered":"2024-09-21T10:28:56.817337+0000","last_unstale":"2024-09-21T10:36:43.665891+0000","last_undegraded":"2024-09-21T10:36:43.665891+0000","last_fullsized":"2024-09-21T10:36:43.665891+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:31.718364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2109,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.676409+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:36:43.676409+0000","last_peered":"2024-09-21T10:36:43.676409+0000","last_clean":"2024-09-21T10:36:43.676409+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:36:43.676409+0000","last_undegraded":"2024-09-21T10:36:43.676409+0000","last_fullsized":"2024-09-21T10:36:43.676409+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":1988,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666616+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:36:43.666616+0000","last_peered":"2024-09-21T10:36:43.666616+0000","last_clean":"2024-09-21T10:36:43.666616+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:36:43.666616+0000","last_undegraded":"2024-09-21T10:36:43.666616+0000","last_fullsized":"2024-09-21T10:36:43.666616+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":462,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.665501+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:36:42.665501+0000","last_peered":"2024-09-21T10:36:42.665501+0000","last_clean":"2024-09-21T10:36:42.665501+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:36:42.665501+0000","last_undegraded":"2024-09-21T10:36:42.665501+0000","last_fullsized":"2024-09-21T10:36:42.665501+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2111,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668032+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:36:43.668032+0000","last_peered":"2024-09-21T10:36:43.668032+0000","last_clean":"2024-09-21T10:36:43.668032+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:36:43.668032+0000","last_undegraded":"2024-09-21T10:36:43.668032+0000","last_fullsized":"2024-09-21T10:36:43.668032+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1343,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986190+0000","last_change":"2024-09-21T10:36:21.854217+0000","last_active":"2024-09-21T10:36:37.986190+0000","last_peered":"2024-09-21T10:36:37.986190+0000","last_clean":"2024-09-21T10:36:37.986190+0000","last_became_active":"2024-09-21T10:36:21.853490+0000","last_became_peered":"2024-09-21T10:36:21.853490+0000","last_unstale":"2024-09-21T10:36:37.986190+0000","last_undegraded":"2024-09-21T10:36:37.986190+0000","last_fullsized":"2024-09-21T10:36:37.986190+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:17.399224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2097,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668041+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:36:43.668041+0000","last_peered":"2024-09-21T10:36:43.668041+0000","last_clean":"2024-09-21T10:36:43.668041+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:36:43.668041+0000","last_undegraded":"2024-09-21T10:36:43.668041+0000","last_fullsized":"2024-09-21T10:36:43.668041+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":462,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.669017+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:36:42.669017+0000","last_peered":"2024-09-21T10:36:42.669017+0000","last_clean":"2024-09-21T10:36:42.669017+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:36:42.669017+0000","last_undegraded":"2024-09-21T10:36:42.669017+0000","last_fullsized":"2024-09-21T10:36:42.669017+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2102,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667708+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:36:43.667708+0000","last_peered":"2024-09-21T10:36:43.667708+0000","last_clean":"2024-09-21T10:36:43.667708+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:36:43.667708+0000","last_undegraded":"2024-09-21T10:36:43.667708+0000","last_fullsized":"2024-09-21T10:36:43.667708+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2097,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668176+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:36:43.668176+0000","last_peered":"2024-09-21T10:36:43.668176+0000","last_clean":"2024-09-21T10:36:43.668176+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:36:43.668176+0000","last_undegraded":"2024-09-21T10:36:43.668176+0000","last_fullsized":"2024-09-21T10:36:43.668176+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":86,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987047+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:36:37.987047+0000","last_peered":"2024-09-21T10:36:37.987047+0000","last_clean":"2024-09-21T10:36:37.987047+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:36:37.987047+0000","last_undegraded":"2024-09-21T10:36:37.987047+0000","last_fullsized":"2024-09-21T10:36:37.987047+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":96,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986912+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:36:37.986912+0000","last_peered":"2024-09-21T10:36:37.986912+0000","last_clean":"2024-09-21T10:36:37.986912+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:36:37.986912+0000","last_undegraded":"2024-09-21T10:36:37.986912+0000","last_fullsized":"2024-09-21T10:36:37.986912+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2094,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668078+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:36:43.668078+0000","last_peered":"2024-09-21T10:36:43.668078+0000","last_clean":"2024-09-21T10:36:43.668078+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:36:43.668078+0000","last_undegraded":"2024-09-21T10:36:43.668078+0000","last_fullsized":"2024-09-21T10:36:43.668078+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1214,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667658+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:36:43.667658+0000","last_peered":"2024-09-21T10:36:43.667658+0000","last_clean":"2024-09-21T10:36:43.667658+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:36:43.667658+0000","last_undegraded":"2024-09-21T10:36:43.667658+0000","last_fullsized":"2024-09-21T10:36:43.667658+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":90,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986861+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:36:37.986861+0000","last_peered":"2024-09-21T10:36:37.986861+0000","last_clean":"2024-09-21T10:36:37.986861+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:36:37.986861+0000","last_undegraded":"2024-09-21T10:36:37.986861+0000","last_fullsized":"2024-09-21T10:36:37.986861+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":409,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667242+0000","last_change":"2024-09-21T10:36:21.852991+0000","last_active":"2024-09-21T10:36:43.667242+0000","last_peered":"2024-09-21T10:36:43.667242+0000","last_clean":"2024-09-21T10:36:43.667242+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:36:43.667242+0000","last_undegraded":"2024-09-21T10:36:43.667242+0000","last_fullsized":"2024-09-21T10:36:43.667242+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.562623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.562623+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2105,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.669928+0000","last_change":"2024-09-21T10:30:09.415060+0000","last_active":"2024-09-21T10:36:43.669928+0000","last_peered":"2024-09-21T10:36:43.669928+0000","last_clean":"2024-09-21T10:36:43.669928+0000","last_became_active":"2024-09-21T10:30:09.414412+0000","last_became_peered":"2024-09-21T10:30:09.414412+0000","last_unstale":"2024-09-21T10:36:43.669928+0000","last_undegraded":"2024-09-21T10:36:43.669928+0000","last_fullsized":"2024-09-21T10:36:43.669928+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:25:36.187303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2095,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665289+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:36:43.665289+0000","last_peered":"2024-09-21T10:36:43.665289+0000","last_clean":"2024-09-21T10:36:43.665289+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:36:43.665289+0000","last_undegraded":"2024-09-21T10:36:43.665289+0000","last_fullsized":"2024-09-21T10:36:43.665289+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2025,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.686736+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:36:43.686736+0000","last_peered":"2024-09-21T10:36:43.686736+0000","last_clean":"2024-09-21T10:36:43.686736+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:36:43.686736+0000","last_undegraded":"2024-09-21T10:36:43.686736+0000","last_fullsized":"2024-09-21T10:36:43.686736+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":258,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665380+0000","last_change":"2024-09-21T10:36:21.858765+0000","last_active":"2024-09-21T10:36:43.665380+0000","last_peered":"2024-09-21T10:36:43.665380+0000","last_clean":"2024-09-21T10:36:43.665380+0000","last_became_active":"2024-09-21T10:36:21.858148+0000","last_became_peered":"2024-09-21T10:36:21.858148+0000","last_unstale":"2024-09-21T10:36:43.665380+0000","last_undegraded":"2024-09-21T10:36:43.665380+0000","last_fullsized":"2024-09-21T10:36:43.665380+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":877,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.676466+0000","last_change":"2024-09-21T10:28:56.817541+0000","last_active":"2024-09-21T10:36:43.676466+0000","last_peered":"2024-09-21T10:36:43.676466+0000","last_clean":"2024-09-21T10:36:43.676466+0000","last_became_active":"2024-09-21T10:28:56.817221+0000","last_became_peered":"2024-09-21T10:28:56.817221+0000","last_unstale":"2024-09-21T10:36:43.676466+0000","last_undegraded":"2024-09-21T10:36:43.676466+0000","last_fullsized":"2024-09-21T10:36:43.676466+0000","mapping_epoch":1244,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:14:51.568259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2126,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666946+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:36:43.666946+0000","last_peered":"2024-09-21T10:36:43.666946+0000","last_clean":"2024-09-21T10:36:43.666946+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:36:43.666946+0000","last_undegraded":"2024-09-21T10:36:43.666946+0000","last_fullsized":"2024-09-21T10:36:43.666946+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1323,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.985603+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:36:37.985603+0000","last_peered":"2024-09-21T10:36:37.985603+0000","last_clean":"2024-09-21T10:36:37.985603+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:36:37.985603+0000","last_undegraded":"2024-09-21T10:36:37.985603+0000","last_fullsized":"2024-09-21T10:36:37.985603+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1297,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.665475+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:36:42.665475+0000","last_peered":"2024-09-21T10:36:42.665475+0000","last_clean":"2024-09-21T10:36:42.665475+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:36:42.665475+0000","last_undegraded":"2024-09-21T10:36:42.665475+0000","last_fullsized":"2024-09-21T10:36:42.665475+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2011,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665999+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:36:43.665999+0000","last_peered":"2024-09-21T10:36:43.665999+0000","last_clean":"2024-09-21T10:36:43.665999+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:36:43.665999+0000","last_undegraded":"2024-09-21T10:36:43.665999+0000","last_fullsized":"2024-09-21T10:36:43.665999+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":90,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987507+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:36:37.987507+0000","last_peered":"2024-09-21T10:36:37.987507+0000","last_clean":"2024-09-21T10:36:37.987507+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:36:37.987507+0000","last_undegraded":"2024-09-21T10:36:37.987507+0000","last_fullsized":"2024-09-21T10:36:37.987507+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1496,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.985466+0000","last_change":"2024-09-21T10:33:25.551397+0000","last_active":"2024-09-21T10:36:37.985466+0000","last_peered":"2024-09-21T10:36:37.985466+0000","last_clean":"2024-09-21T10:36:37.985466+0000","last_became_active":"2024-09-21T10:29:57.005035+0000","last_became_peered":"2024-09-21T10:29:57.005035+0000","last_unstale":"2024-09-21T10:36:37.985466+0000","last_undegraded":"2024-09-21T10:36:37.985466+0000","last_fullsized":"2024-09-21T10:36:37.985466+0000","mapping_epoch":1247,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:20.412919+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":86,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986953+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:36:37.986953+0000","last_peered":"2024-09-21T10:36:37.986953+0000","last_clean":"2024-09-21T10:36:37.986953+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:36:37.986953+0000","last_undegraded":"2024-09-21T10:36:37.986953+0000","last_fullsized":"2024-09-21T10:36:37.986953+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2100,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.669625+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:36:43.669625+0000","last_peered":"2024-09-21T10:36:43.669625+0000","last_clean":"2024-09-21T10:36:43.669625+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:36:43.669625+0000","last_undegraded":"2024-09-21T10:36:43.669625+0000","last_fullsized":"2024-09-21T10:36:43.669625+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1997,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665912+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:36:43.665912+0000","last_peered":"2024-09-21T10:36:43.665912+0000","last_clean":"2024-09-21T10:36:43.665912+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:36:43.665912+0000","last_undegraded":"2024-09-21T10:36:43.665912+0000","last_fullsized":"2024-09-21T10:36:43.665912+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":264,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665326+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:36:43.665326+0000","last_peered":"2024-09-21T10:36:43.665326+0000","last_clean":"2024-09-21T10:36:43.665326+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:36:43.665326+0000","last_undegraded":"2024-09-21T10:36:43.665326+0000","last_fullsized":"2024-09-21T10:36:43.665326+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1196,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.665395+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:36:42.665395+0000","last_peered":"2024-09-21T10:36:42.665395+0000","last_clean":"2024-09-21T10:36:42.665395+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:36:42.665395+0000","last_undegraded":"2024-09-21T10:36:42.665395+0000","last_fullsized":"2024-09-21T10:36:42.665395+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1153,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.686745+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:36:43.686745+0000","last_peered":"2024-09-21T10:36:43.686745+0000","last_clean":"2024-09-21T10:36:43.686745+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:36:43.686745+0000","last_undegraded":"2024-09-21T10:36:43.686745+0000","last_fullsized":"2024-09-21T10:36:43.686745+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":90,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987196+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:36:37.987196+0000","last_peered":"2024-09-21T10:36:37.987196+0000","last_clean":"2024-09-21T10:36:37.987196+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:36:37.987196+0000","last_undegraded":"2024-09-21T10:36:37.987196+0000","last_fullsized":"2024-09-21T10:36:37.987196+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1161,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.688671+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:36:43.688671+0000","last_peered":"2024-09-21T10:36:43.688671+0000","last_clean":"2024-09-21T10:36:43.688671+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:36:43.688671+0000","last_undegraded":"2024-09-21T10:36:43.688671+0000","last_fullsized":"2024-09-21T10:36:43.688671+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1303,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.664629+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:36:42.664629+0000","last_peered":"2024-09-21T10:36:42.664629+0000","last_clean":"2024-09-21T10:36:42.664629+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:36:42.664629+0000","last_undegraded":"2024-09-21T10:36:42.664629+0000","last_fullsized":"2024-09-21T10:36:42.664629+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1457,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.985521+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:36:37.985521+0000","last_peered":"2024-09-21T10:36:37.985521+0000","last_clean":"2024-09-21T10:36:37.985521+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:36:37.985521+0000","last_undegraded":"2024-09-21T10:36:37.985521+0000","last_fullsized":"2024-09-21T10:36:37.985521+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1213,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665286+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:36:43.665286+0000","last_peered":"2024-09-21T10:36:43.665286+0000","last_clean":"2024-09-21T10:36:43.665286+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:36:43.665286+0000","last_undegraded":"2024-09-21T10:36:43.665286+0000","last_fullsized":"2024-09-21T10:36:43.665286+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":214,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.664783+0000","last_change":"2024-09-21T10:36:21.862280+0000","last_active":"2024-09-21T10:36:43.664783+0000","last_peered":"2024-09-21T10:36:43.664783+0000","last_clean":"2024-09-21T10:36:43.664783+0000","last_became_active":"2024-09-21T10:36:21.851690+0000","last_became_peered":"2024-09-21T10:36:21.851690+0000","last_unstale":"2024-09-21T10:36:43.664783+0000","last_undegraded":"2024-09-21T10:36:43.664783+0000","last_fullsized":"2024-09-21T10:36:43.664783+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:45:28.774299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":21,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986151+0000","last_change":"2024-09-21T10:36:21.850449+0000","last_active":"2024-09-21T10:36:37.986151+0000","last_peered":"2024-09-21T10:36:37.986151+0000","last_clean":"2024-09-21T10:36:37.986151+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:36:37.986151+0000","last_undegraded":"2024-09-21T10:36:37.986151+0000","last_fullsized":"2024-09-21T10:36:37.986151+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.479064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.479064+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":34,"num_write":7,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":26835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2125,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667550+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:36:43.667550+0000","last_peered":"2024-09-21T10:36:43.667550+0000","last_clean":"2024-09-21T10:36:43.667550+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:36:43.667550+0000","last_undegraded":"2024-09-21T10:36:43.667550+0000","last_fullsized":"2024-09-21T10:36:43.667550+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":86,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987476+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:36:37.987476+0000","last_peered":"2024-09-21T10:36:37.987476+0000","last_clean":"2024-09-21T10:36:37.987476+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:36:37.987476+0000","last_undegraded":"2024-09-21T10:36:37.987476+0000","last_fullsized":"2024-09-21T10:36:37.987476+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":481,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.669214+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:36:42.669214+0000","last_peered":"2024-09-21T10:36:42.669214+0000","last_clean":"2024-09-21T10:36:42.669214+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:36:42.669214+0000","last_undegraded":"2024-09-21T10:36:42.669214+0000","last_fullsized":"2024-09-21T10:36:42.669214+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1275,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.664454+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:36:42.664454+0000","last_peered":"2024-09-21T10:36:42.664454+0000","last_clean":"2024-09-21T10:36:42.664454+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:36:42.664454+0000","last_undegraded":"2024-09-21T10:36:42.664454+0000","last_fullsized":"2024-09-21T10:36:42.664454+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1281,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987023+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:36:37.987023+0000","last_peered":"2024-09-21T10:36:37.987023+0000","last_clean":"2024-09-21T10:36:37.987023+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:36:37.987023+0000","last_undegraded":"2024-09-21T10:36:37.987023+0000","last_fullsized":"2024-09-21T10:36:37.987023+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":264,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.664902+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:36:43.664902+0000","last_peered":"2024-09-21T10:36:43.664902+0000","last_clean":"2024-09-21T10:36:43.664902+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:36:43.664902+0000","last_undegraded":"2024-09-21T10:36:43.664902+0000","last_fullsized":"2024-09-21T10:36:43.664902+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1196,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.664968+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:36:43.664968+0000","last_peered":"2024-09-21T10:36:43.664968+0000","last_clean":"2024-09-21T10:36:43.664968+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:36:43.664968+0000","last_undegraded":"2024-09-21T10:36:43.664968+0000","last_fullsized":"2024-09-21T10:36:43.664968+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1327,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986358+0000","last_change":"2024-09-21T10:36:21.880497+0000","last_active":"2024-09-21T10:36:37.986358+0000","last_peered":"2024-09-21T10:36:37.986358+0000","last_clean":"2024-09-21T10:36:37.986358+0000","last_became_active":"2024-09-21T10:36:21.858526+0000","last_became_peered":"2024-09-21T10:36:21.858526+0000","last_unstale":"2024-09-21T10:36:37.986358+0000","last_undegraded":"2024-09-21T10:36:37.986358+0000","last_fullsized":"2024-09-21T10:36:37.986358+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:40:03.207886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":83,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986005+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:36:37.986005+0000","last_peered":"2024-09-21T10:36:37.986005+0000","last_clean":"2024-09-21T10:36:37.986005+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:36:37.986005+0000","last_undegraded":"2024-09-21T10:36:37.986005+0000","last_fullsized":"2024-09-21T10:36:37.986005+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":86,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987277+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:36:37.987277+0000","last_peered":"2024-09-21T10:36:37.987277+0000","last_clean":"2024-09-21T10:36:37.987277+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:36:37.987277+0000","last_undegraded":"2024-09-21T10:36:37.987277+0000","last_fullsized":"2024-09-21T10:36:37.987277+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1340,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986290+0000","last_change":"2024-09-21T10:36:21.882404+0000","last_active":"2024-09-21T10:36:37.986290+0000","last_peered":"2024-09-21T10:36:37.986290+0000","last_clean":"2024-09-21T10:36:37.986290+0000","last_became_active":"2024-09-21T10:36:21.862863+0000","last_became_peered":"2024-09-21T10:36:21.862863+0000","last_unstale":"2024-09-21T10:36:37.986290+0000","last_undegraded":"2024-09-21T10:36:37.986290+0000","last_fullsized":"2024-09-21T10:36:37.986290+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:22:32.045848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":44,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":73,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986691+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:36:37.986691+0000","last_peered":"2024-09-21T10:36:37.986691+0000","last_clean":"2024-09-21T10:36:37.986691+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:36:37.986691+0000","last_undegraded":"2024-09-21T10:36:37.986691+0000","last_fullsized":"2024-09-21T10:36:37.986691+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1352,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.664982+0000","last_change":"2024-09-21T10:30:09.424136+0000","last_active":"2024-09-21T10:36:42.664982+0000","last_peered":"2024-09-21T10:36:42.664982+0000","last_clean":"2024-09-21T10:36:42.664982+0000","last_became_active":"2024-09-21T10:30:09.414027+0000","last_became_peered":"2024-09-21T10:30:09.414027+0000","last_unstale":"2024-09-21T10:36:42.664982+0000","last_undegraded":"2024-09-21T10:36:42.664982+0000","last_fullsized":"2024-09-21T10:36:42.664982+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:31:28.506487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1987,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.666413+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:36:43.666413+0000","last_peered":"2024-09-21T10:36:43.666413+0000","last_clean":"2024-09-21T10:36:43.666413+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:36:43.666413+0000","last_undegraded":"2024-09-21T10:36:43.666413+0000","last_fullsized":"2024-09-21T10:36:43.666413+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":262,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665186+0000","last_change":"2024-09-21T10:36:21.857235+0000","last_active":"2024-09-21T10:36:43.665186+0000","last_peered":"2024-09-21T10:36:43.665186+0000","last_clean":"2024-09-21T10:36:43.665186+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:36:43.665186+0000","last_undegraded":"2024-09-21T10:36:43.665186+0000","last_fullsized":"2024-09-21T10:36:43.665186+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:20.658196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:20.658196+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1191,"reported_epoch":1336,"state":"active+clean","last_fresh":"2024-09-21T10:36:42.665452+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:36:42.665452+0000","last_peered":"2024-09-21T10:36:42.665452+0000","last_clean":"2024-09-21T10:36:42.665452+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:36:42.665452+0000","last_undegraded":"2024-09-21T10:36:42.665452+0000","last_fullsized":"2024-09-21T10:36:42.665452+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2106,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.668492+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:36:43.668492+0000","last_peered":"2024-09-21T10:36:43.668492+0000","last_clean":"2024-09-21T10:36:43.668492+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:36:43.668492+0000","last_undegraded":"2024-09-21T10:36:43.668492+0000","last_fullsized":"2024-09-21T10:36:43.668492+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":73,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987041+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:36:37.987041+0000","last_peered":"2024-09-21T10:36:37.987041+0000","last_clean":"2024-09-21T10:36:37.987041+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:36:37.987041+0000","last_undegraded":"2024-09-21T10:36:37.987041+0000","last_fullsized":"2024-09-21T10:36:37.987041+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2106,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.667336+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:36:43.667336+0000","last_peered":"2024-09-21T10:36:43.667336+0000","last_clean":"2024-09-21T10:36:43.667336+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:36:43.667336+0000","last_undegraded":"2024-09-21T10:36:43.667336+0000","last_fullsized":"2024-09-21T10:36:43.667336+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":86,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.987101+0000","last_change":"2024-09-21T10:30:09.405230+0000","last_active":"2024-09-21T10:36:37.987101+0000","last_peered":"2024-09-21T10:36:37.987101+0000","last_clean":"2024-09-21T10:36:37.987101+0000","last_became_active":"2024-09-21T10:30:09.404929+0000","last_became_peered":"2024-09-21T10:30:09.404929+0000","last_unstale":"2024-09-21T10:36:37.987101+0000","last_undegraded":"2024-09-21T10:36:37.987101+0000","last_fullsized":"2024-09-21T10:36:37.987101+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:01.454535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":106,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.986934+0000","last_change":"2024-09-21T10:33:30.245355+0000","last_active":"2024-09-21T10:36:37.986934+0000","last_peered":"2024-09-21T10:36:37.986934+0000","last_clean":"2024-09-21T10:36:37.986934+0000","last_became_active":"2024-09-21T10:33:20.902088+0000","last_became_peered":"2024-09-21T10:33:20.902088+0000","last_unstale":"2024-09-21T10:36:37.986934+0000","last_undegraded":"2024-09-21T10:36:37.986934+0000","last_fullsized":"2024-09-21T10:36:37.986934+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:30.245301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:30.245301+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:52:43.434664+0000","scrub_duration":0.0018729759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1342,"reported_epoch":1335,"state":"active+clean","last_fresh":"2024-09-21T10:36:37.985977+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:36:37.985977+0000","last_peered":"2024-09-21T10:36:37.985977+0000","last_clean":"2024-09-21T10:36:37.985977+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:36:37.985977+0000","last_undegraded":"2024-09-21T10:36:37.985977+0000","last_fullsized":"2024-09-21T10:36:37.985977+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1995,"reported_epoch":1337,"state":"active+clean","last_fresh":"2024-09-21T10:36:43.665009+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:36:43.665009+0000","last_peered":"2024-09-21T10:36:43.665009+0000","last_clean":"2024-09-21T10:36:43.665009+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:36:43.665009+0000","last_undegraded":"2024-09-21T10:36:43.665009+0000","last_fullsized":"2024-09-21T10:36:43.665009+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":47,"num_pg":26,"stat_sum":{"num_bytes":2140340248,"num_objects":32661,"num_object_clones":0,"num_object_copies":97983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9120,"num_objects_unfound":0,"num_objects_dirty":32661,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":164909,"num_write_kb":4889089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18989,"num_bytes_recovered":1244332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7472754688,"data_stored":7472742472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":153164,"ondisk_log_size":153164,"up":78,"acting":76,"num_store_stats":8},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":670873,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":102,"num_write":21,"num_write_kb":222,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":998553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":6246,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398006,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295596,"kb_used_data":701540,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92449556,"statfs":{"total":95995035648,"available":94668345344,"internally_reserved":0,"allocated":718376960,"data_stored":712723449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049666,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663236,"kb_used_data":1318908,"kb_used_omap":242,"kb_used_meta":344077,"kb_avail":92081916,"statfs":{"total":95995035648,"available":94291881984,"internally_reserved":0,"allocated":1350561792,"data_stored":1344893547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":352335071},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790464,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1090476,"kb_used_data":500576,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92654676,"statfs":{"total":95995035648,"available":94878388224,"internally_reserved":0,"allocated":512589824,"data_stored":506936764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395570,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644624,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1612908,"kb_used_data":1059300,"kb_used_omap":67,"kb_used_meta":553532,"kb_avail":92132244,"statfs":{"total":95995035648,"available":94343417856,"internally_reserved":0,"allocated":1084723200,"data_stored":1079074716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":566817629},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527642,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1468352,"kb_used_data":867448,"kb_used_omap":75,"kb_used_meta":600756,"kb_avail":92276800,"statfs":{"total":95995035648,"available":94491443200,"internally_reserved":0,"allocated":888266752,"data_stored":882603094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":615174298},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274795,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2210036,"kb_used_data":1586600,"kb_used_omap":51,"kb_used_meta":623308,"kb_avail":91535116,"statfs":{"total":95995035648,"available":93731958784,"internally_reserved":0,"allocated":1624678400,"data_stored":1619025325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":638267733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354034,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1043304,"kb_used_data":607712,"kb_used_omap":66,"kb_used_meta":435517,"kb_avail":92701848,"statfs":{"total":95995035648,"available":94926692352,"internally_reserved":0,"allocated":622297088,"data_stored":616652172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":445969732},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":5418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":597950464,"data_stored":597950464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1591148544,"data_stored":1591148544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":865341440,"data_stored":865337368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1051918336,"data_stored":1051918336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":849481728,"data_stored":849477656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":496697344,"data_stored":496697344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1317081088,"data_stored":1317077016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":47,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":703135744,"data_stored":703135744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:36:51.497 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:36:51.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_2 pg_num 36 2024-09-21T10:36:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:51 smithi145 ceph-mon[89359]: pgmap v2145: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 754 op/s; 4593/94818 objects degraded (4.844%); 8722/94818 objects misplaced (9.199%); 13 MiB/s, 201 objects/s recovering 2024-09-21T10:36:51.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:51 smithi119 ceph-mon[105288]: pgmap v2145: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 754 op/s; 4593/94818 objects degraded (4.844%); 8722/94818 objects misplaced (9.199%); 13 MiB/s, 201 objects/s recovering 2024-09-21T10:36:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:51 smithi119 ceph-mon[101432]: pgmap v2145: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 754 op/s; 4593/94818 objects degraded (4.844%); 8722/94818 objects misplaced (9.199%); 13 MiB/s, 201 objects/s recovering 2024-09-21T10:36:51.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:51.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:51.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:51.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:51.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:51.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:52.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:52.052 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:52.154 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:52.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:52.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:52.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:52.350 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:36:52.350 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230678 45.e 45:79bfb7a2:::benchmark_data_smithi119_168706_object100508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.808009+0000", 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.53808169299998, 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.097576466, 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230678 2024-09-21T10:36:52.351 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808009+0000", 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808009+0000", 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.352 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808008+0000", 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808013+0000", 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0470000000000001e-06 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808022+0000", 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0080000000000008e-06 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.353 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808026+0000", 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9910000000000004e-06 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808055+0000", 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8611000000000001e-05 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808176+0000", 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000121403 2024-09-21T10:36:52.354 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.808311+0000", 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013477699999999999 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.812893+0000", 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045818250000000003 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.813982+0000", 2024-09-21T10:36:52.355 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010895270000000001 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.817794+0000", 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038114450000000001 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840263+0000", 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.022469335999999999 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.356 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840275+0000", 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1874e-05 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905517+0000", 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.065242453 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905585+0000", 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8132000000000004e-05 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.357 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230690 45.e 45:74649e72:::benchmark_data_smithi119_168706_object100520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.824956+0000", 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.52113459400005, 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.080745302, 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230690 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.358 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824956+0000", 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824956+0000", 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824955+0000", 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.359 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824960+0000", 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8300000000000003e-06 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824969+0000", 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1929999999999999e-06 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.824974+0000", 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.78e-06 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.360 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.825032+0000", 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8323999999999999e-05 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.825177+0000", 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014489799999999999 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.825262+0000", 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5150999999999998e-05 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.361 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840324+0000", 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0150616 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840331+0000", 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4789999999999999e-06 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905636+0000", 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.065304714 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.362 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905701+0000", 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5591000000000003e-05 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230693 45.e 45:7aa067b6:::benchmark_data_smithi119_168706_object100523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.828211+0000", 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.51787936899996, 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.077542488, 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.363 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230693 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.364 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828211+0000", 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828211+0000", 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828209+0000", 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.365 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828217+0000", 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8979999999999996e-06 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828237+0000", 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0010999999999999e-05 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828245+0000", 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2429999999999999e-06 2024-09-21T10:36:52.366 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828977+0000", 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00073276999999999999 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.829121+0000", 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014400599999999999 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.829260+0000", 2024-09-21T10:36:52.367 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013883499999999999 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840365+0000", 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011104862 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840373+0000", 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5220000000000001e-06 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905717+0000", 2024-09-21T10:36:52.368 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0653447140000001 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905754+0000", 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6402999999999997e-05 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230694 45.e 45:79bbaa33:::benchmark_data_smithi119_168706_object100524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1209)", 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.828798+0000", 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.51729299099998, 2024-09-21T10:36:52.369 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.077047415, 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230694 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828798+0000", 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.370 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828798+0000", 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828797+0000", 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828802+0000", 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4580000000000001e-06 2024-09-21T10:36:52.371 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828825+0000", 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2708999999999999e-05 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.828832+0000", 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.374e-06 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.834438+0000", 2024-09-21T10:36:52.372 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0056056700000000001 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.834557+0000", 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000119207 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3,5", 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.834634+0000", 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7002000000000006e-05 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840428+0000", 2024-09-21T10:36:52.373 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0057940259999999999 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840436+0000", 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6480000000000003e-06 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905784+0000", 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0653486219999999 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.374 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905845+0000", 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0616999999999999e-05 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230699 45.e 45:73169608:::benchmark_data_smithi119_168706_object100529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.840651+0000", 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.50543976500001, 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1005979770000001, 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.375 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230699 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840651+0000", 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840651+0000", 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.376 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840649+0000", 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840655+0000", 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2260000000000002e-06 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840666+0000", 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0587e-05 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.377 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840672+0000", 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840762+0000", 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0346999999999994e-05 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.840772+0000", 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0108999999999999e-05 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.378 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.905891+0000", 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.065118842 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.906014+0000", 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012271100000000001 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.906152+0000", 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000138419 2024-09-21T10:36:52.379 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.922422+0000", 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016269520999999999 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941090+0000", 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018668609999999999 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941186+0000", 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5977999999999995e-05 2024-09-21T10:36:52.380 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941225+0000", 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8651e-05 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941249+0000", 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4031000000000001e-05 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.381 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230700 45.e 45:76633324:::benchmark_data_smithi119_168706_object100530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.842302+0000", 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.50378891699995, 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0991607729999999, 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230700 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.382 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842302+0000", 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842302+0000", 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842301+0000", 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.383 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842306+0000", 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9490000000000004e-06 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842316+0000", 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0369e-05 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842320+0000", 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3359999999999997e-06 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.384 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842353+0000", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2666999999999997e-05 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.842358+0000", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2329999999999997e-06 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.910012+0000", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0676535069999999 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.385 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.910143+0000", 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013090700000000001 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.910276+0000", 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000133215 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.922427+0000", 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012150988 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.386 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941113+0000", 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018686472999999999 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941409+0000", 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029531300000000001 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941444+0000", 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5355000000000003e-05 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.387 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941462+0000", 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8413000000000001e-05 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230756 45.e 45:7f9a0f5a:::benchmark_data_smithi119_168706_object100586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.919711+0000", 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.42637943600005, 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0218782360000001, 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.388 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230756 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919711+0000", 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.389 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919711+0000", 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919710+0000", 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919717+0000", 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7500000000000003e-06 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.390 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.391 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.391 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919731+0000", 2024-09-21T10:36:52.391 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.399e-05 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919737+0000", 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1330000000000004e-06 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919778+0000", 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0710000000000002e-05 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.392 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.919785+0000", 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.562e-06 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.913413+0000", 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99362865300000003 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.913493+0000", 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9919999999999994e-05 2024-09-21T10:36:52.393 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.913567+0000", 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3306999999999995e-05 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.928030+0000", 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014463877 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941141+0000", 2024-09-21T10:36:52.394 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013110157000000001 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941504+0000", 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00036338499999999998 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941551+0000", 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6860999999999997e-05 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.395 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941589+0000", 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8544999999999998e-05 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230758 45.e 45:7f2353b2:::benchmark_data_smithi119_168706_object100588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.920715+0000", 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.42537546599999, 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.021031107, 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.396 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230758 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920715+0000", 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920715+0000", 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.397 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920714+0000", 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920719+0000", 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1290000000000004e-06 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920732+0000", 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2548000000000001e-05 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.398 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920737+0000", 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3800000000000002e-06 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920779+0000", 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2024999999999998e-05 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.399 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.920784+0000", 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7940000000000002e-06 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.916661+0000", 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99587639999999999 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.916752+0000", 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1729000000000002e-05 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:52.400 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.916849+0000", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6251999999999999e-05 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.928037+0000", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011188849000000001 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941283+0000", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013245883999999999 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941697+0000", 2024-09-21T10:36:52.401 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041416700000000001 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941726+0000", 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8523000000000001e-05 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941746+0000", 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0319e-05 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.402 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.35976.0:230770 45.e 45:7a02883a:::benchmark_data_smithi119_168706_object100600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1210)", 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:27:17.938360+0000", 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 574.40773030699995, 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0035411750000001, 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.35976", 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1347856014", 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230770 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.403 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938360+0000", 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938360+0000", 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938359+0000", 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.404 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938365+0000", 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8340000000000004e-06 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938379+0000", 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4242e-05 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938386+0000", 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2319999999999998e-06 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.405 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938451+0000", 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5400000000000004e-05 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:17.938459+0000", 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9850000000000006e-06 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.919620+0000", 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98116114600000004 2024-09-21T10:36:52.406 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.919697+0000", 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6692000000000004e-05 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.919783+0000", 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5957000000000004e-05 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.933063+0000", 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013280001 2024-09-21T10:36:52.407 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941307+0000", 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0082444389999999992 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941856+0000", 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00054824699999999997 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941881+0000", 2024-09-21T10:36:52.408 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5786e-05 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:27:18.941901+0000", 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0157999999999999e-05 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:179480 47.13 47:cd216aad:::benchmark_data_smithi119_238957_object54651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:01.347978+0000", 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 50.998112857999999, 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10888231600000001, 2024-09-21T10:36:52.409 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 179480 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347978+0000", 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.410 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347978+0000", 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347976+0000", 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347984+0000", 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.96e-06 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.411 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348002+0000", 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8287999999999998e-05 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348009+0000", 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3989999999999998e-06 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348062+0000", 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2398999999999998e-05 2024-09-21T10:36:52.412 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348228+0000", 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000166377 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348762+0000", 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00053391799999999998 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.351455+0000", 2024-09-21T10:36:52.413 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026928989999999999 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.351538+0000", 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2875999999999997e-05 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.456764+0000", 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.105226323 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.456819+0000", 2024-09-21T10:36:52.414 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5229999999999998e-05 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.456860+0000", 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0645999999999997e-05 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:186102 47.e 47:7755fc4e:::benchmark_data_smithi119_238957_object61273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:07.606247+0000", 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 44.739844052999999, 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.109293795, 2024-09-21T10:36:52.415 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 186102 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606247+0000", 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.416 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606247+0000", 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606245+0000", 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606252+0000", 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5509999999999996e-06 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.417 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606265+0000", 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3397999999999999e-05 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606271+0000", 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9410000000000004e-06 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.418 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606315+0000", 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4279000000000001e-05 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606426+0000", 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011014 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606580+0000", 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000154174 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.419 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610203+0000", 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036228240000000002 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.614098+0000", 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038949929999999998 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715431+0000", 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10133310099999999 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.420 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715505+0000", 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4307000000000005e-05 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715540+0000", 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.421 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:186106 47.e 47:733d1a38:::benchmark_data_smithi119_238957_object61277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:07.608957+0000", 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 44.737133475999997, 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.106723203, 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 186106 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.422 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608957+0000", 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608957+0000", 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608956+0000", 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.423 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608961+0000", 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8539999999999996e-06 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608974+0000", 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3225e-05 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608980+0000", 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0800000000000002e-06 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.424 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610511+0000", 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001531078 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610630+0000", 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118622 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610750+0000", 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120294 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.425 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.619388+0000", 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086377269999999996 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.619483+0000", 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4858999999999994e-05 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715576+0000", 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.096093637999999995 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.426 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715645+0000", 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8706000000000003e-05 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715680+0000", 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5246000000000001e-05 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:186108 47.e 47:78e14e17:::benchmark_data_smithi119_238957_object61279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:07.612337+0000", 2024-09-21T10:36:52.427 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 44.733753759999999, 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10348737600000001, 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 186108 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.612337+0000", 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.428 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.612337+0000", 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.612335+0000", 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.612345+0000", 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7929999999999992e-06 2024-09-21T10:36:52.429 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.612368+0000", 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3065999999999999e-05 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.612374+0000", 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9490000000000001e-06 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.614136+0000", 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017619059999999999 2024-09-21T10:36:52.430 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.614262+0000", 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012609200000000001 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.614385+0000", 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012302400000000001 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.619474+0000", 2024-09-21T10:36:52.431 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0050890730000000004 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.619491+0000", 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6195999999999999e-05 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715722+0000", 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.096231065000000005 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715779+0000", 2024-09-21T10:36:52.432 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7000999999999999e-05 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715824+0000", 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.566e-05 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:197701 47.b 47:d43e12d0:::benchmark_data_smithi119_238957_object72872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.915681+0000", 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.430409636999997, 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15200201699999999, 2024-09-21T10:36:52.433 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 197701 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915681+0000", 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.434 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915681+0000", 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915680+0000", 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915686+0000", 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.435 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915694+0000", 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2169999999999994e-06 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915698+0000", 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.924e-06 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915730+0000", 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2397999999999997e-05 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.436 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915812+0000", 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2003999999999996e-05 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915887+0000", 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4777999999999995e-05 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.920574+0000", 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046868259999999998 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067316+0000", 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.146742226 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067434+0000", 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011733 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067562+0000", 2024-09-21T10:36:52.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012795300000000001 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067683+0000", 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000121478 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46418.0:197720 47.15 e1327/1317 47:ab0f04b9:::benchmark_data_smithi119_238957_object72891:head v 1327'7981, mlcod=1305'4220)", 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.934882+0000", 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.411208440999999, 2024-09-21T10:36:52.439 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.104638546, 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934882+0000", 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934882+0000", 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.440 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934881+0000", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934891+0000", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9490000000000008e-06 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934902+0000", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0753e-05 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934912+0000", 2024-09-21T10:36:52.441 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8770000000000005e-06 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934948+0000", 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6606000000000002e-05 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.934973+0000", 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4751000000000001e-05 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.442 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.039473+0000", 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.104499859 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.039521+0000", 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7719000000000002e-05 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:197721 47.13 47:cdfee054:::benchmark_data_smithi119_238957_object72892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.935578+0000", 2024-09-21T10:36:52.443 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.410512887000003, 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.106628818, 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 197721 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935578+0000", 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.444 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935578+0000", 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935576+0000", 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.445 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935584+0000", 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1489999999999994e-06 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935594+0000", 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8889999999999997e-06 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935599+0000", 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8980000000000004e-06 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.446 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935639+0000", 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9348999999999999e-05 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935715+0000", 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6186999999999994e-05 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.936207+0000", 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049191600000000001 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.447 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.041981+0000", 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10577399699999999 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042011+0000", 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0502e-05 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042140+0000", 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012901500000000001 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.448 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042177+0000", 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7181999999999997e-05 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042207+0000", 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9240000000000001e-05 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop_reply(client.46418.0:197721 47.13 e1327/1322 ondisk, result = 0)", 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.940012+0000", 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 33.406078334, 2024-09-21T10:36:52.449 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10197073600000001, 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.940012+0000", 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.940012+0000", 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.450 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.940011+0000", 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.940022+0000", 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0285999999999999e-05 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.039459+0000", 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.099437385000000003 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.451 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.039473+0000", 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3505e-05 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.041968+0000", 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024952049999999999 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.041975+0000", 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2470000000000002e-06 2024-09-21T10:36:52.452 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.041983+0000", 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0620000000000001e-06 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:214870 47.e 47:716e9391:::benchmark_data_smithi119_238957_object90041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1335)", 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:42.378835+0000", 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 9.9672552930000009, 2024-09-21T10:36:52.453 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.093303579999999997, 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 214870 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378835+0000", 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.454 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378835+0000", 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378833+0000", 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378841+0000", 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1239999999999999e-06 2024-09-21T10:36:52.455 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378854+0000", 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3234000000000001e-05 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378859+0000", 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7359999999999998e-06 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.378911+0000", 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1233000000000002e-05 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.456 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.379041+0000", 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013030800000000001 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.379121+0000", 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0078000000000001e-05 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.382075+0000", 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029539050000000002 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.457 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.458 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.382691+0000", 2024-09-21T10:36:52.458 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00061614500000000002 2024-09-21T10:36:52.458 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.472047+0000", 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.089355453000000001 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.472099+0000", 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2203999999999997e-05 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:42.472139+0000", 2024-09-21T10:36:52.459 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0074999999999998e-05 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:220677 47.13 47:cb4c36ad:::benchmark_data_smithi119_238957_object95848:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:49.473432+0000", 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.8726590930000002, 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.103713349, 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.460 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 220677 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473432+0000", 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473432+0000", 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473431+0000", 2024-09-21T10:36:52.461 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473436+0000", 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0030000000000002e-06 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473447+0000", 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1511999999999999e-05 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.462 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473453+0000", 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5629999999999999e-06 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473494+0000", 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1185e-05 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473601+0000", 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010741700000000001 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:36:52.463 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.474126+0000", 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00052434200000000002 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.478292+0000", 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041660309999999997 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.479966+0000", 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016743859999999999 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.464 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.577059+0000", 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.097093090000000007 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.577108+0000", 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9153999999999997e-05 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.577145+0000", 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6585000000000003e-05 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.465 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:220678 47.13 47:cc7dfd99:::benchmark_data_smithi119_238957_object95849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:49.473919+0000", 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 2.8721714380000001, 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10334270199999999, 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 220678 2024-09-21T10:36:52.466 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473919+0000", 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473919+0000", 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473918+0000", 2024-09-21T10:36:52.467 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473924+0000", 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5929999999999996e-06 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473938+0000", 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4598e-05 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:52.468 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.473945+0000", 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.353e-06 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.476118+0000", 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021728630000000001 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.476205+0000", 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7091999999999999e-05 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.469 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.476484+0000", 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027948399999999998 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.478429+0000", 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0019444 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.480065+0000", 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016361819999999999 2024-09-21T10:36:52.470 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.577183+0000", 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.097118721000000005 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.577225+0000", 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1683000000000001e-05 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:49.577262+0000", 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6695e-05 2024-09-21T10:36:52.471 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.472 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.472 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.472 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:52.472 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:52.472 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:36:52.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:52.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:52.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:52.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:52.761 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:52.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:52.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:52.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:52.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:53.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:53.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:53.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:53.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:53.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.293 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:53.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:53.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:53.416 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:36:53.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:53.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.570 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:53.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:53 smithi119 ceph-mon[105288]: pgmap v2146: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 4593/88362 objects degraded (5.198%); 7980/88362 objects misplaced (9.031%); 11 MiB/s, 174 objects/s recovering 2024-09-21T10:36:53.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:53 smithi119 ceph-mon[101432]: pgmap v2146: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 4593/88362 objects degraded (5.198%); 7980/88362 objects misplaced (9.031%); 11 MiB/s, 174 objects/s recovering 2024-09-21T10:36:53.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:53.698 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:53.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:53.770 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:53 smithi145 ceph-mon[89359]: pgmap v2146: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 1.8 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 780 op/s; 4593/88362 objects degraded (5.198%); 7980/88362 objects misplaced (9.031%); 11 MiB/s, 174 objects/s recovering 2024-09-21T10:36:53.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:53.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:53.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:53.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:53.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:54.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:54.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:54.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:54.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:54.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:36:54.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:54.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:54.537 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1282553429' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:54.537 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:54.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:54.635 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:54.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.783 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:54.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:36:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:36:54] "GET /metrics HTTP/1.1" 200 36921 "" "Prometheus/2.51.0" 2024-09-21T10:36:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1282553429' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:54.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1282553429' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:54.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:36:54.884 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:54.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:54.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:54.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:55.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:55.049 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:55.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:55.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:55.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:55.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:55.311 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:55.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:55.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:55.441 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:55.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:55 smithi145 ceph-mon[89359]: pgmap v2147: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 4295/81003 objects degraded (5.302%); 7011/81003 objects misplaced (8.655%); 13 MiB/s, 214 objects/s recovering 2024-09-21T10:36:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:55 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:36:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:36:55.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:55 smithi145 ceph-mon[89359]: osdmap e1338: 8 total, 8 up, 7 in 2024-09-21T10:36:55.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:55 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 4295/81003 objects degraded (5.302%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:55.580 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[105288]: pgmap v2147: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 4295/81003 objects degraded (5.302%); 7011/81003 objects misplaced (8.655%); 13 MiB/s, 214 objects/s recovering 2024-09-21T10:36:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:36:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:36:55.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[105288]: osdmap e1338: 8 total, 8 up, 7 in 2024-09-21T10:36:55.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 4295/81003 objects degraded (5.302%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[101432]: pgmap v2147: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.6 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 4295/81003 objects degraded (5.302%); 7011/81003 objects misplaced (8.655%); 13 MiB/s, 214 objects/s recovering 2024-09-21T10:36:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:36:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:36:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[101432]: osdmap e1338: 8 total, 8 up, 7 in 2024-09-21T10:36:55.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:55 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 4295/81003 objects degraded (5.302%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:36:55.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.729 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:55.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:55.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:55.852 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:36:55.852 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226336 47.d 47:bfa8194d:::benchmark_data_smithi119_238957_object101507:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:55.839347+0000", 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0099391299999999991, 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0099621640000000008, 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226336 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.853 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839347+0000", 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839347+0000", 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.854 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839346+0000", 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839352+0000", 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4989999999999999e-06 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839363+0000", 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1028e-05 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.855 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839369+0000", 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9580000000000004e-06 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.842438+0000", 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0030688209999999998 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.845963+0000", 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035246750000000001 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.856 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2,4", 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.846094+0000", 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013105500000000001 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop_reply(client.46418.0:226329 47.f e1338/1285)", 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:55.849329+0000", 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4294967295.9999571, 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7621999999999998e-05, 2024-09-21T10:36:55.857 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849329+0000", 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849329+0000", 2024-09-21T10:36:55.858 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849328+0000", 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849333+0000", 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1220000000000003e-06 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:55.859 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849349+0000", 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.556e-05 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849355+0000", 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8690000000000001e-06 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226327 47.d 47:b172994c:::benchmark_data_smithi119_238957_object101498:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:55.835099+0000", 2024-09-21T10:36:55.860 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.01418668, 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01434263, 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226327 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835099+0000", 2024-09-21T10:36:55.861 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835099+0000", 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835097+0000", 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.862 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835104+0000", 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7970000000000003e-06 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835119+0000", 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4864e-05 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835125+0000", 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7590000000000003e-06 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.863 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835173+0000", 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8058e-05 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835304+0000", 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131369 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2,4", 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835429+0000", 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012480100000000001 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.864 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.842358+0000", 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069294430000000004 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.842387+0000", 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8782999999999999e-05 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.865 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226329 47.f 47:f35f46c3:::benchmark_data_smithi119_238957_object101500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:55.835249+0000", 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.014036602, 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.014283128000000001, 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226329 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.866 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835249+0000", 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835249+0000", 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835248+0000", 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.867 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835256+0000", 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9629999999999998e-06 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835269+0000", 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3183999999999999e-05 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835276+0000", 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0389999999999999e-06 2024-09-21T10:36:55.868 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835421+0000", 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014519000000000001 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835594+0000", 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017232799999999999 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,7", 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.835750+0000", 2024-09-21T10:36:55.869 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015587000000000001 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839442+0000", 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036928949999999999 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.839608+0000", 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016535500000000001 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.870 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849443+0000", 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0098347049999999991 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.849518+0000", 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5545999999999998e-05 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226333 47.d 47:b5d8ca77:::benchmark_data_smithi119_238957_object101504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:55.838268+0000", 2024-09-21T10:36:55.871 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.011017816, 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011341802, 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226333 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838268+0000", 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.872 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838268+0000", 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838267+0000", 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:36:55.873 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838273+0000", 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1750000000000002e-06 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838288+0000", 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5177999999999999e-05 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838295+0000", 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7410000000000003e-06 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.874 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838628+0000", 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033309000000000002 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838727+0000", 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9188e-05 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2,4", 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.838852+0000", 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012493199999999999 2024-09-21T10:36:55.875 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:55.842418+0000", 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035662789999999999 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 5 2024-09-21T10:36:55.876 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:36:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:55.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:55.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:55.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:56.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:56.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:56.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:56.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:56.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:56.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:56.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:56.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:56.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:56.631 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:36:56.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:56.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:56.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:56.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:56.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:57.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:57.010 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:36:57.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:57.134 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:57.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:57.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:57.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:57.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:57.384 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:57.469 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:57 smithi145 ceph-mon[89359]: pgmap v2149: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 1.5 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 4295/74649 objects degraded (5.754%); 6268/74649 objects misplaced (8.397%); 12 MiB/s, 190 objects/s recovering 2024-09-21T10:36:57.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:57.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:57.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:57.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:57.750 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:57.772 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:57 smithi119 ceph-mon[101432]: pgmap v2149: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 1.5 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 4295/74649 objects degraded (5.754%); 6268/74649 objects misplaced (8.397%); 12 MiB/s, 190 objects/s recovering 2024-09-21T10:36:57.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:57 smithi119 ceph-mon[105288]: pgmap v2149: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfilling, 6 active+remapped+backfill_wait, 110 active+clean; 1.5 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 890 op/s; 4295/74649 objects degraded (5.754%); 6268/74649 objects misplaced (8.397%); 12 MiB/s, 190 objects/s recovering 2024-09-21T10:36:57.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:57.896 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:57.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:57.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:36:58.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:58.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:36:58.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:36:58.144 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:58.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:36:58.182 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:58.246 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:58.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:58.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:58.395 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:58.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:58 smithi145 ceph-mon[89359]: osdmap e1339: 8 total, 8 up, 7 in 2024-09-21T10:36:58.557 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:58.582 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:36:58.656 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:58 smithi119 ceph-mon[105288]: osdmap e1339: 8 total, 8 up, 7 in 2024-09-21T10:36:58.657 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:58 smithi119 ceph-mon[101432]: osdmap e1339: 8 total, 8 up, 7 in 2024-09-21T10:36:58.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:58.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:58.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:36:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.837 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:58.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:36:58.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:58.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:58.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:59.069 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:59.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:59.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:36:59.235 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:36:59.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:36:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:59.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:36:59.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:36:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:36:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:36:59.580 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:36:59.580 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:36:59.580 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:36:59.580 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:36:59.580 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:36:59.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:36:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:59 smithi119 ceph-mon[101432]: pgmap v2151: 119 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 4033/71091 objects degraded (5.673%); 5801/71091 objects misplaced (8.160%); 12 MiB/s, 186 objects/s recovering 2024-09-21T10:36:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:36:59 smithi119 ceph-mon[101432]: osdmap e1340: 8 total, 8 up, 7 in 2024-09-21T10:36:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:59 smithi119 ceph-mon[105288]: pgmap v2151: 119 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 4033/71091 objects degraded (5.673%); 5801/71091 objects misplaced (8.160%); 12 MiB/s, 186 objects/s recovering 2024-09-21T10:36:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:36:59 smithi119 ceph-mon[105288]: osdmap e1340: 8 total, 8 up, 7 in 2024-09-21T10:36:59.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:36:59.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:59.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:59 smithi145 ceph-mon[89359]: pgmap v2151: 119 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 6 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 988 op/s; 4033/71091 objects degraded (5.673%); 5801/71091 objects misplaced (8.160%); 12 MiB/s, 186 objects/s recovering 2024-09-21T10:36:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:36:59 smithi145 ceph-mon[89359]: osdmap e1340: 8 total, 8 up, 7 in 2024-09-21T10:36:59.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:36:59.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:36:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:36:59.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:36:59.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:00.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:00.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:00.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:00.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:00.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:37:00.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.328 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:00.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:00.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:00.481 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:37:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:37:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:37:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:00 smithi145 ceph-mon[89359]: osdmap e1341: 8 total, 8 up, 7 in 2024-09-21T10:37:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 4033/71091 objects degraded (5.673%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:00.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:00.610 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:00 smithi119 ceph-mon[101432]: osdmap e1341: 8 total, 8 up, 7 in 2024-09-21T10:37:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 4033/71091 objects degraded (5.673%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:00 smithi119 ceph-mon[105288]: osdmap e1341: 8 total, 8 up, 7 in 2024-09-21T10:37:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 4033/71091 objects degraded (5.673%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:00.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:00.744 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:00.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:00.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:00.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:00.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:01.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:01.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:01.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:01.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:01.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:01.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:01.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:01.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:01.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.440 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:01.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:01.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:01.565 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:01.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:01.688 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:01 smithi119 ceph-mon[101432]: pgmap v2154: 119 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 4033/68622 objects degraded (5.877%); 4779/68622 objects misplaced (6.964%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:37:01.688 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:01 smithi119 ceph-mon[105288]: pgmap v2154: 119 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 4033/68622 objects degraded (5.877%); 4779/68622 objects misplaced (6.964%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:37:01.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:01 smithi145 ceph-mon[89359]: pgmap v2154: 119 pgs: 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 110 active+clean; 1.4 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 771 op/s; 4033/68622 objects degraded (5.877%); 4779/68622 objects misplaced (6.964%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:37:01.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:01.824 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:01.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:01.949 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:01.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:02.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:02.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:02.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:02.215 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:02.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:02.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:02.473 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:02.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:02.549 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:37:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:37:02.587 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:37:02.587 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:37:02.612 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:02.630 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:37:02.630 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:37:02.640 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:02.669 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:37:02.669 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:37:02.703 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:37:02.703 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:37:02.734 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:37:02.734 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:37:02.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:02.749 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:37:02.749 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:37:02.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:02.766 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:37:02.766 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:37:02.856 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:37:02.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:02.934 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:02.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:02.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:03.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:03.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:03.200 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:37:03.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:03.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:03.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:03.331 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:03.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:03.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:03.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:03.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:03 smithi119 ceph-mon[105288]: pgmap v2155: 119 pgs: 1 active+recovering+undersized+remapped, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 110 active+clean; 1.3 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 594 op/s; 4033/63945 objects degraded (6.307%); 4083/63945 objects misplaced (6.385%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:37:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2353214617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:03 smithi119 ceph-mon[101432]: pgmap v2155: 119 pgs: 1 active+recovering+undersized+remapped, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 110 active+clean; 1.3 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 594 op/s; 4033/63945 objects degraded (6.307%); 4083/63945 objects misplaced (6.385%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:37:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2353214617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:03.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:03 smithi145 ceph-mon[89359]: pgmap v2155: 119 pgs: 1 active+recovering+undersized+remapped, 2 peering, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 110 active+clean; 1.3 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 594 op/s; 4033/63945 objects degraded (6.307%); 4083/63945 objects misplaced (6.385%); 16 MiB/s, 258 objects/s recovering 2024-09-21T10:37:03.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2353214617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:03.634 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:03.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:03.732 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:03.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:03.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:03.861 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:03.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:03.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:04.002 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:04.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:04.137 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:04.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:04.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:04.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:04.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:04.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:04.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:04.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:04.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:04.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:04.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:37:04] "GET /metrics HTTP/1.1" 200 36913 "" "Prometheus/2.51.0" 2024-09-21T10:37:04.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:04.933 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:04.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:05.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:05.053 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:37:05.053 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:37:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.068 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:05.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:05.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:05.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:05.301 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:05.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:05.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:05.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:05.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:05.609 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:05 smithi119 ceph-mon[105288]: pgmap v2156: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 810 op/s; 3805/58035 objects degraded (6.556%); 3846/58035 objects misplaced (6.627%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:37:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:05 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3805/58035 objects degraded (6.556%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:05 smithi119 ceph-mon[101432]: pgmap v2156: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 810 op/s; 3805/58035 objects degraded (6.556%); 3846/58035 objects misplaced (6.627%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:37:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:05 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3805/58035 objects degraded (6.556%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:05.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:05.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:05 smithi145 ceph-mon[89359]: pgmap v2156: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.2 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 810 op/s; 3805/58035 objects degraded (6.556%); 3846/58035 objects misplaced (6.627%); 15 MiB/s, 241 objects/s recovering 2024-09-21T10:37:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:05 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3805/58035 objects degraded (6.556%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:05.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:05.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:05.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:05.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:06.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:06.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:06.144 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:06.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:06.284 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:06.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:06.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:06.513 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:06.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:06.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:37:06.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:06.681 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:06.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:06.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:06.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:06.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:06.934 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:37:06.943 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:06.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:06.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:07.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:07.065 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:07.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:07.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:07.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:07.346 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:07.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:07.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:07.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:07.606 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:07.649 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:07 smithi145 ceph-mon[89359]: pgmap v2157: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 3805/52365 objects degraded (7.266%); 3261/52365 objects misplaced (6.227%); 9.4 MiB/s, 171 objects/s recovering 2024-09-21T10:37:07.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:07.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:07.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:07 smithi119 ceph-mon[101432]: pgmap v2157: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 3805/52365 objects degraded (7.266%); 3261/52365 objects misplaced (6.227%); 9.4 MiB/s, 171 objects/s recovering 2024-09-21T10:37:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:07 smithi119 ceph-mon[105288]: pgmap v2157: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1.1 GiB data, 7.8 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 779 op/s; 3805/52365 objects degraded (7.266%); 3261/52365 objects misplaced (6.227%); 9.4 MiB/s, 171 objects/s recovering 2024-09-21T10:37:07.899 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:07.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:07.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:08.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:08.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:08.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:08.156 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:08.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.301 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:08.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:08.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:08.434 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:08.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:08.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:08.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:08.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:08.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:08.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:08.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:08.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:08.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:08.996 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:37:08.996 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:37:09.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:09.103 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:09.205 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.241 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:09.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:09.342 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:09.366 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:09.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:09.490 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:09.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:09.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:09.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:09.674 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:09 smithi145 ceph-mon[89359]: pgmap v2158: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1017 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 793 op/s; 3569/48867 objects degraded (7.303%); 3261/48867 objects misplaced (6.673%); 8.8 MiB/s, 159 objects/s recovering 2024-09-21T10:37:09.707 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:37:09.718 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:09.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:09 smithi119 ceph-mon[105288]: pgmap v2158: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1017 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 793 op/s; 3569/48867 objects degraded (7.303%); 3261/48867 objects misplaced (6.673%); 8.8 MiB/s, 159 objects/s recovering 2024-09-21T10:37:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:09 smithi119 ceph-mon[101432]: pgmap v2158: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 1017 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 793 op/s; 3569/48867 objects degraded (7.303%); 3261/48867 objects misplaced (6.673%); 8.8 MiB/s, 159 objects/s recovering 2024-09-21T10:37:09.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:09.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:09.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:10.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:10.039 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:37:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:10.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:10.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:10.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:10.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:10.299 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:10.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:10.442 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:10.463 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:37:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:37:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:37:10.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:10.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:10.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:10.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:10.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:10 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3569/48867 objects degraded (7.303%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:10.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/128494899' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T10:37:10.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T10:37:10.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:10.846 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:10.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:10 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3569/48867 objects degraded (7.303%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/128494899' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T10:37:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T10:37:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:10 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3569/48867 objects degraded (7.303%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/128494899' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T10:37:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"} : dispatch 2024-09-21T10:37:10.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:10.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:10.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:10.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:11.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:11.112 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:11.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:11.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.246 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:11.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:11.308 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 47 pg_num to 36 2024-09-21T10:37:11.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:11.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:11.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:11.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[105288]: pgmap v2159: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 950 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 3569/45639 objects degraded (7.820%); 3005/45639 objects misplaced (6.584%); 3.6 MiB/s, 75 objects/s recovering 2024-09-21T10:37:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"}]': finished 2024-09-21T10:37:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[105288]: osdmap e1342: 8 total, 8 up, 7 in 2024-09-21T10:37:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:11.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:11.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:11.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[101432]: pgmap v2159: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 950 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 3569/45639 objects degraded (7.820%); 3005/45639 objects misplaced (6.584%); 3.6 MiB/s, 75 objects/s recovering 2024-09-21T10:37:11.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"}]': finished 2024-09-21T10:37:11.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[101432]: osdmap e1342: 8 total, 8 up, 7 in 2024-09-21T10:37:11.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:11.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:11.660 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:11 smithi145 ceph-mon[89359]: pgmap v2159: 119 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 950 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 741 op/s; 3569/45639 objects degraded (7.820%); 3005/45639 objects misplaced (6.584%); 3.6 MiB/s, 75 objects/s recovering 2024-09-21T10:37:11.661 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "36"}]': finished 2024-09-21T10:37:11.661 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:11 smithi145 ceph-mon[89359]: osdmap e1342: 8 total, 8 up, 7 in 2024-09-21T10:37:11.661 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:11.661 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:11.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:11.814 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:11.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:11.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:11.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:11.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:12.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:12.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:12.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:12.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:12.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:12.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:12.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:12.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:12.505 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:37:12.505 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:37:12.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:12.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:37:12.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:12.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "36"} : dispatch 2024-09-21T10:37:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:37:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:37:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:12.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:12.871 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:12.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:12.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:12.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:12.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:12.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:13.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:13.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:13.153 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:13.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:13.242 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:13.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:13.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:13.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:13.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:13.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:13 smithi119 ceph-mon[105288]: pgmap v2161: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 809 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 834 op/s; 3569/38895 objects degraded (9.176%); 2370/38895 objects misplaced (6.093%); 2.0 MiB/s, 67 objects/s recovering 2024-09-21T10:37:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:13 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T10:37:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:13 smithi119 ceph-mon[105288]: osdmap e1343: 8 total, 8 up, 7 in 2024-09-21T10:37:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:13 smithi119 ceph-mon[101432]: pgmap v2161: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 809 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 834 op/s; 3569/38895 objects degraded (9.176%); 2370/38895 objects misplaced (6.093%); 2.0 MiB/s, 67 objects/s recovering 2024-09-21T10:37:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:13 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T10:37:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:13 smithi119 ceph-mon[101432]: osdmap e1343: 8 total, 8 up, 7 in 2024-09-21T10:37:13.627 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:13.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:13.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:13.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:13 smithi145 ceph-mon[89359]: pgmap v2161: 119 pgs: 1 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 112 active+clean; 809 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 834 op/s; 3569/38895 objects degraded (9.176%); 2370/38895 objects misplaced (6.093%); 2.0 MiB/s, 67 objects/s recovering 2024-09-21T10:37:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:13 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T10:37:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:13 smithi145 ceph-mon[89359]: osdmap e1343: 8 total, 8 up, 7 in 2024-09-21T10:37:13.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:13.888 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:13.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:13.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:14.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:14.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:14.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:14.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:37:14.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:37:14.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:14.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:14.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:14.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:14.482 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:37:14.500 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:14 smithi145 ceph-mon[89359]: 47.b continuing backfill to osd.2 from (1321'8872,1342'14956] MIN to 1342'14956 2024-09-21T10:37:14.500 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:14 smithi145 ceph-mon[89359]: osdmap e1344: 8 total, 8 up, 7 in 2024-09-21T10:37:14.500 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:14 smithi145 ceph-mon[89359]: pgmap v2164: 129 pgs: 3 peering, 6 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 113 active+clean; 638 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 3289/30654 objects degraded (10.729%); 1811/30654 objects misplaced (5.908%) 2024-09-21T10:37:14.519 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:14.540 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:14.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:14.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:14.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:14.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:14.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:14.787 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:37:14] "GET /metrics HTTP/1.1" 200 36913 "" "Prometheus/2.51.0" 2024-09-21T10:37:14.788 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-mon[101432]: 47.b continuing backfill to osd.2 from (1321'8872,1342'14956] MIN to 1342'14956 2024-09-21T10:37:14.788 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-mon[101432]: osdmap e1344: 8 total, 8 up, 7 in 2024-09-21T10:37:14.788 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-mon[101432]: pgmap v2164: 129 pgs: 3 peering, 6 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 113 active+clean; 638 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 3289/30654 objects degraded (10.729%); 1811/30654 objects misplaced (5.908%) 2024-09-21T10:37:14.788 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-mon[105288]: 47.b continuing backfill to osd.2 from (1321'8872,1342'14956] MIN to 1342'14956 2024-09-21T10:37:14.788 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-mon[105288]: osdmap e1344: 8 total, 8 up, 7 in 2024-09-21T10:37:14.788 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:14 smithi119 ceph-mon[105288]: pgmap v2164: 129 pgs: 3 peering, 6 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 5 active+remapped+backfill_wait, 113 active+clean; 638 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 3289/30654 objects degraded (10.729%); 1811/30654 objects misplaced (5.908%) 2024-09-21T10:37:14.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:14.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:14.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:14.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:14.944 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:15.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:15.030 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:15.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:15.154 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:15.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:15.277 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:15.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:15.379 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:15.501 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:37:15.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:15.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:15.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:15.647 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:15.654 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:15 smithi145 ceph-mon[89359]: osdmap e1345: 8 total, 8 up, 7 in 2024-09-21T10:37:15.654 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:15 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3289/30654 objects degraded (10.729%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:15.655 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4196430459' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:37:15.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:15.782 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:15 smithi119 ceph-mon[101432]: osdmap e1345: 8 total, 8 up, 7 in 2024-09-21T10:37:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:15 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3289/30654 objects degraded (10.729%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4196430459' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:37:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:15 smithi119 ceph-mon[105288]: osdmap e1345: 8 total, 8 up, 7 in 2024-09-21T10:37:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:15 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3289/30654 objects degraded (10.729%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:37:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4196430459' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:37:15.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:15.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:15.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:16.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:16.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:16.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:16.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:16.272 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:37:16.272 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:37:16.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:16.311 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:16.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:16.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:16.552 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:16.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:37:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4196430459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:37:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[101432]: 47.1b continuing backfill to osd.2 from (1321'8872,1342'14956] MIN to 1342'14956 2024-09-21T10:37:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[101432]: osdmap e1346: 8 total, 8 up, 7 in 2024-09-21T10:37:16.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[101432]: pgmap v2167: 129 pgs: 3 peering, 3 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 116 active+clean; 568 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 3289/27294 objects degraded (12.050%); 1478/27294 objects misplaced (5.415%) 2024-09-21T10:37:16.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:37:16.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4196430459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:37:16.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[105288]: 47.1b continuing backfill to osd.2 from (1321'8872,1342'14956] MIN to 1342'14956 2024-09-21T10:37:16.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[105288]: osdmap e1346: 8 total, 8 up, 7 in 2024-09-21T10:37:16.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:16 smithi119 ceph-mon[105288]: pgmap v2167: 129 pgs: 3 peering, 3 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 116 active+clean; 568 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 3289/27294 objects degraded (12.050%); 1478/27294 objects misplaced (5.415%) 2024-09-21T10:37:16.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:16.705 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:16 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:37:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4196430459' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:37:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:16 smithi145 ceph-mon[89359]: 47.1b continuing backfill to osd.2 from (1321'8872,1342'14956] MIN to 1342'14956 2024-09-21T10:37:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:16 smithi145 ceph-mon[89359]: osdmap e1346: 8 total, 8 up, 7 in 2024-09-21T10:37:16.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:16 smithi145 ceph-mon[89359]: pgmap v2167: 129 pgs: 3 peering, 3 unknown, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 116 active+clean; 568 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 3289/27294 objects degraded (12.050%); 1478/27294 objects misplaced (5.415%) 2024-09-21T10:37:16.808 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:16.832 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:16.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:16.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:37:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:16.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:16.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:17.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:17.127 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:17.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:17.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:17.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:17.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:17.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:17.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:17.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:17.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:17.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:17.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:17.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:17.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:17.869 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:17.881 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:37:17.881 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:37:17.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:17.932 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:37:17.932 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:37:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:17.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:17.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:17.975 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:37:17.975 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:37:17.977 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:17.978 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:18.011 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:37:18.011 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:37:18.039 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:37:18.040 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:37:18.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:18.075 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:37:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:37:18.088 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:37:18.088 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:37:18.088 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:37:18.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-21T10:37:18.105 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:37:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:37:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:18.140 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:37:18.140 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:37:18.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:18.250 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:18.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:18.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:18.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:18.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:18.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:18.759 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:37:18.774 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:18.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:18.902 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:18.926 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:18.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:19.027 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:19.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:18 smithi119 ceph-mon[101432]: pgmap v2168: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 117 active+clean; 553 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 3153/26598 objects degraded (11.854%); 1478/26598 objects misplaced (5.557%) 2024-09-21T10:37:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4142243642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:18 smithi119 ceph-mon[105288]: pgmap v2168: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 117 active+clean; 553 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 3153/26598 objects degraded (11.854%); 1478/26598 objects misplaced (5.557%) 2024-09-21T10:37:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4142243642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:19.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:19.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:19.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:18 smithi145 ceph-mon[89359]: pgmap v2168: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 peering, 2 unknown, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 117 active+clean; 553 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 3153/26598 objects degraded (11.854%); 1478/26598 objects misplaced (5.557%) 2024-09-21T10:37:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4142243642' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:19.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:19.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:19.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:19.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:19.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:19.564 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:19.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:19.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:19.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:19.837 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:19.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:19.965 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:19.996 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:19.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:19.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:20.097 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 3153/26598 objects degraded (11.854%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:37:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 3153/26598 objects degraded (11.854%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:37:20.125 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:20.195 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 3153/26598 objects degraded (11.854%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T10:37:20.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:20.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:20.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:20.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:20.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:20.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:37:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:37:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:37:20.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:20.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:20.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:20.820 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:20.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:20.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:20.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:21.041 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:20 smithi145 ceph-mon[89359]: pgmap v2169: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 495 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 163 B/s rd, 77 MiB/s wr, 2.98k op/s; 3153/23820 objects degraded (13.237%); 1554/23820 objects misplaced (6.524%); 9.6 MiB/s, 225 objects/s recovering 2024-09-21T10:37:21.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:21.088 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:21.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:20 smithi119 ceph-mon[105288]: pgmap v2169: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 495 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 163 B/s rd, 77 MiB/s wr, 2.98k op/s; 3153/23820 objects degraded (13.237%); 1554/23820 objects misplaced (6.524%); 9.6 MiB/s, 225 objects/s recovering 2024-09-21T10:37:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:20 smithi119 ceph-mon[101432]: pgmap v2169: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 495 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 163 B/s rd, 77 MiB/s wr, 2.98k op/s; 3153/23820 objects degraded (13.237%); 1554/23820 objects misplaced (6.524%); 9.6 MiB/s, 225 objects/s recovering 2024-09-21T10:37:21.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:21.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:21.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:21.365 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:21.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:21.467 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:21.500 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:21.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:21.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:21.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:21.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:21.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:21.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:22.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:22.056 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:22.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:22.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:22.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:22.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:22.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:22.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:22.467 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:22.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:22.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:22.692 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:22.732 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:22.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:22.834 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:22.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:22.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:22.989 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:22.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:22.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:23.091 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:22 smithi119 ceph-mon[101432]: pgmap v2170: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 333 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 127 B/s rd, 60 MiB/s wr, 2.54k op/s; 3153/16047 objects degraded (19.649%); 883/16047 objects misplaced (5.503%); 7.5 MiB/s, 211 objects/s recovering 2024-09-21T10:37:23.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:22 smithi119 ceph-mon[105288]: pgmap v2170: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 333 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 127 B/s rd, 60 MiB/s wr, 2.54k op/s; 3153/16047 objects degraded (19.649%); 883/16047 objects misplaced (5.503%); 7.5 MiB/s, 211 objects/s recovering 2024-09-21T10:37:23.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:23.234 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:23.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:22 smithi145 ceph-mon[89359]: pgmap v2170: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 333 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 127 B/s rd, 60 MiB/s wr, 2.54k op/s; 3153/16047 objects degraded (19.649%); 883/16047 objects misplaced (5.503%); 7.5 MiB/s, 211 objects/s recovering 2024-09-21T10:37:23.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:23.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:23.492 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:23.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:23.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:23.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:23.679 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Removed 124826 objects 2024-09-21T10:37:23.679 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Clean up completed and total clean up time :137.707 2024-09-21T10:37:23.767 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:23.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:23.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:23.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:23.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:24.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:24.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:24.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:24.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:24.277 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:24.379 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:24.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:24.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:24.501 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:24.522 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:24.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:24.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:24.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:24.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:24.769 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:24.859 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:37:24] "GET /metrics HTTP/1.1" 200 36892 "" "Prometheus/2.51.0" 2024-09-21T10:37:24.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:24.893 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:24.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:24.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:25.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:25.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-mon[105288]: pgmap v2171: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 139 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 110 B/s rd, 52 MiB/s wr, 2.54k op/s; 2769/6726 objects degraded (41.169%); 508/6726 objects misplaced (7.553%); 7.8 MiB/s, 204 objects/s recovering 2024-09-21T10:37:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-mon[105288]: 47.11 scrub starts 2024-09-21T10:37:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 2769/6726 objects degraded (41.169%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:37:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-mon[101432]: pgmap v2171: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 139 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 110 B/s rd, 52 MiB/s wr, 2.54k op/s; 2769/6726 objects degraded (41.169%); 508/6726 objects misplaced (7.553%); 7.8 MiB/s, 204 objects/s recovering 2024-09-21T10:37:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-mon[101432]: 47.11 scrub starts 2024-09-21T10:37:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:24 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 2769/6726 objects degraded (41.169%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:37:25.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:25.142 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:25.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:25.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:24 smithi145 ceph-mon[89359]: pgmap v2171: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 139 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 110 B/s rd, 52 MiB/s wr, 2.54k op/s; 2769/6726 objects degraded (41.169%); 508/6726 objects misplaced (7.553%); 7.8 MiB/s, 204 objects/s recovering 2024-09-21T10:37:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:24 smithi145 ceph-mon[89359]: 47.11 scrub starts 2024-09-21T10:37:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:24 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 2769/6726 objects degraded (41.169%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:37:25.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:25.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:25.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:25.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:25.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:25.634 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:25.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:25.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:25.759 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:25.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:25.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:25.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:25.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:26.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:26.110 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:26.132 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:26.234 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:26.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:26.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:26.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:26.480 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:26.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:26.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:26.627 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:26.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:26.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:26.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:26.976 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:26.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:26.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:27.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:27.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:26 smithi119 ceph-mon[101432]: pgmap v2172: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 31 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 100 B/s rd, 47 MiB/s wr, 2.37k op/s; 2769/1560 objects degraded (177.500%); 61/1560 objects misplaced (3.910%); 7.0 MiB/s, 201 objects/s recovering 2024-09-21T10:37:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:26 smithi119 ceph-mon[105288]: pgmap v2172: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 31 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 100 B/s rd, 47 MiB/s wr, 2.37k op/s; 2769/1560 objects degraded (177.500%); 61/1560 objects misplaced (3.910%); 7.0 MiB/s, 201 objects/s recovering 2024-09-21T10:37:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:27.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:27.244 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:26 smithi145 ceph-mon[89359]: pgmap v2172: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 31 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 100 B/s rd, 47 MiB/s wr, 2.37k op/s; 2769/1560 objects degraded (177.500%); 61/1560 objects misplaced (3.910%); 7.0 MiB/s, 201 objects/s recovering 2024-09-21T10:37:27.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:27.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:27.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:27.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:27.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:27.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:27.642 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-21T10:37:27.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-21T10:37:27.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:27.742 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:27.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:27.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:27.869 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:27.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:27.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:27.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:28.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:28.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:28.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:28.237 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:28.339 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:28.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:28.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:28.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:28.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:28.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:28.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:28.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:28.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:28.860 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:28.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:28.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:28.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:29.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:29.107 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:29.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:29.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:28 smithi145 ceph-mon[89359]: pgmap v2173: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 23 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 85 B/s rd, 40 MiB/s wr, 2.02k op/s; 121/1179 objects degraded (10.263%); 61/1179 objects misplaced (5.174%); 6.2 MiB/s, 177 objects/s recovering 2024-09-21T10:37:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:28 smithi145 ceph-mon[89359]: osdmap e1347: 8 total, 8 up, 7 in 2024-09-21T10:37:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:28 smithi145 ceph-mon[89359]: 47.1 scrub starts 2024-09-21T10:37:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:28 smithi145 ceph-mon[89359]: 47.1 scrub ok 2024-09-21T10:37:29.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:29.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[105288]: pgmap v2173: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 23 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 85 B/s rd, 40 MiB/s wr, 2.02k op/s; 121/1179 objects degraded (10.263%); 61/1179 objects misplaced (5.174%); 6.2 MiB/s, 177 objects/s recovering 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[105288]: osdmap e1347: 8 total, 8 up, 7 in 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[105288]: 47.1 scrub starts 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[105288]: 47.1 scrub ok 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[101432]: pgmap v2173: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 23 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 85 B/s rd, 40 MiB/s wr, 2.02k op/s; 121/1179 objects degraded (10.263%); 61/1179 objects misplaced (5.174%); 6.2 MiB/s, 177 objects/s recovering 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[101432]: osdmap e1347: 8 total, 8 up, 7 in 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[101432]: 47.1 scrub starts 2024-09-21T10:37:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:28 smithi119 ceph-mon[101432]: 47.1 scrub ok 2024-09-21T10:37:29.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:29.481 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:29.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:29.606 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:29.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:29.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:29.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:29.858 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:29.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:29.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:29.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:30.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:30.108 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:30.113 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:30.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:37:30.148 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:30.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:29 smithi119 ceph-mon[105288]: osdmap e1348: 8 total, 8 up, 7 in 2024-09-21T10:37:30.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 121/1179 objects degraded (10.263%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:37:30.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:29 smithi119 ceph-mon[101432]: osdmap e1348: 8 total, 8 up, 7 in 2024-09-21T10:37:30.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 121/1179 objects degraded (10.263%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:37:30.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:30.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:29 smithi145 ceph-mon[89359]: osdmap e1348: 8 total, 8 up, 7 in 2024-09-21T10:37:30.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 121/1179 objects degraded (10.263%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-21T10:37:30.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:30.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:30.337 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:30.359 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:30.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:30.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:37:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:37:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:37:30.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:30.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:30.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:30.738 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:30.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:30.861 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:30.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:30.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:30.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:30.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:31.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:31.112 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:31.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:31.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:30 smithi145 ceph-mon[89359]: pgmap v2176: 129 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 11 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 1.9 MiB/s, 109 objects/s recovering 2024-09-21T10:37:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:30 smithi145 ceph-mon[89359]: 47.0 scrub starts 2024-09-21T10:37:31.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:31.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:30 smithi119 ceph-mon[105288]: pgmap v2176: 129 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 11 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 1.9 MiB/s, 109 objects/s recovering 2024-09-21T10:37:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:30 smithi119 ceph-mon[105288]: 47.0 scrub starts 2024-09-21T10:37:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:30 smithi119 ceph-mon[101432]: pgmap v2176: 129 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 2 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 120 active+clean; 11 MiB data, 5.3 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 413 op/s; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 1.9 MiB/s, 109 objects/s recovering 2024-09-21T10:37:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:30 smithi119 ceph-mon[101432]: 47.0 scrub starts 2024-09-21T10:37:31.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:31.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:31.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:31.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:31.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:31.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:31.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:31.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:31.866 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:31.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:31.993 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:31.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:32.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:32.121 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:32.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:32.244 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:32.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:32.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:32.369 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:32.470 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:32.492 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:32.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:32.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:32.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:32.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:32.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:32.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:32.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:32.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:33.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:33.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:33.154 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:32 smithi119 ceph-mon[105288]: pgmap v2177: 129 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 21 op/s; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 360 KiB/s, 172 objects/s recovering 2024-09-21T10:37:33.154 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:33.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:32 smithi119 ceph-mon[101432]: pgmap v2177: 129 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 21 op/s; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 360 KiB/s, 172 objects/s recovering 2024-09-21T10:37:33.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:33.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:33.237 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:32 smithi145 ceph-mon[89359]: pgmap v2177: 129 pgs: 1 active+recovery_wait+remapped, 1 active+recovery_wait, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 0 B/s wr, 21 op/s; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 360 KiB/s, 172 objects/s recovering 2024-09-21T10:37:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:33.339 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:33.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:33.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:33.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:33.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:33.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:33.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:33.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:33.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:33.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:33.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:34.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:34.142 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:34.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:34.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:34.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:34.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:34.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:34.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:34.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:34.625 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:34.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:34.752 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:34.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:34.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:34.857 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:37:34] "GET /metrics HTTP/1.1" 200 36903 "" "Prometheus/2.51.0" 2024-09-21T10:37:34.875 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:34.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:34.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:35.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:35.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-mon[105288]: pgmap v2178: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 360 KiB/s, 295 objects/s recovering 2024-09-21T10:37:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 121/570 objects degraded (21.228%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:37:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-mon[105288]: osdmap e1349: 8 total, 8 up, 7 in 2024-09-21T10:37:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-mon[101432]: pgmap v2178: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 360 KiB/s, 295 objects/s recovering 2024-09-21T10:37:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 121/570 objects degraded (21.228%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:37:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:34 smithi119 ceph-mon[101432]: osdmap e1349: 8 total, 8 up, 7 in 2024-09-21T10:37:35.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.154 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:35.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:35.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:35.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:34 smithi145 ceph-mon[89359]: pgmap v2178: 129 pgs: 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 5 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 58/570 objects misplaced (10.175%); 360 KiB/s, 295 objects/s recovering 2024-09-21T10:37:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:34 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 121/570 objects degraded (21.228%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:37:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:34 smithi145 ceph-mon[89359]: osdmap e1349: 8 total, 8 up, 7 in 2024-09-21T10:37:35.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:35.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:35.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:35.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:35.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:35.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:35.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:35.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:35.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:35.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:36.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:36.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[105288]: 47.21 scrub starts 2024-09-21T10:37:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[105288]: 1.13 scrub starts 2024-09-21T10:37:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[105288]: 1.13 scrub ok 2024-09-21T10:37:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[105288]: 47.21 scrub ok 2024-09-21T10:37:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[105288]: osdmap e1350: 8 total, 8 up, 7 in 2024-09-21T10:37:36.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[101432]: 47.21 scrub starts 2024-09-21T10:37:36.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[101432]: 1.13 scrub starts 2024-09-21T10:37:36.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[101432]: 1.13 scrub ok 2024-09-21T10:37:36.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[101432]: 47.21 scrub ok 2024-09-21T10:37:36.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:35 smithi119 ceph-mon[101432]: osdmap e1350: 8 total, 8 up, 7 in 2024-09-21T10:37:36.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:36.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:36.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:36.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:36.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:35 smithi145 ceph-mon[89359]: 47.21 scrub starts 2024-09-21T10:37:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:35 smithi145 ceph-mon[89359]: 1.13 scrub starts 2024-09-21T10:37:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:35 smithi145 ceph-mon[89359]: 1.13 scrub ok 2024-09-21T10:37:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:35 smithi145 ceph-mon[89359]: 47.21 scrub ok 2024-09-21T10:37:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:35 smithi145 ceph-mon[89359]: osdmap e1350: 8 total, 8 up, 7 in 2024-09-21T10:37:36.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:36.400 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:37:36.400 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:37:36.401 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-21T10:37:36.401 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:37:36.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:36.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:36.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:36.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:36.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:36.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:36.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:36.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:36.782 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:36.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:36.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:36.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:37.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:37.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:37.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:37.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:36 smithi119 ceph-mon[101432]: pgmap v2181: 129 pgs: 4 peering, 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 20/570 objects misplaced (3.509%); 0 B/s, 334 objects/s recovering 2024-09-21T10:37:37.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:36 smithi119 ceph-mon[101432]: 1.3 scrub starts 2024-09-21T10:37:37.151 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:36 smithi119 ceph-mon[101432]: 1.3 scrub ok 2024-09-21T10:37:37.151 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:36 smithi119 ceph-mon[105288]: pgmap v2181: 129 pgs: 4 peering, 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 20/570 objects misplaced (3.509%); 0 B/s, 334 objects/s recovering 2024-09-21T10:37:37.152 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:36 smithi119 ceph-mon[105288]: 1.3 scrub starts 2024-09-21T10:37:37.152 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:36 smithi119 ceph-mon[105288]: 1.3 scrub ok 2024-09-21T10:37:37.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:37.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:37.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:37.287 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:36 smithi145 ceph-mon[89359]: pgmap v2181: 129 pgs: 4 peering, 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.3 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 20/570 objects misplaced (3.509%); 0 B/s, 334 objects/s recovering 2024-09-21T10:37:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:36 smithi145 ceph-mon[89359]: 1.3 scrub starts 2024-09-21T10:37:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:36 smithi145 ceph-mon[89359]: 1.3 scrub ok 2024-09-21T10:37:37.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:37.412 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:37.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:37.537 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:37.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:37.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:37.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:37.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:37.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:37.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:37.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:38.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:38.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:38.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:38.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:37 smithi119 ceph-mon[101432]: 47.d scrub starts 2024-09-21T10:37:38.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:37 smithi119 ceph-mon[101432]: 47.d scrub ok 2024-09-21T10:37:38.072 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:37 smithi119 ceph-mon[105288]: 47.d scrub starts 2024-09-21T10:37:38.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:37 smithi119 ceph-mon[105288]: 47.d scrub ok 2024-09-21T10:37:38.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.160 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:38.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:38.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:38.286 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:37 smithi145 ceph-mon[89359]: 47.d scrub starts 2024-09-21T10:37:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:37 smithi145 ceph-mon[89359]: 47.d scrub ok 2024-09-21T10:37:38.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.411 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:38.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:38.513 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:38.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:38.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:38.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:38.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:38.767 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:38.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:38.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:38.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:38.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:39.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:39.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:39.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:37:39.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:39.162 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 1.e scrub starts 2024-09-21T10:37:39.162 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 1.e scrub ok 2024-09-21T10:37:39.162 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: pgmap v2182: 129 pgs: 4 peering, 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 20/570 objects misplaced (3.509%); 0 B/s, 287 objects/s recovering 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 1.5 scrub starts 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 1.5 scrub ok 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 47.10 scrub starts 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 47.10 scrub ok 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 1.2 scrub starts 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[101432]: 1.2 scrub ok 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 1.e scrub starts 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 1.e scrub ok 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: pgmap v2182: 129 pgs: 4 peering, 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 20/570 objects misplaced (3.509%); 0 B/s, 287 objects/s recovering 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 1.5 scrub starts 2024-09-21T10:37:39.163 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 1.5 scrub ok 2024-09-21T10:37:39.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 47.10 scrub starts 2024-09-21T10:37:39.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 47.10 scrub ok 2024-09-21T10:37:39.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 1.2 scrub starts 2024-09-21T10:37:39.164 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:38 smithi119 ceph-mon[105288]: 1.2 scrub ok 2024-09-21T10:37:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:39.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:39.291 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 1.e scrub starts 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 1.e scrub ok 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: pgmap v2182: 129 pgs: 4 peering, 1 active+recovery_wait, 1 active+recovery_wait+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 121 active+clean; 11 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 121/570 objects degraded (21.228%); 20/570 objects misplaced (3.509%); 0 B/s, 287 objects/s recovering 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 1.5 scrub starts 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 1.5 scrub ok 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 47.10 scrub starts 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 47.10 scrub ok 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 1.2 scrub starts 2024-09-21T10:37:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:38 smithi145 ceph-mon[89359]: 1.2 scrub ok 2024-09-21T10:37:39.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:39.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:39.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:39.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:39.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:39.665 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:39.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:39.767 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:39.792 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:39.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:39.904 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_2' removed 2024-09-21T10:37:39.917 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:39.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:39.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:40.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:40.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:40.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:40.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:39 smithi145 ceph-mon[89359]: 47.e scrub starts 2024-09-21T10:37:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:39 smithi145 ceph-mon[89359]: 47.e scrub ok 2024-09-21T10:37:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:39 smithi145 ceph-mon[89359]: 47.2 scrub starts 2024-09-21T10:37:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:39 smithi145 ceph-mon[89359]: 47.2 scrub ok 2024-09-21T10:37:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/727513087' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:37:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:39 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[101432]: 47.e scrub starts 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[101432]: 47.e scrub ok 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[101432]: 47.2 scrub starts 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[101432]: 47.2 scrub ok 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/727513087' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:37:40.247 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[105288]: 47.e scrub starts 2024-09-21T10:37:40.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[105288]: 47.e scrub ok 2024-09-21T10:37:40.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[105288]: 47.2 scrub starts 2024-09-21T10:37:40.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[105288]: 47.2 scrub ok 2024-09-21T10:37:40.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/727513087' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:37:40.248 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:39 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:37:40.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:40.289 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:40.349 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:40.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:40.412 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:40.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:37:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:37:40.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:40.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:40.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:40.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:40.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:40.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:moved 6 / 279 (2.15054%) 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:avg 39.8571 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 7.91795 -> 6.33423 (expected baseline 5.84493) 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 26 -> 31 pgs (0.65233 -> 0.777778 * mean) 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 49 -> 47 pgs (1.22939 -> 1.17921 * mean) 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:37:40.908 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:37:40.909 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:37:40.909 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0083 2024-09-21T10:37:40.909 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0828 2024-09-21T10:37:40.909 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.7110 -> 0.7610 2024-09-21T10:37:40.909 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 weight 0.8746 -> 0.8868 2024-09-21T10:37:40.909 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2 weight 0.9500 -> 1.0000 2024-09-21T10:37:40.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:40.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:37:41.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:41.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:41.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: 47.f scrub starts 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: 47.f scrub ok 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: 47.3 scrub starts 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: 47.3 scrub ok 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: pgmap v2183: 129 pgs: 4 peering, 125 active+clean; 10 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 121/570 objects degraded (21.228%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:37:41.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: osdmap e1351: 8 total, 8 up, 7 in 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1367447433' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'49871','2':'65536','6':'58117'}"} : dispatch 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: 47.f scrub starts 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: 47.f scrub ok 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: 47.3 scrub starts 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: 47.3 scrub ok 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: pgmap v2183: 129 pgs: 4 peering, 125 active+clean; 10 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 121/570 objects degraded (21.228%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:37:41.112 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: osdmap e1351: 8 total, 8 up, 7 in 2024-09-21T10:37:41.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1367447433' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:37:41.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:37:41.113 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:40 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'49871','2':'65536','6':'58117'}"} : dispatch 2024-09-21T10:37:41.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:41.156 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:41.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:41.279 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:41.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: 47.f scrub starts 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: 47.f scrub ok 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: 47.3 scrub starts 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: 47.3 scrub ok 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: pgmap v2183: 129 pgs: 4 peering, 125 active+clean; 10 MiB data, 5.2 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 121/570 objects degraded (21.228%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:37:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:37:41.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: osdmap e1351: 8 total, 8 up, 7 in 2024-09-21T10:37:41.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1367447433' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:37:41.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:37:41.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:40 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'49871','2':'65536','6':'58117'}"} : dispatch 2024-09-21T10:37:41.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:41.404 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:41.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:41.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:41.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:41.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:41.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:41.753 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:41.775 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:41.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:41.899 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:41.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:42.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:42.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:42.041 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:37:42.041 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:37:42.041 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-21T10:37:42.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_3 16 2024-09-21T10:37:42.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:42.147 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:42.160 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[105288]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:37:42.160 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:37:42.160 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:37:42.160 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:37:42.160 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'49871','2':'65536','6':'58117'}"}]': finished 2024-09-21T10:37:42.160 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[105288]: osdmap e1352: 8 total, 8 up, 7 in 2024-09-21T10:37:42.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[101432]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:37:42.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:37:42.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:37:42.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:37:42.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'49871','2':'65536','6':'58117'}"}]': finished 2024-09-21T10:37:42.161 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:41 smithi119 ceph-mon[101432]: osdmap e1352: 8 total, 8 up, 7 in 2024-09-21T10:37:42.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:42.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:41 smithi145 ceph-mon[89359]: from='client.36672 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:37:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:41 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:37:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:41 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:37:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:41 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:37:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:41 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'49871','2':'65536','6':'58117'}"}]': finished 2024-09-21T10:37:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:41 smithi145 ceph-mon[89359]: osdmap e1352: 8 total, 8 up, 7 in 2024-09-21T10:37:42.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:42.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:42.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:42.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:42.622 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:42.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:42.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:42.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:42.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:42.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:42.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:42.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:42.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:43.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:43.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:42 smithi119 ceph-mon[105288]: pgmap v2186: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:43.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:42 smithi119 ceph-mon[105288]: osdmap e1353: 8 total, 8 up, 7 in 2024-09-21T10:37:43.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:42 smithi119 ceph-mon[101432]: pgmap v2186: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:43.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:42 smithi119 ceph-mon[101432]: osdmap e1353: 8 total, 8 up, 7 in 2024-09-21T10:37:43.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:43.142 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:43.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:43.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:42 smithi145 ceph-mon[89359]: pgmap v2186: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:42 smithi145 ceph-mon[89359]: osdmap e1353: 8 total, 8 up, 7 in 2024-09-21T10:37:43.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:43.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:43.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:43.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:43.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:43.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:43.592 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:37:43.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-21T10:37:43.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:43.656 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:43.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:43.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:43.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:43.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:43.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:44.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:44.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:44.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:44.153 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:44.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:44.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:44.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:44.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:44.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:44.532 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:44.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:44.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:44.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:44.787 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:37:44] "GET /metrics HTTP/1.1" 200 36903 "" "Prometheus/2.51.0" 2024-09-21T10:37:44.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:44.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:44.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:45.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:45.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:45.107 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:45.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:45.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:45.196 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:44 smithi119 ceph-mon[101432]: pgmap v2188: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:45.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:44 smithi119 ceph-mon[105288]: pgmap v2188: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:45.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:45.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:44 smithi145 ceph-mon[89359]: pgmap v2188: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:45.387 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:45.413 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:45.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:45.540 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:45.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:45.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:45.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:45.792 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:45.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:45.894 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:37:45.894 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:37:45.894 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:37:45.895 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:37:45.895 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:37:45.916 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:45.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:46.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:46.040 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:46.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:46.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:46.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:46.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:46.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:46.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:46.443 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:37:46.444 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:37:46.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:37:46.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:37:46.477 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:37:46.477 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:37:46.493 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:37:46.493 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:37:46.508 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:37:46.509 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:37:46.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:46.524 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:37:46.525 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:37:46.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:46.540 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:37:46.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:37:46.556 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:37:46.556 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:37:46.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:46.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:46.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:46.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:46.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:46.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:46.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:47.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:47.037 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:47.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:47.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:37:47.120 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:47.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:47.161 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:47.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:46 smithi119 ceph-mon[101432]: pgmap v2189: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:47.221 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/34075999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:47.221 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:46 smithi119 ceph-mon[105288]: pgmap v2189: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:47.222 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/34075999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:47.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:47.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:46 smithi145 ceph-mon[89359]: pgmap v2189: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/34075999' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:37:47.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:47.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.409 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:47.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:47.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:47.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:47.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:47.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:47.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:47.783 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:47.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:47.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:47.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:47.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:37:48.008 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:48.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:48.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:48.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:48.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:48.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:48.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:48.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:48.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:48.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:48.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:48.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:48.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:48.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:48.914 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:48.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:49.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:49.037 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:48 smithi145 ceph-mon[89359]: pgmap v2190: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:37:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:48 smithi145 ceph-mon[89359]: 1.14 scrub starts 2024-09-21T10:37:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:48 smithi145 ceph-mon[89359]: 1.14 scrub ok 2024-09-21T10:37:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[105288]: pgmap v2190: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:37:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[105288]: 1.14 scrub starts 2024-09-21T10:37:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[105288]: 1.14 scrub ok 2024-09-21T10:37:49.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:49.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[101432]: pgmap v2190: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s, 0 objects/s recovering 2024-09-21T10:37:49.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:49.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:37:49.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[101432]: 1.14 scrub starts 2024-09-21T10:37:49.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[101432]: 1.14 scrub ok 2024-09-21T10:37:49.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:37:49.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:49.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:49.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:49.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:49.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:49.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:49.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:49.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:49.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:49.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:49.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:49.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:49.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:49.797 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_3' created 2024-09-21T10:37:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.808 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:49.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:49.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:49.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:49.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:50.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/479050811' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:37:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:49 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:37:50.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/479050811' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:37:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:49 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:37:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/479050811' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:37:50.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:49 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-21T10:37:50.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:50.192 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:50.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:50.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:50.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:50.421 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:50.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:37:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:37:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:37:50.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:50.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:50.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-21T10:37:50.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:50.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:50.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:50.775 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:50.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[101432]: 1.11 scrub starts 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[101432]: 1.11 scrub ok 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[101432]: pgmap v2191: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[101432]: osdmap e1354: 8 total, 8 up, 7 in 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[105288]: 1.11 scrub starts 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[105288]: 1.11 scrub ok 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[105288]: pgmap v2191: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:37:50.804 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:37:50.805 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:50 smithi119 ceph-mon[105288]: osdmap e1354: 8 total, 8 up, 7 in 2024-09-21T10:37:50.828 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:50.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:50.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:50.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:50 smithi145 ceph-mon[89359]: 1.11 scrub starts 2024-09-21T10:37:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:50 smithi145 ceph-mon[89359]: 1.11 scrub ok 2024-09-21T10:37:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:50 smithi145 ceph-mon[89359]: pgmap v2191: 93 pgs: 93 active+clean; 608 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 0 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:37:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:50 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-21T10:37:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:50 smithi145 ceph-mon[89359]: osdmap e1354: 8 total, 8 up, 7 in 2024-09-21T10:37:51.053 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:51.074 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:51.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:51.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:51.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:51.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:51.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:51.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:moved 11 / 327 (3.36391%) 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 8.74584 -> 6.79736 (expected baseline 6.32778) 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 33 -> 44 pgs (0.706422 -> 0.941896 * mean) 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.3 with 59 -> 56 pgs (1.263 -> 1.19878 * mean) 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 1 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0076 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0756 2024-09-21T10:37:51.448 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.7610 -> 1.0000 2024-09-21T10:37:51.449 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:37:51.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:51.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:51.581 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:51.683 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:51.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:51.803 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:37:51.445+0000 7fc0be6c7640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-09-21T10:37:51.803 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:51 smithi119 ceph-mon[101432]: osdmap e1355: 8 total, 8 up, 7 in 2024-09-21T10:37:51.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:51 smithi119 ceph-mon[101432]: from='client.46598 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:37:51.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:51.831 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:51.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:51.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:37:51.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:51.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:52.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:52.082 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:52.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:52.116 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:51 smithi119 ceph-mon[105288]: osdmap e1355: 8 total, 8 up, 7 in 2024-09-21T10:37:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:51 smithi119 ceph-mon[105288]: from='client.46598 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:37:52.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:51 smithi145 ceph-mon[89359]: osdmap e1355: 8 total, 8 up, 7 in 2024-09-21T10:37:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:51 smithi145 ceph-mon[89359]: from='client.46598 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:37:52.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:52.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:52.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:52.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:52.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:52.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:52.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:52.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:52.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:52.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:52.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:52.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:52.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:52.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:53.038 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:52 smithi119 ceph-mon[101432]: pgmap v2194: 109 pgs: 9 creating+peering, 7 unknown, 93 active+clean; 501 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:53.038 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:52 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:37:53.038 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:52 smithi119 ceph-mon[101432]: osdmap e1356: 8 total, 8 up, 7 in 2024-09-21T10:37:53.038 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:52 smithi119 ceph-mon[105288]: pgmap v2194: 109 pgs: 9 creating+peering, 7 unknown, 93 active+clean; 501 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:53.038 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:52 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:37:53.039 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:52 smithi119 ceph-mon[105288]: osdmap e1356: 8 total, 8 up, 7 in 2024-09-21T10:37:53.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:53.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:53.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:52 smithi145 ceph-mon[89359]: pgmap v2194: 109 pgs: 9 creating+peering, 7 unknown, 93 active+clean; 501 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:52 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:37:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:52 smithi145 ceph-mon[89359]: osdmap e1356: 8 total, 8 up, 7 in 2024-09-21T10:37:53.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:53.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:53.331 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:37:53.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T10:37:53.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:53.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:53.554 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:53.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:53.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:53.704 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:53.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:53.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:53.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:53.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:53.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:54.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:54.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:54.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:54.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:54.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:54.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:54.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:54.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:54.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:54.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:54.558 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:54.582 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:54.684 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:54.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:54.806 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:37:54] "GET /metrics HTTP/1.1" 200 34465 "" "Prometheus/2.51.0" 2024-09-21T10:37:54.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:54.815 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:37:54.832 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:54.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:54.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:54.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:55.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.079 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:55.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-mon[105288]: pgmap v2196: 109 pgs: 9 creating+peering, 100 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1818792286' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:37:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:37:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-mon[101432]: pgmap v2196: 109 pgs: 9 creating+peering, 100 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1818792286' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:37:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:37:55.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:55.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:54 smithi145 ceph-mon[89359]: pgmap v2196: 109 pgs: 9 creating+peering, 100 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1818792286' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:37:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:37:55.307 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:55.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:55.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:55.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:55.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:55.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:55.583 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:55.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:55.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:55.808 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:55.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:55.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:55.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:55.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:55.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:56.056 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:55 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:37:56.057 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:37:56.057 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:55 smithi119 ceph-mon[105288]: osdmap e1357: 8 total, 8 up, 7 in 2024-09-21T10:37:56.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:55 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:37:56.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:37:56.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:55 smithi119 ceph-mon[101432]: osdmap e1357: 8 total, 8 up, 7 in 2024-09-21T10:37:56.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:56.082 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:56.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:56.204 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:56.271 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:55 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:37:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:37:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:55 smithi145 ceph-mon[89359]: osdmap e1357: 8 total, 8 up, 7 in 2024-09-21T10:37:56.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:56.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:56.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:56.451 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:56.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:56.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:56.676 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:56.702 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:56.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:56.828 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:56.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:56.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:56.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:57.000 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:56 smithi119 ceph-mon[105288]: pgmap v2198: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:57.000 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:56 smithi119 ceph-mon[101432]: pgmap v2198: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:57.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:57.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:57.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:57.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:56 smithi145 ceph-mon[89359]: pgmap v2198: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:57.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:57.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:57.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:57.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:57.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:57.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:57.680 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:37:57.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:37:57.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:57.822 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-21T10:37:57.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:57.836 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2451523308' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:37:57.836 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:57 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:37:57.836 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2451523308' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:37:57.836 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:57 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:37:57.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:57.944 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:37:57.944 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:37:57.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:57.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:58.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:37:58.087 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:37:58.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:58.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2451523308' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:37:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:57 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:37:58.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:58.333 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:58.343 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:37:58.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:58.457 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:58.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:58.585 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:58.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:58.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:58.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:58.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:58.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:58.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:37:58.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:37:58.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:58.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:37:59.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:59.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:37:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:58 smithi119 ceph-mon[105288]: pgmap v2199: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:58 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:37:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:58 smithi119 ceph-mon[105288]: osdmap e1358: 8 total, 8 up, 7 in 2024-09-21T10:37:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:58 smithi119 ceph-mon[101432]: pgmap v2199: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:58 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:37:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:58 smithi119 ceph-mon[101432]: osdmap e1358: 8 total, 8 up, 7 in 2024-09-21T10:37:59.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:37:59.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:37:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:58 smithi145 ceph-mon[89359]: pgmap v2199: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:37:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:58 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:37:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:58 smithi145 ceph-mon[89359]: osdmap e1358: 8 total, 8 up, 7 in 2024-09-21T10:37:59.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:59.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:59.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:37:59.453 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:59.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:37:59.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:37:59.577 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:37:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:37:59.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:59.705 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:59.806 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:37:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.833 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:59.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:37:59.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:37:59.933 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:37:59.957 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:37:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:37:59.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:00.029 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-21T10:38:00.031 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:38:00.031 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:38:00.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:00.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:00.121 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:hints = 1 2024-09-21T10:38:00.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:38:00.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object prefix: benchmark_data_smithi119_267254 2024-09-21T10:38:00.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:38:00.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:38:00.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:00.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:37:59 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:38:00.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:00.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:00.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:00.354 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:37:59 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:38:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:00.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:37:59 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:38:00.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:00.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:38:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:38:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:38:00.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:00.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:00.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.764 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:00.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:00.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:00.894 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:00 smithi119 ceph-mon[101432]: pgmap v2201: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:38:00.894 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:00 smithi119 ceph-mon[105288]: pgmap v2201: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:38:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:00.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:00.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:00.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:01.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:01.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 1 16 3798 3782 236.362 236.375 0.00315048 0.00421734 2024-09-21T10:38:01.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:01.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.176 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:01.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:01.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:00 smithi145 ceph-mon[89359]: pgmap v2201: 109 pgs: 109 active+clean; 475 KiB data, 4.7 GiB used, 621 GiB / 626 GiB avail 2024-09-21T10:38:01.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:01.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:01.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:01.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:01.540 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:38:01.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:38:01.542 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:38:01.542 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:38:01.542 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:38:01.542 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:38:01.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:01.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:01.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:01.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:01.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:01.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:01.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:01.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:01.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:01.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:02.091 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:02.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 2 15 7209 7194 224.796 213.25 0.00252418 0.00443916 2024-09-21T10:38:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:02.229 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:02.259 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3510617158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:02.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:02.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3510617158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:02.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3510617158' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:02.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:02.493 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:02.524 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:02.624 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:02.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:02.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:02.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:02.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:02.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:02.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:03.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:03.054 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:03.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 3 16 10307 10291 214.379 193.562 0.00247685 0.00465679 2024-09-21T10:38:03.156 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:03.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:02 smithi119 ceph-mon[101432]: pgmap v2202: 109 pgs: 109 active+clean; 57 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 7.1 MiB/s wr, 113 op/s 2024-09-21T10:38:03.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:03.194 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:03.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:02 smithi119 ceph-mon[105288]: pgmap v2202: 109 pgs: 109 active+clean; 57 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 7.1 MiB/s wr, 113 op/s 2024-09-21T10:38:03.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:03.194 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:03.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:02 smithi145 ceph-mon[89359]: pgmap v2202: 109 pgs: 109 active+clean; 57 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 7.1 MiB/s wr, 113 op/s 2024-09-21T10:38:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:03.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:03.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:03.434 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:03.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:03.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:03.604 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:03.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:03.742 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:03.844 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:03.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:03.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:03.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:04.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:04.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:04.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 4 16 13525 13509 211.061 201.125 0.00559367 0.00473091 2024-09-21T10:38:04.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:04.251 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:04.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:04.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:04.417 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:04.519 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:04.547 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:04.591 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:04.591 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:38:04] "GET /metrics HTTP/1.1" 200 36852 "" "Prometheus/2.51.0" 2024-09-21T10:38:04.592 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:04.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:04.687 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:04.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:04.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:04.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:04.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:04.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:05.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:05.155 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 5 16 16515 16499 206.221 186.875 0.00449117 0.00484194 2024-09-21T10:38:05.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:05.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:05.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:05.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:05.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:05.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:05.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:05.587 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:05 smithi119 ceph-mon[101432]: pgmap v2203: 109 pgs: 109 active+clean; 340 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 38 MiB/s wr, 608 op/s 2024-09-21T10:38:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:05 smithi119 ceph-mon[105288]: pgmap v2203: 109 pgs: 109 active+clean; 340 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 38 MiB/s wr, 608 op/s 2024-09-21T10:38:05.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:38:05.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:05.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:05 smithi145 ceph-mon[89359]: pgmap v2203: 109 pgs: 109 active+clean; 340 MiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 38 MiB/s wr, 608 op/s 2024-09-21T10:38:05.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:05.851 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:05.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:05.976 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:05.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:06.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:06.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:06.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 6 16 18936 18920 197.067 151.312 0.00433323 0.00506785 2024-09-21T10:38:06.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:06.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:06.344 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:06.371 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:06.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:06.473 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:06.511 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:06.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:06.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:06.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.774 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:06.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:06.875 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:06.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:06.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:07.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:07.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:07.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 7 16 21314 21298 190.145 148.625 0.00236771 0.00525356 2024-09-21T10:38:07.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:07.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:07.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:07.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:07.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:07.452 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:07.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:07.554 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:07.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:07 smithi119 ceph-mon[105288]: pgmap v2204: 109 pgs: 109 active+clean; 832 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:38:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:07 smithi119 ceph-mon[101432]: pgmap v2204: 109 pgs: 109 active+clean; 832 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:38:07.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:07.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:07 smithi145 ceph-mon[89359]: pgmap v2204: 109 pgs: 109 active+clean; 832 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:38:07.834 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:07.862 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:07.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:07.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:07.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:08.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:08.122 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 8 16 23865 23849 186.305 159.438 0.00594834 0.00536256 2024-09-21T10:38:08.137 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:08.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:08.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:08.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:08.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:08.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.540 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:08.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:08.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:08.659 INFO:teuthology.orchestra.run.smithi119.stdout:moved 4 / 327 (1.22324%) 2024-09-21T10:38:08.659 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:38:08.659 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 8.74584 -> 8.24126 (expected baseline 6.32778) 2024-09-21T10:38:08.659 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 33 -> 36 pgs (0.706422 -> 0.770642 * mean) 2024-09-21T10:38:08.659 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.3 with 59 -> 58 pgs (1.263 -> 1.24159 * mean) 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0113 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.1130 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.7610 -> 0.8110 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6 weight 0.8868 -> 0.9284 2024-09-21T10:38:08.660 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:38:08.691 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:08.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:08.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:08.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:08.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:08 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:38:08.652+0000 7fc0be6c7640 -1 mgr.server reply reply (2) No such file or directory no change 2024-09-21T10:38:08.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:08.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:08.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:09.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:09.071 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:09.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 9 16 25406 25390 176.304 96.3125 0.00136479 0.00547612 2024-09-21T10:38:09.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:09.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:09.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:09.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:09.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:09.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:09.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:09.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:09.604 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:09.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:09.732 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:09 smithi145 ceph-mon[89359]: pgmap v2205: 109 pgs: 109 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:38:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/841930237' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:09 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:09.834 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:09.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:09.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:09 smithi119 ceph-mon[101432]: pgmap v2205: 109 pgs: 109 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:38:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/841930237' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:09.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:09 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:09 smithi119 ceph-mon[105288]: pgmap v2205: 109 pgs: 109 active+clean; 884 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:38:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/841930237' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:09 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:38:09.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:09.998 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:09.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:10.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:10.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 10 16 27462 27446 171.522 128.5 0.00611221 0.00582456 2024-09-21T10:38:10.136 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:10.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:10.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:10.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:10.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:10.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:10.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:10.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:10.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:10.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:38:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:38:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:38:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:10 smithi145 ceph-mon[89359]: from='client.46631 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:38:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:10 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:38:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:38:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:10 smithi145 ceph-mon[89359]: osdmap e1359: 8 total, 8 up, 7 in 2024-09-21T10:38:10.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:10.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:10.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:10.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[101432]: from='client.46631 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:38:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:38:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:38:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[101432]: osdmap e1359: 8 total, 8 up, 7 in 2024-09-21T10:38:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[105288]: from='client.46631 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:38:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:38:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:38:10.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:10 smithi119 ceph-mon[105288]: osdmap e1359: 8 total, 8 up, 7 in 2024-09-21T10:38:10.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:10.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:10.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:11.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:11.071 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:11.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 11 16 29601 29585 168.082 133.688 0.00379809 0.00594441 2024-09-21T10:38:11.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:11.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:11.308 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:11.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:11.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:11.480 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:11.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:11.616 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:11.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:11.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:11 smithi145 ceph-mon[89359]: pgmap v2207: 109 pgs: 109 active+clean; 1.4 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-21T10:38:11.857 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:11.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:11 smithi119 ceph-mon[101432]: pgmap v2207: 109 pgs: 109 active+clean; 1.4 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-21T10:38:11.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:11 smithi119 ceph-mon[105288]: pgmap v2207: 109 pgs: 109 active+clean; 1.4 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-21T10:38:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:11.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:11.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:12.020 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:12.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:12.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 12 16 31635 31619 164.668 127.125 0.00697294 0.00606794 2024-09-21T10:38:12.161 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:12.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:12.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:12.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:12.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:12.517 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:12.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:12.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:12.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:12.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:12.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:12.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:12.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:12 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:38:12.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:12 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:38:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:12 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:38:12.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:12.945 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:12.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:13.047 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:13.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:13.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:13.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 13 15 33092 33077 159.01 91.125 0.00351973 0.00628415 2024-09-21T10:38:13.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:13.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:13.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:13.343 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:13.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:13.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:13.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:13.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:13.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:13 smithi119 ceph-mon[101432]: pgmap v2208: 109 pgs: 109 active+clean; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-21T10:38:13.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:13 smithi119 ceph-mon[105288]: pgmap v2208: 109 pgs: 109 active+clean; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-21T10:38:13.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:13.748 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:13 smithi145 ceph-mon[89359]: pgmap v2208: 109 pgs: 109 active+clean; 1.6 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 153 MiB/s wr, 2.45k op/s 2024-09-21T10:38:13.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:13.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:13.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:13.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.029 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:14.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:14.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 14 16 34817 34801 155.348 107.75 0.00639918 0.00642903 2024-09-21T10:38:14.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:14.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:14.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:14.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:14.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:14.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:38:14] "GET /metrics HTTP/1.1" 200 36852 "" "Prometheus/2.51.0" 2024-09-21T10:38:14.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:14.709 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:14.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:14.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:14.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:14.983 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:14.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:15.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:15.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:15.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:15.123 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 15 16 36898 36882 153.662 130.062 0.00447185 0.00650237 2024-09-21T10:38:15.222 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:15.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:15.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:15.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:15.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:15.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:15.568 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:15.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:15.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:15.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:15 smithi145 ceph-mon[89359]: pgmap v2209: 109 pgs: 109 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-21T10:38:15.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:15.850 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:15.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:15 smithi119 ceph-mon[101432]: pgmap v2209: 109 pgs: 109 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-21T10:38:15.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:15 smithi119 ceph-mon[105288]: pgmap v2209: 109 pgs: 109 active+clean; 1.8 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 150 MiB/s wr, 2.40k op/s 2024-09-21T10:38:15.952 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:15.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:15.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:15.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:16.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:16.123 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:16.124 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 16 16 38947 38931 152.054 128.062 0.00850113 0.00657046 2024-09-21T10:38:16.225 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:16.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:16.369 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:16.404 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:16.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:16.507 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:16.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:16.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:16.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:16.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:16.826 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:16.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:16.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:16.944 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:38:16.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-21T10:38:16.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:16.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:17.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:17.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:17.124 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 17 16 40880 40864 150.215 120.812 0.00866189 0.00665187 2024-09-21T10:38:17.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:17.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:17.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:17.394 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:17.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:17.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:17.529 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:17.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:17.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:17.690 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:38:17.690 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:38:17.690 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.690 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:41896 48.f e1359/1354 48:fd3bfbd9:::benchmark_data_smithi119_267254_object41895:head v 1359'2628, mlcod=1359'2626)", 2024-09-21T10:38:17.690 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:17.687456+0000", 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00063136699999999997, 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00065692699999999997, 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687456+0000", 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.691 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687456+0000", 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687457+0000", 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3799999999999997e-07 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687741+0000", 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028425299999999999 2024-09-21T10:38:17.692 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687749+0000", 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0010000000000001e-06 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687754+0000", 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2490000000000001e-06 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687876+0000", 2024-09-21T10:38:17.693 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121569 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687892+0000", 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5803000000000001e-05 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:41895 48.c e1359/1354)", 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:17.687770+0000", 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00031812300000000002, 2024-09-21T10:38:17.694 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00039515699999999998, 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687770+0000", 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687770+0000", 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:17.695 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687770+0000", 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9099999999999999e-07 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687794+0000", 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3638999999999998e-05 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:17.696 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687800+0000", 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.331e-06 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687805+0000", 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.507e-06 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:41900 48.0 e1359/1354 48:0ec24654:::benchmark_data_smithi119_267254_object41899:head v 1359'2576, mlcod=1359'2573)", 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:17.687625+0000", 2024-09-21T10:38:17.697 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00046305800000000002, 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000582644, 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687625+0000", 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687625+0000", 2024-09-21T10:38:17.698 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687625+0000", 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7899999999999998e-07 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687646+0000", 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1056e-05 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:17.699 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687655+0000", 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9649999999999997e-06 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687662+0000", 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4359999999999999e-06 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687761+0000", 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9547000000000001e-05 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:17.700 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:17.687781+0000", 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9270999999999999e-05 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 3 2024-09-21T10:38:17.701 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:38:17.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:17 smithi145 ceph-mon[89359]: pgmap v2210: 109 pgs: 109 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:38:17.822 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:17.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:17 smithi119 ceph-mon[101432]: pgmap v2210: 109 pgs: 109 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:38:17.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:17 smithi119 ceph-mon[105288]: pgmap v2210: 109 pgs: 109 active+clean; 2.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 134 MiB/s wr, 2.15k op/s 2024-09-21T10:38:17.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:17.961 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:17.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:18.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:18.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:18.124 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 18 16 42713 42697 148.234 114.562 0.00523564 0.00674155 2024-09-21T10:38:18.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:18.234 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:18.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:38:18.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:18.382 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:18.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:18.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:18.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:18.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:18.639 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:38:18.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:18.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:18.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:18.769 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:18.769 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:18.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:18.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:18.940 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:18.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:19.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:19.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:19.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:19.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 19 16 44442 44426 146.105 108.062 0.0131971 0.00683912 2024-09-21T10:38:19.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:19.211 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:19.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:19.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.356 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:19.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:19.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:19.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:19.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:19.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:19.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:19.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:19.761 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:19 smithi145 ceph-mon[89359]: pgmap v2211: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T10:38:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:19 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:19 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:19.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:19 smithi119 ceph-mon[101432]: pgmap v2211: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T10:38:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:19.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:19 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:19 smithi119 ceph-mon[105288]: pgmap v2211: 109 pgs: 109 active+clean; 2.2 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 133 MiB/s wr, 2.12k op/s 2024-09-21T10:38:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:19.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:19 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:19.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:19.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:19.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:20.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:20.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:20.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:38:20.125349+0000 min lat: 0.00108018 max lat: 0.372048 avg lat: 0.00691061 2024-09-21T10:38:20.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:38:20.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 20 16 46294 46278 144.587 115.75 0.0209951 0.00691061 2024-09-21T10:38:20.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:20.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:20.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:20.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:20.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:20.438 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:20.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:20.542 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:38:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:38:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:38:20.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:20 smithi119 ceph-mon[101432]: pgmap v2212: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:38:20.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:20 smithi119 ceph-mon[105288]: pgmap v2212: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:38:20.577 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:20.679 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:20.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:20.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:20 smithi145 ceph-mon[89359]: pgmap v2212: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 116 MiB/s wr, 1.86k op/s 2024-09-21T10:38:20.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:20.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:20.954 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:20.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:20.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:21.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:21.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 21 16 47899 47883 142.479 100.312 0.00748249 0.0070137 2024-09-21T10:38:21.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:21.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:21.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:21.369 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:21.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:21.543 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:21.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:21.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:21.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:21.808 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:21.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:21.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:21.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:21.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:22.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:22.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:22.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 22 16 49139 49123 139.525 77.5 0.0110646 0.00716232 2024-09-21T10:38:22.185 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:22.221 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:22.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.368 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:22.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:22.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:22.490 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:22.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:22.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:22.619 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:22.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:22.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:22.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:22.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:22.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:22.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:22.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:22.961 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:22.986 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:38:22.986 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:38:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:22.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:23.013 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:23.024 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:23.024 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:38:23.025 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:38:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:23.058 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:38:23.059 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:38:23.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:22 smithi119 ceph-mon[101432]: pgmap v2213: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-21T10:38:23.062 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:22 smithi119 ceph-mon[105288]: pgmap v2213: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-21T10:38:23.099 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:38:23.099 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:38:23.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:23.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 23 16 50471 50455 137.078 83.25 0.00734477 0.00728871 2024-09-21T10:38:23.133 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:38:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:38:23.158 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:38:23.158 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:38:23.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:23.187 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:38:23.187 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:38:23.217 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:38:23.217 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:38:23.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:22 smithi145 ceph-mon[89359]: pgmap v2213: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 115 MiB/s wr, 1.85k op/s 2024-09-21T10:38:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:23.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:23.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:23.448 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:23.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:23.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:23.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:23.829 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:23.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:23.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:23.968 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:23.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:24.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1678808340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:24.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1678808340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:24.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:24.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 24 16 52159 52143 135.762 105.5 0.00855418 0.00736112 2024-09-21T10:38:24.132 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:24.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:24.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:24.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:24.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1678808340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:24.370 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:24.406 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:24.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:24.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:24.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:38:24] "GET /metrics HTTP/1.1" 200 36865 "" "Prometheus/2.51.0" 2024-09-21T10:38:24.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:24.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:24.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:24.819 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:24.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:24.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:24.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:24.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:25.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:24 smithi119 ceph-mon[105288]: pgmap v2214: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T10:38:25.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:24 smithi119 ceph-mon[101432]: pgmap v2214: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T10:38:25.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:25.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:25.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:25.126 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 25 16 53741 53725 134.287 98.875 0.00261468 0.00744152 2024-09-21T10:38:25.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:25.229 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:24 smithi145 ceph-mon[89359]: pgmap v2214: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 116 MiB/s wr, 1.85k op/s 2024-09-21T10:38:25.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:25.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:25.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:25.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:25.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:25.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:25.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:25.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:25.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:25.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:25.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:25.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:25.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:26.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:26.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:26.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:38:26.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:26.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 26 16 55399 55383 133.107 103.625 0.00428459 0.00750836 2024-09-21T10:38:26.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:26.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:26.322 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:26.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:26.470 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:26.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:26.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:26.600 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:26.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:26.733 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:26.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:26.875 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:26.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:26.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:27.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:27.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:26 smithi119 ceph-mon[101432]: pgmap v2215: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-21T10:38:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:26 smithi119 ceph-mon[105288]: pgmap v2215: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-21T10:38:27.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 27 16 56854 56838 131.545 90.9375 0.0088246 0.00759703 2024-09-21T10:38:27.136 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:27.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:27.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:27.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:26 smithi145 ceph-mon[89359]: pgmap v2215: 109 pgs: 109 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 117 MiB/s wr, 1.88k op/s 2024-09-21T10:38:27.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:27.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:27.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:27.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:27.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:27.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:27.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:27.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:27.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:27.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:28.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:28.115 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:28.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 28 16 58368 58352 130.226 94.625 0.00977735 0.00767335 2024-09-21T10:38:28.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:28.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:28.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:28.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.375 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:28.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:28.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:28.516 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:28.617 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:28.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:28.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:28.769 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:28.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:28.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:28.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:29.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:29.030 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:28 smithi119 ceph-mon[101432]: pgmap v2216: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-21T10:38:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:28 smithi119 ceph-mon[105288]: pgmap v2216: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-21T10:38:29.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 29 16 59816 59800 128.856 90.5 0.00796096 0.0077551 2024-09-21T10:38:29.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:29.163 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:29.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:29.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:28 smithi145 ceph-mon[89359]: pgmap v2216: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 92 MiB/s wr, 1.48k op/s 2024-09-21T10:38:29.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:29.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:29.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:29.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:29.565 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:29.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:29.695 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:29.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:29.832 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:29.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:29.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:29.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:30.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:30.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:30.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 30 16 61387 61371 127.834 98.1875 0.0107221 0.00781829 2024-09-21T10:38:30.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:30.240 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:30.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:30.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:30.510 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:38:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:38:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:38:30.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:30.649 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:30.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.794 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:30.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:30.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:30.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:30.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:30.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:30.984 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:30 smithi119 ceph-mon[101432]: pgmap v2217: 109 pgs: 109 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:38:30.985 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:30 smithi119 ceph-mon[105288]: pgmap v2217: 109 pgs: 109 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:38:31.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:31.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:31.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 31 16 62791 62775 126.54 87.75 0.00561988 0.00789731 2024-09-21T10:38:31.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:31.217 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:30 smithi145 ceph-mon[89359]: pgmap v2217: 109 pgs: 109 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-21T10:38:31.319 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:31.345 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:31.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:31.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:31.447 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:38:31.447 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:38:31.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:31.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:31.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:31.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:31.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:31.755 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:31.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:31.854 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:38:31.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:31.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:31.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:31.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:31.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:32.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:32.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:31 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2153427033' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:38:32.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:31 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:38:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:31 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2153427033' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:38:32.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:31 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:38:32.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:32.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 32 16 63947 63931 124.844 72.25 0.00286269 0.007975 2024-09-21T10:38:32.138 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:32.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:32.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:31 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2153427033' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:38:32.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:31 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:38:32.362 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:32.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:32.492 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:32.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:32.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:32.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:32.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:32.791 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:32.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:32.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:38:32.893 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:32.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:32.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:33.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:33.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[101432]: pgmap v2218: 109 pgs: 109 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-21T10:38:33.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[101432]: osdmap e1360: 8 total, 8 up, 7 in 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[105288]: pgmap v2218: 109 pgs: 109 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[105288]: osdmap e1360: 8 total, 8 up, 7 in 2024-09-21T10:38:33.120 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:33.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:33.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 33 16 65060 65044 123.169 69.5625 0.0325274 0.00811273 2024-09-21T10:38:33.224 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:33.252 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:32 smithi145 ceph-mon[89359]: pgmap v2218: 109 pgs: 109 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-21T10:38:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:32 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:38:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:32 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:38:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:32 smithi145 ceph-mon[89359]: osdmap e1360: 8 total, 8 up, 7 in 2024-09-21T10:38:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:33.301 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:33.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:33.406 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:33.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:33.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:33.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.686 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:33.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:33.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:33.841 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:33.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:moved 3 / 327 (0.917431%) 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 8.74584 -> 8.34462 (expected baseline 6.32778) 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 33 -> 36 pgs (0.706422 -> 0.770642 * mean) 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.3 with 59 -> 59 pgs (1.263 -> 1.263 * mean) 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:38:33.866 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:38:33.867 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:38:33.867 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0254 2024-09-21T10:38:33.867 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.2539 2024-09-21T10:38:33.867 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.7610 -> 0.8110 2024-09-21T10:38:33.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:33.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:33.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:34.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:34.100 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:33 smithi119 ceph-mon[101432]: from='client.55831 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:38:34.101 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'53147'}"} : dispatch 2024-09-21T10:38:34.101 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:33 smithi119 ceph-mon[105288]: from='client.55831 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:38:34.101 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'53147'}"} : dispatch 2024-09-21T10:38:34.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:34.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 34 16 66386 66370 121.983 82.875 0.0164575 0.00819292 2024-09-21T10:38:34.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:34.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:33 smithi145 ceph-mon[89359]: from='client.55831 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:38:34.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'53147'}"} : dispatch 2024-09-21T10:38:34.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:34.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:34.501 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:34.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:34.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:34.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:34.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:34.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:34.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:38:34] "GET /metrics HTTP/1.1" 200 36866 "" "Prometheus/2.51.0" 2024-09-21T10:38:34.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:34.950 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:34.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:35.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:35.082 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:35.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 35 16 67888 67872 121.18 93.875 0.00932657 0.00824727 2024-09-21T10:38:35.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:35.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:34 smithi145 ceph-mon[89359]: pgmap v2220: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-21T10:38:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'53147'}"}]': finished 2024-09-21T10:38:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:34 smithi145 ceph-mon[89359]: osdmap e1361: 8 total, 8 up, 7 in 2024-09-21T10:38:35.319 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:35.350 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-mon[105288]: pgmap v2220: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-21T10:38:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'53147'}"}]': finished 2024-09-21T10:38:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-mon[105288]: osdmap e1361: 8 total, 8 up, 7 in 2024-09-21T10:38:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-mon[101432]: pgmap v2220: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 93 MiB/s wr, 1.50k op/s 2024-09-21T10:38:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'53147'}"}]': finished 2024-09-21T10:38:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:34 smithi119 ceph-mon[101432]: osdmap e1361: 8 total, 8 up, 7 in 2024-09-21T10:38:35.422 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:35.452 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:35.483 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:35.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:35.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:35.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:35.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:35.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:38:35.863 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:35.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:35.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:35.917 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:38:35.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T10:38:35.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:36.024 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:36.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:35 smithi119 ceph-mon[101432]: osdmap e1362: 8 total, 8 up, 7 in 2024-09-21T10:38:36.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:35 smithi119 ceph-mon[105288]: osdmap e1362: 8 total, 8 up, 7 in 2024-09-21T10:38:36.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:36.129 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 36 16 69266 69250 120.206 86.125 0.00895394 0.00831372 2024-09-21T10:38:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:36.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:35 smithi145 ceph-mon[89359]: osdmap e1362: 8 total, 8 up, 7 in 2024-09-21T10:38:36.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:36.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:36.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:36.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:36.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:36.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:36.759 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.802 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:36.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:36.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:36.931 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:36.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:36.948 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:38:36.948 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:38:36.967 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:38:36.968 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:38:37.009 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:38:37.009 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:38:37.009 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:36 smithi119 ceph-mon[105288]: pgmap v2223: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:38:37.010 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/987818533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:37.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:36 smithi119 ceph-mon[101432]: pgmap v2223: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:38:37.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/987818533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:37.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:37.046 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:38:37.047 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:38:37.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:37.079 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:38:37.080 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:38:37.109 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:38:37.109 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:38:37.127 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 37 16 70607 70591 119.222 83.8125 0.0153405 0.00838149 2024-09-21T10:38:37.139 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:38:37.139 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:38:37.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:37.173 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:38:37.173 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:38:37.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:36 smithi145 ceph-mon[89359]: pgmap v2223: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:38:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/987818533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:38:37.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:37.340 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:37.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:37.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:37.482 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:37.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:37.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:37.605 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:37.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:37.734 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:37.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:37.872 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:37.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:37.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:38.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:38.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:38.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 38 16 71860 71844 118.146 78.3125 0.00815983 0.00845929 2024-09-21T10:38:38.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:38.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:38.276 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:37 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-21T10:38:38.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:37 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T10:38:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:37 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-21T10:38:38.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:37 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T10:38:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:37 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]} : dispatch 2024-09-21T10:38:38.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:37 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]} : dispatch 2024-09-21T10:38:38.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:38.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:38.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:38.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:38.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:38.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:38.686 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:38.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:38.815 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:38.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:38.938 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:38.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:39.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:39.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:39.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 39 16 73124 73108 117.142 79 0.00732459 0.0085315 2024-09-21T10:38:39.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:39.217 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:38 smithi145 ceph-mon[89359]: pgmap v2224: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:38:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-21T10:38:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T10:38:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:38 smithi145 ceph-mon[89359]: osdmap e1363: 8 total, 8 up, 7 in 2024-09-21T10:38:39.319 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:39.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[105288]: pgmap v2224: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[105288]: osdmap e1363: 8 total, 8 up, 7 in 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[101432]: pgmap v2224: 109 pgs: 109 active+clean; 4.1 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 0 B/s, 0 objects/s recovering 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 5]}]': finished 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-21T10:38:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:38 smithi119 ceph-mon[101432]: osdmap e1363: 8 total, 8 up, 7 in 2024-09-21T10:38:39.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:39.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:39.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:39.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:39.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:39.750 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:39.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:39.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:39.873 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:39.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:39.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:39.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:40.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:40.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:40.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:38:40.127211+0000 min lat: 0.00108018 max lat: 0.372048 avg lat: 0.00856527 2024-09-21T10:38:40.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:38:40.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 40 16 73929 73913 115.471 50.3125 0.00257221 0.00856527 2024-09-21T10:38:40.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:39 smithi145 ceph-mon[89359]: osdmap e1364: 8 total, 8 up, 7 in 2024-09-21T10:38:40.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:40.260 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:40.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:39 smithi119 ceph-mon[105288]: osdmap e1364: 8 total, 8 up, 7 in 2024-09-21T10:38:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:39 smithi119 ceph-mon[101432]: osdmap e1364: 8 total, 8 up, 7 in 2024-09-21T10:38:40.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:40.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:40.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:38:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:38:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:38:40.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:40.672 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:40.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:40.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:40.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:41.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:41.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:41.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 41 16 74952 74936 114.214 63.9375 0.0487063 0.00874305 2024-09-21T10:38:41.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:41.202 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:41.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:40 smithi145 ceph-mon[89359]: pgmap v2227: 109 pgs: 2 peering, 107 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:38:41.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:41.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.346 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:41.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:40 smithi119 ceph-mon[101432]: pgmap v2227: 109 pgs: 2 peering, 107 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:38:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:40 smithi119 ceph-mon[105288]: pgmap v2227: 109 pgs: 2 peering, 107 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:38:41.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:41.484 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:41.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:41.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:41.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.748 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:41.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:41.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:41.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:41.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:41.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:42.023 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:42.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:42.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 42 16 76358 76342 113.587 87.875 0.0121411 0.00879951 2024-09-21T10:38:42.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:42.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:42.287 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:42.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:42.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:42.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:42.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:42.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:42.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:42.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:42.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:42.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:42.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:42.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:43.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:43.096 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:43.128 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 43 16 77590 77574 112.736 77 0.0168294 0.0088649 2024-09-21T10:38:43.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:43.234 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:43.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:42 smithi145 ceph-mon[89359]: pgmap v2228: 109 pgs: 2 peering, 107 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:38:43.307 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:42 smithi119 ceph-mon[105288]: pgmap v2228: 109 pgs: 2 peering, 107 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:38:43.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:42 smithi119 ceph-mon[101432]: pgmap v2228: 109 pgs: 2 peering, 107 active+clean; 4.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 78 MiB/s wr, 1.24k op/s; 3 B/s, 0 objects/s recovering 2024-09-21T10:38:43.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:43.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:43.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.514 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:43.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:43.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:43.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:43.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:43.780 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:43.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:43.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:43.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:44.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:44.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:44.129 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 44 16 78908 78892 112.045 82.375 0.0109842 0.00891893 2024-09-21T10:38:44.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:44.179 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:44.281 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:44.309 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:44.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:44.444 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:44.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:44.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:44.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:38:44] "GET /metrics HTTP/1.1" 200 36866 "" "Prometheus/2.51.0" 2024-09-21T10:38:44.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:44.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:44.823 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:44.855 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:44.957 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:44.988 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:44.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:45.023 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:44 smithi119 ceph-mon[105288]: pgmap v2229: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:45.023 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:44 smithi119 ceph-mon[101432]: pgmap v2229: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:45.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:45.116 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:45.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:45.129 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 45 16 80216 80200 111.371 81.75 0.0154524 0.00897409 2024-09-21T10:38:45.195 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:38:45.195 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:38:45.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55558 47.2 47:47e6821c:::benchmark_data_smithi119_238957_object55557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406694+0000", 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 282.78271670999999, 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0386517150000001, 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.196 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55558 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406694+0000", 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406694+0000", 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406696+0000", 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8679999999999999e-06 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406739+0000", 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3121999999999998e-05 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406752+0000", 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3101e-05 2024-09-21T10:38:45.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406760+0000", 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8059999999999995e-06 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.521176+0000", 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.11441596499999999 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.521184+0000", 2024-09-21T10:38:45.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8680000000000005e-06 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.445172+0000", 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.923987856 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.445345+0000", 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000173129 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55566 47.17 47:ee11e6db:::benchmark_data_smithi119_238957_object55565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406775+0000", 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 282.782635529, 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.074695475, 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55566 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.201 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406775+0000", 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406775+0000", 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406775+0000", 2024-09-21T10:38:45.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7899999999999999e-07 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406796+0000", 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0157e-05 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406802+0000", 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9530000000000002e-06 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406808+0000", 2024-09-21T10:38:45.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1100000000000002e-06 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.532613+0000", 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.12580530000000001 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.532620+0000", 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8730000000000001e-06 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.476517+0000", 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.94389676099999997 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for degraded object", 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.478070+0000", 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015535970000000001 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481398+0000", 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033275240000000001 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481470+0000", 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2720999999999997e-05 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55577 47.7 47:e42c9d18:::benchmark_data_smithi119_238957_object55576:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.472961+0000", 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 282.71644901899998, 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0052156160000001, 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.206 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55577 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.472961+0000", 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.472961+0000", 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.472963+0000", 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.826e-06 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.473003+0000", 2024-09-21T10:38:45.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0073000000000001e-05 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.473017+0000", 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3740000000000001e-05 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.473028+0000", 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0856000000000001e-05 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.532383+0000", 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.059354900000000002 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.532393+0000", 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7489999999999993e-06 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.470261+0000", 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.93786851699999996 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.470409+0000", 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014749199999999999 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,4", 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.470566+0000", 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015735799999999999 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.473598+0000", 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030318039999999999 2024-09-21T10:38:45.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.475328+0000", 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00172999 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.478090+0000", 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027623069999999999 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.478135+0000", 2024-09-21T10:38:45.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4864000000000003e-05 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.478177+0000", 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.214e-05 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:69028 47.10 47.ec858290 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e1295)", 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:10.467107+0000", 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 274.72230365899998, 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99923426500000001, 2024-09-21T10:38:45.213 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69028 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.467107+0000", 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.467107+0000", 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.467108+0000", 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.531e-06 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.467133+0000", 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4609000000000001e-05 2024-09-21T10:38:45.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.467145+0000", 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2172e-05 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.467153+0000", 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8290000000000008e-06 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.469497+0000", 2024-09-21T10:38:45.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023436170000000001 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.469510+0000", 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3361e-05 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:11.466328+0000", 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.99681837799999995 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:11.466341+0000", 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2768e-05 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:69063 47.10 47.8127e470 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1295)", 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:10.495413+0000", 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 274.69399746800002, 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.97096388600000005, 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.218 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 69063 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495413+0000", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495413+0000", 2024-09-21T10:38:45.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495413+0000", 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3900000000000005e-07 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495436+0000", 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.23e-05 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495442+0000", 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5060000000000001e-06 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495448+0000", 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5729999999999998e-06 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495536+0000", 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7989000000000001e-05 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:10.495543+0000", 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9290000000000001e-06 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:11.466369+0000", 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.97082602699999998 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:11.466377+0000", 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0229999999999992e-06 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.222 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.223 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.223 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:197708 47.c 47:3d3a25d1:::benchmark_data_smithi119_238957_object72879:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:38:45.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:18.916597+0000", 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 146.27281382999999, 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16163629900000001, 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.224 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 197708 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.916597+0000", 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.916597+0000", 2024-09-21T10:38:45.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.916598+0000", 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1170000000000001e-06 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.916602+0000", 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4349999999999999e-06 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.226 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.916613+0000", 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.128e-05 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.916621+0000", 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.289e-06 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.920920+0000", 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042990820000000001 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.921079+0000", 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015956999999999999 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,6", 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.921212+0000", 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000132626 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.925188+0000", 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039758529999999997 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:18.925301+0000", 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000113307 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:19.078146+0000", 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15284445999999999 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:19.078187+0000", 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1718000000000003e-05 2024-09-21T10:38:45.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:19.078233+0000", 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5562000000000002e-05 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:225023 47.16 47:6b4265f8:::benchmark_data_smithi119_238957_object100194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1337)", 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:54.175560+0000", 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 111.013850385, 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16525937199999999, 2024-09-21T10:38:45.230 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 225023 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.175560+0000", 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.175560+0000", 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.175561+0000", 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2410000000000001e-06 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.175565+0000", 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8430000000000003e-06 2024-09-21T10:38:45.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.175574+0000", 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2140000000000002e-06 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.175581+0000", 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1909999999999997e-06 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.309652+0000", 2024-09-21T10:38:45.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.13407128200000001 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.309900+0000", 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024853899999999999 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.310593+0000", 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00069308599999999998 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.317387+0000", 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006793805 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.321638+0000", 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004250467 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.340081+0000", 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018443721999999999 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.340154+0000", 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2150999999999994e-05 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:54.340819+0000", 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066583099999999997 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:226782 47.18 47:1bf6c0eb:::benchmark_data_smithi119_238957_object101953:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:56.336251+0000", 2024-09-21T10:38:45.236 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 108.853159421, 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23808374400000001, 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 226782 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336251+0000", 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336251+0000", 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336252+0000", 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.277e-06 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336256+0000", 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.45e-06 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336267+0000", 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1474000000000001e-05 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336275+0000", 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3089999999999998e-06 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336392+0000", 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011767799999999999 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336520+0000", 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012770300000000001 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3,4", 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.336627+0000", 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106801 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.345118+0000", 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.008491017 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.345158+0000", 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0148000000000003e-05 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.345953+0000", 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00079519499999999995 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.574153+0000", 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.228200128 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.574215+0000", 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2070999999999996e-05 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.574335+0000", 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119493 2024-09-21T10:38:45.242 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:226798 47.8 47:154659e1:::benchmark_data_smithi119_238957_object101969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:56.350721+0000", 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 108.838689448, 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.22379323100000001, 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.243 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 226798 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350721+0000", 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350721+0000", 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350722+0000", 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.108e-06 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350728+0000", 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5389999999999998e-06 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350743+0000", 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5752000000000001e-05 2024-09-21T10:38:45.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350751+0000", 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7090000000000002e-06 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350857+0000", 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106401 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.350974+0000", 2024-09-21T10:38:45.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116601 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3,4", 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.351094+0000", 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119732 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.354000+0000", 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029066500000000002 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.354969+0000", 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00096830799999999997 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.355178+0000", 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020897600000000001 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.574376+0000", 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.219197908 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.574422+0000", 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5898e-05 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.574514+0000", 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2649e-05 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:226809 47.a 47:5d16b31e:::benchmark_data_smithi119_238957_object101980:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:38:45.249 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:56.370419+0000", 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 108.818991013, 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20683858199999999, 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 226809 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.250 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370419+0000", 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370419+0000", 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370420+0000", 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9800000000000003e-07 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370423+0000", 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8729999999999999e-06 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370430+0000", 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.533e-06 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370434+0000", 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9929999999999997e-06 2024-09-21T10:38:45.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370477+0000", 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3699000000000003e-05 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370644+0000", 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016674300000000001 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:38:45.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.370759+0000", 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011477299999999999 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.373935+0000", 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031762090000000002 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.373951+0000", 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5475e-05 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.577170+0000", 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.203219124 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.577219+0000", 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9461e-05 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.577258+0000", 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8900999999999999e-05 2024-09-21T10:38:45.255 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:226814 47.a 47:54bb68a6:::benchmark_data_smithi119_238957_object101985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:56.384744+0000", 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 108.804666889, 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19725078200000001, 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.256 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 226814 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384744+0000", 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384744+0000", 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.257 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384745+0000", 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4160000000000001e-06 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384750+0000", 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.95e-06 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384760+0000", 2024-09-21T10:38:45.258 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6909999999999992e-06 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384766+0000", 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.494e-06 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384821+0000", 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5158999999999998e-05 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.259 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.384968+0000", 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014667799999999999 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.385094+0000", 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012646099999999999 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387921+0000", 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028262669999999999 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.260 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.391155+0000", 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003234598 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.581904+0000", 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19074917899999999 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.581953+0000", 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8983999999999997e-05 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.261 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.581994+0000", 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0905000000000002e-05 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:226816 47.a 47:56924858:::benchmark_data_smithi119_238957_object101987:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:56.387919+0000", 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 108.80149187799999, 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.194183206, 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.262 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 226816 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387919+0000", 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.263 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387919+0000", 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387920+0000", 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.759e-06 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387927+0000", 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9269999999999999e-06 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.264 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387939+0000", 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1582e-05 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387946+0000", 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7469999999999999e-06 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.265 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.387993+0000", 2024-09-21T10:38:45.266 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7160000000000002e-05 2024-09-21T10:38:45.266 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.266 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.266 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.266 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.388076+0000", 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3374999999999994e-05 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.388150+0000", 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3553999999999994e-05 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.390542+0000", 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023922259999999999 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.267 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.391326+0000", 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00078370000000000002 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.582029+0000", 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19070350799999999 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.582070+0000", 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1090999999999997e-05 2024-09-21T10:38:45.268 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:56.582102+0000", 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1576999999999999e-05 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:74613 48.7 48:e4f80a60:::benchmark_data_smithi119_267254_object74612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.688164+0000", 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.5012461249999998, 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.17490684200000001, 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.269 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 74613 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688164+0000", 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.270 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688164+0000", 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688166+0000", 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.666e-06 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688200+0000", 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3720000000000002e-05 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.271 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688212+0000", 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1945e-05 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688219+0000", 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.644e-06 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688291+0000", 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2089999999999996e-05 2024-09-21T10:38:45.272 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688692+0000", 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040066899999999998 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.688821+0000", 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012890300000000001 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.694821+0000", 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059997649999999998 2024-09-21T10:38:45.273 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701832+0000", 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0070111490000000004 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.862971+0000", 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16113870699999999 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.863022+0000", 2024-09-21T10:38:45.274 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0967000000000002e-05 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.863071+0000", 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9617000000000003e-05 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:74619 48.7 48:e64c3b62:::benchmark_data_smithi119_267254_object74618:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.695297+0000", 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4941137910000002, 2024-09-21T10:38:45.275 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.17297446599999999, 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 74619 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695297+0000", 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.276 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695297+0000", 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695298+0000", 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8619999999999999e-06 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695320+0000", 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1882000000000002e-05 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.277 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695333+0000", 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2446000000000001e-05 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695339+0000", 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6100000000000002e-06 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695681+0000", 2024-09-21T10:38:45.278 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00034115799999999999 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695823+0000", 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014228400000000001 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695971+0000", 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014789999999999999 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.862934+0000", 2024-09-21T10:38:45.279 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.166963378 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.864751+0000", 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001816914 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868171+0000", 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034197720000000002 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.280 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868230+0000", 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9061000000000003e-05 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868271+0000", 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1199000000000003e-05 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:74621 48.7 48:e7c74823:::benchmark_data_smithi119_267254_object74620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.695595+0000", 2024-09-21T10:38:45.281 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4938158460000004, 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.172789575, 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 74621 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695595+0000", 2024-09-21T10:38:45.282 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695595+0000", 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695595+0000", 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0299999999999998e-07 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.283 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695618+0000", 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2745999999999999e-05 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695629+0000", 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0705e-05 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.695635+0000", 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3940000000000001e-06 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.284 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698752+0000", 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031164130000000002 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698901+0000", 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000149815 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.699118+0000", 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000217106 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.285 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.863084+0000", 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.163965049 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.864774+0000", 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001690275 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868306+0000", 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003532444 2024-09-21T10:38:45.286 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868351+0000", 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4317000000000003e-05 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868384+0000", 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3608000000000001e-05 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.287 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:74628 48.7 48:ec3d0770:::benchmark_data_smithi119_267254_object74627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.698929+0000", 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4904811179999999, 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.176833412, 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 74628 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.288 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698929+0000", 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698929+0000", 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698931+0000", 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.341e-06 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.289 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698954+0000", 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2877000000000001e-05 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698964+0000", 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0376e-05 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.698973+0000", 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8619999999999992e-06 2024-09-21T10:38:45.290 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701879+0000", 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029057739999999999 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.702006+0000", 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012761000000000001 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.702126+0000", 2024-09-21T10:38:45.291 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011966 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.866359+0000", 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16423300599999999 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.866371+0000", 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2517999999999999e-05 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.292 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.875649+0000", 2024-09-21T10:38:45.293 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092778449999999998 2024-09-21T10:38:45.293 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:45.293 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.293 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.293 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.875706+0000", 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6501e-05 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.875763+0000", 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7042e-05 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:74627 48.f e1364/1354 48:f18df3e1:::benchmark_data_smithi119_267254_object74626:head v 1364'4660, mlcod=1364'4658)", 2024-09-21T10:38:45.294 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.701855+0000", 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4875550640000004, 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16631768899999999, 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701855+0000", 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.295 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701855+0000", 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701856+0000", 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0979999999999999e-06 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701882+0000", 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5877e-05 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.296 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701894+0000", 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1829e-05 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701901+0000", 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0910000000000004e-06 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.702192+0000", 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000291095 2024-09-21T10:38:45.297 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.702209+0000", 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6699000000000001e-05 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868130+0000", 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16592069600000001 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868173+0000", 2024-09-21T10:38:45.298 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3303999999999997e-05 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:74630 48.6 e1364/1354 48:64e53894:::benchmark_data_smithi119_267254_object74629:head v 1364'4710, mlcod=1364'4708)", 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.701916+0000", 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4874947399999998, 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16156706000000001, 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701916+0000", 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701916+0000", 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701916+0000", 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.03e-07 2024-09-21T10:38:45.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701950+0000", 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4116999999999999e-05 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701956+0000", 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2999999999999998e-06 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.701961+0000", 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7439999999999998e-06 2024-09-21T10:38:45.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.702232+0000", 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00027053999999999999 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.860924+0000", 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15869267400000001 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.863440+0000", 2024-09-21T10:38:45.302 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025157759999999999 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.863483+0000", 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2605999999999998e-05 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:74632 48.f e1364/1354 48:f2da6939:::benchmark_data_smithi119_267254_object74631:head v 1364'4661, mlcod=1364'4658)", 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.703895+0000", 2024-09-21T10:38:45.303 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4855150630000002, 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16431285400000001, 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.703895+0000", 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.703895+0000", 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.304 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.703896+0000", 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0999999999999997e-07 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.703919+0000", 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3192000000000001e-05 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.860918+0000", 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15699844900000001 2024-09-21T10:38:45.305 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.860939+0000", 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1447e-05 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.862935+0000", 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019959299999999999 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.862951+0000", 2024-09-21T10:38:45.306 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6297999999999999e-05 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868177+0000", 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052253040000000001 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.868208+0000", 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1424000000000003e-05 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.307 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:74635 48.7 48:ea16eed1:::benchmark_data_smithi119_267254_object74634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:40.705619+0000", 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 4.4837919099999999, 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.182835412, 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 74635 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.308 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.705619+0000", 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.705619+0000", 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.705620+0000", 2024-09-21T10:38:45.309 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9360000000000002e-06 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.705645+0000", 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4890999999999999e-05 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.860937+0000", 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.155292021 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:45.310 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.860950+0000", 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2692e-05 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.878541+0000", 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.017591032999999999 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.878661+0000", 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011981699999999999 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.311 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.878779+0000", 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000117864 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.884546+0000", 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057672060000000004 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.884645+0000", 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8855000000000001e-05 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.312 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.888344+0000", 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036995840000000001 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.888411+0000", 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6538000000000006e-05 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.888454+0000", 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2975000000000001e-05 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.313 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.314 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:45.314 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:45.314 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:38:45.314 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:45.315 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:44 smithi145 ceph-mon[89359]: pgmap v2229: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:45.317 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:45.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:45.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:38:45.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:45.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:45.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:45.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:45.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:45.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:45.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:45.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:45.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:45.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:45.901 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:45.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:46.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:46.037 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:46.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:46.129 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 46 15 81341 81326 110.48 70.375 0.0262363 0.00904615 2024-09-21T10:38:46.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:46.176 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:46.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:46.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:46.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.434 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:46.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:46.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:46.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:38:46.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:46.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:46.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:46.708 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:46.810 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:46.837 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:46.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:46.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:46.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:47.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:47.110 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:46 smithi119 ceph-mon[101432]: pgmap v2230: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:47.110 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:46 smithi119 ceph-mon[105288]: pgmap v2230: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:47.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:47.133 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 47 16 82547 82531 109.721 75.3125 0.0100609 0.009108 2024-09-21T10:38:47.221 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:47.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:47.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:46 smithi145 ceph-mon[89359]: pgmap v2230: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:47.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:47.384 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:47.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.522 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:47.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:47.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:47.644 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:47.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:47.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:47.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:47.910 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:47.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:48.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:48.134 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 48 16 83584 83568 108.785 64.8125 0.0149081 0.00918715 2024-09-21T10:38:48.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.183 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:48.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:48.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:47 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:47 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:47 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:38:48.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:48.428 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:48.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:48.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:48.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:48.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:48.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:48.810 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:48.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:48.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:48.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:49.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:49.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:49.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:49.134 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 49 16 84441 84425 107.658 53.5625 0.00976596 0.00928393 2024-09-21T10:38:49.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:49.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:49.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:48 smithi145 ceph-mon[89359]: pgmap v2231: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:49.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:49.349 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:48 smithi119 ceph-mon[101432]: pgmap v2231: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:48 smithi119 ceph-mon[105288]: pgmap v2231: 109 pgs: 109 active+clean; 4.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s; 2 B/s, 0 objects/s recovering 2024-09-21T10:38:49.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.482 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:49.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:49.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:49.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:49.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:49.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:49.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:49.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:49.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:50.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.070 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:50.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:50.137 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 50 16 85502 85486 106.825 66.3125 0.0100639 0.00935565 2024-09-21T10:38:50.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:50.230 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:50.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:50.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:50.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:50.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:50.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:50.498 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:38:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:38:50.599 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:50.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:50.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:50.762 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:50.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:50.908 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:50.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:51.010 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:51.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[101432]: pgmap v2232: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:38:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:38:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:38:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[105288]: pgmap v2232: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:38:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:38:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:38:51.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.137 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 51 16 86671 86655 106.163 73.0625 0.0139368 0.00941474 2024-09-21T10:38:51.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:51.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:51.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-mon[89359]: pgmap v2232: 109 pgs: 109 active+clean; 5.1 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:38:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:38:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:38:51.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.331 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:51.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:51.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:51.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:51.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:51.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:51.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:51.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:51.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:51.739 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:51.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:51.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:51.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:51.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:51.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:51.979 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:51.979 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:38:52.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:52.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:52.137 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 52 16 87819 87803 105.501 71.75 0.0162977 0.00947338 2024-09-21T10:38:52.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:52.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:52.277 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:52.379 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:52.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:52.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:52.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:52.547 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:52.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:52.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:52.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:52.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:52.816 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:52.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:52.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:52.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:52.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:53.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:53.086 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:53.100 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:52 smithi119 ceph-mon[105288]: pgmap v2233: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:38:53.100 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:52 smithi119 ceph-mon[101432]: pgmap v2233: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:38:53.137 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 53 16 88942 88926 104.835 70.1875 0.0145061 0.0095331 2024-09-21T10:38:53.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:53.224 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:52 smithi145 ceph-mon[89359]: pgmap v2233: 109 pgs: 109 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-21T10:38:53.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:53.356 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:53.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:53.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:53.494 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:53.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:53.626 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:53.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:53.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:53.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:53.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3855672265' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:53.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:53 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:53.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:53.832 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:38:53.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:53.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:53.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:53.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:54.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:54.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3855672265' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:53 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:54.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:54.137 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 54 16 90131 90115 104.27 74.3125 0.00870009 0.00958573 2024-09-21T10:38:54.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:54.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3855672265' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:53 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:38:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:38:54.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:54.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:54.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:54.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:54.546 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:54.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:54.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:54.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:54.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:54.811 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:38:54] "GET /metrics HTTP/1.1" 200 36866 "" "Prometheus/2.51.0" 2024-09-21T10:38:54.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:54.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:54.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:54.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:55.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:55.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:55.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[101432]: pgmap v2234: 109 pgs: 109 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[101432]: osdmap e1365: 8 total, 8 up, 7 in 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[105288]: pgmap v2234: 109 pgs: 109 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:38:55.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:54 smithi119 ceph-mon[105288]: osdmap e1365: 8 total, 8 up, 7 in 2024-09-21T10:38:55.137 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 55 16 90998 90982 103.359 54.1875 0.0264138 0.0096677 2024-09-21T10:38:55.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:55.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:54 smithi145 ceph-mon[89359]: pgmap v2234: 109 pgs: 109 active+clean; 5.4 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:38:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:54 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:38:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:54 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:38:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:38:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:54 smithi145 ceph-mon[89359]: osdmap e1365: 8 total, 8 up, 7 in 2024-09-21T10:38:55.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:55.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:55.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:55.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:55.491 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:55.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:55.633 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:55.734 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:55.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:55.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:55.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:55.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:56.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:56.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:56.138 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 56 16 92099 92083 102.742 68.8125 0.0110168 0.00972814 2024-09-21T10:38:56.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:56.177 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:56.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:56.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:56.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:56.454 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:56.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:56.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:56.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:38:56.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:38:56.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:56.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:56 smithi119 ceph-mon[101432]: pgmap v2236: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:38:56.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:56 smithi119 ceph-mon[105288]: pgmap v2236: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:38:56.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:56.871 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:56.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:56.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:57.012 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:57.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:38:57.146 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 57 16 93228 93212 102.178 70.5625 0.00901278 0.00978233 2024-09-21T10:38:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:57.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:38:57.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:57.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:56 smithi145 ceph-mon[89359]: pgmap v2236: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:38:57.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:57.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:57.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:57.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.558 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:57.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:57.660 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:57.698 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:57.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:38:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.843 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:57.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:38:57.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:57.987 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:57.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:58.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:58.128 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:58.140 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 58 16 94384 94368 101.657 72.25 0.0104921 0.00983201 2024-09-21T10:38:58.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.274 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:58.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:58.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:58.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:38:58.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:58.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:58.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:94856 48.4 48:2dc1b75e:::benchmark_data_smithi119_267254_object94855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:58.571833+0000", 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012434153999999999, 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012468809000000001, 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:38:58.646 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 94856 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.571833+0000", 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:58.647 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.571833+0000", 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.571834+0000", 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2750000000000001e-06 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.571929+0000", 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4736999999999998e-05 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.648 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.571943+0000", 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4239999999999999e-05 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.571953+0000", 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0929999999999992e-06 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.572021+0000", 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8034999999999995e-05 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.649 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.572146+0000", 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012593700000000001 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.572263+0000", 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116952 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.579258+0000", 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0069946050000000001 2024-09-21T10:38:58.650 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.579670+0000", 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00041202200000000001 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:94861 48.4 48:26d00004:::benchmark_data_smithi119_267254_object94860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:58.581513+0000", 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0027538580000000001, 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028648050000000002, 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:38:58.651 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 94861 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581513+0000", 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.652 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581513+0000", 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581516+0000", 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2069999999999998e-06 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581539+0000", 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3102e-05 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581558+0000", 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9126e-05 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581565+0000", 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.542e-06 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581622+0000", 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6653999999999997e-05 2024-09-21T10:38:58.654 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581753+0000", 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013092900000000001 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:58.581885+0000", 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013222700000000001 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:38:58.655 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 2 2024-09-21T10:38:58.656 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:38:58.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:58.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:58.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:58.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:58.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:58.943 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:58.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:59.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:38:59.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:38:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:58 smithi119 ceph-mon[105288]: pgmap v2237: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:38:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:58 smithi119 ceph-mon[101432]: pgmap v2237: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:38:59.140 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 59 16 95517 95501 101.135 70.8125 0.0168714 0.00988278 2024-09-21T10:38:59.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:38:59.218 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:38:59.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:38:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:58 smithi145 ceph-mon[89359]: pgmap v2237: 109 pgs: 109 active+clean; 5.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:38:59.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:59.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:59.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:59.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:59.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:59.588 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:38:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:38:59.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:38:59.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:38:59.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:38:59.839 INFO:teuthology.orchestra.run.smithi119.stdout:moved 8 / 327 (2.44648%) 2024-09-21T10:38:59.839 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:38:59.839 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 7.66652 -> 6.04068 (expected baseline 6.32778) 2024-09-21T10:38:59.839 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.0 with 36 -> 44 pgs (0.770642 -> 0.941896 * mean) 2024-09-21T10:38:59.839 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.3 with 57 -> 54 pgs (1.22018 -> 1.15596 * mean) 2024-09-21T10:38:59.839 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:38:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:38:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 1 2024-09-21T10:38:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:38:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0343 2024-09-21T10:38:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.3427 2024-09-21T10:38:59.840 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0 weight 0.8110 -> 1.0000 2024-09-21T10:38:59.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:38:59.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:38:59.903 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:38:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:38:59.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:00.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:00.039 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:00.098 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:38:59 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536'}"} : dispatch 2024-09-21T10:39:00.098 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:38:59 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536'}"} : dispatch 2024-09-21T10:39:00.140 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:39:00.139584+0000 min lat: 0.00108018 max lat: 0.470495 avg lat: 0.00992911 2024-09-21T10:39:00.140 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:39:00.140 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 60 16 96432 96416 100.402 57.1875 0.00500155 0.00992911 2024-09-21T10:39:00.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:00.163 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:00.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:38:59 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536'}"} : dispatch 2024-09-21T10:39:00.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:00.266 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:39:00.266 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:39:00.266 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:39:00.266 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:39:00.266 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:39:00.267 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:39:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:39:00.268 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:39:00.288 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:00.390 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:00.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:00.513 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:00.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:00.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:39:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:39:00.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:00.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:00.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:00.780 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:00.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:00.903 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:00.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:00.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:01.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:01.026 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:01.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[101432]: from='client.36753 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[101432]: pgmap v2238: 109 pgs: 109 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536'}"}]': finished 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[101432]: osdmap e1366: 8 total, 8 up, 7 in 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3532119523' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[101432]: osdmap e1367: 8 total, 8 up, 7 in 2024-09-21T10:39:01.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[105288]: from='client.36753 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:39:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[105288]: pgmap v2238: 109 pgs: 109 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-09-21T10:39:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536'}"}]': finished 2024-09-21T10:39:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[105288]: osdmap e1366: 8 total, 8 up, 7 in 2024-09-21T10:39:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3532119523' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:01.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:00 smithi119 ceph-mon[105288]: osdmap e1367: 8 total, 8 up, 7 in 2024-09-21T10:39:01.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:01.140 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 61 16 96450 96434 98.7751 1.125 0.00346518 0.00992808 2024-09-21T10:39:01.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:01.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:01.273 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-mon[89359]: from='client.36753 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:39:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-mon[89359]: pgmap v2238: 109 pgs: 109 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 73 MiB/s wr, 1.18k op/s 2024-09-21T10:39:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536'}"}]': finished 2024-09-21T10:39:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-mon[89359]: osdmap e1366: 8 total, 8 up, 7 in 2024-09-21T10:39:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3532119523' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:00 smithi145 ceph-mon[89359]: osdmap e1367: 8 total, 8 up, 7 in 2024-09-21T10:39:01.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:01.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:01.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:01.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:01.579 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:01.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:01.641 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:01.743 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:01.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:01.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:01.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:01.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:01.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:02.063 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:02.111 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:02.136 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:02.141 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 62 16 96468 96452 97.1992 1.125 2.20758 0.0100888 2024-09-21T10:39:02.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:02.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:02.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:02.406 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:02.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:02.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:02.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:02.669 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:02.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:02.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:02.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:02.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:02.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:02 smithi145 ceph-mon[89359]: pgmap v2241: 109 pgs: 1 peering, 108 active+clean; 5.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-21T10:39:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:02 smithi145 ceph-mon[89359]: osdmap e1368: 8 total, 8 up, 7 in 2024-09-21T10:39:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:39:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:03.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:03.092 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[105288]: pgmap v2241: 109 pgs: 1 peering, 108 active+clean; 5.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-21T10:39:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[105288]: osdmap e1368: 8 total, 8 up, 7 in 2024-09-21T10:39:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:39:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:03.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[101432]: pgmap v2241: 109 pgs: 1 peering, 108 active+clean; 5.8 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 899 op/s 2024-09-21T10:39:03.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[101432]: osdmap e1368: 8 total, 8 up, 7 in 2024-09-21T10:39:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:39:03.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:03.141 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 63 16 97516 97500 96.696 65.5 0.00839794 0.0103354 2024-09-21T10:39:03.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:03.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:03.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:03.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:03.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:03.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:03.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:03.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:39:03.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:03.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:03.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:03.851 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:03.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:03.993 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:03.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:04.095 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:04.125 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:04.141 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 64 15 98523 98508 96.1695 63 0.0072401 0.0103922 2024-09-21T10:39:04.226 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:04.259 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:04.360 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:04.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:04.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:04.520 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:04.543 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:39:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:39:04] "GET /metrics HTTP/1.1" 200 36852 "" "Prometheus/2.51.0" 2024-09-21T10:39:04.622 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:04.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:04.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:04.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:04.941 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:04.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:05.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:05.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:05.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:04 smithi119 ceph-mon[101432]: pgmap v2243: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 107 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 5909/291033 objects misplaced (2.030%); 256 KiB/s, 4 objects/s recovering 2024-09-21T10:39:05.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:04 smithi119 ceph-mon[105288]: pgmap v2243: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 107 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 5909/291033 objects misplaced (2.030%); 256 KiB/s, 4 objects/s recovering 2024-09-21T10:39:05.141 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 65 16 99559 99543 95.6852 64.6875 0.00926385 0.0104458 2024-09-21T10:39:05.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:05.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:05.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:04 smithi145 ceph-mon[89359]: pgmap v2243: 109 pgs: 1 active+recovering+undersized+remapped, 1 peering, 107 active+clean; 5.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 75 MiB/s wr, 1.20k op/s; 5909/291033 objects misplaced (2.030%); 256 KiB/s, 4 objects/s recovering 2024-09-21T10:39:05.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:05.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:05.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:05.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.506 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:05.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:05.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:05.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:05.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:05.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:05.788 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:05.866 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:39:05.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:05.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:05.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:06.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:06.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:05 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3457025902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:39:06.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:05 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:39:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:05 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3457025902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:39:06.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:05 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:39:06.142 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 66 16 100468 100452 95.0963 56.8125 0.0099098 0.0105105 2024-09-21T10:39:06.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:06.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:06.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:05 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3457025902' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:39:06.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:05 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:39:06.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:06.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:06.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:06.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:06.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:06.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:06.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:06.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:06.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:06.888 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:39:06.888 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:39:06.949 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:06.977 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:06.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:07.000 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[105288]: pgmap v2244: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 36 MiB/s wr, 573 op/s; 5909/294012 objects misplaced (2.010%); 256 KiB/s, 4 objects/s recovering 2024-09-21T10:39:07.001 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:07.001 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:39:07.001 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[105288]: osdmap e1369: 8 total, 8 up, 7 in 2024-09-21T10:39:07.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[101432]: pgmap v2244: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 36 MiB/s wr, 573 op/s; 5909/294012 objects misplaced (2.010%); 256 KiB/s, 4 objects/s recovering 2024-09-21T10:39:07.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:07.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:39:07.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:06 smithi119 ceph-mon[101432]: osdmap e1369: 8 total, 8 up, 7 in 2024-09-21T10:39:07.019 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:39:07.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:39:07.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.130 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:07.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:07.142 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 67 16 101436 101420 94.5799 60.5 0.0309518 0.0105683 2024-09-21T10:39:07.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:07.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:07.323 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:06 smithi145 ceph-mon[89359]: pgmap v2244: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 36 MiB/s wr, 573 op/s; 5909/294012 objects misplaced (2.010%); 256 KiB/s, 4 objects/s recovering 2024-09-21T10:39:07.323 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:06 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:07.323 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:06 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:39:07.324 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:06 smithi145 ceph-mon[89359]: osdmap e1369: 8 total, 8 up, 7 in 2024-09-21T10:39:07.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:07.456 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:07.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:07.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:07.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:07.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:07.728 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:07.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:07.829 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:07.864 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:07.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:07.966 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:07.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:07.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:08.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:08.126 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:08.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:08.142 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 68 16 102488 102472 94.156 65.75 0.00894542 0.0106148 2024-09-21T10:39:08.166 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:08.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:39:08.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.281 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:08.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:08.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:08.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:08.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:08.555 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:08.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:08.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:08.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:08.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:08.935 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:08.963 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:08.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:09.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:09.098 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:08 smithi119 ceph-mon[101432]: pgmap v2246: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 31 MiB/s wr, 494 op/s; 5909/294912 objects misplaced (2.004%); 222 KiB/s, 4 objects/s recovering 2024-09-21T10:39:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:08 smithi119 ceph-mon[105288]: pgmap v2246: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 31 MiB/s wr, 494 op/s; 5909/294912 objects misplaced (2.004%); 222 KiB/s, 4 objects/s recovering 2024-09-21T10:39:09.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:09.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:09.142 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 69 16 103369 103353 93.5892 55.0625 0.0198284 0.0106796 2024-09-21T10:39:09.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:09.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:09.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:08 smithi145 ceph-mon[89359]: pgmap v2246: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 31 MiB/s wr, 494 op/s; 5909/294912 objects misplaced (2.004%); 222 KiB/s, 4 objects/s recovering 2024-09-21T10:39:09.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:09.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:09.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:09.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:39:09.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:09.560 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:09.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:09.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:09.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:09.667 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:09.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:09.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:09.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:09.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:09.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:09.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:10.031 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:10.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:10.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 70 16 104346 104330 93.1187 61.0625 0.0134903 0.0107336 2024-09-21T10:39:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.173 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:10.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:10.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:10.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:10.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:10.452 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:39:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:39:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:39:10.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:10.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:10.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:10.827 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:10.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:10.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:10.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:10.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:11.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:11.028 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:10 smithi119 ceph-mon[101432]: pgmap v2247: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 967 op/s; 5801/307857 objects misplaced (1.884%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:39:11.028 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:10 smithi119 ceph-mon[105288]: pgmap v2247: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 967 op/s; 5801/307857 objects misplaced (1.884%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:39:11.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:11.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:11.146 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 71 16 105288 105272 92.6365 58.875 0.0105318 0.0107896 2024-09-21T10:39:11.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:11.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:11.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:10 smithi145 ceph-mon[89359]: pgmap v2247: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 60 MiB/s wr, 967 op/s; 5801/307857 objects misplaced (1.884%); 1.0 MiB/s, 16 objects/s recovering 2024-09-21T10:39:11.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:11.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:11.498 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:11.520 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:11.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:11.658 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:11.760 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:11.798 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:11.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:11.937 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:11.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:12.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:39:12.071 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:39:12.072 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:39:12.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:12.147 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 72 16 106121 106105 92.073 52.0625 0.0241898 0.0108542 2024-09-21T10:39:12.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:12.207 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/219836247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:12.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:12.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/219836247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/219836247' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:12.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:12.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:12.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:12.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:12.707 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:12.736 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:12.838 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:12.870 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:12.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:12.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:12.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:13.000 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:13.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:13.101 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:13.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:13.147 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 73 16 107184 107168 91.7219 66.4375 0.00555819 0.0108952 2024-09-21T10:39:13.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:13.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:13.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:12 smithi145 ceph-mon[89359]: pgmap v2248: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 51 MiB/s wr, 810 op/s; 5801/308775 objects misplaced (1.879%); 851 KiB/s, 13 objects/s recovering 2024-09-21T10:39:13.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:13.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:12 smithi119 ceph-mon[101432]: pgmap v2248: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 51 MiB/s wr, 810 op/s; 5801/308775 objects misplaced (1.879%); 851 KiB/s, 13 objects/s recovering 2024-09-21T10:39:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:12 smithi119 ceph-mon[105288]: pgmap v2248: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 51 MiB/s wr, 810 op/s; 5801/308775 objects misplaced (1.879%); 851 KiB/s, 13 objects/s recovering 2024-09-21T10:39:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:13.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:13.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:13.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:13.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:13.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:13.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:13.808 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:13.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:13.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:13.945 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:13.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:14.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:14.147 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 74 16 108153 108137 91.3008 60.5625 0.0172258 0.0109471 2024-09-21T10:39:14.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:14.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:14.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.350 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:14.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:14.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:14.480 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:14.581 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:14.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:14.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:14.752 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:14.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:14.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:39:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:39:14] "GET /metrics HTTP/1.1" 200 36852 "" "Prometheus/2.51.0" 2024-09-21T10:39:14.884 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:14.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:14.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:15.021 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:15.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:15.148 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 75 16 109196 109180 90.9513 65.1875 0.0181822 0.0109895 2024-09-21T10:39:15.168 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:15.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:14 smithi145 ceph-mon[89359]: pgmap v2249: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57 MiB/s wr, 912 op/s; 5678/318414 objects misplaced (1.783%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:39:15.298 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:15.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:14 smithi119 ceph-mon[101432]: pgmap v2249: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57 MiB/s wr, 912 op/s; 5678/318414 objects misplaced (1.783%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:39:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:14 smithi119 ceph-mon[105288]: pgmap v2249: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57 MiB/s wr, 912 op/s; 5678/318414 objects misplaced (1.783%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:39:15.399 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:15.428 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:15.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:15.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:15.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:15.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:15.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:15.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:15.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:15.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:15.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:15.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:16.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:16.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:16.148 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 76 16 110220 110204 90.5967 64 0.0207815 0.0110327 2024-09-21T10:39:16.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:16.241 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:16.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:16.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:16.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:16.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:16.480 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:16.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:16.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:16.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:16.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:16.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:16.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:16.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:16.975 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:17.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:17.021 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:16 smithi119 ceph-mon[105288]: pgmap v2250: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 968 op/s; 5678/323073 objects misplaced (1.757%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73888 48.4 48:2e75d43f:::benchmark_data_smithi119_267254_object73887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.693234+0000", 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 37.321778438999999, 2024-09-21T10:39:17.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46585072999999999, 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73888 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693234+0000", 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693234+0000", 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693236+0000", 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6199999999999999e-06 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693259+0000", 2024-09-21T10:39:17.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3017e-05 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693273+0000", 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3974999999999999e-05 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693280+0000", 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3730000000000004e-06 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693339+0000", 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9147999999999997e-05 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693482+0000", 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014265400000000001 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693623+0000", 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014130600000000001 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158578+0000", 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46495520600000001 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158619+0000", 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0404999999999997e-05 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158998+0000", 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037948099999999998 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159047+0000", 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8637999999999997e-05 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159085+0000", 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7907000000000001e-05 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.028 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73892 48.4 48:26a06aae:::benchmark_data_smithi119_267254_object73891:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.694971+0000", 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 37.320041082000003, 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46426756200000002, 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73892 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.694971+0000", 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.694971+0000", 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.694973+0000", 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.875e-06 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695005+0000", 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2094000000000001e-05 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695018+0000", 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2794e-05 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695026+0000", 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.696896+0000", 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001869894 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.697028+0000", 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013275900000000001 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.697158+0000", 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129932 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158588+0000", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46142990699999997 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158722+0000", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013418999999999999 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159132+0000", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040954899999999999 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159180+0000", 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8374000000000001e-05 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159239+0000", 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8587e-05 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73895 48.a 48:52a5ab4b:::benchmark_data_smithi119_267254_object73894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.695044+0000", 2024-09-21T10:39:17.034 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 37.319967896000001, 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46376412299999997, 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73895 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695044+0000", 2024-09-21T10:39:17.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695044+0000", 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695045+0000", 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0299999999999999e-07 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695066+0000", 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1246999999999998e-05 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695073+0000", 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2069999999999998e-06 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695081+0000", 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8520000000000004e-06 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695138+0000", 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7051e-05 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695285+0000", 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014705400000000001 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695412+0000", 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000126391 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698128+0000", 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027160639999999998 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698382+0000", 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025434299999999997 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158704+0000", 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46032213100000002 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158758+0000", 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3431999999999999e-05 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158809+0000", 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0847999999999998e-05 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73897 48.a 48:5db11cab:::benchmark_data_smithi119_267254_object73896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.698883+0000", 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 37.316129836000002, 2024-09-21T10:39:17.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46637884099999999, 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73897 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698883+0000", 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698883+0000", 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698884+0000", 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.353e-06 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698905+0000", 2024-09-21T10:39:17.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1542000000000001e-05 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698915+0000", 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3179999999999996e-06 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158511+0000", 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.45959662899999998 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158894+0000", 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038238299999999997 2024-09-21T10:39:17.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159034+0000", 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014023599999999999 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159212+0000", 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017811199999999999 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165101+0000", 2024-09-21T10:39:17.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005888908 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165144+0000", 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2777000000000001e-05 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165156+0000", 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1970999999999999e-05 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165211+0000", 2024-09-21T10:39:17.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5526999999999999e-05 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165261+0000", 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0084999999999999e-05 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105281 48.e 48:75bdb000:::benchmark_data_smithi119_267254_object105280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.137052+0000", 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8779607189999998, 2024-09-21T10:39:17.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.080676064000000006, 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105281 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.137052+0000", 2024-09-21T10:39:17.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.137052+0000", 2024-09-21T10:39:17.048 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:16 smithi119 ceph-mon[101432]: pgmap v2250: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 968 op/s; 5678/323073 objects misplaced (1.757%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:39:17.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.137053+0000", 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2589999999999999e-06 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.137073+0000", 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0159000000000001e-05 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.137085+0000", 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2016e-05 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.137092+0000", 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8549999999999996e-06 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.140470+0000", 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033783120000000001 2024-09-21T10:39:17.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.141207+0000", 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00073645999999999998 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.141325+0000", 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118286 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.146695+0000", 2024-09-21T10:39:17.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0053701210000000003 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148293+0000", 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001597626 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.217625+0000", 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.069332697999999998 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.217680+0000", 2024-09-21T10:39:17.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4824999999999998e-05 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.217728+0000", 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7447000000000002e-05 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:105288 48.7 e1369/1354 48:e76a8196:::benchmark_data_smithi119_267254_object105287:head v 1369'6704, mlcod=1369'6702)", 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.145620+0000", 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.869392618, 2024-09-21T10:39:17.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.071976882000000006, 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145620+0000", 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145620+0000", 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145620+0000", 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5100000000000001e-07 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145643+0000", 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2524e-05 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145650+0000", 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8040000000000002e-06 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145656+0000", 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.541e-06 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145792+0000", 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000136095 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.145822+0000", 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0052999999999998e-05 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.217547+0000", 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.071724580999999996 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.217597+0000", 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9932999999999998e-05 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.057 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105292 48.0 48:0c7744fa:::benchmark_data_smithi119_267254_object105291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.148601+0000", 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8664117510000002, 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.065041510999999996, 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105292 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148601+0000", 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148601+0000", 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148605+0000", 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7380000000000002e-06 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148625+0000", 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9556999999999999e-05 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148635+0000", 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7249999999999992e-06 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.148641+0000", 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1190000000000002e-06 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.150985+0000", 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023441970000000001 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.151115+0000", 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129796 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.151257+0000", 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014272800000000001 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.155984+0000", 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047263629999999999 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.156243+0000", 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025890899999999998 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.213473+0000", 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.057230396000000003 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.213583+0000", 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109651 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.213642+0000", 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9332e-05 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105299 48.a 48:52a300fb:::benchmark_data_smithi119_267254_object105298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.152346+0000", 2024-09-21T10:39:17.063 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8626663260000003, 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.071447452999999994, 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105299 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152346+0000", 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.064 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152346+0000", 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152348+0000", 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5400000000000001e-06 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152371+0000", 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3129000000000002e-05 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152383+0000", 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2089000000000001e-05 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152389+0000", 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5010000000000001e-06 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152464+0000", 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4567999999999995e-05 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152613+0000", 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014890500000000001 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.152735+0000", 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121999 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.214601+0000", 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.061866179 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.214648+0000", 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6869000000000001e-05 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.223557+0000", 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0089095500000000005 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.223741+0000", 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000183306 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.223794+0000", 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2818000000000002e-05 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:105291 48.7 e1369/1354 48:e09fa548:::benchmark_data_smithi119_267254_object105290:head v 1369'6705, mlcod=1369'6702)", 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.153993+0000", 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8610189669999997, 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.066617448999999995, 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.069 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.153993+0000", 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.153993+0000", 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.153994+0000", 2024-09-21T10:39:17.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5899999999999995e-07 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.154019+0000", 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4229000000000001e-05 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.154030+0000", 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1293e-05 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.154037+0000", 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8870000000000003e-06 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.154114+0000", 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7161000000000002e-05 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.154134+0000", 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0043000000000001e-05 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.220482+0000", 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.066348156000000005 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.220611+0000", 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012882099999999999 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:105296 48.7 e1369/1354 48:e9009788:::benchmark_data_smithi119_267254_object105295:head v 1369'6706, mlcod=1369'6702)", 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.157868+0000", 2024-09-21T10:39:17.073 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.8571443910000003, 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.076022682999999994, 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.157868+0000", 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.074 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.157868+0000", 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.157869+0000", 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.065e-06 2024-09-21T10:39:17.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.157897+0000", 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7936000000000002e-05 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.212064+0000", 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.054166832999999998 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.212080+0000", 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6429000000000001e-05 2024-09-21T10:39:17.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.214599+0000", 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002519046 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.214647+0000", 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7444000000000002e-05 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.233440+0000", 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.018793687999999999 2024-09-21T10:39:17.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.233891+0000", 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000450242 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105552 48.e 48:7890892f:::benchmark_data_smithi119_267254_object105551:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.450538+0000", 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.5644746930000002, 2024-09-21T10:39:17.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.068770748000000007, 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105552 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.450538+0000", 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.450538+0000", 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.450539+0000", 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.412e-06 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.450560+0000", 2024-09-21T10:39:17.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1362999999999999e-05 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.450571+0000", 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0661e-05 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.450578+0000", 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2760000000000003e-06 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.453919+0000", 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033407179999999999 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.454072+0000", 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000153034 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.454193+0000", 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121227 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.457605+0000", 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034119689999999999 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.458373+0000", 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00076723700000000004 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519207+0000", 2024-09-21T10:39:17.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.060834566 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519263+0000", 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6064999999999999e-05 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519308+0000", 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5219999999999997e-05 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105566 48.4 48:23a99fdc:::benchmark_data_smithi119_267254_object105565:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.469098+0000", 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.5459148389999999, 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.066576472999999997, 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105566 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.085 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.469098+0000", 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.469098+0000", 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.469100+0000", 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1289999999999999e-06 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.469140+0000", 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0008e-05 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.469155+0000", 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5418999999999999e-05 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.469163+0000", 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1650000000000006e-06 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.518969+0000", 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.049806054000000002 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519104+0000", 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013418300000000001 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519242+0000", 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000138436 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.530685+0000", 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011442610000000001 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.533563+0000", 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028788310000000001 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.535582+0000", 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020183990000000001 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.535635+0000", 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2738999999999998e-05 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.535674+0000", 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9499999999999998e-05 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:105569 48.7 e1369/1354 48:e4905a49:::benchmark_data_smithi119_267254_object105568:head v 1369'6717, mlcod=1369'6715)", 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.470240+0000", 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.5447722319999997, 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.061702603000000002, 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.470240+0000", 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.470240+0000", 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.470241+0000", 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4399999999999999e-07 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.470264+0000", 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2836e-05 2024-09-21T10:39:17.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.518720+0000", 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.048456394 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.518730+0000", 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0336000000000001e-05 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519483+0000", 2024-09-21T10:39:17.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075205200000000002 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.519502+0000", 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9788000000000001e-05 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.531891+0000", 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012389043000000001 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.531943+0000", 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1409999999999997e-05 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105874 48.4 48:250b2854:::benchmark_data_smithi119_267254_object105873:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.810504+0000", 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.2045079059999999, 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.108360848, 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.095 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105874 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810504+0000", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810504+0000", 2024-09-21T10:39:17.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810506+0000", 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.167e-06 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810537+0000", 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1649000000000001e-05 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810548+0000", 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0739e-05 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810562+0000", 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3762e-05 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810632+0000", 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0339000000000005e-05 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.810881+0000", 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024892099999999998 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.811185+0000", 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000304202 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814621+0000", 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003435329 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.815062+0000", 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044116499999999998 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.918756+0000", 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.10369441 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.918822+0000", 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6335999999999996e-05 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.918865+0000", 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2828999999999999e-05 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105879 48.e 48:70887b76:::benchmark_data_smithi119_267254_object105878:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.814865+0000", 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.2001477740000004, 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.102481629, 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.101 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.102 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.102 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.103 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:17.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:17.103 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105879 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814865+0000", 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814865+0000", 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814866+0000", 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3570000000000001e-06 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814891+0000", 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4737999999999999e-05 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814901+0000", 2024-09-21T10:39:17.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0209e-05 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814908+0000", 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8549999999999996e-06 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.814953+0000", 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5303999999999999e-05 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.815081+0000", 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012814099999999999 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.815174+0000", 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2983000000000001e-05 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.817984+0000", 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028100920000000001 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818324+0000", 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00033939700000000002 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.917256+0000", 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.098932668000000001 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.917304+0000", 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7534000000000003e-05 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.917346+0000", 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2351000000000002e-05 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105880 48.0 48:0c8bb4ed:::benchmark_data_smithi119_267254_object105879:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.818414+0000", 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.1965980600000004, 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.086904801000000004, 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.109 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105880 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818414+0000", 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818414+0000", 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818416+0000", 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7120000000000001e-06 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818455+0000", 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9307999999999997e-05 2024-09-21T10:39:17.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818463+0000", 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3699999999999997e-06 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.818467+0000", 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0539999999999996e-06 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894317+0000", 2024-09-21T10:39:17.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.075849982999999996 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894490+0000", 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017316400000000001 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894624+0000", 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013390900000000001 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.904765+0000", 2024-09-21T10:39:17.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010141397 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.904805+0000", 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9981999999999998e-05 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.905214+0000", 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000408699 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.905270+0000", 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5924000000000002e-05 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.905319+0000", 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9298999999999997e-05 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:105881 48.a 48:58084aac:::benchmark_data_smithi119_267254_object105880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:11.823048+0000", 2024-09-21T10:39:17.115 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.1919643620000002, 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.076196574000000003, 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 105881 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.823048+0000", 2024-09-21T10:39:17.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.823048+0000", 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.823049+0000", 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.13e-07 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.823083+0000", 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4387999999999997e-05 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894274+0000", 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.071191438999999995 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894292+0000", 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7672999999999998e-05 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.118 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894550+0000", 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025792300000000002 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894674+0000", 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012437599999999999 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.894893+0000", 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021831199999999999 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.898766+0000", 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038734970000000001 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.899112+0000", 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00034595999999999997 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.899145+0000", 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3241999999999998e-05 2024-09-21T10:39:17.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.899200+0000", 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4777000000000001e-05 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:11.899245+0000", 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4474000000000001e-05 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:107187 48.e 48:701e9dde:::benchmark_data_smithi119_267254_object107186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:13.148042+0000", 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.8669703819999999, 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.065732785000000002, 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 107187 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148042+0000", 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148042+0000", 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148045+0000", 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.543e-06 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148072+0000", 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7591000000000001e-05 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148087+0000", 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5153999999999999e-05 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148095+0000", 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2830000000000004e-06 2024-09-21T10:39:17.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148160+0000", 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4968000000000005e-05 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148306+0000", 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014607699999999999 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.148433+0000", 2024-09-21T10:39:17.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012693599999999999 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152117+0000", 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003684082 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152600+0000", 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00048289600000000002 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213659+0000", 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.061059812999999998 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213724+0000", 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4465e-05 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213775+0000", 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0977000000000003e-05 2024-09-21T10:39:17.127 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:107188 48.4 48:2a1e3a4f:::benchmark_data_smithi119_267254_object107187:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:13.149731+0000", 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.8652817189999999, 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.064044119999999996, 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.128 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 107188 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149731+0000", 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149731+0000", 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149732+0000", 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.773e-06 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149757+0000", 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4147000000000001e-05 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149773+0000", 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6327000000000001e-05 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149781+0000", 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3250000000000008e-06 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.149852+0000", 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1076999999999997e-05 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.150001+0000", 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000148393 2024-09-21T10:39:17.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.150138+0000", 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000136938 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.156331+0000", 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061937249999999998 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.156363+0000", 2024-09-21T10:39:17.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2125999999999997e-05 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213677+0000", 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.057313496999999998 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213726+0000", 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9230000000000001e-05 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213775+0000", 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8562e-05 2024-09-21T10:39:17.133 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:107189 48.4 48:2558b078:::benchmark_data_smithi119_267254_object107188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1369)", 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:13.152418+0000", 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 3.8625947219999999, 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.061482095, 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:17.134 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 107189 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152418+0000", 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152418+0000", 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152420+0000", 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9740000000000001e-06 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152472+0000", 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2280000000000001e-05 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152486+0000", 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4484999999999999e-05 2024-09-21T10:39:17.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.152495+0000", 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7860000000000002e-06 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.153302+0000", 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080669800000000001 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.153441+0000", 2024-09-21T10:39:17.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013908099999999999 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.153534+0000", 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2824000000000004e-05 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.156461+0000", 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002927275 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.157473+0000", 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010119689999999999 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213819+0000", 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.056345783000000003 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213870+0000", 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1532000000000001e-05 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:17.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:13.213900+0000", 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9408e-05 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:17.140 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:39:17.148 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 77 16 111151 111135 90.1759 58.1875 0.00955521 0.0110845 2024-09-21T10:39:17.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:17.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:16 smithi145 ceph-mon[89359]: pgmap v2250: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 968 op/s; 5678/323073 objects misplaced (1.757%); 1.4 MiB/s, 23 objects/s recovering 2024-09-21T10:39:17.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:17.373 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:17.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:17.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:17.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:17.618 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:17.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:17.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:17.792 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:17.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:17.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:18.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:18.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:39:18.068 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:18.148 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 78 16 112046 112030 89.737 55.9375 0.018724 0.0111381 2024-09-21T10:39:18.170 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:18.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3133492280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:39:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:39:18.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:18.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:18.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3133492280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:39:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:39:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3133492280' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:39:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:39:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:18.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:18.493 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:18.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:18.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:18.649 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:39:18.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:18.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:18.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:18.904 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:18.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:19.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:19.045 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:19.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:19.147 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:19.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 79 16 113043 113027 89.3898 62.3125 0.0189379 0.0111818 2024-09-21T10:39:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.181 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:19.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:19.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:18 smithi145 ceph-mon[89359]: pgmap v2251: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 52 MiB/s wr, 838 op/s; 5678/323985 objects misplaced (1.753%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:39:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3133492280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:39:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:18 smithi145 ceph-mon[89359]: osdmap e1370: 8 total, 8 up, 7 in 2024-09-21T10:39:19.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:19.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:18 smithi119 ceph-mon[101432]: pgmap v2251: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 52 MiB/s wr, 838 op/s; 5678/323985 objects misplaced (1.753%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:39:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3133492280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:39:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:18 smithi119 ceph-mon[101432]: osdmap e1370: 8 total, 8 up, 7 in 2024-09-21T10:39:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:18 smithi119 ceph-mon[105288]: pgmap v2251: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 52 MiB/s wr, 838 op/s; 5678/323985 objects misplaced (1.753%); 1.2 MiB/s, 19 objects/s recovering 2024-09-21T10:39:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3133492280' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:39:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:18 smithi119 ceph-mon[105288]: osdmap e1370: 8 total, 8 up, 7 in 2024-09-21T10:39:19.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:19.456 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:19.558 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:19.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:19.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.723 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:19.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:19.824 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:19.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:19.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:39:19.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:19.979 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:19.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:20.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:20.129 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:20.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:20.148 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:39:20.147842+0000 min lat: 0.00108018 max lat: 2.27176 avg lat: 0.0112235 2024-09-21T10:39:20.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:39:20.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 80 16 114047 114031 89.0568 62.75 0.016133 0.0112235 2024-09-21T10:39:20.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.275 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:20.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:20.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:20.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:20.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:20.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:20.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:39:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:39:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:39:20.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:20.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:20.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:20.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:20.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:20.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:20.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:21.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:21.066 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:21.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 81 16 115015 114999 88.7043 60.5 0.0110621 0.011268 2024-09-21T10:39:21.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:21.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:20 smithi145 ceph-mon[89359]: pgmap v2253: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s; 5553/336765 objects misplaced (1.649%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:21.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:21.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.335 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:21.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:20 smithi119 ceph-mon[101432]: pgmap v2253: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s; 5553/336765 objects misplaced (1.649%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:20 smithi119 ceph-mon[105288]: pgmap v2253: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s; 5553/336765 objects misplaced (1.649%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:21.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:21.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:21.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:39:21.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:21.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:21.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:21.794 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:21.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:21.948 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:21.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:22.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:22.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:22.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:22.101 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:22.101 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:22.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 82 16 115917 115901 88.3099 56.375 0.00828198 0.0113182 2024-09-21T10:39:22.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:22.227 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:22.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:22.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:22.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:22.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:22.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:22.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:22.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:22.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:22.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:22.931 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:22.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:23.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:23.034 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:22 smithi119 ceph-mon[105288]: pgmap v2254: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s; 5553/337683 objects misplaced (1.644%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:23.034 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:22 smithi119 ceph-mon[101432]: pgmap v2254: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s; 5553/337683 objects misplaced (1.644%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:23.059 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:23.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 83 16 116893 116877 87.9809 61 0.0197089 0.0113602 2024-09-21T10:39:23.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:22 smithi145 ceph-mon[89359]: pgmap v2254: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 60 MiB/s wr, 963 op/s; 5553/337683 objects misplaced (1.644%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:23.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:23.340 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:23.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:23.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:23.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:23.601 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:23.702 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:23.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:23.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:23.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:23.872 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:23.974 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:23.996 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:23.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:24.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:24.141 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:24.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:24.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 84 16 117769 117753 87.5853 54.75 0.00772303 0.0114125 2024-09-21T10:39:24.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:24.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:24.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:24.398 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:24.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:24.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:24.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:24.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:24.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:24.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:24.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:24.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:39:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:39:24] "GET /metrics HTTP/1.1" 200 36872 "" "Prometheus/2.51.0" 2024-09-21T10:39:24.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:24.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:24.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:24.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:25.031 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:25.061 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:25.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 85 16 118654 118638 87.2057 55.3125 0.0156651 0.0114615 2024-09-21T10:39:25.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:25.194 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:25.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:25.207 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:24 smithi119 ceph-mon[101432]: pgmap v2255: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.1 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s; 5438/347574 objects misplaced (1.565%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:39:25.207 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1098813202' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:25.207 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:24 smithi119 ceph-mon[105288]: pgmap v2255: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.1 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s; 5438/347574 objects misplaced (1.565%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:39:25.207 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1098813202' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:24 smithi145 ceph-mon[89359]: pgmap v2255: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.1 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 61 MiB/s wr, 971 op/s; 5438/347574 objects misplaced (1.565%); 1.5 MiB/s, 23 objects/s recovering 2024-09-21T10:39:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1098813202' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:25.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:25.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:25.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:25.460 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:25.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:25.582 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:25.683 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:25.714 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:25.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:25.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:25.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:25.977 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:25.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:26.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:26.110 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:26.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 86 16 119541 119525 86.8363 55.4375 0.0396343 0.0115113 2024-09-21T10:39:26.211 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:26.228 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:25 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:26.228 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:25 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1098813202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:39:26.228 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:25 smithi119 ceph-mon[101432]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-21T10:39:26.229 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:25 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:26.229 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:25 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1098813202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:39:26.229 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:25 smithi119 ceph-mon[105288]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-21T10:39:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:26.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:25 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:25 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1098813202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:39:26.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:25 smithi145 ceph-mon[89359]: osdmap e1371: 8 total, 8 up, 7 in 2024-09-21T10:39:26.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:26.384 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:26.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:26.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:26.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:26.618 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:26.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:26.752 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:26.790 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:26.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:26.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:26.916 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:26.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:26.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:27.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:27.031 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:39:27.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:27.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:27.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:26 smithi119 ceph-mon[101432]: pgmap v2257: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 5438/352149 objects misplaced (1.544%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:39:27.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:26 smithi119 ceph-mon[105288]: pgmap v2257: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 5438/352149 objects misplaced (1.544%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:39:27.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:27.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 87 16 120475 120459 86.5092 58.375 0.0127543 0.0115545 2024-09-21T10:39:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:27.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:27.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:27.290 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:26 smithi145 ceph-mon[89359]: pgmap v2257: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 73 MiB/s wr, 1.17k op/s; 5438/352149 objects misplaced (1.544%); 1.9 MiB/s, 29 objects/s recovering 2024-09-21T10:39:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:27.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:27.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:27.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:27.615 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:27.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:27.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:27.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:27.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:27.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:27.989 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:28.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:28.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:28.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 88 16 121346 121330 86.1447 54.4375 0.0263199 0.011604 2024-09-21T10:39:28.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:28.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:28.328 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:28.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:28.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:28.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:28.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:28.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:28.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:121996 48.f 48:fbac8cd1:::benchmark_data_smithi119_267254_object121995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1371)", 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:28.856061+0000", 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0081823460000000001, 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082190519999999993, 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:28.867 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 121996 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856061+0000", 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856061+0000", 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.868 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856063+0000", 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4309999999999999e-06 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856085+0000", 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2461999999999999e-05 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856097+0000", 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1626e-05 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.869 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856104+0000", 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1330000000000001e-06 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856157+0000", 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.291e-05 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856289+0000", 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000132571 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.870 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.856421+0000", 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013136800000000001 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859591+0000", 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031699330000000002 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.861868+0000", 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022776290000000002 2024-09-21T10:39:28.871 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:121995 48.e e1371/1354 48:7eeab8fd:::benchmark_data_smithi119_267254_object121994:head v 1371'7721, mlcod=1371'7718)", 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:28.859939+0000", 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0043040700000000001, 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044219100000000003, 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:28.872 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859939+0000", 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859939+0000", 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859941+0000", 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.598e-06 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.873 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859965+0000", 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3702000000000001e-05 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859983+0000", 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7963999999999999e-05 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.859991+0000", 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1049999999999995e-06 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.874 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.860105+0000", 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011379600000000001 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.860123+0000", 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8450000000000001e-05 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:121978 48.9 48:9139addb:::benchmark_data_smithi119_267254_object121977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1371)", 2024-09-21T10:39:28.875 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:28.841324+0000", 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.022919055000000001, 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0230929, 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 121978 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.876 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841324+0000", 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841324+0000", 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841326+0000", 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.72e-06 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.877 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841388+0000", 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2255000000000006e-05 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841404+0000", 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5322999999999999e-05 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841411+0000", 2024-09-21T10:39:28.878 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5020000000000003e-06 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841468+0000", 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6326999999999998e-05 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841618+0000", 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015037699999999999 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 5,7", 2024-09-21T10:39:28.879 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.841766+0000", 2024-09-21T10:39:28.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000148536 2024-09-21T10:39:28.880 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.880 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.880 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:28.880 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.848871+0000", 2024-09-21T10:39:28.880 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0071048409999999998 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.849295+0000", 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00042358199999999998 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:121984 48.f 48:ffa2babd:::benchmark_data_smithi119_267254_object121983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1371)", 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:39:28.844356+0000", 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.019887552999999999, 2024-09-21T10:39:28.881 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020132415000000001, 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 121984 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844356+0000", 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.882 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844356+0000", 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844358+0000", 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.579e-06 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844375+0000", 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7336000000000001e-05 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844387+0000", 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2057e-05 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844394+0000", 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4699999999999996e-06 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844450+0000", 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5266000000000002e-05 2024-09-21T10:39:28.884 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844592+0000", 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014198699999999999 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.844788+0000", 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000195945 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.854030+0000", 2024-09-21T10:39:28.885 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092421240000000009 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:39:28.854322+0000", 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029275500000000001 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 4 2024-09-21T10:39:28.886 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:39:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:28.887 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:28.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:28.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.030 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:29.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:28 smithi119 ceph-mon[105288]: pgmap v2258: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 5438/352992 objects misplaced (1.541%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:28 smithi119 ceph-mon[101432]: pgmap v2258: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 5438/352992 objects misplaced (1.541%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:29.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:29.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 89 16 122259 122243 85.818 57.0625 0.08244 0.0116454 2024-09-21T10:39:29.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:29.260 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:29.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:28 smithi145 ceph-mon[89359]: pgmap v2258: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.2 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s; 5438/352992 objects misplaced (1.541%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:29.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:29.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:29.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:29.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:29.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:29.679 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:29.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:29.808 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:29.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:29.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:30.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:30.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:30.149 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total time run: 90.0179 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total writes made: 123207 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Write size: 65536 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object size: 65536 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Bandwidth (MB/sec): 85.5434 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Bandwidth: 40.4688 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max bandwidth (MB/sec): 236.375 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min bandwidth (MB/sec): 1.125 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average IOPS: 1368 2024-09-21T10:39:30.150 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev IOPS: 647.501 2024-09-21T10:39:30.151 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max IOPS: 3782 2024-09-21T10:39:30.151 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min IOPS: 18 2024-09-21T10:39:30.151 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average Latency(s): 0.0116849 2024-09-21T10:39:30.151 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Latency(s): 0.0276997 2024-09-21T10:39:30.151 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max latency(s): 2.27176 2024-09-21T10:39:30.151 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min latency(s): 0.00108018 2024-09-21T10:39:30.154 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:39:30.186 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:30.216 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:30.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:39:30.318 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:30.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:30.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:30.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:30.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:30.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:39:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:39:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:39:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:39:30.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:30.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:30.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:30.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:30.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:30.874 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:30.902 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:30.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:31.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:31.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:31.084 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:30 smithi119 ceph-mon[105288]: pgmap v2259: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 928 op/s; 5311/364629 objects misplaced (1.457%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:31.084 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2445291507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:31.085 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:30 smithi119 ceph-mon[101432]: pgmap v2259: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 928 op/s; 5311/364629 objects misplaced (1.457%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:31.085 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2445291507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:31.104 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:39:31.104 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:39:31.104 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:39:31.104 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:39:31.104 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:39:31.104 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:39:31.105 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:39:31.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:31.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:30 smithi145 ceph-mon[89359]: pgmap v2259: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 928 op/s; 5311/364629 objects misplaced (1.457%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2445291507' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:31.301 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:31.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:31.403 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:31.442 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:31.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:31.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:31.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:31.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:31.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:31.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:31.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:31.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:31.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:32.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:32.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:32.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:32.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:32.315 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:32.327 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:32.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:32.412 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:32.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:32.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:32.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:32.684 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:32.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:32.814 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:32.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:32.916 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:32.949 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:32.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:33.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:32 smithi119 ceph-mon[101432]: pgmap v2260: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 930 op/s; 5311/365436 objects misplaced (1.453%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:33.020 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:33.020 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:32 smithi119 ceph-mon[105288]: pgmap v2260: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 930 op/s; 5311/365436 objects misplaced (1.453%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:33.021 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:33.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:33.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:33.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:33.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:33.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:32 smithi145 ceph-mon[89359]: pgmap v2260: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 58 MiB/s wr, 930 op/s; 5311/365436 objects misplaced (1.453%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:33.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:33.339 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:33.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:33.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:33.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:33.622 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:33.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:33.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:39:33.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:33.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:34.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:34.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:34.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:34.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:34.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:34.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:34.483 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:34.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:34.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:34.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:34.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:34.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:39:34] "GET /metrics HTTP/1.1" 200 36871 "" "Prometheus/2.51.0" 2024-09-21T10:39:34.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:34.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:34.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:34.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:34.993 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:39:35.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:35.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:35.146 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:35.248 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:35.279 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:35.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:34 smithi145 ceph-mon[89359]: pgmap v2261: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 102 B/s rd, 45 MiB/s wr, 925 op/s; 5195/362829 objects misplaced (1.432%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2657346034' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:39:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:34 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:39:35.340 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-mon[105288]: pgmap v2261: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 102 B/s rd, 45 MiB/s wr, 925 op/s; 5195/362829 objects misplaced (1.432%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:35.340 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2657346034' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:39:35.340 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:39:35.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-mon[101432]: pgmap v2261: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 102 B/s rd, 45 MiB/s wr, 925 op/s; 5195/362829 objects misplaced (1.432%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:35.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2657346034' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:39:35.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:34 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:39:35.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:35.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:35.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:35.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:35.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:35.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:35.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:35.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:35.812 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:35.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:35.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:36.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:36.089 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:36.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:36.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:36.232 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:36.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:36 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:39:36.259 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:36 smithi119 ceph-mon[101432]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-21T10:39:36.259 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:36 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:39:36.259 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:36 smithi119 ceph-mon[105288]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-21T10:39:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:36 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:39:36.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:36 smithi145 ceph-mon[89359]: osdmap e1372: 8 total, 8 up, 7 in 2024-09-21T10:39:36.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:36.375 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:36.477 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:36.502 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:36.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:36.631 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:36.650 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:39:36.650 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:39:36.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:36.760 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:36.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:36.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:36.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:36.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:36.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:37.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:37.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:37.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:37.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:37 smithi145 ceph-mon[89359]: pgmap v2263: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 37 MiB/s wr, 921 op/s; 5195/359613 objects misplaced (1.445%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:37.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:37 smithi119 ceph-mon[105288]: pgmap v2263: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 37 MiB/s wr, 921 op/s; 5195/359613 objects misplaced (1.445%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:37 smithi119 ceph-mon[101432]: pgmap v2263: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 37 MiB/s wr, 921 op/s; 5195/359613 objects misplaced (1.445%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:37.397 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:37.426 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:37.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:37.568 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:37.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:37.702 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:37.804 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:37.832 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:37.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:37.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:37.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:38.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:38.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:38.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:38.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:38.651 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:38.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:38.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:38.781 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:38.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:38.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:38.926 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:38.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:38.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:39:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:39:39.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:39.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:39.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:39.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:39.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:39 smithi145 ceph-mon[89359]: pgmap v2264: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 35 MiB/s wr, 923 op/s; 5195/358692 objects misplaced (1.448%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3394947546' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:39:39.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:39.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:39.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:39 smithi119 ceph-mon[101432]: pgmap v2264: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 35 MiB/s wr, 923 op/s; 5195/358692 objects misplaced (1.448%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:39.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3394947546' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:39:39.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:39 smithi119 ceph-mon[105288]: pgmap v2264: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 102 B/s rd, 35 MiB/s wr, 923 op/s; 5195/358692 objects misplaced (1.448%); 1.5 MiB/s, 24 objects/s recovering 2024-09-21T10:39:39.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3394947546' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:39:39.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:39.496 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:39.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:39.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:39.625 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:39.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:39.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:39.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:39:39.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:39.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:39.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:39.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:39.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:40.023 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:40.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:40.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:40.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:40.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:40.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:39:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:39:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:39:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:40 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3394947546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:39:40.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:40 smithi145 ceph-mon[89359]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-21T10:39:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.319 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:40.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:40 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3394947546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:39:40.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:40 smithi119 ceph-mon[101432]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-21T10:39:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:40 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:39:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3394947546' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:39:40.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:40 smithi119 ceph-mon[105288]: osdmap e1373: 8 total, 8 up, 7 in 2024-09-21T10:39:40.420 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:40.448 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:40.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:40.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:40.589 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:40.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:40.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:40.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:40.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:40.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:40.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:40.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:41.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:41.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:41.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:41 smithi119 ceph-mon[105288]: pgmap v2266: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 127 B/s rd, 11 MiB/s wr, 1.17k op/s; 5069/345741 objects misplaced (1.466%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:39:41.117 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:41.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:41 smithi119 ceph-mon[101432]: pgmap v2266: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 127 B/s rd, 11 MiB/s wr, 1.17k op/s; 5069/345741 objects misplaced (1.466%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:39:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:41.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:41.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:41.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:41 smithi145 ceph-mon[89359]: pgmap v2266: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 127 B/s rd, 11 MiB/s wr, 1.17k op/s; 5069/345741 objects misplaced (1.466%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:39:41.301 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:41.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:41.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:41.575 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:41.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:41.704 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:41.764 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:39:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:39:41.778 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:39:41.778 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:39:41.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:41.823 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:39:41.823 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:39:41.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:41.858 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:39:41.858 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:39:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:41.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:39:41.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:39:41.915 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:39:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:39:41.952 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:39:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:39:41.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:41.982 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:39:41.982 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:39:41.991 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:41.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:42.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:42.130 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:42.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:42.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:42 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1598450496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:42.233 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:42 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1598450496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:42.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:42.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:42 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1598450496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:42.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:42.400 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:42.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:42.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:42.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:42.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:42.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:42.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:42.794 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:42.884 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:39:42.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:42.924 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:42.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:43.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:43.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:43.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:43.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:43.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:43.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:43 smithi119 ceph-mon[101432]: pgmap v2267: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.6 MiB/s wr, 803 op/s; 5069/344781 objects misplaced (1.470%); 1008 KiB/s, 15 objects/s recovering 2024-09-21T10:39:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:43 smithi119 ceph-mon[105288]: pgmap v2267: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.6 MiB/s wr, 803 op/s; 5069/344781 objects misplaced (1.470%); 1008 KiB/s, 15 objects/s recovering 2024-09-21T10:39:43.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:43.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:43 smithi145 ceph-mon[89359]: pgmap v2267: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 7.0 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.6 MiB/s wr, 803 op/s; 5069/344781 objects misplaced (1.470%); 1008 KiB/s, 15 objects/s recovering 2024-09-21T10:39:43.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:43.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:43.716 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:43.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:43.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:43.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:43.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:43.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:44.020 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:44.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:44.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:44.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:44.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:44.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:44.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:44.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:44.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:44.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:44.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:44.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:44.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:44.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:39:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:39:44] "GET /metrics HTTP/1.1" 200 36871 "" "Prometheus/2.51.0" 2024-09-21T10:39:44.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:44.966 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:44.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:45.067 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:45.105 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:45.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:45.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.233 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:45.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:45.334 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:45.360 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:45 smithi119 ceph-mon[105288]: pgmap v2268: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 983 op/s; 4924/334605 objects misplaced (1.472%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:39:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:45 smithi119 ceph-mon[101432]: pgmap v2268: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 983 op/s; 4924/334605 objects misplaced (1.472%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:39:45.462 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.491 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:45.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:45 smithi145 ceph-mon[89359]: pgmap v2268: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 983 op/s; 4924/334605 objects misplaced (1.472%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:39:45.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:45.632 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:45.734 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.773 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:45.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:45.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:45.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:45.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:45.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:46.008 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:46.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:46.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:46.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:46.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:46.274 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.334 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:46.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:46.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:46.473 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:46.575 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:46.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:46.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:46.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:46.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:46.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:46.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:46.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:47.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:47.057 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:47.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:47.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:47.222 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:47 smithi119 ceph-mon[105288]: pgmap v2269: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 990 op/s; 4924/329889 objects misplaced (1.493%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:39:47.222 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:47 smithi119 ceph-mon[101432]: pgmap v2269: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 990 op/s; 4924/329889 objects misplaced (1.493%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:39:47.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:47.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:47.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:47.454 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:47 smithi145 ceph-mon[89359]: pgmap v2269: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 990 op/s; 4924/329889 objects misplaced (1.493%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:39:47.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:47.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:47.702 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:47.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:47.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:47.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:47.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:47.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:47.980 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:48.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:48.062 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:39:48.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:48.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:48.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:48.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:48.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:39:48.337 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1934052959' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:39:48.337 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:48.337 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1934052959' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:39:48.337 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:48.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:48.420 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:48.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1934052959' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:39:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:39:48.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:48.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:48.691 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:48.793 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:48.821 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:48.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:48.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:48.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:49.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:49.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:49.189 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:49.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:49.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:49.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:49.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:49 smithi119 ceph-mon[101432]: pgmap v2270: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 987 op/s; 4924/329040 objects misplaced (1.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:39:49.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1934052959' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:39:49.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:49 smithi119 ceph-mon[101432]: osdmap e1374: 8 total, 8 up, 7 in 2024-09-21T10:39:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:49 smithi119 ceph-mon[105288]: pgmap v2270: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 987 op/s; 4924/329040 objects misplaced (1.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:39:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1934052959' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:39:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:49 smithi119 ceph-mon[105288]: osdmap e1374: 8 total, 8 up, 7 in 2024-09-21T10:39:49.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:49.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:49 smithi145 ceph-mon[89359]: pgmap v2270: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.7 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 0 B/s wr, 987 op/s; 4924/329040 objects misplaced (1.496%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:39:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1934052959' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:39:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:49 smithi145 ceph-mon[89359]: osdmap e1374: 8 total, 8 up, 7 in 2024-09-21T10:39:49.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.617 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:49.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:49.718 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:49.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:49.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:49.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:39:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:49.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:49.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:49.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:50.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:50.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:50.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:50.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:50.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:50.271 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:50.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:50.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:50.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:50.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:50.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:39:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:39:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:39:50.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:50.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:50.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:50.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:50.877 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:50.892 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:50.916 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:50.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:50.962 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:51.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:51.045 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:51.083 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:39:51.147 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:51.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:51.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:51.303 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:51.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[105288]: pgmap v2272: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 993 op/s; 4780/315921 objects misplaced (1.513%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:39:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1971898372' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:39:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[101432]: pgmap v2272: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 993 op/s; 4780/315921 objects misplaced (1.513%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:39:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1971898372' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:51.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:51 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:39:51.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:51.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:51 smithi145 ceph-mon[89359]: pgmap v2272: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 993 op/s; 4780/315921 objects misplaced (1.513%); 1.8 MiB/s, 28 objects/s recovering 2024-09-21T10:39:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:51 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1971898372' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:51 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:39:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:51 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:39:51.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:51.587 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:51.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:51.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:51.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:51.861 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:51.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:51.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:51.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:52.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:52.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:52.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:52.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:52.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:52.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[101432]: osdmap e1375: 8 total, 8 up, 7 in 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:39:52.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:52 smithi119 ceph-mon[105288]: osdmap e1375: 8 total, 8 up, 7 in 2024-09-21T10:39:52.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:52.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:52.524 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:52 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:39:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:52 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:39:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:52 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:39:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:52 smithi145 ceph-mon[89359]: osdmap e1375: 8 total, 8 up, 7 in 2024-09-21T10:39:52.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:52.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:52.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.791 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:52.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:52.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:52.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:52.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:53.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:53.055 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:53.157 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:53.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:53.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:53.297 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:53.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:53 smithi119 ceph-mon[105288]: pgmap v2274: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 4780/315009 objects misplaced (1.517%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:39:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:53 smithi119 ceph-mon[101432]: pgmap v2274: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 4780/315009 objects misplaced (1.517%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:39:53.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:53.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:53 smithi145 ceph-mon[89359]: pgmap v2274: 109 pgs: 1 active+recovering+undersized+remapped, 108 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 816 op/s; 4780/315009 objects misplaced (1.517%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:39:53.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:53.591 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:53.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:53.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:53.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:53.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:53.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:53.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:53.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:53.998 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:53.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:54.099 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:54.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:54.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:54.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:54.248 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:54 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 104/305133 objects degraded (0.034%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:39:54.249 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:54 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 104/305133 objects degraded (0.034%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:39:54.280 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:54.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.416 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:54.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:54.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:54 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 104/305133 objects degraded (0.034%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:39:54.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:54.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:54.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:54.700 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:54.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:54.802 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:54.832 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:39:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:39:54] "GET /metrics HTTP/1.1" 200 36876 "" "Prometheus/2.51.0" 2024-09-21T10:39:54.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:54.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:54.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:54.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:55.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:55.130 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:39:55.130 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:39:55.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:39:55.130 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55556 47.4 47:20ae005e:::benchmark_data_smithi119_238957_object55555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406246+0000", 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 352.71900176000003, 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.049347604, 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55556 2024-09-21T10:39:55.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406246+0000", 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406246+0000", 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406248+0000", 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3640000000000002e-06 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406272+0000", 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3833999999999999e-05 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406292+0000", 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9598000000000001e-05 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406300+0000", 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5469999999999997e-06 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406522+0000", 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00022146899999999999 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406528+0000", 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1809999999999998e-06 2024-09-21T10:39:55.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589630+0000", 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.183101813 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589635+0000", 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.604e-06 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.455474+0000", 2024-09-21T10:39:55.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.86583890500000005 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.455594+0000", 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119289 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55563 47.14 47:2837fb2c:::benchmark_data_smithi119_238957_object55562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406329+0000", 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 352.71891864899999, 2024-09-21T10:39:55.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0804854319999999, 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55563 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406329+0000", 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406329+0000", 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406330+0000", 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7899999999999998e-07 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406351+0000", 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1458999999999999e-05 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406362+0000", 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1031e-05 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406368+0000", 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6409999999999999e-06 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589708+0000", 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.183339892 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589716+0000", 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9189999999999999e-06 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.477620+0000", 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.88790453000000003 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for degraded object", 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.480345+0000", 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002724831 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.486756+0000", 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064112860000000004 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.486815+0000", 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8264000000000003e-05 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55569 47.4 47:24811331:::benchmark_data_smithi119_238957_object55568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:39:55.141 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 352.71887178200001, 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.061292364, 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55569 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0699999999999998e-07 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406395+0000", 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9015999999999999e-05 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406405+0000", 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1989999999999995e-06 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406410+0000", 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8450000000000002e-06 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406663+0000", 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025364300000000001 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406668+0000", 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6809999999999997e-06 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589689+0000", 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.18302143100000001 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589693+0000", 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7129999999999999e-06 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.455625+0000", 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.86593157700000001 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for degraded object", 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.458580+0000", 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029557210000000001 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.467600+0000", 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090200220000000008 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.467668+0000", 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8108999999999995e-05 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55575 47.15 47:afa8fef1:::benchmark_data_smithi119_238957_object55574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.411557+0000", 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 352.71369062799999, 2024-09-21T10:39:55.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0780909969999999, 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55575 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411557+0000", 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411557+0000", 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411559+0000", 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1749999999999999e-06 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411587+0000", 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7732000000000001e-05 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411612+0000", 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5049999999999999e-05 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411622+0000", 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4900000000000006e-06 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.571507+0000", 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15988506099999999 2024-09-21T10:39:55.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.571519+0000", 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2184000000000001e-05 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481556+0000", 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.91003687 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481709+0000", 2024-09-21T10:39:55.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015347599999999999 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481865+0000", 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000155339 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.483549+0000", 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00168486 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.485261+0000", 2024-09-21T10:39:55.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017114330000000001 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.489542+0000", 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042811730000000001 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.489605+0000", 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3213999999999995e-05 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.489648+0000", 2024-09-21T10:39:55.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2939999999999999e-05 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188940 47.14 47:2b2878f2:::benchmark_data_smithi119_238957_object64111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.337463+0000", 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.78778489600001, 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0420609350000001, 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:39:55.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188940 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337463+0000", 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337463+0000", 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337465+0000", 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7710000000000001e-06 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337471+0000", 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337487+0000", 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5948e-05 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337495+0000", 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3529999999999995e-06 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337561+0000", 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5837e-05 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337705+0000", 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014370100000000001 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338300+0000", 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059517599999999999 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.342262+0000", 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039625090000000003 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352397+0000", 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010134513 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352407+0000", 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0586e-05 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.379449+0000", 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0270420339999999 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.379524+0000", 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4417000000000003e-05 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188944 47.14 47:28ba1a9b:::benchmark_data_smithi119_238957_object64115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:39:55.160 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.338123+0000", 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.78712459900001, 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.057266204, 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188944 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338123+0000", 2024-09-21T10:39:55.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338123+0000", 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338124+0000", 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.004e-06 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338128+0000", 2024-09-21T10:39:55.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1649999999999999e-06 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338141+0000", 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2663999999999999e-05 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338149+0000", 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8849999999999999e-06 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.342528+0000", 2024-09-21T10:39:55.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043793360000000002 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.342735+0000", 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020680300000000001 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.343233+0000", 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049773700000000005 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352434+0000", 2024-09-21T10:39:55.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092011950000000006 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352441+0000", 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1470000000000004e-06 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387576+0000", 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0351347230000001 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for missing object", 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387697+0000", 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121187 2024-09-21T10:39:55.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395338+0000", 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076414100000000004 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395389+0000", 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0948000000000001e-05 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.166 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188958 47.14 47:28b3f8ea:::benchmark_data_smithi119_238957_object64129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.358429+0000", 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.76681845600001, 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0371616340000001, 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188958 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358429+0000", 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358429+0000", 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358431+0000", 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4479999999999999e-06 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358436+0000", 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7099999999999998e-06 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358452+0000", 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6008999999999999e-05 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358459+0000", 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3420000000000002e-06 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.170 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:55.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:55.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358518+0000", 2024-09-21T10:39:55.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9611000000000002e-05 2024-09-21T10:39:55.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358529+0000", 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0013000000000001e-05 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387780+0000", 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.029251371 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387899+0000", 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119255 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.388392+0000", 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049286499999999999 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395300+0000", 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006907631 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395441+0000", 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014154599999999999 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395491+0000", 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9666000000000002e-05 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395522+0000", 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1401e-05 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395591+0000", 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8765999999999999e-05 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.174 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188969 47.14 47:2833d4a7:::benchmark_data_smithi119_238957_object64140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.369661+0000", 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.75558646299999, 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0431977539999999, 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.175 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188969 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369661+0000", 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369661+0000", 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369663+0000", 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4100000000000001e-06 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369667+0000", 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9890000000000003e-06 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369677+0000", 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9340000000000001e-06 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369683+0000", 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.947e-06 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378389+0000", 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087062170000000005 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378398+0000", 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8729999999999996e-06 2024-09-21T10:39:55.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.390476+0000", 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.012078727 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.390552+0000", 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5896999999999995e-05 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.390766+0000", 2024-09-21T10:39:55.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021314899999999999 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395307+0000", 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045415730000000001 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.403326+0000", 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080186179999999999 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412690+0000", 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.009364341 2024-09-21T10:39:55.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412746+0000", 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6301999999999997e-05 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412859+0000", 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011277699999999999 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.181 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188973 47.14 47:2ce0866e:::benchmark_data_smithi119_238957_object64144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.375700+0000", 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.74954790199999, 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.03734028, 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188973 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375700+0000", 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375700+0000", 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375702+0000", 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7659999999999999e-06 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375707+0000", 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3190000000000002e-06 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375724+0000", 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6906e-05 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375732+0000", 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5900000000000002e-06 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378545+0000", 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028136979999999999 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378551+0000", 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1399999999999997e-06 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.392161+0000", 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0136094010000001 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.392229+0000", 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7960000000000007e-05 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.392419+0000", 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019038200000000001 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395456+0000", 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003036969 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.403365+0000", 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079085119999999995 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412904+0000", 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095398970000000003 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412952+0000", 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7236e-05 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413040+0000", 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8504000000000005e-05 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188993 47.14 47:2a03ebbf:::benchmark_data_smithi119_238957_object64164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.393087+0000", 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.7321608, 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0201839669999999, 2024-09-21T10:39:55.188 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188993 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393087+0000", 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393087+0000", 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393088+0000", 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2810000000000001e-06 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393093+0000", 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8849999999999998e-06 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393108+0000", 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4514e-05 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393116+0000", 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4810000000000006e-06 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393325+0000", 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020838399999999999 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393334+0000", 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7089999999999997e-06 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.393584+0000", 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0002499899999999 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.393648+0000", 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4220999999999994e-05 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.393901+0000", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025227499999999999 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395461+0000", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015599240000000001 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.407039+0000", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011578128 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413113+0000", 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060744789999999998 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413169+0000", 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5862000000000001e-05 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413271+0000", 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101834 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:189012 47.14 47:2d28305b:::benchmark_data_smithi119_238957_object64183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.410915+0000", 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 224.71433242699999, 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0085174750000001, 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 189012 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410915+0000", 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410915+0000", 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410917+0000", 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.916e-06 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410921+0000", 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0960000000000003e-06 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410931+0000", 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.91e-06 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410938+0000", 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8009999999999998e-06 2024-09-21T10:39:55.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.411001+0000", 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2686000000000003e-05 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.411010+0000", 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4700000000000008e-06 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395623+0000", 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98461289399999996 2024-09-21T10:39:55.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395693+0000", 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9830000000000001e-05 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395949+0000", 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025582600000000001 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.405124+0000", 2024-09-21T10:39:55.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0091752299999999995 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.407085+0000", 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019611939999999999 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.419297+0000", 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012211539 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.419335+0000", 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8654e-05 2024-09-21T10:39:55.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.419433+0000", 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7429000000000005e-05 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:215054 47.4 47:2419daf5:::benchmark_data_smithi119_238957_object90225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1335)", 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:42.661029+0000", 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 192.46421860300001, 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0238612810000001, 2024-09-21T10:39:55.201 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 215054 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661029+0000", 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661029+0000", 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661031+0000", 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.454e-06 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661034+0000", 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6490000000000001e-06 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661043+0000", 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7250000000000003e-06 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661051+0000", 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0439999999999996e-06 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661624+0000", 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057317900000000005 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661740+0000", 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116045 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,4,6", 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661886+0000", 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014524899999999999 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.665341+0000", 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034553710000000001 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.671135+0000", 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057939430000000002 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.671150+0000", 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4844000000000001e-05 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:43.684799+0000", 2024-09-21T10:39:55.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0136492100000001 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:43.684891+0000", 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1568000000000001e-05 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:83853 48.6 48:6e8d7be1:::benchmark_data_smithi119_267254_object83852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:48.363487+0000", 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 66.761761141999997, 2024-09-21T10:39:55.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.243881986, 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 83853 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.363487+0000", 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.363487+0000", 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.363488+0000", 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4799999999999997e-07 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.363510+0000", 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1900999999999999e-05 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.363521+0000", 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1763000000000001e-05 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.363528+0000", 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0350000000000004e-06 2024-09-21T10:39:55.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.370039+0000", 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006510839 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.370183+0000", 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000143435 2024-09-21T10:39:55.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.370309+0000", 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012623499999999999 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.373357+0000", 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030481990000000001 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374010+0000", 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000653059 2024-09-21T10:39:55.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.607119+0000", 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23310921200000001 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.607255+0000", 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000136122 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.607369+0000", 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000113238 2024-09-21T10:39:55.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:83866 48.f 48:f62d76e1:::benchmark_data_smithi119_267254_object83865:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:48.373898+0000", 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 66.751350293000002, 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.244848544, 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.214 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 83866 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.373898+0000", 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.373898+0000", 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.373899+0000", 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.023e-06 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.373919+0000", 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0275000000000002e-05 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374021+0000", 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102534 2024-09-21T10:39:55.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374029+0000", 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6140000000000003e-06 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374070+0000", 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1035000000000003e-05 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374157+0000", 2024-09-21T10:39:55.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7010999999999998e-05 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374269+0000", 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000112386 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.379604+0000", 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0053341059999999999 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.379884+0000", 2024-09-21T10:39:55.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028034799999999998 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.618649+0000", 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23876544299999999 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.618705+0000", 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5373e-05 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.618746+0000", 2024-09-21T10:39:55.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1396000000000001e-05 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:83868 48.f 48:fe2e2541:::benchmark_data_smithi119_267254_object83867:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:48.374570+0000", 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 66.750677362000005, 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25024063299999999, 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.220 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 83868 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374570+0000", 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374570+0000", 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374571+0000", 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3799999999999996e-07 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374594+0000", 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2381000000000001e-05 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374607+0000", 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2738999999999999e-05 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.374613+0000", 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.754e-06 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.376770+0000", 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021569549999999999 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.376903+0000", 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013232900000000001 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.377041+0000", 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013844500000000001 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.379741+0000", 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027003719999999999 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.379919+0000", 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017755599999999999 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.624681+0000", 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24476181299999999 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.624776+0000", 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5379999999999998e-05 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:48.624811+0000", 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4971000000000001e-05 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:90425 48.f 48:fcae2d76:::benchmark_data_smithi119_267254_object90424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:54.384499+0000", 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 60.740748453000002, 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.248613059, 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 90425 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.226 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384499+0000", 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384499+0000", 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384500+0000", 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9200000000000001e-07 2024-09-21T10:39:55.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384545+0000", 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5396999999999999e-05 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384553+0000", 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.328e-06 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384558+0000", 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1939999999999997e-06 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384605+0000", 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6835999999999997e-05 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384742+0000", 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000137665 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.384882+0000", 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013948699999999999 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.388853+0000", 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039709539999999996 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.390493+0000", 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016402739999999999 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.633002+0000", 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24250860799999999 2024-09-21T10:39:55.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.633063+0000", 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1488000000000005e-05 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.633112+0000", 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9336000000000003e-05 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:90434 48.6 48:604dc6a6:::benchmark_data_smithi119_267254_object90433:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:54.394517+0000", 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 60.730730399000002, 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24961629599999999, 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 90434 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394517+0000", 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394517+0000", 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394519+0000", 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4699999999999999e-06 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394655+0000", 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013656399999999999 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394669+0000", 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3920000000000001e-05 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394677+0000", 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4220000000000003e-06 2024-09-21T10:39:55.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394750+0000", 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3085999999999998e-05 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394865+0000", 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000115003 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394962+0000", 2024-09-21T10:39:55.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7152999999999996e-05 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.400517+0000", 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055550850000000004 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.404345+0000", 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038273830000000002 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.643801+0000", 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23945658 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.643967+0000", 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000165513 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.644134+0000", 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016711699999999999 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.237 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:90435 48.f 48:f116c812:::benchmark_data_smithi119_267254_object90434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:54.394697+0000", 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 60.730550581999999, 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24633121999999999, 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.238 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 90435 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394697+0000", 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394697+0000", 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394698+0000", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4799999999999998e-07 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394715+0000", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6767000000000001e-05 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394722+0000", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.075e-06 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394727+0000", 2024-09-21T10:39:55.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6389999999999997e-06 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394783+0000", 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5810999999999998e-05 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394921+0000", 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013771399999999999 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:39:55.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395072+0000", 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000150834 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.397699+0000", 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0026276210000000001 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.398502+0000", 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080289399999999996 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.640919+0000", 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24241702700000001 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.640976+0000", 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6950000000000002e-05 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.641028+0000", 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2339999999999997e-05 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.243 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:90437 48.6 48:606339cf:::benchmark_data_smithi119_267254_object90436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:54.394745+0000", 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 60.730503216999999, 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.249658035, 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 90437 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.244 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394745+0000", 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394745+0000", 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394745+0000", 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6499999999999999e-07 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394765+0000", 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9718e-05 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394771+0000", 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7020000000000004e-06 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.394776+0000", 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.87e-06 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.397481+0000", 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002704335 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.397625+0000", 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000143869 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.397794+0000", 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000169242 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.400549+0000", 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027555380000000001 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.404391+0000", 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038413380000000001 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.644188+0000", 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23979710300000001 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.644314+0000", 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012593899999999999 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.644403+0000", 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8916e-05 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:90438 48.6 48:60c67c7c:::benchmark_data_smithi119_267254_object90437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:54.395913+0000", 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 60.729335284000001, 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24919843899999999, 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:39:55.250 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 90438 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395913+0000", 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395913+0000", 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.251 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395914+0000", 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.145e-06 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395933+0000", 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9607e-05 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395944+0000", 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0682999999999999e-05 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.252 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.395951+0000", 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0940000000000002e-06 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.400577+0000", 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046255350000000001 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.400717+0000", 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013999699999999999 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.253 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.400871+0000", 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015446700000000001 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.404325+0000", 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034540069999999998 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.404442+0000", 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116477 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.254 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.644439+0000", 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23999757399999999 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.645006+0000", 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00056715699999999999 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:54.645111+0000", 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104696 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.255 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:39:55.256 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:39:55.256 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:39:55.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:55.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:55 smithi145 ceph-mon[89359]: pgmap v2275: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 104/305133 objects degraded (0.034%); 4535/305133 objects misplaced (1.486%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:39:55.303 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:55.388 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:55 smithi119 ceph-mon[105288]: pgmap v2275: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 104/305133 objects degraded (0.034%); 4535/305133 objects misplaced (1.486%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:39:55.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:55 smithi119 ceph-mon[101432]: pgmap v2275: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 104/305133 objects degraded (0.034%); 4535/305133 objects misplaced (1.486%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:39:55.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:55.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:55.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:55.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:55.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:55.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:55.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:55.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:55.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:55.989 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:55.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:55.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:56.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:56.123 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:56.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:56.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:56.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:56.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:56.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:56.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:56.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:56.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:56.751 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:56.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:56.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:56.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:39:56.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:56.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:56.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:57.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:57.033 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:39:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:57.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:57.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:39:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.203 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:57.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:39:57.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:57.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:57.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:57.456 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:39:57.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:57 smithi119 ceph-mon[105288]: pgmap v2276: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 104/300399 objects degraded (0.035%); 4535/300399 objects misplaced (1.510%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:39:57.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1655167719' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:39:57.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:57 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:39:57.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:57 smithi119 ceph-mon[101432]: pgmap v2276: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 104/300399 objects degraded (0.035%); 4535/300399 objects misplaced (1.510%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:39:57.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1655167719' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:39:57.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:57 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:39:57.491 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:57.509 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:39:57.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:57 smithi145 ceph-mon[89359]: pgmap v2276: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 104/300399 objects degraded (0.035%); 4535/300399 objects misplaced (1.510%); 2.2 MiB/s, 35 objects/s recovering 2024-09-21T10:39:57.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1655167719' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:39:57.546 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:57 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:39:57.593 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:39:57.624 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:39:57.726 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:39:57.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:39:57.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:57.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:57.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:58.010 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:58.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:58.139 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:58.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:39:58.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:58.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:58 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:39:58.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:58 smithi145 ceph-mon[89359]: osdmap e1376: 8 total, 8 up, 7 in 2024-09-21T10:39:58.301 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:58.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:58.426 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:58 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:39:58.426 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:58 smithi119 ceph-mon[101432]: osdmap e1376: 8 total, 8 up, 7 in 2024-09-21T10:39:58.428 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:58 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:39:58.428 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:58 smithi119 ceph-mon[105288]: osdmap e1376: 8 total, 8 up, 7 in 2024-09-21T10:39:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.443 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:58.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:58.503 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:39:58.503 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:39:58.531 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:39:58.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:58.545 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:39:58.546 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:39:58.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:58.577 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:39:58.577 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:39:58.612 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:39:58.613 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:39:58.640 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:39:58.641 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:39:58.668 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:39:58.669 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:39:58.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.697 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:58.702 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:39:58.702 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:39:58.731 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:39:58.732 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:39:58.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:39:58.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:39:58.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:58.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:58.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:59.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:39:59.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:39:59.203 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:59.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:59.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:39:59.370 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:59.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:39:59.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:59.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:59.537 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:59 smithi145 ceph-mon[89359]: pgmap v2278: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 104/299520 objects degraded (0.035%); 4535/299520 objects misplaced (1.514%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:39:59.537 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:39:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1716482732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:59.611 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:59 smithi119 ceph-mon[101432]: pgmap v2278: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 104/299520 objects degraded (0.035%); 4535/299520 objects misplaced (1.514%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:39:59.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:39:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1716482732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:59.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:59 smithi119 ceph-mon[105288]: pgmap v2278: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 682 op/s; 104/299520 objects degraded (0.035%); 4535/299520 objects misplaced (1.514%); 1.1 MiB/s, 17 objects/s recovering 2024-09-21T10:39:59.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:39:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1716482732' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:39:59.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:39:59.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:39:59.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:39:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:39:59.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:39:59.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:39:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:39:59.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:00.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:00.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:00.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:00.191 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:00.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:00.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:00.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:00.470 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:00.470 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:00.471 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:40:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:40:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:40:00.471 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:40:00.471 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:00 smithi145 ceph-mon[89359]: Health detail: HEALTH_WARN Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded 2024-09-21T10:40:00.471 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:00 smithi145 ceph-mon[89359]: [WRN] PG_DEGRADED: Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded 2024-09-21T10:40:00.471 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:00 smithi145 ceph-mon[89359]: pg 48.1 is active+recovering+undersized+degraded+remapped, acting [7,6] 2024-09-21T10:40:00.473 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:00.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:00.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:00.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[105288]: Health detail: HEALTH_WARN Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[105288]: [WRN] PG_DEGRADED: Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[105288]: pg 48.1 is active+recovering+undersized+degraded+remapped, acting [7,6] 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[101432]: [WRN] PG_DEGRADED: Degraded data redundancy: 104/299520 objects degraded (0.035%), 1 pg degraded 2024-09-21T10:40:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:00 smithi119 ceph-mon[101432]: pg 48.1 is active+recovering+undersized+degraded+remapped, acting [7,6] 2024-09-21T10:40:00.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:00.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:00.850 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:00.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:00.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:00.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:00.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:01.021 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:01.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:01.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:01.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:01.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:01.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:01.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:01.390 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:01.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:01.492 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:01 smithi119 ceph-mon[101432]: pgmap v2279: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 277/287301 objects degraded (0.096%); 4231/287301 objects misplaced (1.473%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:40:01.492 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:01 smithi119 ceph-mon[105288]: pgmap v2279: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 277/287301 objects degraded (0.096%); 4231/287301 objects misplaced (1.473%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:40:01.515 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:01 smithi145 ceph-mon[89359]: pgmap v2279: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.06k op/s; 277/287301 objects degraded (0.096%); 4231/287301 objects misplaced (1.473%); 2.0 MiB/s, 31 objects/s recovering 2024-09-21T10:40:01.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:01.553 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:40:01.585 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:01.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:01.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:01.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:01.851 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:01.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:01.974 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:01.988 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:01.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:02.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.122 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:02.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:02.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:02.258 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:02.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:02.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:02.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:02.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:02.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:02.682 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:02.682 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:02 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:02.682 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:02 smithi119 ceph-mon[101432]: pgmap v2280: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 954 op/s; 277/286347 objects degraded (0.097%); 4231/286347 objects misplaced (1.478%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:40:02.682 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:02.683 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:02 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:02.683 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:02 smithi119 ceph-mon[105288]: pgmap v2280: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 954 op/s; 277/286347 objects degraded (0.097%); 4231/286347 objects misplaced (1.478%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:40:02.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:02.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:02.816 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:02.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:02.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:02.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:02.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:02.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:02 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:02 smithi145 ceph-mon[89359]: pgmap v2280: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 954 op/s; 277/286347 objects degraded (0.097%); 4231/286347 objects misplaced (1.478%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:40:03.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:03.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:03.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:03.222 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:03.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:03.350 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:03.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:03.452 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:03.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:03.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:03.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:03.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:03.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:03.857 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:03.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:03.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:03.935 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:03.935 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:03.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:04.069 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:04.070 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:04.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:04.080 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:04.080 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:04.080 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:04.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:04.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:04.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:04.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.294 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:04.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:04.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:04.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:40:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:04.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:04.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:04.560 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:04.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:04.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:40:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:40:04] "GET /metrics HTTP/1.1" 200 36877 "" "Prometheus/2.51.0" 2024-09-21T10:40:04.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:04.704 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:04.726 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:40:04.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:04.849 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:04.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:04.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:04.979 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:04.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:04 smithi145 ceph-mon[89359]: pgmap v2281: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 461/276315 objects degraded (0.167%); 3910/276315 objects misplaced (1.415%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:05.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:04 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 277/286347 objects degraded (0.097%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:40:05.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:05.086 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:05.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:05.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:04 smithi119 ceph-mon[101432]: pgmap v2281: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 461/276315 objects degraded (0.167%); 3910/276315 objects misplaced (1.415%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:05.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:04 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 277/286347 objects degraded (0.097%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:40:05.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:04 smithi119 ceph-mon[105288]: pgmap v2281: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 461/276315 objects degraded (0.167%); 3910/276315 objects misplaced (1.415%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:05.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:04 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 277/286347 objects degraded (0.097%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:40:05.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:05.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:05.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:05.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:05.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:05.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:05.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:05.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:05.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:05.677 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:05.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:05.807 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:05.908 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:05.938 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:05.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:06.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:06.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:06.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:06.208 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:06.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:06.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:06.369 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:06.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:06.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:06.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:06.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:06.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:06.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:06.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:06.860 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:40:06.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:06.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:06.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:06.988 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:06 smithi145 ceph-mon[89359]: pgmap v2282: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 959 op/s; 461/271596 objects degraded (0.170%); 3910/271596 objects misplaced (1.440%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:06.988 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3046853701' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:06.988 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:06 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:07.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:07.039 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:07.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:06 smithi119 ceph-mon[105288]: pgmap v2282: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 959 op/s; 461/271596 objects degraded (0.170%); 3910/271596 objects misplaced (1.440%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3046853701' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:06 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:06 smithi119 ceph-mon[101432]: pgmap v2282: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 959 op/s; 461/271596 objects degraded (0.170%); 3910/271596 objects misplaced (1.440%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3046853701' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:06 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:40:07.135 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:07.135 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:07.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:07.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:07.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:07.307 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:07.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:07.444 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:07.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:07.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:07.680 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:07.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:07.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:07.843 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:07.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:07.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:07.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:08.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:08.108 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:08.196 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:07 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:08.196 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:40:08.196 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:07 smithi145 ceph-mon[89359]: osdmap e1377: 8 total, 8 up, 7 in 2024-09-21T10:40:08.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:08.251 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:08.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:08.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:07 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:40:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:07 smithi119 ceph-mon[105288]: osdmap e1377: 8 total, 8 up, 7 in 2024-09-21T10:40:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:07 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:40:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:07 smithi119 ceph-mon[101432]: osdmap e1377: 8 total, 8 up, 7 in 2024-09-21T10:40:08.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:08.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:08.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:08.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:08.614 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:08.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:08.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:08.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.789 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:08.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:08.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:08.923 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:08.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:09.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:09.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:09.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:09.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:09.200 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:08 smithi119 ceph-mon[101432]: pgmap v2284: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 461/270729 objects degraded (0.170%); 3910/270729 objects misplaced (1.444%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:09.202 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:08 smithi119 ceph-mon[105288]: pgmap v2284: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 461/270729 objects degraded (0.170%); 3910/270729 objects misplaced (1.444%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:09.283 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:09.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:08 smithi145 ceph-mon[89359]: pgmap v2284: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 461/270729 objects degraded (0.170%); 3910/270729 objects misplaced (1.444%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:40:09.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:09.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:09.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:09.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:09.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:09.588 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:09.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:09.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:09.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:09.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:09.850 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:09.897 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:40:09.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:09.988 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:09.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:10.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:10.116 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:10.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:09 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 461/270729 objects degraded (0.170%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:10.154 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/81043647' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:40:10.154 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:09 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 461/270729 objects degraded (0.170%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:10.154 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/81043647' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:40:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:09 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 461/270729 objects degraded (0.170%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:10.214 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/81043647' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:40:10.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:10.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:10.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:10.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:10.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:10.485 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:40:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:40:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:40:10.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:10.609 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:10.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:10.711 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:10.732 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:10.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:10.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:10.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:10.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:10.979 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:10.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:11.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:11.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:11.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:10 smithi119 ceph-mon[101432]: pgmap v2285: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 981 op/s; 666/257823 objects degraded (0.258%); 3567/257823 objects misplaced (1.384%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:40:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/81043647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:40:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:10 smithi119 ceph-mon[101432]: osdmap e1378: 8 total, 8 up, 7 in 2024-09-21T10:40:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:10 smithi119 ceph-mon[105288]: pgmap v2285: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 981 op/s; 666/257823 objects degraded (0.258%); 3567/257823 objects misplaced (1.384%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:40:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/81043647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:40:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:10 smithi119 ceph-mon[105288]: osdmap e1378: 8 total, 8 up, 7 in 2024-09-21T10:40:11.156 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:10 smithi145 ceph-mon[89359]: pgmap v2285: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 5.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 981 op/s; 666/257823 objects degraded (0.258%); 3567/257823 objects misplaced (1.384%); 1.7 MiB/s, 27 objects/s recovering 2024-09-21T10:40:11.156 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/81043647' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:40:11.157 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:10 smithi145 ceph-mon[89359]: osdmap e1378: 8 total, 8 up, 7 in 2024-09-21T10:40:11.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:40:11.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:11.225 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:11.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:40:11.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:40:11.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:11.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:11.437 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:11.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:11.472 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:11.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:11.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:11.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:11.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:11.719 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:11.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:11.841 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:11.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:11.965 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:11.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:12.066 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:12.089 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:12.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:12.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:12.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:11 smithi145 ceph-mon[89359]: osdmap e1379: 8 total, 8 up, 7 in 2024-09-21T10:40:12.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:12.346 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:12.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:11 smithi119 ceph-mon[105288]: osdmap e1379: 8 total, 8 up, 7 in 2024-09-21T10:40:12.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:11 smithi119 ceph-mon[101432]: osdmap e1379: 8 total, 8 up, 7 in 2024-09-21T10:40:12.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:12.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:12.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:12.630 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:12.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:12.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:12.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:12.907 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:12.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:13.008 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:13.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:13.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:12 smithi145 ceph-mon[89359]: pgmap v2288: 109 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 764 op/s; 666/257823 objects degraded (0.258%); 3567/257823 objects misplaced (1.384%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:40:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:12 smithi145 ceph-mon[89359]: 48.7 starting backfill to osd.6 from (0'0,0'0] MAX to 1377'10380 2024-09-21T10:40:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:12 smithi145 ceph-mon[89359]: osdmap e1380: 8 total, 8 up, 7 in 2024-09-21T10:40:13.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:13.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:13.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:12 smithi119 ceph-mon[101432]: pgmap v2288: 109 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 764 op/s; 666/257823 objects degraded (0.258%); 3567/257823 objects misplaced (1.384%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:40:13.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:12 smithi119 ceph-mon[101432]: 48.7 starting backfill to osd.6 from (0'0,0'0] MAX to 1377'10380 2024-09-21T10:40:13.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:12 smithi119 ceph-mon[101432]: osdmap e1380: 8 total, 8 up, 7 in 2024-09-21T10:40:13.231 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:12 smithi119 ceph-mon[105288]: pgmap v2288: 109 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 764 op/s; 666/257823 objects degraded (0.258%); 3567/257823 objects misplaced (1.384%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:40:13.232 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:12 smithi119 ceph-mon[105288]: 48.7 starting backfill to osd.6 from (0'0,0'0] MAX to 1377'10380 2024-09-21T10:40:13.232 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:12 smithi119 ceph-mon[105288]: osdmap e1380: 8 total, 8 up, 7 in 2024-09-21T10:40:13.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:13.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:13.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:13.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:13.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:13.599 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:13.606 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:13.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:13.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:13.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:13.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:13.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:13.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:13.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:14.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:14.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:14.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:14.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:14.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.307 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:14.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:14.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:14.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:14.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:14.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:14.484 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:40:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:14.514 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:40:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:14.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:14.549 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:40:14.549 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:14.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:14.579 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:40:14.579 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:14.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:14.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:14.780 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:40:14] "GET /metrics HTTP/1.1" 200 36877 "" "Prometheus/2.51.0" 2024-09-21T10:40:14.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:14.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:14.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:14.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:15.013 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:15.073 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:15.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[105288]: pgmap v2290: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 107 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 723/252249 objects degraded (0.287%); 3210/252249 objects misplaced (1.273%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:40:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2972180281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 723/252249 objects degraded (0.287%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:15.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:15.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[101432]: pgmap v2290: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 107 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 723/252249 objects degraded (0.287%); 3210/252249 objects misplaced (1.273%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:40:15.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2972180281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:15.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 723/252249 objects degraded (0.287%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:15.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:15.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:15.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:15.232 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:14 smithi145 ceph-mon[89359]: pgmap v2290: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 107 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 1.03k op/s; 723/252249 objects degraded (0.287%); 3210/252249 objects misplaced (1.273%); 4.6 MiB/s, 72 objects/s recovering 2024-09-21T10:40:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2972180281' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:14 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 723/252249 objects degraded (0.287%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:15.334 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:15.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:15.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:15.507 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:15.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:15.637 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:15.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:15.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:15.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:15.888 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:15.920 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:15.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:16.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:16.056 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:16.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:16.157 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:16.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:16.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.328 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:16.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:165808 48.1 48:83a6240d:::benchmark_data_smithi119_267254_object42598:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1380)", 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:16.307521+0000", 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.023161877000000001, 2024-09-21T10:40:16.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.023190908999999999, 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 165808 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.307521+0000", 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.307521+0000", 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.307520+0000", 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.307528+0000", 2024-09-21T10:40:16.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6879999999999999e-06 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.307538+0000", 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0370000000000001e-05 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.307544+0000", 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5600000000000001e-06 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.322441+0000", 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01489774 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.322569+0000", 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012797700000000001 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.322706+0000", 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013682200000000001 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:16.337 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:165827 48.1 48.1d0cd11 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1380)", 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:16.327513+0000", 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0031703930000000001, 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032720420000000002, 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 165827 2024-09-21T10:40:16.338 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327513+0000", 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327513+0000", 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:16.339 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327511+0000", 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327519+0000", 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3669999999999999e-06 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327535+0000", 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6039e-05 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.340 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327541+0000", 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6599999999999998e-06 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:165828 48.1 48.499cc3b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1380)", 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:16.327551+0000", 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.003132531, 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032893570000000001, 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:16.341 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 165828 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327551+0000", 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327551+0000", 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.342 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327550+0000", 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327554+0000", 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5420000000000001e-06 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327560+0000", 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.438e-06 2024-09-21T10:40:16.343 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327565+0000", 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2399999999999998e-06 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop_reply(client.46619.0:165808 48.1 e1380/1367)", 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:16.327556+0000", 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0031273709999999999, 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330127, 2024-09-21T10:40:16.344 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327556+0000", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327556+0000", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327555+0000", 2024-09-21T10:40:16.345 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327560+0000", 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5090000000000004e-06 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327569+0000", 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9069999999999996e-06 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:16.346 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:16.327573+0000", 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1509999999999997e-06 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 4 2024-09-21T10:40:16.347 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:40:16.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:16.461 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:16.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.602 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:16.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:16.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:16.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:16.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:16.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:16.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:16.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:17.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:17.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:17.091 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:16 smithi119 ceph-mon[101432]: pgmap v2291: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 723/249072 objects degraded (0.290%); 8411/249072 objects misplaced (3.377%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:40:17.091 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:16 smithi119 ceph-mon[105288]: pgmap v2291: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 723/249072 objects degraded (0.290%); 8411/249072 objects misplaced (3.377%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:40:17.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:17.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:17.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:17.230 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:16 smithi145 ceph-mon[89359]: pgmap v2291: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 486 op/s; 723/249072 objects degraded (0.290%); 8411/249072 objects misplaced (3.377%); 3.1 MiB/s, 49 objects/s recovering 2024-09-21T10:40:17.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:17.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:17.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:17.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:17.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:17.543 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:17.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:40:17.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:17.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:17.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:17.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:17.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:17.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:18.001 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:18.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:18.017 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:18.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:18.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:18.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:18.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:18.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:18.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:18.565 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:18.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:18.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:18.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:18.825 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:18.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:18.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:18.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:19.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:19.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[101432]: pgmap v2292: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 408 op/s; 723/248136 objects degraded (0.291%); 8411/248136 objects misplaced (3.390%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[105288]: pgmap v2292: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 408 op/s; 723/248136 objects degraded (0.291%); 8411/248136 objects misplaced (3.390%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.111 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:19.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:19.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:18 smithi145 ceph-mon[89359]: pgmap v2292: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 408 op/s; 723/248136 objects degraded (0.291%); 8411/248136 objects misplaced (3.390%); 2.4 MiB/s, 37 objects/s recovering 2024-09-21T10:40:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:19.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:19.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:19.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.525 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:19.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:19.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:40:19.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:19.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:19.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:19.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:19.821 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:40:19.829 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1364027260' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:40:19.829 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:19 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:40:19.829 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:19 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 723/248136 objects degraded (0.291%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:19.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:19.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:19.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:19.968 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:40:19.968 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:40:19.968 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:40:19.968 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:40:19.969 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:40:20.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:20.084 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:20.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1364027260' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:40:20.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:19 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:40:20.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:19 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 723/248136 objects degraded (0.291%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:20.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1364027260' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:40:20.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:19 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:40:20.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:19 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 723/248136 objects degraded (0.291%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:20.185 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:20.243 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:20.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.383 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:20.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:20.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:20.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:40:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:40:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:40:20.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:20.654 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:20.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:40:20.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:20.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:20.821 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:40:20.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:20.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:21.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:20 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:40:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:20 smithi145 ceph-mon[89359]: osdmap e1381: 8 total, 8 up, 7 in 2024-09-21T10:40:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:20 smithi145 ceph-mon[89359]: pgmap v2294: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 723/235470 objects degraded (0.307%); 8264/235470 objects misplaced (3.510%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:40:21.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/455496761' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:21.097 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.098 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:21.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[101432]: osdmap e1381: 8 total, 8 up, 7 in 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[101432]: pgmap v2294: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 723/235470 objects degraded (0.307%); 8264/235470 objects misplaced (3.510%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/455496761' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[105288]: osdmap e1381: 8 total, 8 up, 7 in 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[105288]: pgmap v2294: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 929 op/s; 723/235470 objects degraded (0.307%); 8264/235470 objects misplaced (3.510%); 3.5 MiB/s, 55 objects/s recovering 2024-09-21T10:40:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/455496761' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:21.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:40:21.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:21.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:21.337 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.382 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:21.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:21.394 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:21.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.520 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:21.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:21.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:21.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:21.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:21.778 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:21.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:21.900 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:21.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:22.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:22.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:22.115 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:21 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:22.115 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/455496761' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:22.115 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:21 smithi145 ceph-mon[89359]: osdmap e1382: 8 total, 8 up, 7 in 2024-09-21T10:40:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:21 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/455496761' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:21 smithi119 ceph-mon[105288]: osdmap e1382: 8 total, 8 up, 7 in 2024-09-21T10:40:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:21 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/455496761' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:21 smithi119 ceph-mon[101432]: osdmap e1382: 8 total, 8 up, 7 in 2024-09-21T10:40:22.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:22.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:22.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:22.304 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:22.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:22.434 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:22.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:22.566 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:22.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:22.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.702 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:22.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:22.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:22.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:22.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:22.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:22.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:23.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:23.100 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:23.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:22 smithi119 ceph-mon[101432]: pgmap v2296: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 723/234618 objects degraded (0.308%); 7980/234618 objects misplaced (3.401%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:40:23.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:22 smithi119 ceph-mon[105288]: pgmap v2296: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 723/234618 objects degraded (0.308%); 7980/234618 objects misplaced (3.401%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:40:23.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:23.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:23.263 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:22 smithi145 ceph-mon[89359]: pgmap v2296: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 733 op/s; 723/234618 objects degraded (0.308%); 7980/234618 objects misplaced (3.401%); 1.1 MiB/s, 18 objects/s recovering 2024-09-21T10:40:23.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:23.375 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:23.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:23.512 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:23.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:23.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:23.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:23.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:23.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:23.878 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:23.905 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:23.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:24.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:24.033 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:24.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:24.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:24.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:24.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:24.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:24.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:24.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:24.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:24.584 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:24.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:24.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:24.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:24.833 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:40:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:40:24] "GET /metrics HTTP/1.1" 200 36858 "" "Prometheus/2.51.0" 2024-09-21T10:40:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:24.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:24.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:24.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:25.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:25.017 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:24 smithi145 ceph-mon[89359]: pgmap v2297: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 970 op/s; 782/225729 objects degraded (0.346%); 7738/225729 objects misplaced (3.428%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:40:25.017 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:24 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 782/225729 objects degraded (0.346%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:25.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:25.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:24 smithi119 ceph-mon[101432]: pgmap v2297: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 970 op/s; 782/225729 objects degraded (0.346%); 7738/225729 objects misplaced (3.428%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:40:25.113 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:24 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 782/225729 objects degraded (0.346%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:25.114 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:24 smithi119 ceph-mon[105288]: pgmap v2297: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 970 op/s; 782/225729 objects degraded (0.346%); 7738/225729 objects misplaced (3.428%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:40:25.114 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:24 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 782/225729 objects degraded (0.346%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:25.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:25.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:179480 47.13 47:cd216aad:::benchmark_data_smithi119_238957_object54651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:40:25.166 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:01.347978+0000", 2024-09-21T10:40:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 263.81191982399997, 2024-09-21T10:40:25.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10888231600000001, 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 179480 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347978+0000", 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347978+0000", 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347976+0000", 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.347984+0000", 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.96e-06 2024-09-21T10:40:25.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348002+0000", 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8287999999999998e-05 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348009+0000", 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3989999999999998e-06 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348062+0000", 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2398999999999998e-05 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348228+0000", 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000166377 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.348762+0000", 2024-09-21T10:40:25.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00053391799999999998 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.351455+0000", 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026928989999999999 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.351538+0000", 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2875999999999997e-05 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.456764+0000", 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.105226323 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.456819+0000", 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5229999999999998e-05 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:01.456860+0000", 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0645999999999997e-05 2024-09-21T10:40:25.175 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:186102 47.e 47:7755fc4e:::benchmark_data_smithi119_238957_object61273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:07.606247+0000", 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 257.55365101899997, 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.109293795, 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.176 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 186102 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606247+0000", 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606247+0000", 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606245+0000", 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606252+0000", 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5509999999999996e-06 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606265+0000", 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3397999999999999e-05 2024-09-21T10:40:25.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606271+0000", 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9410000000000004e-06 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606315+0000", 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4279000000000001e-05 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606426+0000", 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011014 2024-09-21T10:40:25.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.606580+0000", 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000154174 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610203+0000", 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036228240000000002 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.614098+0000", 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0038949929999999998 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715431+0000", 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10133310099999999 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715505+0000", 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4307000000000005e-05 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715540+0000", 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5410000000000001e-05 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:186106 47.e 47:733d1a38:::benchmark_data_smithi119_238957_object61277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:07.608957+0000", 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 257.55094044200001, 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.106723203, 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.182 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 186106 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608957+0000", 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608957+0000", 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608956+0000", 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608961+0000", 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8539999999999996e-06 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608974+0000", 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3225e-05 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.608980+0000", 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0800000000000002e-06 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610511+0000", 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001531078 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610630+0000", 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118622 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.610750+0000", 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000120294 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.619388+0000", 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0086377269999999996 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.619483+0000", 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4858999999999994e-05 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715576+0000", 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.096093637999999995 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715645+0000", 2024-09-21T10:40:25.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8706000000000003e-05 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:07.715680+0000", 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5246000000000001e-05 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:197701 47.b 47:d43e12d0:::benchmark_data_smithi119_238957_object72872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.915681+0000", 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 246.24421660300001, 2024-09-21T10:40:25.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15200201699999999, 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 197701 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915681+0000", 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915681+0000", 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915680+0000", 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915686+0000", 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9399999999999999e-06 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915694+0000", 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2169999999999994e-06 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915698+0000", 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.924e-06 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915730+0000", 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2397999999999997e-05 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915812+0000", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2003999999999996e-05 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.915887+0000", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4777999999999995e-05 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.920574+0000", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0046868259999999998 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067316+0000", 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.146742226 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067434+0000", 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011733 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067562+0000", 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012795300000000001 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.067683+0000", 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000121478 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:197721 47.13 47:cdfee054:::benchmark_data_smithi119_238957_object72892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1327)", 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:18.935578+0000", 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 246.224319853, 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.106628818, 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 197721 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935578+0000", 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935578+0000", 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935576+0000", 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935584+0000", 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1489999999999994e-06 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935594+0000", 2024-09-21T10:40:25.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8889999999999997e-06 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935599+0000", 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8980000000000004e-06 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935639+0000", 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9348999999999999e-05 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.935715+0000", 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6186999999999994e-05 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:18.936207+0000", 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049191600000000001 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.041981+0000", 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10577399699999999 2024-09-21T10:40:25.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042011+0000", 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0502e-05 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042140+0000", 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012901500000000001 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042177+0000", 2024-09-21T10:40:25.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7181999999999997e-05 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:19.042207+0000", 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9240000000000001e-05 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226789 47.e 47:70d5f70c:::benchmark_data_smithi119_238957_object101960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:56.336615+0000", 2024-09-21T10:40:25.201 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.82328257699999, 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.245457536, 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226789 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336615+0000", 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336615+0000", 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336613+0000", 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336625+0000", 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2061e-05 2024-09-21T10:40:25.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336644+0000", 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9349999999999999e-05 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336654+0000", 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7990000000000005e-06 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336704+0000", 2024-09-21T10:40:25.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0090999999999999e-05 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336823+0000", 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011842500000000001 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336930+0000", 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010705 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.340115+0000", 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00318534 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.342634+0000", 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025185920000000001 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.581991+0000", 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.239357232 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582029+0000", 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7951999999999998e-05 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582073+0000", 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3699000000000003e-05 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226796 47.e 47:7b665d5d:::benchmark_data_smithi119_238957_object101967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:40:25.207 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:56.349363+0000", 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 208.81053465100001, 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23279523299999999, 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226796 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349363+0000", 2024-09-21T10:40:25.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349363+0000", 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349361+0000", 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349376+0000", 2024-09-21T10:40:25.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.466e-05 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349388+0000", 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349393+0000", 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6319999999999999e-06 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349437+0000", 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4230000000000002e-05 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349563+0000", 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012641 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349639+0000", 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5679000000000005e-05 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.353138+0000", 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034988020000000001 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.353630+0000", 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049189000000000004 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582107+0000", 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.22847714799999999 2024-09-21T10:40:25.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582136+0000", 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9453000000000001e-05 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582158+0000", 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1905000000000001e-05 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.213 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:230876 47.b 47:d81b5366:::benchmark_data_smithi119_238957_object106047:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1341)", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:37:02.184799+0000", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 202.975098879, 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.12443298799999999, 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230876 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184799+0000", 2024-09-21T10:40:25.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184799+0000", 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184798+0000", 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184803+0000", 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2769999999999996e-06 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184812+0000", 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5589999999999994e-06 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184826+0000", 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3720000000000001e-05 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184871+0000", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4471999999999997e-05 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184990+0000", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000119064 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,4,6", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.185101+0000", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011158399999999999 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.188215+0000", 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031135500000000001 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.188223+0000", 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1100000000000003e-06 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.188392+0000", 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000168643 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309053+0000", 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.12066115099999999 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309117+0000", 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3843000000000005e-05 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309232+0000", 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011513700000000001 2024-09-21T10:40:25.219 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:230878 47.e 47:79cd3af5:::benchmark_data_smithi119_238957_object106049:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1341)", 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:37:02.184835+0000", 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 202.975062933, 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.124528526, 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230878 2024-09-21T10:40:25.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184835+0000", 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184835+0000", 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184834+0000", 2024-09-21T10:40:25.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184837+0000", 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2759999999999999e-06 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184840+0000", 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8469999999999998e-06 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184843+0000", 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.452e-06 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184880+0000", 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6729e-05 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.184985+0000", 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000105728 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.185101+0000", 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115876 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.188322+0000", 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003220962 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.189622+0000", 2024-09-21T10:40:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001299555 2024-09-21T10:40:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309287+0000", 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.119665454 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309327+0000", 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9298999999999998e-05 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309363+0000", 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6644999999999999e-05 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:230890 47.b 47:d8601318:::benchmark_data_smithi119_238957_object106061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1341)", 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:37:02.200168+0000", 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 202.95972993800001, 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10924196999999999, 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 230890 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200168+0000", 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200168+0000", 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200166+0000", 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200172+0000", 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1370000000000001e-06 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200182+0000", 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8840000000000006e-06 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200185+0000", 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8840000000000004e-06 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200247+0000", 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1655999999999994e-05 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200329+0000", 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2222e-05 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,4,6", 2024-09-21T10:40:25.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.200409+0000", 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9287000000000002e-05 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.202831+0000", 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024224469999999999 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.202923+0000", 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2322000000000001e-05 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.202965+0000", 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1635999999999999e-05 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309269+0000", 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.106303911 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309313+0000", 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4348999999999998e-05 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:37:02.309410+0000", 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6387999999999999e-05 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:49531 48.b 48:dc8908d6:::benchmark_data_smithi119_267254_object49530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1359)", 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:22.424222+0000", 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 122.735675816, 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.115546649, 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.233 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 49531 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424222+0000", 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424222+0000", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424221+0000", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424423+0000", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000201848 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424462+0000", 2024-09-21T10:40:25.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9137000000000002e-05 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424470+0000", 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8220000000000007e-06 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424522+0000", 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2185000000000003e-05 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424799+0000", 2024-09-21T10:40:25.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027753800000000003 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.424918+0000", 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118972 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427018+0000", 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020995039999999999 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427293+0000", 2024-09-21T10:40:25.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00027563400000000002 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.539597+0000", 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.112303983 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.539685+0000", 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7721999999999998e-05 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.539768+0000", 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3288999999999995e-05 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:49537 48.b 48:d3726101:::benchmark_data_smithi119_267254_object49536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1359)", 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:22.427801+0000", 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 122.732096627, 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.112356068, 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.239 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 49537 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427801+0000", 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427801+0000", 2024-09-21T10:40:25.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427799+0000", 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427845+0000", 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5847000000000003e-05 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.241 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427855+0000", 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6399999999999992e-06 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427859+0000", 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0319999999999997e-06 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427888+0000", 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9187999999999999e-05 2024-09-21T10:40:25.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.427987+0000", 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8782000000000004e-05 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.428082+0000", 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4838000000000002e-05 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.429752+0000", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001670314 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.430264+0000", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051211500000000005 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.540069+0000", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.109804551 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.540126+0000", 2024-09-21T10:40:25.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7666e-05 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:22.540157+0000", 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0549999999999997e-05 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:117661 48.b 48:de3ca175:::benchmark_data_smithi119_267254_object117660:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1370)", 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:23.917397+0000", 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 61.242500915999997, 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.12586161800000001, 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.245 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 117661 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917397+0000", 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917397+0000", 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917396+0000", 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917434+0000", 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7295999999999997e-05 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917441+0000", 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2139999999999999e-06 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917444+0000", 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6869999999999999e-06 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917486+0000", 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2376000000000002e-05 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917614+0000", 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012808900000000001 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.917761+0000", 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000147446 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.920515+0000", 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002753943 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:23.920925+0000", 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00041008800000000002 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.043190+0000", 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.122264096 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.043222+0000", 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2647000000000001e-05 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.043258+0000", 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6047000000000002e-05 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:118354 48.1 48:8ee7a202:::benchmark_data_smithi119_267254_object118353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1370)", 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:24.805707+0000", 2024-09-21T10:40:25.251 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.354190422999999, 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.111915946, 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 118354 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.805707+0000", 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.805707+0000", 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.805706+0000", 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.805790+0000", 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3818000000000003e-05 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.805802+0000", 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2771e-05 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.805807+0000", 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0370000000000002e-06 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.821072+0000", 2024-09-21T10:40:25.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015264761999999999 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.821215+0000", 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014237700000000001 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.821360+0000", 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000144957 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.255 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.833623+0000", 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012263427 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.833630+0000", 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2880000000000004e-06 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.914657+0000", 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.081026476 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.914718+0000", 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0739999999999998e-05 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.917623+0000", 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029056020000000002 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:118394 48.1 48:8e34eca8:::benchmark_data_smithi119_267254_object118393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1370)", 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:24.848411+0000", 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.311486145000003, 2024-09-21T10:40:25.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.124643854, 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 118394 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848411+0000", 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848411+0000", 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848410+0000", 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848470+0000", 2024-09-21T10:40:25.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0353999999999998e-05 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848483+0000", 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2299e-05 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848487+0000", 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5190000000000003e-06 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848534+0000", 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6485e-05 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848634+0000", 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100744 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.848735+0000", 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100484 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.851502+0000", 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00276736 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.851999+0000", 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049686999999999995 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.971429+0000", 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.119429994 2024-09-21T10:40:25.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.971470+0000", 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0908000000000002e-05 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.973055+0000", 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001585181 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.263 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:118404 48.1 48:87405601:::benchmark_data_smithi119_267254_object118403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1370)", 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:24.858332+0000", 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.301565412999999, 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.11645225100000001, 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 118404 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858332+0000", 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858332+0000", 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858331+0000", 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858363+0000", 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1946000000000002e-05 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858373+0000", 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0115000000000001e-05 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858377+0000", 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5240000000000002e-06 2024-09-21T10:40:25.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858422+0000", 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4229e-05 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858620+0000", 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019826800000000001 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.858782+0000", 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016239499999999999 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.862713+0000", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039305119999999997 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.862812+0000", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9407999999999994e-05 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.973099+0000", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.110286547 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.973133+0000", 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4100999999999998e-05 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.974784+0000", 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016515569999999999 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:118412 48.1 48:8d4911d3:::benchmark_data_smithi119_267254_object118411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1370)", 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:24.867662+0000", 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.292235921, 2024-09-21T10:40:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.10883944600000001, 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 118412 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867662+0000", 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867662+0000", 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867660+0000", 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867717+0000", 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7114000000000002e-05 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.271 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867734+0000", 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7099999999999999e-05 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867741+0000", 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.272e-06 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.867787+0000", 2024-09-21T10:40:25.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5717000000000003e-05 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.868027+0000", 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00023950499999999999 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.868222+0000", 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019572099999999999 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.871628+0000", 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034053349999999998 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.871754+0000", 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012673200000000001 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.974840+0000", 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.103085283 2024-09-21T10:40:25.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.974877+0000", 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7585e-05 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.976501+0000", 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001623871 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.275 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:118422 48.1 48:869a7b6f:::benchmark_data_smithi119_267254_object118421:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1370)", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:24.877775+0000", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 60.282122909000002, 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.117250278, 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 118422 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.877775+0000", 2024-09-21T10:40:25.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.877775+0000", 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.877774+0000", 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.877834+0000", 2024-09-21T10:40:25.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0093000000000002e-05 2024-09-21T10:40:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.278 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.877850+0000", 2024-09-21T10:40:25.278 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6133999999999998e-05 2024-09-21T10:40:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.877856+0000", 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.579e-06 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.882358+0000", 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045017219999999997 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.882458+0000", 2024-09-21T10:40:25.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9574999999999994e-05 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.882564+0000", 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010594699999999999 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.892774+0000", 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010210024 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.892812+0000", 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.875e-05 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.992761+0000", 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.099948910000000002 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.992804+0000", 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2985000000000003e-05 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:24.995025+0000", 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022206470000000001 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:119043 48.b 48:dbf30873:::benchmark_data_smithi119_267254_object119042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1371)", 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:25.550951+0000", 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 59.608946961999997, 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.125423433, 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.283 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 119043 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.550951+0000", 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.550951+0000", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.550949+0000", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.551016+0000", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6323999999999997e-05 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.551026+0000", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0016e-05 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.551031+0000", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1030000000000001e-06 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.551064+0000", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.286e-05 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.551159+0000", 2024-09-21T10:40:25.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5024000000000004e-05 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.551254+0000", 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5251e-05 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.555365+0000", 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041108869999999997 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.555490+0000", 2024-09-21T10:40:25.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012530899999999999 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.676303+0000", 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.120812479 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.676337+0000", 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3914000000000001e-05 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.676374+0000", 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7469999999999999e-05 2024-09-21T10:40:25.288 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:119047 48.b 48:df39945c:::benchmark_data_smithi119_267254_object119046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1371)", 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:39:25.555929+0000", 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 59.603968187, 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.12064596499999999, 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 119047 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.289 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.555929+0000", 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.555929+0000", 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.555928+0000", 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.556026+0000", 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7598999999999998e-05 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.556036+0000", 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0139000000000001e-05 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.556040+0000", 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9369999999999997e-06 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.556074+0000", 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3874999999999997e-05 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.556175+0000", 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010114400000000001 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.556276+0000", 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100623 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.558879+0000", 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026037040000000001 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.559074+0000", 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000194539 2024-09-21T10:40:25.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.676509+0000", 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.117435093 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.676542+0000", 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2911999999999999e-05 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:39:25.676575+0000", 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3590999999999997e-05 2024-09-21T10:40:25.294 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.295 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.295 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.295 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:25.295 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:25.295 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:40:25.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:25.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:25.458 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:25.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:25.600 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:25.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:25.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:25.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:25.869 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:25.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:25.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:25.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:26.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:26.141 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:26.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:26.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:26.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:26.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:40:26.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:26.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:26.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:26.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:26.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:26.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.701 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:26.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:26.713 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:26.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:26.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:26.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:26.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:40:26.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:26.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:27.005 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:26 smithi119 ceph-mon[105288]: pgmap v2298: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 782/221421 objects degraded (0.353%); 7445/221421 objects misplaced (3.362%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:40:27.005 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:26 smithi119 ceph-mon[101432]: pgmap v2298: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 782/221421 objects degraded (0.353%); 7445/221421 objects misplaced (3.362%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:40:27.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:27.113 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:27.171 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:26 smithi145 ceph-mon[89359]: pgmap v2298: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 782/221421 objects degraded (0.353%); 7445/221421 objects misplaced (3.362%); 2.6 MiB/s, 41 objects/s recovering 2024-09-21T10:40:27.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:27.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:27.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:27.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:27.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:27.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:27.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:27.564 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:27.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:27.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:27.799 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:27.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:27.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:27.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:27.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:27.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:28.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:28.088 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:28.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:28.189 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:28.211 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:28.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:28.347 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:28.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:28.477 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:28.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:28.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:28.616 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:28.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:28.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:28.784 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:40:28.784 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:40:28.784 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:177234 48.2 48:494fe2bb:::benchmark_data_smithi119_267254_object54024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.772953+0000", 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0077256089999999996, 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077553789999999997, 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 177234 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.785 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.772953+0000", 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.772953+0000", 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.772953+0000", 2024-09-21T10:40:28.786 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.772958+0000", 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0560000000000004e-06 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.772971+0000", 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2978e-05 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.787 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.772976+0000", 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6570000000000002e-06 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773026+0000", 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9688999999999998e-05 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773137+0000", 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011099000000000001 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:40:28.788 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773245+0000", 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010801100000000001 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:177235 48.a e1382/1354 48:5d5bb101:::benchmark_data_smithi119_267254_object54025:head v 1382'11270, mlcod=1382'11268)", 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.773459+0000", 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0072203500000000004, 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073373229999999998, 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.789 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773459+0000", 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773459+0000", 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773458+0000", 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.790 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773463+0000", 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2800000000000003e-06 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773478+0000", 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4667e-05 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773484+0000", 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5060000000000003e-06 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.791 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773516+0000", 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2282e-05 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.773533+0000", 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7557000000000001e-05 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:177238 48.2 48:4d72f6b2:::benchmark_data_smithi119_267254_object54028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:40:28.792 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.774909+0000", 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0057699869999999999, 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005953257, 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 177238 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774909+0000", 2024-09-21T10:40:28.793 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774909+0000", 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774908+0000", 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774913+0000", 2024-09-21T10:40:28.794 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7939999999999999e-06 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774924+0000", 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0987e-05 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774930+0000", 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2909999999999998e-06 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:28.795 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.776303+0000", 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013738000000000001 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.776408+0000", 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010432 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.776511+0000", 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103687 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.796 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:177239 48.2 48.276f9902 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.774957+0000", 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0057222970000000003, 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059703769999999998, 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 177239 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.797 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774957+0000", 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774957+0000", 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774956+0000", 2024-09-21T10:40:28.798 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774959+0000", 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.23e-06 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774966+0000", 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.81e-06 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.774970+0000", 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8240000000000001e-06 2024-09-21T10:40:28.799 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop_reply(client.46619.0:177234 48.2 e1382/1354)", 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.777471+0000", 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0032082769999999998, 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003508504, 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.777471+0000", 2024-09-21T10:40:28.800 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.777471+0000", 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.777470+0000", 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.801 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.777475+0000", 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5160000000000002e-06 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.777489+0000", 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3570000000000001e-05 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.777495+0000", 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4559999999999999e-06 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.802 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:177243 48.b e1382/1354 48:dbf3b470:::benchmark_data_smithi119_267254_object54033:head v 1382'11054, mlcod=1382'11052)", 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.778459+0000", 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0022201669999999999, 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002565145, 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778459+0000", 2024-09-21T10:40:28.803 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778459+0000", 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778458+0000", 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778464+0000", 2024-09-21T10:40:28.804 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9710000000000001e-06 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778474+0000", 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6479999999999998e-06 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778479+0000", 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3669999999999996e-06 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778564+0000", 2024-09-21T10:40:28.805 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4312000000000002e-05 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.778581+0000", 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7544e-05 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop_reply(client.46619.0:177234 48.2 e1382/1354)", 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.780688+0000", 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4294967295.9999914, 2024-09-21T10:40:28.806 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00038607700000000002, 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780688+0000", 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780688+0000", 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.807 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780686+0000", 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780693+0000", 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4250000000000003e-06 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780705+0000", 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2299e-05 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.808 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780711+0000", 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5280000000000003e-06 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop_reply(client.46619.0:177238 48.2 e1382/1354)", 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.780717+0000", 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4294967295.9999623, 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039700899999999998, 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "queued for pg", 2024-09-21T10:40:28.809 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780717+0000", 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780717+0000", 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:28.810 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780716+0000", 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967296 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780719+0000", 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0059999999999999e-06 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780724+0000", 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1120000000000004e-06 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:28.811 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.780727+0000", 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0429999999999999e-06 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 8 2024-09-21T10:40:28.812 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:40:28.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:28.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:28.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:28.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:29.018 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:28 smithi145 ceph-mon[89359]: pgmap v2299: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 622 op/s; 782/220524 objects degraded (0.355%); 7445/220524 objects misplaced (3.376%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:40:29.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:28 smithi119 ceph-mon[105288]: pgmap v2299: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 622 op/s; 782/220524 objects degraded (0.355%); 7445/220524 objects misplaced (3.376%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:40:29.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:28 smithi119 ceph-mon[101432]: pgmap v2299: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 622 op/s; 782/220524 objects degraded (0.355%); 7445/220524 objects misplaced (3.376%); 1.4 MiB/s, 22 objects/s recovering 2024-09-21T10:40:29.120 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:29.153 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:29.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:29.278 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:29.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:29.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:29.411 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:29.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:29.557 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:29.608 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:40:29.608 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:29.644 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:40:29.656 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:29.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:29.677 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:29.677 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:40:29.677 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:29.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:29.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:29.722 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:40:29.722 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:29.763 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:40:29.763 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:29.791 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:40:29.791 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:29.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:29.824 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:40:29.825 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:29.859 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:40:29.859 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:29.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:29.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:29.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:29.893 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:40:29.893 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:29.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:29.987 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1774952561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:29 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 782/220524 objects degraded (0.355%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:30.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1774952561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:29 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 782/220524 objects degraded (0.355%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1774952561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:29 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 782/220524 objects degraded (0.355%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:30.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:30.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:30.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:30.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:30.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:30.445 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:40:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:40:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:40:30.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:30.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:30.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:30.668 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:30.770 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:30.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:30.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:30.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:30.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:31.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:31.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:31.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:30 smithi119 ceph-mon[105288]: pgmap v2300: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 856/208590 objects degraded (0.410%); 7187/208590 objects misplaced (3.446%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:40:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:30 smithi119 ceph-mon[101432]: pgmap v2300: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 856/208590 objects degraded (0.410%); 7187/208590 objects misplaced (3.446%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:40:31.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:31.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:31.236 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:30 smithi145 ceph-mon[89359]: pgmap v2300: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 895 op/s; 856/208590 objects degraded (0.410%); 7187/208590 objects misplaced (3.446%); 2.3 MiB/s, 36 objects/s recovering 2024-09-21T10:40:31.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:31.324 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:31.426 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:31.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:31.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:31.597 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:31.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:31.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:31.738 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:31.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:31.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:31.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:31.905 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:40:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=cleanup t=2024-09-21T10:40:31.698199627Z level=info msg="Completed cleanup jobs" duration=60.366415ms 2024-09-21T10:40:31.907 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:31.907 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:31.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:32.008 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:32.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:32.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:32.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:32.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:32.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:32.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:32.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:32.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:32.560 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:32.662 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:32.693 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:32.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:40:32.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:32.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:32.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:32.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:33.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:32 smithi145 ceph-mon[89359]: pgmap v2301: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 856/207714 objects degraded (0.412%); 6895/207714 objects misplaced (3.319%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:40:33.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:33.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:33.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:32 smithi119 ceph-mon[101432]: pgmap v2301: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 856/207714 objects degraded (0.412%); 6895/207714 objects misplaced (3.319%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:40:33.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:32 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:33.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:32 smithi119 ceph-mon[105288]: pgmap v2301: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 815 op/s; 856/207714 objects degraded (0.412%); 6895/207714 objects misplaced (3.319%); 2.1 MiB/s, 33 objects/s recovering 2024-09-21T10:40:33.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:32 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:33.118 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:33.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:33.161 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:33.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:33.251 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:33.353 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:33.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:33.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:33.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:33.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:33.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:33.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:33.807 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:33.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:33.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:33.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:34.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:34.067 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:34.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:34.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:34.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:34.271 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.271 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.271 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:40:34.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.301 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:34.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:34.430 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.431 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.431 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:40:34.431 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:40:34.431 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:40:34.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:34.473 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:34.575 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:34.609 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:34.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:34.753 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:34.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:34.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:40:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:40:34] "GET /metrics HTTP/1.1" 200 36861 "" "Prometheus/2.51.0" 2024-09-21T10:40:34.885 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:34.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:34.986 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:35.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:35.060 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:35.060 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:35.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:35.262 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:35.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:35 smithi119 ceph-mon[105288]: pgmap v2302: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 953/197961 objects degraded (0.481%); 6610/197961 objects misplaced (3.339%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:35.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:35 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 953/197961 objects degraded (0.481%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:35.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:35 smithi119 ceph-mon[101432]: pgmap v2302: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 953/197961 objects degraded (0.481%); 6610/197961 objects misplaced (3.339%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:35.273 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:35 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 953/197961 objects degraded (0.481%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:35.299 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:35.397 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:35 smithi145 ceph-mon[89359]: pgmap v2302: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 953/197961 objects degraded (0.481%); 6610/197961 objects misplaced (3.339%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:35.397 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:35 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 953/197961 objects degraded (0.481%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:35.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:35.444 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:35.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:35.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:35.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:35.684 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:35.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:35.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:35.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:40:35.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:35.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:35.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:35.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:40:35.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:36.038 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:36.087 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:40:36.092 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:40:36.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:36.147 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:36.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:36.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:36.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:36.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/746870080' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:40:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/693651185' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:36.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:36 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/746870080' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:40:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/693651185' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:36.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:36 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:36.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:36.428 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:36.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:36.444 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/746870080' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:40:36.444 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/693651185' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:36.444 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:36 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:40:36.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:36.562 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:36.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:36.664 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:36.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:36.807 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:36.837 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:36.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:36.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:36.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:37.074 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:37.106 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:37.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:37.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:37.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:37.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[105288]: pgmap v2303: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 953/193317 objects degraded (0.493%); 6269/193317 objects misplaced (3.243%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:40:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/746870080' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:40:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:40:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[105288]: osdmap e1383: 8 total, 8 up, 7 in 2024-09-21T10:40:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[101432]: pgmap v2303: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 953/193317 objects degraded (0.493%); 6269/193317 objects misplaced (3.243%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:40:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/746870080' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:40:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:40:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:37 smithi119 ceph-mon[101432]: osdmap e1383: 8 total, 8 up, 7 in 2024-09-21T10:40:37.415 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:37 smithi145 ceph-mon[89359]: pgmap v2303: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 953/193317 objects degraded (0.493%); 6269/193317 objects misplaced (3.243%); 1.9 MiB/s, 30 objects/s recovering 2024-09-21T10:40:37.415 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:37 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:40:37.415 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/746870080' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:40:37.415 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:37 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:40:37.416 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:37 smithi145 ceph-mon[89359]: osdmap e1383: 8 total, 8 up, 7 in 2024-09-21T10:40:37.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:37.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:37.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:37.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:37.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:37.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:37.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:37.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:37.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:37.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:38.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:38.147 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:38.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:38.285 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:38.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:38.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:38.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:38.543 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:38.587 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:38.587 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:38.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:38.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:38.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:38.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:38.817 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:40:38.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:40:38.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:38.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:38.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:39.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:39.092 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:39.100 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:39 smithi119 ceph-mon[101432]: pgmap v2305: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 934 op/s; 953/192477 objects degraded (0.495%); 6269/192477 objects misplaced (3.257%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:40:39.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:39.222 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:39 smithi145 ceph-mon[89359]: pgmap v2305: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 934 op/s; 953/192477 objects degraded (0.495%); 6269/192477 objects misplaced (3.257%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:40:39.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:39.356 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:39 smithi119 ceph-mon[105288]: pgmap v2305: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 934 op/s; 953/192477 objects degraded (0.495%); 6269/192477 objects misplaced (3.257%); 2.3 MiB/s, 37 objects/s recovering 2024-09-21T10:40:39.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:39.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:39.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:39.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:39.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:40:39.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:39.771 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:39.803 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:39.817 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:39.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:39.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:39.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:40.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:40.053 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:40.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:40.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:40.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:40.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:40.226 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:40 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 953/192477 objects degraded (0.495%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:40.227 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:40 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 953/192477 objects degraded (0.495%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:40.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:40.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:40.361 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:40:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:40:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:40:40.361 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:40 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 953/192477 objects degraded (0.495%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:40.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:40.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:40.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:40.617 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:40.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:40.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:40.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:40.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:40.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:40.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:40.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:41.029 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:41.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:41.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:41.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:41.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:41 smithi145 ceph-mon[89359]: pgmap v2306: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 924 op/s; 1033/180846 objects degraded (0.571%); 5990/180846 objects misplaced (3.312%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:40:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:41.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:41 smithi119 ceph-mon[105288]: pgmap v2306: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 924 op/s; 1033/180846 objects degraded (0.571%); 5990/180846 objects misplaced (3.312%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:40:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:41 smithi119 ceph-mon[101432]: pgmap v2306: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 924 op/s; 1033/180846 objects degraded (0.571%); 5990/180846 objects misplaced (3.312%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:40:41.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:41.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:41.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:41.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:41.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:41.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:40:41.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:41.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:41.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:41.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:41.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:41.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:42.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:42.037 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:40:42.037 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:40:42.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:42.172 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:42.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:42.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:42.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:42.459 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:42.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:42.589 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:42.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:42.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:40:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.733 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:42.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:42.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:42.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:42.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:42.980 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:43.020 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:43.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:43.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:43.124 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:40:43.145 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:43 smithi119 ceph-mon[101432]: pgmap v2307: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 1033/180033 objects degraded (0.574%); 5719/180033 objects misplaced (3.177%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:40:43.145 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:43 smithi119 ceph-mon[105288]: pgmap v2307: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 1033/180033 objects degraded (0.574%); 5719/180033 objects misplaced (3.177%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:40:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:43.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:43.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:43.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:43.314 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:40:43.314 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:43.355 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:40:43.355 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:43.371 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:43 smithi145 ceph-mon[89359]: pgmap v2307: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.7 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 1033/180033 objects degraded (0.574%); 5719/180033 objects misplaced (3.177%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:40:43.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:43.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:40:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:43.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:43.454 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:40:43.454 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:43.494 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:40:43.494 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:43.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:43.529 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:40:43.529 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:43.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:43.564 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:40:43.564 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:43.598 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:40:43.598 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:43.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:43.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:43.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:43.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:43.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:43.851 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:40:43.853 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:40:43.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:43.973 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:43.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:44.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:44.211 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:44.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:44.348 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:44.374 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:44.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:44.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:44.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1122021038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:44.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1122021038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1122021038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:44.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:44.642 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:44.780 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:44.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:44.862 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:44.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:40:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:40:44] "GET /metrics HTTP/1.1" 200 36861 "" "Prometheus/2.51.0" 2024-09-21T10:40:44.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:44.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:44.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:45.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:45.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:45.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:45.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:45.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:45.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:45 smithi119 ceph-mon[101432]: pgmap v2308: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 1121/169986 objects degraded (0.659%); 5425/169986 objects misplaced (3.191%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:40:45.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:45 smithi119 ceph-mon[101432]: from='client.36858 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:45.384 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:45 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1121/169986 objects degraded (0.659%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:45.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:45 smithi119 ceph-mon[105288]: pgmap v2308: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 1121/169986 objects degraded (0.659%); 5425/169986 objects misplaced (3.191%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:40:45.384 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:45 smithi119 ceph-mon[105288]: from='client.36858 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:45.385 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:45 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1121/169986 objects degraded (0.659%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:45.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:45.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.430 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:45.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:45.531 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:45 smithi145 ceph-mon[89359]: pgmap v2308: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 932 op/s; 1121/169986 objects degraded (0.659%); 5425/169986 objects misplaced (3.191%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:40:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:45 smithi145 ceph-mon[89359]: from='client.36858 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:45 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1121/169986 objects degraded (0.659%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:45.560 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:45.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:45.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:45.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.816 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:45.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:45.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:45.952 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:45.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:46.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:46.087 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:46.188 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:46.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:46.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:46.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:46.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:46.500 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:46.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:46.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:46.716 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2308,"stamp":"2024-09-21T10:40:43.823169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3712118074,"num_objects":56662,"num_object_clones":0,"num_object_copies":169986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1121,"num_objects_misplaced":5425,"num_objects_unfound":0,"num_objects_dirty":56662,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189843,"num_write_kb":7885280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3163,"num_bytes_recovered":204544687,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105763,"ondisk_log_size":105763,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16787516,"kb_used_data":11312324,"kb_used_omap":507,"kb_used_meta":5474564,"kb_avail":639428548,"statfs":{"total":671965249536,"available":654774833152,"internally_reserved":0,"allocated":11583819776,"data_stored":11541839308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":5605954079},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-611123200,"num_objects":-9325,"num_object_clones":0,"num_object_copies":-27975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":168,"num_objects_misplaced":-1185,"num_objects_unfound":0,"num_objects_dirty":-9325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9325,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":26542080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004416"},"pg_stats":[{"pgid":"48.f","version":"1383'11912","reported_seq":11982,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.725096+0000","last_change":"2024-09-21T10:37:50.818738+0000","last_active":"2024-09-21T10:40:43.725096+0000","last_peered":"2024-09-21T10:40:43.725096+0000","last_clean":"2024-09-21T10:40:43.725096+0000","last_became_active":"2024-09-21T10:37:50.818409+0000","last_became_peered":"2024-09-21T10:37:50.818409+0000","last_unstale":"2024-09-21T10:40:43.725096+0000","last_undegraded":"2024-09-21T10:40:43.725096+0000","last_fullsized":"2024-09-21T10:40:43.725096+0000","mapping_epoch":1354,"log_start":"1368'6300","ondisk_log_start":"1368'6300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5612,"log_dups_size":0,"ondisk_log_size":5612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:24:13.373038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224133120,"num_objects":3420,"num_object_clones":0,"num_object_copies":10260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3420,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11912,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.e","version":"1383'11784","reported_seq":11854,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:39.364465+0000","last_change":"2024-09-21T10:37:50.816783+0000","last_active":"2024-09-21T10:40:39.364465+0000","last_peered":"2024-09-21T10:40:39.364465+0000","last_clean":"2024-09-21T10:40:39.364465+0000","last_became_active":"2024-09-21T10:37:50.816482+0000","last_became_peered":"2024-09-21T10:37:50.816482+0000","last_unstale":"2024-09-21T10:40:39.364465+0000","last_undegraded":"2024-09-21T10:40:39.364465+0000","last_fullsized":"2024-09-21T10:40:39.364465+0000","mapping_epoch":1354,"log_start":"1359'4000","ondisk_log_start":"1359'4000","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:57.011771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247857152,"num_objects":3782,"num_object_clones":0,"num_object_copies":11346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11784,"num_write_kb":498112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.d","version":"1383'11937","reported_seq":11975,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.928013+0000","last_change":"2024-09-21T10:37:50.816596+0000","last_active":"2024-09-21T10:40:42.928013+0000","last_peered":"2024-09-21T10:40:42.928013+0000","last_clean":"2024-09-21T10:40:42.928013+0000","last_became_active":"2024-09-21T10:37:50.816060+0000","last_became_peered":"2024-09-21T10:37:50.816060+0000","last_unstale":"2024-09-21T10:40:42.928013+0000","last_undegraded":"2024-09-21T10:40:42.928013+0000","last_fullsized":"2024-09-21T10:40:42.928013+0000","mapping_epoch":1354,"log_start":"1368'6274","ondisk_log_start":"1368'6274","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:09.093088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":232652824,"num_objects":3551,"num_object_clones":0,"num_object_copies":10653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3551,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11937,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.c","version":"1383'12147","reported_seq":12217,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.674159+0000","last_change":"2024-09-21T10:37:50.814694+0000","last_active":"2024-09-21T10:40:43.674159+0000","last_peered":"2024-09-21T10:40:43.674159+0000","last_clean":"2024-09-21T10:40:43.674159+0000","last_became_active":"2024-09-21T10:37:50.814424+0000","last_became_peered":"2024-09-21T10:37:50.814424+0000","last_unstale":"2024-09-21T10:40:43.674159+0000","last_undegraded":"2024-09-21T10:40:43.674159+0000","last_fullsized":"2024-09-21T10:40:43.674159+0000","mapping_epoch":1354,"log_start":"1369'6500","ondisk_log_start":"1369'6500","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5647,"log_dups_size":0,"ondisk_log_size":5647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:22:22.559518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225116160,"num_objects":3435,"num_object_clones":0,"num_object_copies":10305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12147,"num_write_kb":498624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.b","version":"1383'11897","reported_seq":11933,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.802648+0000","last_change":"2024-09-21T10:37:50.809808+0000","last_active":"2024-09-21T10:40:43.802648+0000","last_peered":"2024-09-21T10:40:43.802648+0000","last_clean":"2024-09-21T10:40:43.802648+0000","last_became_active":"2024-09-21T10:37:50.809451+0000","last_became_peered":"2024-09-21T10:37:50.809451+0000","last_unstale":"2024-09-21T10:40:43.802648+0000","last_undegraded":"2024-09-21T10:40:43.802648+0000","last_fullsized":"2024-09-21T10:40:43.802648+0000","mapping_epoch":1354,"log_start":"1364'5600","ondisk_log_start":"1364'5600","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6297,"log_dups_size":0,"ondisk_log_size":6297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:46.455275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222232576,"num_objects":3391,"num_object_clones":0,"num_object_copies":10173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11897,"num_write_kb":489216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.a","version":"1383'11902","reported_seq":11972,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:39.322305+0000","last_change":"2024-09-21T10:37:50.806280+0000","last_active":"2024-09-21T10:40:39.322305+0000","last_peered":"2024-09-21T10:40:39.322305+0000","last_clean":"2024-09-21T10:40:39.322305+0000","last_became_active":"2024-09-21T10:37:50.805997+0000","last_became_peered":"2024-09-21T10:37:50.805997+0000","last_unstale":"2024-09-21T10:40:39.322305+0000","last_undegraded":"2024-09-21T10:40:39.322305+0000","last_fullsized":"2024-09-21T10:40:39.322305+0000","mapping_epoch":1354,"log_start":"1361'4200","ondisk_log_start":"1361'4200","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7702,"log_dups_size":0,"ondisk_log_size":7702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.061122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250085376,"num_objects":3816,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11902,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.9","version":"1383'11712","reported_seq":11782,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.730227+0000","last_change":"2024-09-21T10:37:50.814434+0000","last_active":"2024-09-21T10:40:43.730227+0000","last_peered":"2024-09-21T10:40:43.730227+0000","last_clean":"2024-09-21T10:40:43.730227+0000","last_became_active":"2024-09-21T10:37:50.814150+0000","last_became_peered":"2024-09-21T10:37:50.814150+0000","last_unstale":"2024-09-21T10:40:43.730227+0000","last_undegraded":"2024-09-21T10:40:43.730227+0000","last_fullsized":"2024-09-21T10:40:43.730227+0000","mapping_epoch":1354,"log_start":"1368'6100","ondisk_log_start":"1368'6100","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5612,"log_dups_size":0,"ondisk_log_size":5612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:18.301417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216399872,"num_objects":3302,"num_object_clones":0,"num_object_copies":9906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11712,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.8","version":"1383'11816","reported_seq":11852,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.674299+0000","last_change":"2024-09-21T10:37:50.812704+0000","last_active":"2024-09-21T10:40:43.674299+0000","last_peered":"2024-09-21T10:40:43.674299+0000","last_clean":"2024-09-21T10:40:43.674299+0000","last_became_active":"2024-09-21T10:37:50.812368+0000","last_became_peered":"2024-09-21T10:37:50.812368+0000","last_unstale":"2024-09-21T10:40:43.674299+0000","last_undegraded":"2024-09-21T10:40:43.674299+0000","last_fullsized":"2024-09-21T10:40:43.674299+0000","mapping_epoch":1354,"log_start":"1361'4100","ondisk_log_start":"1361'4100","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7716,"log_dups_size":0,"ondisk_log_size":7716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:22.704564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817792,"num_objects":3522,"num_object_clones":0,"num_object_copies":10566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11816,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"48.7","version":"1383'12058","reported_seq":12132,"reported_epoch":1383,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:40:40.562115+0000","last_change":"2024-09-21T10:40:12.260234+0000","last_active":"2024-09-21T10:40:40.562115+0000","last_peered":"2024-09-21T10:40:40.562115+0000","last_clean":"2024-09-21T10:40:09.890233+0000","last_became_active":"2024-09-21T10:40:12.242243+0000","last_became_peered":"2024-09-21T10:40:12.242243+0000","last_unstale":"2024-09-21T10:40:40.562115+0000","last_undegraded":"2024-09-21T10:40:40.562115+0000","last_fullsized":"2024-09-21T10:40:40.562115+0000","mapping_epoch":1379,"log_start":"1368'6300","ondisk_log_start":"1368'6300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5758,"log_dups_size":0,"ondisk_log_size":5758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243793920,"num_objects":3720,"num_object_clones":0,"num_object_copies":11160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3720,"num_objects_unfound":0,"num_objects_dirty":3720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":3720}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"48.6","version":"1383'11969","reported_seq":12039,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:43.728731+0000","last_change":"2024-09-21T10:37:50.811591+0000","last_active":"2024-09-21T10:40:43.728731+0000","last_peered":"2024-09-21T10:40:43.728731+0000","last_clean":"2024-09-21T10:40:43.728731+0000","last_became_active":"2024-09-21T10:37:50.811260+0000","last_became_peered":"2024-09-21T10:37:50.811260+0000","last_unstale":"2024-09-21T10:40:43.728731+0000","last_undegraded":"2024-09-21T10:40:43.728731+0000","last_fullsized":"2024-09-21T10:40:43.728731+0000","mapping_epoch":1354,"log_start":"1369'6400","ondisk_log_start":"1369'6400","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:19.966736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222625792,"num_objects":3397,"num_object_clones":0,"num_object_copies":10191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":491712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.5","version":"1383'11844","reported_seq":11881,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.972047+0000","last_change":"2024-09-21T10:37:50.816067+0000","last_active":"2024-09-21T10:40:42.972047+0000","last_peered":"2024-09-21T10:40:42.972047+0000","last_clean":"2024-09-21T10:40:42.972047+0000","last_became_active":"2024-09-21T10:37:50.815783+0000","last_became_peered":"2024-09-21T10:37:50.815783+0000","last_unstale":"2024-09-21T10:40:42.972047+0000","last_undegraded":"2024-09-21T10:40:42.972047+0000","last_fullsized":"2024-09-21T10:40:42.972047+0000","mapping_epoch":1354,"log_start":"1368'6110","ondisk_log_start":"1368'6110","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5734,"log_dups_size":0,"ondisk_log_size":5734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:03.096851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228982784,"num_objects":3494,"num_object_clones":0,"num_object_copies":10482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11844,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.4","version":"1383'11601","reported_seq":11671,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:39.364942+0000","last_change":"2024-09-21T10:37:50.810595+0000","last_active":"2024-09-21T10:40:39.364942+0000","last_peered":"2024-09-21T10:40:39.364942+0000","last_clean":"2024-09-21T10:40:39.364942+0000","last_became_active":"2024-09-21T10:37:50.810178+0000","last_became_peered":"2024-09-21T10:37:50.810178+0000","last_unstale":"2024-09-21T10:40:39.364942+0000","last_undegraded":"2024-09-21T10:40:39.364942+0000","last_fullsized":"2024-09-21T10:40:39.364942+0000","mapping_epoch":1354,"log_start":"1359'3900","ondisk_log_start":"1359'3900","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:30.334590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242679808,"num_objects":3703,"num_object_clones":0,"num_object_copies":11109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11601,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.3","version":"1383'11735","reported_seq":11805,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.538105+0000","last_change":"2024-09-21T10:37:50.808064+0000","last_active":"2024-09-21T10:40:42.538105+0000","last_peered":"2024-09-21T10:40:42.538105+0000","last_clean":"2024-09-21T10:40:42.538105+0000","last_became_active":"2024-09-21T10:37:50.807674+0000","last_became_peered":"2024-09-21T10:37:50.807674+0000","last_unstale":"2024-09-21T10:40:42.538105+0000","last_undegraded":"2024-09-21T10:40:42.538105+0000","last_fullsized":"2024-09-21T10:40:42.538105+0000","mapping_epoch":1354,"log_start":"1364'4700","ondisk_log_start":"1364'4700","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7035,"log_dups_size":0,"ondisk_log_size":7035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:21.766427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227737600,"num_objects":3475,"num_object_clones":0,"num_object_copies":10425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11735,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"48.2","version":"1383'12201","reported_seq":12238,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:42.960419+0000","last_change":"2024-09-21T10:37:50.816843+0000","last_active":"2024-09-21T10:40:42.960419+0000","last_peered":"2024-09-21T10:40:42.960419+0000","last_clean":"2024-09-21T10:40:42.960419+0000","last_became_active":"2024-09-21T10:37:50.816490+0000","last_became_peered":"2024-09-21T10:37:50.816490+0000","last_unstale":"2024-09-21T10:40:42.960419+0000","last_undegraded":"2024-09-21T10:40:42.960419+0000","last_fullsized":"2024-09-21T10:40:42.960419+0000","mapping_epoch":1354,"log_start":"1369'6491","ondisk_log_start":"1369'6491","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5710,"log_dups_size":0,"ondisk_log_size":5710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:33.243791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233766912,"num_objects":3567,"num_object_clones":0,"num_object_copies":10701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12201,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.1","version":"1383'11804","reported_seq":18066,"reported_epoch":1383,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:40:43.783337+0000","last_change":"2024-09-21T10:39:49.804470+0000","last_active":"2024-09-21T10:40:43.783337+0000","last_peered":"2024-09-21T10:40:43.783337+0000","last_clean":"2024-09-21T10:38:59.823246+0000","last_became_active":"2024-09-21T10:39:02.073182+0000","last_became_peered":"2024-09-21T10:39:02.073182+0000","last_unstale":"2024-09-21T10:40:43.783337+0000","last_undegraded":"2024-09-21T10:39:49.800872+0000","last_fullsized":"2024-09-21T10:39:01.839371+0000","mapping_epoch":1367,"log_start":"1364'5500","ondisk_log_start":"1364'5500","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6304,"log_dups_size":0,"ondisk_log_size":6304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:06.547860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220594176,"num_objects":3366,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1121,"num_objects_misplaced":1705,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11804,"num_write_kb":485440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3107,"num_bytes_recovered":203620352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,6],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1705},{"shards":"6,7","objects":540}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.0","version":"1383'11461","reported_seq":11531,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:39.358737+0000","last_change":"2024-09-21T10:37:50.812734+0000","last_active":"2024-09-21T10:40:39.358737+0000","last_peered":"2024-09-21T10:40:39.358737+0000","last_clean":"2024-09-21T10:40:39.358737+0000","last_became_active":"2024-09-21T10:37:50.812438+0000","last_became_peered":"2024-09-21T10:37:50.812438+0000","last_unstale":"2024-09-21T10:40:39.358737+0000","last_undegraded":"2024-09-21T10:40:39.358737+0000","last_fullsized":"2024-09-21T10:40:39.358737+0000","mapping_epoch":1354,"log_start":"1359'3700","ondisk_log_start":"1359'3700","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7761,"log_dups_size":0,"ondisk_log_size":7761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:25.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242155520,"num_objects":3695,"num_object_clones":0,"num_object_copies":11085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11461,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":50,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099864+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:40:36.099864+0000","last_peered":"2024-09-21T10:40:36.099864+0000","last_clean":"2024-09-21T10:40:36.099864+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:40:36.099864+0000","last_undegraded":"2024-09-21T10:40:36.099864+0000","last_fullsized":"2024-09-21T10:40:36.099864+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1277,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830224+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:40:20.830224+0000","last_peered":"2024-09-21T10:40:20.830224+0000","last_clean":"2024-09-21T10:40:20.830224+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:40:20.830224+0000","last_undegraded":"2024-09-21T10:40:20.830224+0000","last_fullsized":"2024-09-21T10:40:20.830224+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2087,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106329+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:40:36.106329+0000","last_peered":"2024-09-21T10:40:36.106329+0000","last_clean":"2024-09-21T10:40:36.106329+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:40:36.106329+0000","last_undegraded":"2024-09-21T10:40:36.106329+0000","last_fullsized":"2024-09-21T10:40:36.106329+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2106,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.107181+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:40:36.107181+0000","last_peered":"2024-09-21T10:40:36.107181+0000","last_clean":"2024-09-21T10:40:36.107181+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:40:36.107181+0000","last_undegraded":"2024-09-21T10:40:36.107181+0000","last_fullsized":"2024-09-21T10:40:36.107181+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":151,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823631+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:40:20.823631+0000","last_peered":"2024-09-21T10:40:20.823631+0000","last_clean":"2024-09-21T10:40:20.823631+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:40:20.823631+0000","last_undegraded":"2024-09-21T10:40:20.823631+0000","last_fullsized":"2024-09-21T10:40:20.823631+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1289,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106486+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:40:36.106486+0000","last_peered":"2024-09-21T10:40:36.106486+0000","last_clean":"2024-09-21T10:40:36.106486+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:40:36.106486+0000","last_undegraded":"2024-09-21T10:40:36.106486+0000","last_fullsized":"2024-09-21T10:40:36.106486+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":151,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823414+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:40:20.823414+0000","last_peered":"2024-09-21T10:40:20.823414+0000","last_clean":"2024-09-21T10:40:20.823414+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:40:20.823414+0000","last_undegraded":"2024-09-21T10:40:20.823414+0000","last_fullsized":"2024-09-21T10:40:20.823414+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1376,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.107077+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:40:36.107077+0000","last_peered":"2024-09-21T10:40:36.107077+0000","last_clean":"2024-09-21T10:40:36.107077+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:40:36.107077+0000","last_undegraded":"2024-09-21T10:40:36.107077+0000","last_fullsized":"2024-09-21T10:40:36.107077+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":44,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100043+0000","last_change":"2024-09-21T10:39:00.847895+0000","last_active":"2024-09-21T10:40:36.100043+0000","last_peered":"2024-09-21T10:40:36.100043+0000","last_clean":"2024-09-21T10:40:36.100043+0000","last_became_active":"2024-09-21T10:39:00.847533+0000","last_became_peered":"2024-09-21T10:39:00.847533+0000","last_unstale":"2024-09-21T10:40:36.100043+0000","last_undegraded":"2024-09-21T10:40:36.100043+0000","last_fullsized":"2024-09-21T10:40:36.100043+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:55.080913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":73,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.109868+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:40:36.109868+0000","last_peered":"2024-09-21T10:40:36.109868+0000","last_clean":"2024-09-21T10:40:36.109868+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:40:36.109868+0000","last_undegraded":"2024-09-21T10:40:36.109868+0000","last_fullsized":"2024-09-21T10:40:36.109868+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":149,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.121882+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:40:36.121882+0000","last_peered":"2024-09-21T10:40:36.121882+0000","last_clean":"2024-09-21T10:40:36.121882+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:40:36.121882+0000","last_undegraded":"2024-09-21T10:40:36.121882+0000","last_fullsized":"2024-09-21T10:40:36.121882+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2095,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.108749+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:40:36.108749+0000","last_peered":"2024-09-21T10:40:36.108749+0000","last_clean":"2024-09-21T10:40:36.108749+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:40:36.108749+0000","last_undegraded":"2024-09-21T10:40:36.108749+0000","last_fullsized":"2024-09-21T10:40:36.108749+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2194,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.121581+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:40:36.121581+0000","last_peered":"2024-09-21T10:40:36.121581+0000","last_clean":"2024-09-21T10:40:36.121581+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:40:36.121581+0000","last_undegraded":"2024-09-21T10:40:36.121581+0000","last_fullsized":"2024-09-21T10:40:36.121581+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2203,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099771+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:40:36.099771+0000","last_peered":"2024-09-21T10:40:36.099771+0000","last_clean":"2024-09-21T10:40:36.099771+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:40:36.099771+0000","last_undegraded":"2024-09-21T10:40:36.099771+0000","last_fullsized":"2024-09-21T10:40:36.099771+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2121,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106532+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:40:36.106532+0000","last_peered":"2024-09-21T10:40:36.106532+0000","last_clean":"2024-09-21T10:40:36.106532+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:40:36.106532+0000","last_undegraded":"2024-09-21T10:40:36.106532+0000","last_fullsized":"2024-09-21T10:40:36.106532+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1341,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830709+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:40:20.830709+0000","last_peered":"2024-09-21T10:40:20.830709+0000","last_clean":"2024-09-21T10:40:20.830709+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:40:20.830709+0000","last_undegraded":"2024-09-21T10:40:20.830709+0000","last_fullsized":"2024-09-21T10:40:20.830709+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2100,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106946+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:40:36.106946+0000","last_peered":"2024-09-21T10:40:36.106946+0000","last_clean":"2024-09-21T10:40:36.106946+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:40:36.106946+0000","last_undegraded":"2024-09-21T10:40:36.106946+0000","last_fullsized":"2024-09-21T10:40:36.106946+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2213,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100049+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:40:36.100049+0000","last_peered":"2024-09-21T10:40:36.100049+0000","last_clean":"2024-09-21T10:40:36.100049+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:40:36.100049+0000","last_undegraded":"2024-09-21T10:40:36.100049+0000","last_fullsized":"2024-09-21T10:40:36.100049+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":230,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.110006+0000","last_change":"2024-09-21T10:37:47.788643+0000","last_active":"2024-09-21T10:40:36.110006+0000","last_peered":"2024-09-21T10:40:36.110006+0000","last_clean":"2024-09-21T10:40:36.110006+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:40:36.110006+0000","last_undegraded":"2024-09-21T10:40:36.110006+0000","last_fullsized":"2024-09-21T10:40:36.110006+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:39.188916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2083,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.107392+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:40:36.107392+0000","last_peered":"2024-09-21T10:40:36.107392+0000","last_clean":"2024-09-21T10:40:36.107392+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:40:36.107392+0000","last_undegraded":"2024-09-21T10:40:36.107392+0000","last_fullsized":"2024-09-21T10:40:36.107392+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823998+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:40:20.823998+0000","last_peered":"2024-09-21T10:40:20.823998+0000","last_clean":"2024-09-21T10:40:20.823998+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:40:20.823998+0000","last_undegraded":"2024-09-21T10:40:20.823998+0000","last_fullsized":"2024-09-21T10:40:20.823998+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1744,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.123280+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:40:36.123280+0000","last_peered":"2024-09-21T10:40:36.123280+0000","last_clean":"2024-09-21T10:40:36.123280+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:40:36.123280+0000","last_undegraded":"2024-09-21T10:40:36.123280+0000","last_fullsized":"2024-09-21T10:40:36.123280+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2186,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122142+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:40:36.122142+0000","last_peered":"2024-09-21T10:40:36.122142+0000","last_clean":"2024-09-21T10:40:36.122142+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:40:36.122142+0000","last_undegraded":"2024-09-21T10:40:36.122142+0000","last_fullsized":"2024-09-21T10:40:36.122142+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":398,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824752+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:40:20.824752+0000","last_peered":"2024-09-21T10:40:20.824752+0000","last_clean":"2024-09-21T10:40:20.824752+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:40:20.824752+0000","last_undegraded":"2024-09-21T10:40:20.824752+0000","last_fullsized":"2024-09-21T10:40:20.824752+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1242,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829988+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:40:20.829988+0000","last_peered":"2024-09-21T10:40:20.829988+0000","last_clean":"2024-09-21T10:40:20.829988+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:40:20.829988+0000","last_undegraded":"2024-09-21T10:40:20.829988+0000","last_fullsized":"2024-09-21T10:40:20.829988+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":342,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.109223+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:40:36.109223+0000","last_peered":"2024-09-21T10:40:36.109223+0000","last_clean":"2024-09-21T10:40:36.109223+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:40:36.109223+0000","last_undegraded":"2024-09-21T10:40:36.109223+0000","last_fullsized":"2024-09-21T10:40:36.109223+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1276,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823003+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:40:20.823003+0000","last_peered":"2024-09-21T10:40:20.823003+0000","last_clean":"2024-09-21T10:40:20.823003+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:40:20.823003+0000","last_undegraded":"2024-09-21T10:40:20.823003+0000","last_fullsized":"2024-09-21T10:40:20.823003+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1337,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830640+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:40:20.830640+0000","last_peered":"2024-09-21T10:40:20.830640+0000","last_clean":"2024-09-21T10:40:20.830640+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:40:20.830640+0000","last_undegraded":"2024-09-21T10:40:20.830640+0000","last_fullsized":"2024-09-21T10:40:20.830640+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":77,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099611+0000","last_change":"2024-09-21T10:37:41.926686+0000","last_active":"2024-09-21T10:40:36.099611+0000","last_peered":"2024-09-21T10:40:36.099611+0000","last_clean":"2024-09-21T10:40:36.099611+0000","last_became_active":"2024-09-21T10:37:41.922706+0000","last_became_peered":"2024-09-21T10:37:41.922706+0000","last_unstale":"2024-09-21T10:40:36.099611+0000","last_undegraded":"2024-09-21T10:40:36.099611+0000","last_fullsized":"2024-09-21T10:40:36.099611+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:08.955843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":132,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.114309+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:40:36.114309+0000","last_peered":"2024-09-21T10:40:36.114309+0000","last_clean":"2024-09-21T10:40:36.114309+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:40:36.114309+0000","last_undegraded":"2024-09-21T10:40:36.114309+0000","last_fullsized":"2024-09-21T10:40:36.114309+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":39,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.822763+0000","last_change":"2024-09-21T10:39:00.847173+0000","last_active":"2024-09-21T10:40:20.822763+0000","last_peered":"2024-09-21T10:40:20.822763+0000","last_clean":"2024-09-21T10:40:20.822763+0000","last_became_active":"2024-09-21T10:39:00.846851+0000","last_became_peered":"2024-09-21T10:39:00.846851+0000","last_unstale":"2024-09-21T10:40:20.822763+0000","last_undegraded":"2024-09-21T10:40:20.822763+0000","last_fullsized":"2024-09-21T10:40:20.822763+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:47.133721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2203,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122368+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:40:36.122368+0000","last_peered":"2024-09-21T10:40:36.122368+0000","last_clean":"2024-09-21T10:40:36.122368+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:40:36.122368+0000","last_undegraded":"2024-09-21T10:40:36.122368+0000","last_fullsized":"2024-09-21T10:40:36.122368+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2082,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.105741+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:40:36.105741+0000","last_peered":"2024-09-21T10:40:36.105741+0000","last_clean":"2024-09-21T10:40:36.105741+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:40:36.105741+0000","last_undegraded":"2024-09-21T10:40:36.105741+0000","last_fullsized":"2024-09-21T10:40:36.105741+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":508,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829790+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:40:20.829790+0000","last_peered":"2024-09-21T10:40:20.829790+0000","last_clean":"2024-09-21T10:40:20.829790+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:40:20.829790+0000","last_undegraded":"2024-09-21T10:40:20.829790+0000","last_fullsized":"2024-09-21T10:40:20.829790+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2205,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.101320+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:40:36.101320+0000","last_peered":"2024-09-21T10:40:36.101320+0000","last_clean":"2024-09-21T10:40:36.101320+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:40:36.101320+0000","last_undegraded":"2024-09-21T10:40:36.101320+0000","last_fullsized":"2024-09-21T10:40:36.101320+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1400,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824847+0000","last_change":"2024-09-21T10:37:41.922996+0000","last_active":"2024-09-21T10:40:20.824847+0000","last_peered":"2024-09-21T10:40:20.824847+0000","last_clean":"2024-09-21T10:40:20.824847+0000","last_became_active":"2024-09-21T10:37:41.922703+0000","last_became_peered":"2024-09-21T10:37:41.922703+0000","last_unstale":"2024-09-21T10:40:20.824847+0000","last_undegraded":"2024-09-21T10:40:20.824847+0000","last_fullsized":"2024-09-21T10:40:20.824847+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:23.614770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2191,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122292+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:40:36.122292+0000","last_peered":"2024-09-21T10:40:36.122292+0000","last_clean":"2024-09-21T10:40:36.122292+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:40:36.122292+0000","last_undegraded":"2024-09-21T10:40:36.122292+0000","last_fullsized":"2024-09-21T10:40:36.122292+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":508,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830565+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:40:20.830565+0000","last_peered":"2024-09-21T10:40:20.830565+0000","last_clean":"2024-09-21T10:40:20.830565+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:40:20.830565+0000","last_undegraded":"2024-09-21T10:40:20.830565+0000","last_fullsized":"2024-09-21T10:40:20.830565+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2196,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122280+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:40:36.122280+0000","last_peered":"2024-09-21T10:40:36.122280+0000","last_clean":"2024-09-21T10:40:36.122280+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:40:36.122280+0000","last_undegraded":"2024-09-21T10:40:36.122280+0000","last_fullsized":"2024-09-21T10:40:36.122280+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2191,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122520+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:40:36.122520+0000","last_peered":"2024-09-21T10:40:36.122520+0000","last_clean":"2024-09-21T10:40:36.122520+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:40:36.122520+0000","last_undegraded":"2024-09-21T10:40:36.122520+0000","last_fullsized":"2024-09-21T10:40:36.122520+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823541+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:40:20.823541+0000","last_peered":"2024-09-21T10:40:20.823541+0000","last_clean":"2024-09-21T10:40:20.823541+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:40:20.823541+0000","last_undegraded":"2024-09-21T10:40:20.823541+0000","last_fullsized":"2024-09-21T10:40:20.823541+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":143,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823097+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:40:20.823097+0000","last_peered":"2024-09-21T10:40:20.823097+0000","last_clean":"2024-09-21T10:40:20.823097+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:40:20.823097+0000","last_undegraded":"2024-09-21T10:40:20.823097+0000","last_fullsized":"2024-09-21T10:40:20.823097+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2188,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122427+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:40:36.122427+0000","last_peered":"2024-09-21T10:40:36.122427+0000","last_clean":"2024-09-21T10:40:36.122427+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:40:36.122427+0000","last_undegraded":"2024-09-21T10:40:36.122427+0000","last_fullsized":"2024-09-21T10:40:36.122427+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1308,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100627+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:40:36.100627+0000","last_peered":"2024-09-21T10:40:36.100627+0000","last_clean":"2024-09-21T10:40:36.100627+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:40:36.100627+0000","last_undegraded":"2024-09-21T10:40:36.100627+0000","last_fullsized":"2024-09-21T10:40:36.100627+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":137,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823246+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:40:20.823246+0000","last_peered":"2024-09-21T10:40:20.823246+0000","last_clean":"2024-09-21T10:40:20.823246+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:40:20.823246+0000","last_undegraded":"2024-09-21T10:40:20.823246+0000","last_fullsized":"2024-09-21T10:40:20.823246+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":511,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099455+0000","last_change":"2024-09-21T10:37:38.606346+0000","last_active":"2024-09-21T10:40:36.099455+0000","last_peered":"2024-09-21T10:40:36.099455+0000","last_clean":"2024-09-21T10:40:36.099455+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:40:36.099455+0000","last_undegraded":"2024-09-21T10:40:36.099455+0000","last_fullsized":"2024-09-21T10:40:36.099455+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:16.461221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2206,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122517+0000","last_change":"2024-09-21T10:37:41.924374+0000","last_active":"2024-09-21T10:40:36.122517+0000","last_peered":"2024-09-21T10:40:36.122517+0000","last_clean":"2024-09-21T10:40:36.122517+0000","last_became_active":"2024-09-21T10:37:41.922589+0000","last_became_peered":"2024-09-21T10:37:41.922589+0000","last_unstale":"2024-09-21T10:40:36.122517+0000","last_undegraded":"2024-09-21T10:40:36.122517+0000","last_fullsized":"2024-09-21T10:40:36.122517+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:51.987384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2188,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.130406+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:40:36.130406+0000","last_peered":"2024-09-21T10:40:36.130406+0000","last_clean":"2024-09-21T10:40:36.130406+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:40:36.130406+0000","last_undegraded":"2024-09-21T10:40:36.130406+0000","last_fullsized":"2024-09-21T10:40:36.130406+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2119,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.103641+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:40:36.103641+0000","last_peered":"2024-09-21T10:40:36.103641+0000","last_clean":"2024-09-21T10:40:36.103641+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:40:36.103641+0000","last_undegraded":"2024-09-21T10:40:36.103641+0000","last_fullsized":"2024-09-21T10:40:36.103641+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":373,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.124343+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:40:36.124343+0000","last_peered":"2024-09-21T10:40:36.124343+0000","last_clean":"2024-09-21T10:40:36.124343+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:40:36.124343+0000","last_undegraded":"2024-09-21T10:40:36.124343+0000","last_fullsized":"2024-09-21T10:40:36.124343+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":44,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099509+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:40:36.099509+0000","last_peered":"2024-09-21T10:40:36.099509+0000","last_clean":"2024-09-21T10:40:36.099509+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:40:36.099509+0000","last_undegraded":"2024-09-21T10:40:36.099509+0000","last_fullsized":"2024-09-21T10:40:36.099509+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2220,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.110970+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:40:36.110970+0000","last_peered":"2024-09-21T10:40:36.110970+0000","last_clean":"2024-09-21T10:40:36.110970+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:40:36.110970+0000","last_undegraded":"2024-09-21T10:40:36.110970+0000","last_fullsized":"2024-09-21T10:40:36.110970+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1370,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824469+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:40:20.824469+0000","last_peered":"2024-09-21T10:40:20.824469+0000","last_clean":"2024-09-21T10:40:20.824469+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:40:20.824469+0000","last_undegraded":"2024-09-21T10:40:20.824469+0000","last_fullsized":"2024-09-21T10:40:20.824469+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1343,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829812+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:40:20.829812+0000","last_peered":"2024-09-21T10:40:20.829812+0000","last_clean":"2024-09-21T10:40:20.829812+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:40:20.829812+0000","last_undegraded":"2024-09-21T10:40:20.829812+0000","last_fullsized":"2024-09-21T10:40:20.829812+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2105,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106574+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:40:36.106574+0000","last_peered":"2024-09-21T10:40:36.106574+0000","last_clean":"2024-09-21T10:40:36.106574+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:40:36.106574+0000","last_undegraded":"2024-09-21T10:40:36.106574+0000","last_fullsized":"2024-09-21T10:40:36.106574+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":137,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823363+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:40:20.823363+0000","last_peered":"2024-09-21T10:40:20.823363+0000","last_clean":"2024-09-21T10:40:20.823363+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:40:20.823363+0000","last_undegraded":"2024-09-21T10:40:20.823363+0000","last_fullsized":"2024-09-21T10:40:20.823363+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1553,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824368+0000","last_change":"2024-09-21T10:37:41.922049+0000","last_active":"2024-09-21T10:40:20.824368+0000","last_peered":"2024-09-21T10:40:20.824368+0000","last_clean":"2024-09-21T10:40:20.824368+0000","last_became_active":"2024-09-21T10:37:41.921709+0000","last_became_peered":"2024-09-21T10:37:41.921709+0000","last_unstale":"2024-09-21T10:40:20.824368+0000","last_undegraded":"2024-09-21T10:40:20.824368+0000","last_fullsized":"2024-09-21T10:40:20.824368+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:12.446157+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823352+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:40:20.823352+0000","last_peered":"2024-09-21T10:40:20.823352+0000","last_clean":"2024-09-21T10:40:20.823352+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:40:20.823352+0000","last_undegraded":"2024-09-21T10:40:20.823352+0000","last_fullsized":"2024-09-21T10:40:20.823352+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2194,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.122430+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:40:36.122430+0000","last_peered":"2024-09-21T10:40:36.122430+0000","last_clean":"2024-09-21T10:40:36.122430+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:40:36.122430+0000","last_undegraded":"2024-09-21T10:40:36.122430+0000","last_fullsized":"2024-09-21T10:40:36.122430+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2091,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106630+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:40:36.106630+0000","last_peered":"2024-09-21T10:40:36.106630+0000","last_clean":"2024-09-21T10:40:36.106630+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:40:36.106630+0000","last_undegraded":"2024-09-21T10:40:36.106630+0000","last_fullsized":"2024-09-21T10:40:36.106630+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":357,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.124256+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:40:36.124256+0000","last_peered":"2024-09-21T10:40:36.124256+0000","last_clean":"2024-09-21T10:40:36.124256+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:40:36.124256+0000","last_undegraded":"2024-09-21T10:40:36.124256+0000","last_fullsized":"2024-09-21T10:40:36.124256+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1242,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829869+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:40:20.829869+0000","last_peered":"2024-09-21T10:40:20.829869+0000","last_clean":"2024-09-21T10:40:20.829869+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:40:20.829869+0000","last_undegraded":"2024-09-21T10:40:20.829869+0000","last_fullsized":"2024-09-21T10:40:20.829869+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1246,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.103728+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:40:36.103728+0000","last_peered":"2024-09-21T10:40:36.103728+0000","last_clean":"2024-09-21T10:40:36.103728+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:40:36.103728+0000","last_undegraded":"2024-09-21T10:40:36.103728+0000","last_fullsized":"2024-09-21T10:40:36.103728+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":137,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.822988+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:40:20.822988+0000","last_peered":"2024-09-21T10:40:20.822988+0000","last_clean":"2024-09-21T10:40:20.822988+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:40:20.822988+0000","last_undegraded":"2024-09-21T10:40:20.822988+0000","last_fullsized":"2024-09-21T10:40:20.822988+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1254,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.114465+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:40:36.114465+0000","last_peered":"2024-09-21T10:40:36.114465+0000","last_clean":"2024-09-21T10:40:36.114465+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:40:36.114465+0000","last_undegraded":"2024-09-21T10:40:36.114465+0000","last_fullsized":"2024-09-21T10:40:36.114465+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1349,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830839+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:40:20.830839+0000","last_peered":"2024-09-21T10:40:20.830839+0000","last_clean":"2024-09-21T10:40:20.830839+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:40:20.830839+0000","last_undegraded":"2024-09-21T10:40:20.830839+0000","last_fullsized":"2024-09-21T10:40:20.830839+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1504,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.827301+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:40:20.827301+0000","last_peered":"2024-09-21T10:40:20.827301+0000","last_clean":"2024-09-21T10:40:20.827301+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:40:20.827301+0000","last_undegraded":"2024-09-21T10:40:20.827301+0000","last_fullsized":"2024-09-21T10:40:20.827301+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1306,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.112601+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:40:36.112601+0000","last_peered":"2024-09-21T10:40:36.112601+0000","last_clean":"2024-09-21T10:40:36.112601+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:40:36.112601+0000","last_undegraded":"2024-09-21T10:40:36.112601+0000","last_fullsized":"2024-09-21T10:40:36.112601+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":319,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106967+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:40:36.106967+0000","last_peered":"2024-09-21T10:40:36.106967+0000","last_clean":"2024-09-21T10:40:36.106967+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:40:36.106967+0000","last_undegraded":"2024-09-21T10:40:36.106967+0000","last_fullsized":"2024-09-21T10:40:36.106967+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":78,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.825814+0000","last_change":"2024-09-21T10:37:37.360857+0000","last_active":"2024-09-21T10:40:20.825814+0000","last_peered":"2024-09-21T10:40:20.825814+0000","last_clean":"2024-09-21T10:40:20.825814+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:40:20.825814+0000","last_undegraded":"2024-09-21T10:40:20.825814+0000","last_fullsized":"2024-09-21T10:40:20.825814+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.360779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.360779+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:47:25.310501+0000","scrub_duration":0.001559979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2219,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100536+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:40:36.100536+0000","last_peered":"2024-09-21T10:40:36.100536+0000","last_clean":"2024-09-21T10:40:36.100536+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:40:36.100536+0000","last_undegraded":"2024-09-21T10:40:36.100536+0000","last_fullsized":"2024-09-21T10:40:36.100536+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823355+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:40:20.823355+0000","last_peered":"2024-09-21T10:40:20.823355+0000","last_clean":"2024-09-21T10:40:20.823355+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:40:20.823355+0000","last_undegraded":"2024-09-21T10:40:20.823355+0000","last_fullsized":"2024-09-21T10:40:20.823355+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":527,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830683+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:40:20.830683+0000","last_peered":"2024-09-21T10:40:20.830683+0000","last_clean":"2024-09-21T10:40:20.830683+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:40:20.830683+0000","last_undegraded":"2024-09-21T10:40:20.830683+0000","last_fullsized":"2024-09-21T10:40:20.830683+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1321,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830769+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:40:20.830769+0000","last_peered":"2024-09-21T10:40:20.830769+0000","last_clean":"2024-09-21T10:40:20.830769+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:40:20.830769+0000","last_undegraded":"2024-09-21T10:40:20.830769+0000","last_fullsized":"2024-09-21T10:40:20.830769+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1328,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823508+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:40:20.823508+0000","last_peered":"2024-09-21T10:40:20.823508+0000","last_clean":"2024-09-21T10:40:20.823508+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:40:20.823508+0000","last_undegraded":"2024-09-21T10:40:20.823508+0000","last_fullsized":"2024-09-21T10:40:20.823508+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":357,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.109125+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:40:36.109125+0000","last_peered":"2024-09-21T10:40:36.109125+0000","last_clean":"2024-09-21T10:40:36.109125+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:40:36.109125+0000","last_undegraded":"2024-09-21T10:40:36.109125+0000","last_fullsized":"2024-09-21T10:40:36.109125+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1289,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.121671+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:40:36.121671+0000","last_peered":"2024-09-21T10:40:36.121671+0000","last_clean":"2024-09-21T10:40:36.121671+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:40:36.121671+0000","last_undegraded":"2024-09-21T10:40:36.121671+0000","last_fullsized":"2024-09-21T10:40:36.121671+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1388,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.826075+0000","last_change":"2024-09-21T10:38:34.901523+0000","last_active":"2024-09-21T10:40:20.826075+0000","last_peered":"2024-09-21T10:40:20.826075+0000","last_clean":"2024-09-21T10:40:20.826075+0000","last_became_active":"2024-09-21T10:38:34.887435+0000","last_became_peered":"2024-09-21T10:38:34.887435+0000","last_unstale":"2024-09-21T10:40:20.826075+0000","last_undegraded":"2024-09-21T10:40:20.826075+0000","last_fullsized":"2024-09-21T10:40:20.826075+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:27.993702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":130,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824582+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:40:20.824582+0000","last_peered":"2024-09-21T10:40:20.824582+0000","last_clean":"2024-09-21T10:40:20.824582+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:40:20.824582+0000","last_undegraded":"2024-09-21T10:40:20.824582+0000","last_fullsized":"2024-09-21T10:40:20.824582+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823018+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:40:20.823018+0000","last_peered":"2024-09-21T10:40:20.823018+0000","last_clean":"2024-09-21T10:40:20.823018+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:40:20.823018+0000","last_undegraded":"2024-09-21T10:40:20.823018+0000","last_fullsized":"2024-09-21T10:40:20.823018+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1401,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.825892+0000","last_change":"2024-09-21T10:38:34.891682+0000","last_active":"2024-09-21T10:40:20.825892+0000","last_peered":"2024-09-21T10:40:20.825892+0000","last_clean":"2024-09-21T10:40:20.825892+0000","last_became_active":"2024-09-21T10:38:34.885292+0000","last_became_peered":"2024-09-21T10:38:34.885292+0000","last_unstale":"2024-09-21T10:40:20.825892+0000","last_undegraded":"2024-09-21T10:40:20.825892+0000","last_fullsized":"2024-09-21T10:40:20.825892+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:30.111501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":120,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823048+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:40:20.823048+0000","last_peered":"2024-09-21T10:40:20.823048+0000","last_clean":"2024-09-21T10:40:20.823048+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:40:20.823048+0000","last_undegraded":"2024-09-21T10:40:20.823048+0000","last_fullsized":"2024-09-21T10:40:20.823048+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1418,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830539+0000","last_change":"2024-09-21T10:39:00.852998+0000","last_active":"2024-09-21T10:40:20.830539+0000","last_peered":"2024-09-21T10:40:20.830539+0000","last_clean":"2024-09-21T10:40:20.830539+0000","last_became_active":"2024-09-21T10:39:00.846811+0000","last_became_peered":"2024-09-21T10:39:00.846811+0000","last_unstale":"2024-09-21T10:40:20.830539+0000","last_undegraded":"2024-09-21T10:40:20.830539+0000","last_fullsized":"2024-09-21T10:40:20.830539+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:14.374741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2081,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.105406+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:40:36.105406+0000","last_peered":"2024-09-21T10:40:36.105406+0000","last_clean":"2024-09-21T10:40:36.105406+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:40:36.105406+0000","last_undegraded":"2024-09-21T10:40:36.105406+0000","last_fullsized":"2024-09-21T10:40:36.105406+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":364,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.124145+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:40:36.124145+0000","last_peered":"2024-09-21T10:40:36.124145+0000","last_clean":"2024-09-21T10:40:36.124145+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:40:36.124145+0000","last_undegraded":"2024-09-21T10:40:36.124145+0000","last_fullsized":"2024-09-21T10:40:36.124145+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1237,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829616+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:40:20.829616+0000","last_peered":"2024-09-21T10:40:20.829616+0000","last_clean":"2024-09-21T10:40:20.829616+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:40:20.829616+0000","last_undegraded":"2024-09-21T10:40:20.829616+0000","last_fullsized":"2024-09-21T10:40:20.829616+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2200,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099765+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:40:36.099765+0000","last_peered":"2024-09-21T10:40:36.099765+0000","last_clean":"2024-09-21T10:40:36.099765+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:40:36.099765+0000","last_undegraded":"2024-09-21T10:40:36.099765+0000","last_fullsized":"2024-09-21T10:40:36.099765+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":120,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.822869+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:40:20.822869+0000","last_peered":"2024-09-21T10:40:20.822869+0000","last_clean":"2024-09-21T10:40:20.822869+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:40:20.822869+0000","last_undegraded":"2024-09-21T10:40:20.822869+0000","last_fullsized":"2024-09-21T10:40:20.822869+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2199,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099563+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:40:36.099563+0000","last_peered":"2024-09-21T10:40:36.099563+0000","last_clean":"2024-09-21T10:40:36.099563+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:40:36.099563+0000","last_undegraded":"2024-09-21T10:40:36.099563+0000","last_fullsized":"2024-09-21T10:40:36.099563+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":143,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823579+0000","last_change":"2024-09-21T10:38:38.962357+0000","last_active":"2024-09-21T10:40:20.823579+0000","last_peered":"2024-09-21T10:40:20.823579+0000","last_clean":"2024-09-21T10:40:20.823579+0000","last_became_active":"2024-09-21T10:38:38.962017+0000","last_became_peered":"2024-09-21T10:38:38.962017+0000","last_unstale":"2024-09-21T10:40:20.823579+0000","last_undegraded":"2024-09-21T10:40:20.823579+0000","last_fullsized":"2024-09-21T10:40:20.823579+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.355706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":81,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.110659+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:40:36.110659+0000","last_peered":"2024-09-21T10:40:36.110659+0000","last_clean":"2024-09-21T10:40:36.110659+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:40:36.110659+0000","last_undegraded":"2024-09-21T10:40:36.110659+0000","last_fullsized":"2024-09-21T10:40:36.110659+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1389,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829388+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:40:20.829388+0000","last_peered":"2024-09-21T10:40:20.829388+0000","last_clean":"2024-09-21T10:40:20.829388+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:40:20.829388+0000","last_undegraded":"2024-09-21T10:40:20.829388+0000","last_fullsized":"2024-09-21T10:40:20.829388+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2089,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.112327+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:40:36.112327+0000","last_peered":"2024-09-21T10:40:36.112327+0000","last_clean":"2024-09-21T10:40:36.112327+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:40:36.112327+0000","last_undegraded":"2024-09-21T10:40:36.112327+0000","last_fullsized":"2024-09-21T10:40:36.112327+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":48,"num_pg":16,"stat_sum":{"num_bytes":3711631384,"num_objects":56636,"num_object_clones":0,"num_object_copies":169908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1121,"num_objects_misplaced":5425,"num_objects_unfound":0,"num_objects_dirty":56636,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":189780,"num_write_kb":7885249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3107,"num_bytes_recovered":203620352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11353927680,"data_stored":11353915464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103605,"ondisk_log_size":103605,"up":48,"acting":47,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":481141,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6268,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398054,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2923940,"kb_used_data":2019996,"kb_used_omap":1,"kb_used_meta":903934,"kb_avail":90821212,"statfs":{"total":95995035648,"available":93000921088,"internally_reserved":0,"allocated":2068475904,"data_stored":2062471585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925628760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049713,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3225316,"kb_used_data":2285532,"kb_used_omap":242,"kb_used_meta":939533,"kb_avail":90519836,"statfs":{"total":95995035648,"available":92692312064,"internally_reserved":0,"allocated":2340384768,"data_stored":2334380454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":962082015},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790512,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944984,"kb_used_data":1056272,"kb_used_omap":1,"kb_used_meta":888702,"kb_avail":91800168,"statfs":{"total":95995035648,"available":94003372032,"internally_reserved":0,"allocated":1081622528,"data_stored":1075626806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910031192},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644672,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2950420,"kb_used_data":2229900,"kb_used_omap":67,"kb_used_meta":720380,"kb_avail":90794732,"statfs":{"total":95995035648,"available":92973805568,"internally_reserved":0,"allocated":2283417600,"data_stored":2277425449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":737669981},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527689,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2194208,"kb_used_data":1479000,"kb_used_omap":75,"kb_used_meta":715124,"kb_avail":91550944,"statfs":{"total":95995035648,"available":93748166656,"internally_reserved":0,"allocated":1514496000,"data_stored":1508496819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":732287130},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274842,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2412628,"kb_used_data":1667532,"kb_used_omap":51,"kb_used_meta":745036,"kb_avail":91332524,"statfs":{"total":95995035648,"available":93524504576,"internally_reserved":0,"allocated":1707552768,"data_stored":1701560098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":762917205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354081,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1136020,"kb_used_data":574092,"kb_used_omap":66,"kb_used_meta":561853,"kb_avail":92609132,"statfs":{"total":95995035648,"available":94831751168,"internally_reserved":0,"allocated":587870208,"data_stored":581878097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":575337796},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":565510144,"data_stored":565510144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1679097856,"data_stored":1679097856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1474691072,"data_stored":1474691072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2253590528,"data_stored":2253586456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1059192832,"data_stored":1059192832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2292125696,"data_stored":2292121624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2029719552,"data_stored":2029715480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:40:46.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:40:46.739 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:46.772 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:46.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:46.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:46.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:47.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.051 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:47.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:47.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:47.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:47.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:47.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:47.454 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:47.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:47 smithi119 ceph-mon[101432]: pgmap v2309: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 1121/165621 objects degraded (0.677%); 5156/165621 objects misplaced (3.113%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:40:47.472 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:47 smithi119 ceph-mon[105288]: pgmap v2309: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 1121/165621 objects degraded (0.677%); 5156/165621 objects misplaced (3.113%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:40:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:47 smithi145 ceph-mon[89359]: pgmap v2309: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 922 op/s; 1121/165621 objects degraded (0.677%); 5156/165621 objects misplaced (3.113%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:40:47.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:47.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:47.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:47.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:47.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:47.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:40:47.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:47.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:48.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:48.041 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:48.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:48.146 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:48.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:48.285 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:48.315 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:48.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:48.437 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:48.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:48.459 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:48.539 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:40:48.575 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:48.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:48.719 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:48.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:48.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:48.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:48.987 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:48.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:49.023 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:49.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:49.130 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:49.166 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:40:49.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:49.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:49.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:49.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:49.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:49.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:49.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:49 smithi119 ceph-mon[101432]: pgmap v2310: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 1121/164865 objects degraded (0.680%); 5156/164865 objects misplaced (3.127%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:40:49.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2957854074' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:49.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:49 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:49.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:49 smithi119 ceph-mon[105288]: pgmap v2310: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 1121/164865 objects degraded (0.680%); 5156/164865 objects misplaced (3.127%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:40:49.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2957854074' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:49.521 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:49 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:49.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:49 smithi145 ceph-mon[89359]: pgmap v2310: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 808 op/s; 1121/164865 objects degraded (0.680%); 5156/164865 objects misplaced (3.127%); 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:40:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2957854074' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:49 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:40:49.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:49.677 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:49.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:49.819 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:49.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:49.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:49.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:49.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:49.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:50.048 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:50.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:50.189 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:50.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[101432]: osdmap e1384: 8 total, 8 up, 7 in 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1121/164865 objects degraded (0.680%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[105288]: osdmap e1384: 8 total, 8 up, 7 in 2024-09-21T10:40:50.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:50 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1121/164865 objects degraded (0.680%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:50.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:50.367 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:50.445 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:40:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:40:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:40:50.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:50 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:40:50.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:50 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:40:50.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:50 smithi145 ceph-mon[89359]: osdmap e1384: 8 total, 8 up, 7 in 2024-09-21T10:40:50.445 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:50 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1121/164865 objects degraded (0.680%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:50.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:50.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:50.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:50.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:50.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:50.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:50.766 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:50.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:50.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:50.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:51.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:51.032 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:51.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.172 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:51.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:51.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:51.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:51.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:51.430 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:51.478 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:51 smithi145 ceph-mon[89359]: pgmap v2312: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 1131/153405 objects degraded (0.737%); 4891/153405 objects misplaced (3.188%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:51.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:51.570 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:51.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:51 smithi119 ceph-mon[101432]: pgmap v2312: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 1131/153405 objects degraded (0.737%); 4891/153405 objects misplaced (3.188%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:51 smithi119 ceph-mon[105288]: pgmap v2312: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 1131/153405 objects degraded (0.737%); 4891/153405 objects misplaced (3.188%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:51.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:51.698 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:51.799 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:51.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:51.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:51.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:51.981 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:51.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:52.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:52.115 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55572 47.18 47:1dcd9306:::benchmark_data_smithi119_238957_object55571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.408987+0000", 2024-09-21T10:40:52.213 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 409.79890758099998, 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0684545240000001, 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55572 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408987+0000", 2024-09-21T10:40:52.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408987+0000", 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408986+0000", 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.409021+0000", 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4363000000000003e-05 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.409028+0000", 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9979999999999997e-06 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.409031+0000", 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.704e-06 2024-09-21T10:40:52.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.490527+0000", 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.081495740999999997 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.490531+0000", 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0210000000000001e-06 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.464917+0000", 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97438605499999997 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.465031+0000", 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011390799999999999 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.465159+0000", 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012815999999999999 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.467195+0000", 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020360980000000001 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.471262+0000", 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040663349999999999 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.477347+0000", 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0060858429999999996 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.477406+0000", 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8780999999999998e-05 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.477441+0000", 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4968000000000001e-05 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.220 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69028 47.10 47:0941a137:::benchmark_data_smithi119_238957_object69027:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.466563+0000", 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.74133112999999, 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.394462952, 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "reached pg", 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69028 2024-09-21T10:40:52.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.466563+0000", 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.466563+0000", 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.466562+0000", 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467472+0000", 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00091045000000000004 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467488+0000", 2024-09-21T10:40:52.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6191000000000001e-05 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467496+0000", 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2359999999999998e-06 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467555+0000", 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9392000000000003e-05 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467565+0000", 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7140000000000005e-06 2024-09-21T10:40:52.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.860944+0000", 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3933793059999999 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.861026+0000", 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1992999999999999e-05 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69063 47.10 47:0e27e481:::benchmark_data_smithi119_238957_object69062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.467510+0000", 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.740383877, 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4003795530000001, 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69063 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467510+0000", 2024-09-21T10:40:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467510+0000", 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.467510+0000", 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:52.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468261+0000", 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00075173799999999995 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468282+0000", 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0313999999999999e-05 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468288+0000", 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.409e-06 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468323+0000", 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5114000000000003e-05 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468330+0000", 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3790000000000003e-06 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.861980+0000", 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.393650289 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.862152+0000", 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017203699999999999 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.862264+0000", 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112545 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.866596+0000", 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0043316589999999999 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.866610+0000", 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3971000000000001e-05 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.866656+0000", 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5673000000000001e-05 2024-09-21T10:40:52.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.866721+0000", 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5368000000000002e-05 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.866759+0000", 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7719999999999998e-05 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.867890+0000", 2024-09-21T10:40:52.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011311559999999999 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69101 47.10 47:08e8388b:::benchmark_data_smithi119_238957_object69100:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.468299+0000", 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73959529899997, 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.400638294, 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.234 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69101 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468299+0000", 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468299+0000", 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468298+0000", 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468975+0000", 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067703800000000003 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.468995+0000", 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0000000000000002e-05 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469000+0000", 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2619999999999998e-06 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469028+0000", 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7398000000000001e-05 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469031+0000", 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4699999999999998e-06 2024-09-21T10:40:52.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.863595+0000", 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.394564181 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.863742+0000", 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014677599999999999 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.863921+0000", 2024-09-21T10:40:52.238 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017829599999999999 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.866619+0000", 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002698935 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.867944+0000", 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001324041 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.868018+0000", 2024-09-21T10:40:52.239 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4307000000000005e-05 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.868039+0000", 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1271e-05 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.868076+0000", 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7144000000000002e-05 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.240 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.868937+0000", 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00086101799999999996 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69109 47.10 47:0d9a22c2:::benchmark_data_smithi119_238957_object69108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.469008+0000", 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73888587599998, 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4010727250000001, 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.241 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69109 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469008+0000", 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.242 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469008+0000", 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469008+0000", 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469074+0000", 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6488000000000004e-05 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.243 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469082+0000", 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9300000000000003e-06 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469087+0000", 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5580000000000002e-06 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469107+0000", 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0054000000000001e-05 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.244 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469110+0000", 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4340000000000001e-06 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.865194+0000", 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3960833580000001 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.865343+0000", 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000149557 2024-09-21T10:40:52.245 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.865467+0000", 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000123786 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.867900+0000", 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024331259999999999 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.868985+0000", 2024-09-21T10:40:52.246 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00108471 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.869069+0000", 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4124999999999998e-05 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.869123+0000", 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4035e-05 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.247 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.869177+0000", 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3826000000000003e-05 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.870081+0000", 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000904436 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69114 47.10 47:0aa7036a:::benchmark_data_smithi119_238957_object69113:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.469093+0000", 2024-09-21T10:40:52.248 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73880093600002, 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.408746069, 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69114 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469093+0000", 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.249 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469093+0000", 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469093+0000", 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469299+0000", 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00020589 2024-09-21T10:40:52.250 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469312+0000", 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3343000000000001e-05 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469317+0000", 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2170000000000002e-06 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469337+0000", 2024-09-21T10:40:52.251 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9762000000000002e-05 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469340+0000", 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.729e-06 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.870142+0000", 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.400801733 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.252 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.870281+0000", 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013948899999999999 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.870448+0000", 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016693799999999999 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.876167+0000", 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005718987 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.253 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.876205+0000", 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8435e-05 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.876236+0000", 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0888e-05 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.876266+0000", 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9371000000000001e-05 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.254 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.876322+0000", 2024-09-21T10:40:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6808000000000002e-05 2024-09-21T10:40:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.255 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.877839+0000", 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015169859999999999 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69123 47.10 47:0bda38ba:::benchmark_data_smithi119_238957_object69122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.469324+0000", 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73857020899999, 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.409840972, 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.256 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69123 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469324+0000", 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.257 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469324+0000", 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469324+0000", 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469931+0000", 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00060744100000000001 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.258 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469940+0000", 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7309999999999999e-06 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469945+0000", 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0479999999999998e-06 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469970+0000", 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5717000000000001e-05 2024-09-21T10:40:52.259 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469974+0000", 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7340000000000002e-06 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.873454+0000", 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4034793649999999 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.873623+0000", 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016979300000000001 2024-09-21T10:40:52.260 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.873795+0000", 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017208100000000001 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.877904+0000", 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0041089999999999998 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.261 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.877948+0000", 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3631000000000003e-05 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.877956+0000", 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2889999999999998e-06 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.878004+0000", 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7787000000000002e-05 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.262 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.878060+0000", 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6029999999999997e-05 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.879165+0000", 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001104889 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.263 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69154 47.10 47:0d2a5903:::benchmark_data_smithi119_238957_object69153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.469951+0000", 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73794279999998, 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4262207659999999, 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69154 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.264 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469951+0000", 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469951+0000", 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.469951+0000", 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.265 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470132+0000", 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018105700000000001 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470140+0000", 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1519999999999992e-06 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470145+0000", 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6299999999999997e-06 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.266 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470166+0000", 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1679e-05 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470169+0000", 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.88e-06 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.879237+0000", 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.409068057 2024-09-21T10:40:52.267 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.879344+0000", 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000106394 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.879459+0000", 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000115467 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.882990+0000", 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035306790000000001 2024-09-21T10:40:52.268 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883032+0000", 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1891999999999998e-05 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883042+0000", 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9010000000000006e-06 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.894926+0000", 2024-09-21T10:40:52.269 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011883879 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.894979+0000", 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3597999999999997e-05 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.896172+0000", 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011930459999999999 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.270 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69163 47.10 47:0ae0a62b:::benchmark_data_smithi119_238957_object69162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.470151+0000", 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73774282699998, 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.426914053, 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69163 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.271 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470151+0000", 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470151+0000", 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470151+0000", 2024-09-21T10:40:52.272 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470330+0000", 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000179037 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470338+0000", 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7430000000000002e-06 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.273 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470343+0000", 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8879999999999996e-06 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470362+0000", 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9445000000000001e-05 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470366+0000", 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4400000000000001e-06 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.274 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.880461+0000", 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4100954960000001 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.880595+0000", 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000133771 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.880755+0000", 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016061600000000001 2024-09-21T10:40:52.275 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883049+0000", 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022933760000000002 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883092+0000", 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2991000000000002e-05 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883146+0000", 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4626000000000002e-05 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.276 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.896281+0000", 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013134323999999999 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.896318+0000", 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7237999999999998e-05 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.897066+0000", 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00074755500000000001 2024-09-21T10:40:52.277 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69168 47.10 47:0ce2ef47:::benchmark_data_smithi119_238957_object69167:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.470349+0000", 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.737545012, 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4332416509999999, 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.278 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69168 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470349+0000", 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470349+0000", 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.279 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470349+0000", 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470965+0000", 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00061583000000000004 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470971+0000", 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8979999999999999e-06 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.280 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470976+0000", 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8590000000000004e-06 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471012+0000", 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5587000000000003e-05 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471018+0000", 2024-09-21T10:40:52.281 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4910000000000002e-06 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.881896+0000", 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4108771149999999 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.882030+0000", 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013475299999999999 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.282 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.882182+0000", 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015169399999999999 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.884383+0000", 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022009759999999999 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.885693+0000", 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001310475 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.283 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.885747+0000", 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3758999999999998e-05 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.902135+0000", 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016387477000000001 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.902191+0000", 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6205999999999997e-05 2024-09-21T10:40:52.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.903591+0000", 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014000480000000001 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69189 47.10 47:08d24009:::benchmark_data_smithi119_238957_object69188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.470983+0000", 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73691154099998, 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.433723965, 2024-09-21T10:40:52.285 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69189 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470983+0000", 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.286 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470983+0000", 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.470982+0000", 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471233+0000", 2024-09-21T10:40:52.287 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025088500000000001 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471240+0000", 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3560000000000004e-06 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471245+0000", 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5800000000000002e-06 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.288 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471265+0000", 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9964999999999999e-05 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471268+0000", 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0070000000000002e-06 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883206+0000", 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.411937591 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.289 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883349+0000", 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000143545 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.883516+0000", 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000167286 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.885760+0000", 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002243288 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.290 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.885807+0000", 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7741000000000003e-05 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.885864+0000", 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7034999999999999e-05 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.903649+0000", 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017784879999999999 2024-09-21T10:40:52.291 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.903730+0000", 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0588999999999997e-05 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.904707+0000", 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00097674799999999994 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.292 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69202 47.10 47:0e5a6fc9:::benchmark_data_smithi119_238957_object69201:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.471252+0000", 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73664260100003, 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4344938329999999, 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69202 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.293 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471252+0000", 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471252+0000", 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471251+0000", 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.294 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471391+0000", 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014031800000000001 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471399+0000", 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5259999999999996e-06 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471403+0000", 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3080000000000001e-06 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.295 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471424+0000", 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0976e-05 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471427+0000", 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.791e-06 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.884410+0000", 2024-09-21T10:40:52.297 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.412983433 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.884548+0000", 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000137964 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.884739+0000", 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019097999999999999 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.298 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888320+0000", 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035803760000000001 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888377+0000", 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6983e-05 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888437+0000", 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9821999999999997e-05 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.299 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.904772+0000", 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.01633544 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.904847+0000", 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4958999999999998e-05 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.905746+0000", 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00089850200000000002 2024-09-21T10:40:52.300 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69208 47.10 47:0b7b820b:::benchmark_data_smithi119_238957_object69207:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.471410+0000", 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73648474800001, 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.435332278, 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69208 2024-09-21T10:40:52.301 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471410+0000", 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471410+0000", 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.302 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.471409+0000", 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472029+0000", 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00062000499999999997 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472036+0000", 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4019999999999996e-06 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.303 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472041+0000", 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1790000000000001e-06 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472068+0000", 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6897e-05 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472072+0000", 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.843e-06 2024-09-21T10:40:52.304 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.885893+0000", 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.413820673 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.886006+0000", 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000112933 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.305 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.886216+0000", 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000210375 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888449+0000", 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022325380000000001 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888475+0000", 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6418999999999999e-05 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.306 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888525+0000", 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9543000000000003e-05 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.905799+0000", 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.017274094 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.905861+0000", 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2081000000000004e-05 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.307 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.906742+0000", 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00088079499999999997 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69265 47.10 47:086e833e:::benchmark_data_smithi119_238957_object69264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.472047+0000", 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73584768500001, 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.437327633, 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.308 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69265 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472047+0000", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472047+0000", 2024-09-21T10:40:52.309 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472046+0000", 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472371+0000", 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00032470300000000003 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.310 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472381+0000", 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0101999999999999e-05 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472386+0000", 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3539999999999999e-06 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472415+0000", 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8495000000000001e-05 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.311 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472419+0000", 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2100000000000003e-06 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.887185+0000", 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4147664550000001 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.887331+0000", 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000145694 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.312 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.887437+0000", 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010631199999999999 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888535+0000", 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010975959999999999 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.890995+0000", 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002460168 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.313 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.891057+0000", 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1879999999999997e-05 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.907955+0000", 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016898099 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.908015+0000", 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9475000000000002e-05 2024-09-21T10:40:52.314 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.909374+0000", 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013596089999999999 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69266 47.10 47:0be4fe5f:::benchmark_data_smithi119_238957_object69265:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.472394+0000", 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.735500559, 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.43788245, 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.315 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69266 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472394+0000", 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472394+0000", 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.316 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472393+0000", 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472565+0000", 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00017141699999999999 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472572+0000", 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.413e-06 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.317 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472577+0000", 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9840000000000001e-06 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472597+0000", 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0358999999999999e-05 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472600+0000", 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0290000000000001e-06 2024-09-21T10:40:52.318 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888584+0000", 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4159838119999999 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888774+0000", 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018963699999999999 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.888904+0000", 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013018700000000001 2024-09-21T10:40:52.319 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.891068+0000", 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021640019999999999 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.891142+0000", 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3967999999999994e-05 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.320 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.893552+0000", 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024100319999999999 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.909418+0000", 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.015866265000000001 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.909459+0000", 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0435000000000002e-05 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.321 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.910276+0000", 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00081752699999999997 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69279 47.10 47:0c33176e:::benchmark_data_smithi119_238957_object69278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.472584+0000", 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73531020399997, 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.438725195, 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.322 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69279 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472584+0000", 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472584+0000", 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.323 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472584+0000", 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472977+0000", 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00039326699999999999 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472987+0000", 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7089999999999997e-06 2024-09-21T10:40:52.324 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.472994+0000", 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7649999999999993e-06 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473031+0000", 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6758000000000003e-05 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473039+0000", 2024-09-21T10:40:52.325 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9270000000000005e-06 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.889702+0000", 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4166634710000001 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.889853+0000", 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000150612 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.326 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.890033+0000", 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000180077 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.892382+0000", 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023488379999999998 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.893502+0000", 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011197589999999999 2024-09-21T10:40:52.327 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.893612+0000", 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000110046 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.910321+0000", 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016709436000000001 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.910374+0000", 2024-09-21T10:40:52.328 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3260999999999997e-05 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.911309+0000", 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00093480400000000002 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:69331 47.10 47:09a14d40:::benchmark_data_smithi119_238957_object69330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1296)", 2024-09-21T10:40:52.329 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:11.473001+0000", 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 400.73489334999999, 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.439560349, 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 69331 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:40:52.330 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473001+0000", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473001+0000", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473000+0000", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473187+0000", 2024-09-21T10:40:52.331 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018660000000000001 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473194+0000", 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9160000000000004e-06 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473198+0000", 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.121e-06 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.332 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473220+0000", 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1687000000000001e-05 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:11.473223+0000", 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2629999999999999e-06 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.891201+0000", 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.417978484 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.333 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.891334+0000", 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013267800000000001 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,5,6", 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.891475+0000", 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014090899999999999 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.893650+0000", 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021752799999999999 2024-09-21T10:40:52.334 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.894859+0000", 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012085170000000001 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.894890+0000", 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1566999999999998e-05 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.911483+0000", 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016592638999999999 2024-09-21T10:40:52.335 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.911539+0000", 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6029000000000002e-05 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:12.912561+0000", 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010222409999999999 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:40:52.336 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:40:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:52.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:52.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:40:52.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:52.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:52.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:52.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:52.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:52.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:52.757 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:52.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:52.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:52.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:53.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.045 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:53.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:53.120 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:40:53.121 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:40:53.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:53.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:53.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:53.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:53.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:40:53.391 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:53 smithi119 ceph-mon[105288]: pgmap v2313: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 1131/152574 objects degraded (0.741%); 4614/152574 objects misplaced (3.024%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:53.394 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:53 smithi119 ceph-mon[101432]: pgmap v2313: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 1131/152574 objects degraded (0.741%); 4614/152574 objects misplaced (3.024%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:53.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:53.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:53.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:53 smithi145 ceph-mon[89359]: pgmap v2313: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 914 op/s; 1131/152574 objects degraded (0.741%); 4614/152574 objects misplaced (3.024%); 2.9 MiB/s, 46 objects/s recovering 2024-09-21T10:40:53.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:53.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:53.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:53.743 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:53.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:53.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:53.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:53.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:53.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:54.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:54.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:54.179 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2313,"stamp":"2024-09-21T10:40:51.825900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3331747130,"num_objects":50858,"num_object_clones":0,"num_object_copies":152574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1131,"num_objects_misplaced":4614,"num_objects_unfound":0,"num_objects_dirty":50858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195647,"num_write_kb":7885280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3418,"num_bytes_recovered":221256367,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105467,"ondisk_log_size":105467,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15736140,"kb_used_data":10246740,"kb_used_omap":507,"kb_used_meta":5488452,"kb_avail":640479924,"statfs":{"total":671965249536,"available":655851442176,"internally_reserved":0,"allocated":10492661760,"data_stored":10450673974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":5620175391},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,2,2,6,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-599851008,"num_objects":-9153,"num_object_clones":0,"num_object_copies":-27459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":-1105,"num_objects_unfound":0,"num_objects_dirty":-9153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":30212096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003291"},"pg_stats":[{"pgid":"48.f","version":"1383'12167","reported_seq":12237,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:48.713667+0000","last_change":"2024-09-21T10:37:50.818738+0000","last_active":"2024-09-21T10:40:48.713667+0000","last_peered":"2024-09-21T10:40:48.713667+0000","last_clean":"2024-09-21T10:40:48.713667+0000","last_became_active":"2024-09-21T10:37:50.818409+0000","last_became_peered":"2024-09-21T10:37:50.818409+0000","last_unstale":"2024-09-21T10:40:48.713667+0000","last_undegraded":"2024-09-21T10:40:48.713667+0000","last_fullsized":"2024-09-21T10:40:48.713667+0000","mapping_epoch":1354,"log_start":"1369'6600","ondisk_log_start":"1369'6600","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:24:13.373038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207421440,"num_objects":3165,"num_object_clones":0,"num_object_copies":9495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12167,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.e","version":"1384'12356","reported_seq":12428,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.405573+0000","last_change":"2024-09-21T10:37:50.816783+0000","last_active":"2024-09-21T10:40:49.405573+0000","last_peered":"2024-09-21T10:40:49.405573+0000","last_clean":"2024-09-21T10:40:49.405573+0000","last_became_active":"2024-09-21T10:37:50.816482+0000","last_became_peered":"2024-09-21T10:37:50.816482+0000","last_unstale":"2024-09-21T10:40:49.405573+0000","last_undegraded":"2024-09-21T10:40:49.405573+0000","last_fullsized":"2024-09-21T10:40:49.405573+0000","mapping_epoch":1354,"log_start":"1364'4600","ondisk_log_start":"1364'4600","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7756,"log_dups_size":0,"ondisk_log_size":7756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:57.011771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210370560,"num_objects":3210,"num_object_clones":0,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12356,"num_write_kb":498112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.d","version":"1383'12238","reported_seq":12276,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:47.960455+0000","last_change":"2024-09-21T10:37:50.816596+0000","last_active":"2024-09-21T10:40:47.960455+0000","last_peered":"2024-09-21T10:40:47.960455+0000","last_clean":"2024-09-21T10:40:47.960455+0000","last_became_active":"2024-09-21T10:37:50.816060+0000","last_became_peered":"2024-09-21T10:37:50.816060+0000","last_unstale":"2024-09-21T10:40:47.960455+0000","last_undegraded":"2024-09-21T10:40:47.960455+0000","last_fullsized":"2024-09-21T10:40:47.960455+0000","mapping_epoch":1354,"log_start":"1369'6574","ondisk_log_start":"1369'6574","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5664,"log_dups_size":0,"ondisk_log_size":5664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:09.093088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212926488,"num_objects":3250,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12238,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.c","version":"1383'12397","reported_seq":12466,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:48.751476+0000","last_change":"2024-09-21T10:37:50.814694+0000","last_active":"2024-09-21T10:40:48.751476+0000","last_peered":"2024-09-21T10:40:48.751476+0000","last_clean":"2024-09-21T10:40:48.751476+0000","last_became_active":"2024-09-21T10:37:50.814424+0000","last_became_peered":"2024-09-21T10:37:50.814424+0000","last_unstale":"2024-09-21T10:40:48.751476+0000","last_undegraded":"2024-09-21T10:40:48.751476+0000","last_fullsized":"2024-09-21T10:40:48.751476+0000","mapping_epoch":1354,"log_start":"1369'6800","ondisk_log_start":"1369'6800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5597,"log_dups_size":0,"ondisk_log_size":5597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:22:22.559518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208732160,"num_objects":3185,"num_object_clones":0,"num_object_copies":9555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12397,"num_write_kb":498624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.b","version":"1383'12162","reported_seq":12199,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:48.742856+0000","last_change":"2024-09-21T10:37:50.809808+0000","last_active":"2024-09-21T10:40:48.742856+0000","last_peered":"2024-09-21T10:40:48.742856+0000","last_clean":"2024-09-21T10:40:48.742856+0000","last_became_active":"2024-09-21T10:37:50.809451+0000","last_became_peered":"2024-09-21T10:37:50.809451+0000","last_unstale":"2024-09-21T10:40:48.742856+0000","last_undegraded":"2024-09-21T10:40:48.742856+0000","last_fullsized":"2024-09-21T10:40:48.742856+0000","mapping_epoch":1354,"log_start":"1365'5900","ondisk_log_start":"1365'5900","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6262,"log_dups_size":0,"ondisk_log_size":6262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:46.455275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204865536,"num_objects":3126,"num_object_clones":0,"num_object_copies":9378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3126,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12162,"num_write_kb":489216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.a","version":"1384'12523","reported_seq":12595,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.420674+0000","last_change":"2024-09-21T10:37:50.806280+0000","last_active":"2024-09-21T10:40:49.420674+0000","last_peered":"2024-09-21T10:40:49.420674+0000","last_clean":"2024-09-21T10:40:49.420674+0000","last_became_active":"2024-09-21T10:37:50.805997+0000","last_became_peered":"2024-09-21T10:37:50.805997+0000","last_unstale":"2024-09-21T10:40:49.420674+0000","last_undegraded":"2024-09-21T10:40:49.420674+0000","last_fullsized":"2024-09-21T10:40:49.420674+0000","mapping_epoch":1354,"log_start":"1364'4800","ondisk_log_start":"1364'4800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7723,"log_dups_size":0,"ondisk_log_size":7723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.061122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209387520,"num_objects":3195,"num_object_clones":0,"num_object_copies":9585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12523,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.9","version":"1383'11986","reported_seq":12056,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:48.707324+0000","last_change":"2024-09-21T10:37:50.814434+0000","last_active":"2024-09-21T10:40:48.707324+0000","last_peered":"2024-09-21T10:40:48.707324+0000","last_clean":"2024-09-21T10:40:48.707324+0000","last_became_active":"2024-09-21T10:37:50.814150+0000","last_became_peered":"2024-09-21T10:37:50.814150+0000","last_unstale":"2024-09-21T10:40:48.707324+0000","last_undegraded":"2024-09-21T10:40:48.707324+0000","last_fullsized":"2024-09-21T10:40:48.707324+0000","mapping_epoch":1354,"log_start":"1369'6400","ondisk_log_start":"1369'6400","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5586,"log_dups_size":0,"ondisk_log_size":5586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:18.301417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198443008,"num_objects":3028,"num_object_clones":0,"num_object_copies":9084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11986,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.8","version":"1383'12100","reported_seq":12137,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:48.674152+0000","last_change":"2024-09-21T10:37:50.812704+0000","last_active":"2024-09-21T10:40:48.674152+0000","last_peered":"2024-09-21T10:40:48.674152+0000","last_clean":"2024-09-21T10:40:48.674152+0000","last_became_active":"2024-09-21T10:37:50.812368+0000","last_became_peered":"2024-09-21T10:37:50.812368+0000","last_unstale":"2024-09-21T10:40:48.674152+0000","last_undegraded":"2024-09-21T10:40:48.674152+0000","last_fullsized":"2024-09-21T10:40:48.674152+0000","mapping_epoch":1354,"log_start":"1363'4400","ondisk_log_start":"1363'4400","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7700,"log_dups_size":0,"ondisk_log_size":7700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:22.704564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212205568,"num_objects":3238,"num_object_clones":0,"num_object_copies":9714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12100,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"48.7","version":"1384'12604","reported_seq":12680,"reported_epoch":1384,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:40:50.582805+0000","last_change":"2024-09-21T10:40:12.260234+0000","last_active":"2024-09-21T10:40:50.582805+0000","last_peered":"2024-09-21T10:40:50.582805+0000","last_clean":"2024-09-21T10:40:09.890233+0000","last_became_active":"2024-09-21T10:40:12.242243+0000","last_became_peered":"2024-09-21T10:40:12.242243+0000","last_unstale":"2024-09-21T10:40:50.582805+0000","last_undegraded":"2024-09-21T10:40:50.582805+0000","last_fullsized":"2024-09-21T10:40:50.582805+0000","mapping_epoch":1379,"log_start":"1369'6900","ondisk_log_start":"1369'6900","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5704,"log_dups_size":0,"ondisk_log_size":5704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208011264,"num_objects":3174,"num_object_clones":0,"num_object_copies":9522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3174,"num_objects_unfound":0,"num_objects_dirty":3174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12604,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":3174}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"48.6","version":"1383'12232","reported_seq":12302,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:48.744675+0000","last_change":"2024-09-21T10:37:50.811591+0000","last_active":"2024-09-21T10:40:48.744675+0000","last_peered":"2024-09-21T10:40:48.744675+0000","last_clean":"2024-09-21T10:40:48.744675+0000","last_became_active":"2024-09-21T10:37:50.811260+0000","last_became_peered":"2024-09-21T10:37:50.811260+0000","last_unstale":"2024-09-21T10:40:48.744675+0000","last_undegraded":"2024-09-21T10:40:48.744675+0000","last_fullsized":"2024-09-21T10:40:48.744675+0000","mapping_epoch":1354,"log_start":"1369'6600","ondisk_log_start":"1369'6600","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:19.966736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205389824,"num_objects":3134,"num_object_clones":0,"num_object_copies":9402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12232,"num_write_kb":491712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.5","version":"1383'12095","reported_seq":12132,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:47.973065+0000","last_change":"2024-09-21T10:37:50.816067+0000","last_active":"2024-09-21T10:40:47.973065+0000","last_peered":"2024-09-21T10:40:47.973065+0000","last_clean":"2024-09-21T10:40:47.973065+0000","last_became_active":"2024-09-21T10:37:50.815783+0000","last_became_peered":"2024-09-21T10:37:50.815783+0000","last_unstale":"2024-09-21T10:40:47.973065+0000","last_undegraded":"2024-09-21T10:40:47.973065+0000","last_fullsized":"2024-09-21T10:40:47.973065+0000","mapping_epoch":1354,"log_start":"1369'6410","ondisk_log_start":"1369'6410","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:03.096851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212533248,"num_objects":3243,"num_object_clones":0,"num_object_copies":9729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12095,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.4","version":"1384'12150","reported_seq":12222,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.414299+0000","last_change":"2024-09-21T10:37:50.810595+0000","last_active":"2024-09-21T10:40:49.414299+0000","last_peered":"2024-09-21T10:40:49.414299+0000","last_clean":"2024-09-21T10:40:49.414299+0000","last_became_active":"2024-09-21T10:37:50.810178+0000","last_became_peered":"2024-09-21T10:37:50.810178+0000","last_unstale":"2024-09-21T10:40:49.414299+0000","last_undegraded":"2024-09-21T10:40:49.414299+0000","last_fullsized":"2024-09-21T10:40:49.414299+0000","mapping_epoch":1354,"log_start":"1362'4400","ondisk_log_start":"1362'4400","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7750,"log_dups_size":0,"ondisk_log_size":7750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:30.334590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206700544,"num_objects":3154,"num_object_clones":0,"num_object_copies":9462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12150,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.3","version":"1383'11987","reported_seq":12057,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:47.549700+0000","last_change":"2024-09-21T10:37:50.808064+0000","last_active":"2024-09-21T10:40:47.549700+0000","last_peered":"2024-09-21T10:40:47.549700+0000","last_clean":"2024-09-21T10:40:47.549700+0000","last_became_active":"2024-09-21T10:37:50.807674+0000","last_became_peered":"2024-09-21T10:37:50.807674+0000","last_unstale":"2024-09-21T10:40:47.549700+0000","last_undegraded":"2024-09-21T10:40:47.549700+0000","last_fullsized":"2024-09-21T10:40:47.549700+0000","mapping_epoch":1354,"log_start":"1364'5000","ondisk_log_start":"1364'5000","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6987,"log_dups_size":0,"ondisk_log_size":6987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:21.766427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211222528,"num_objects":3223,"num_object_clones":0,"num_object_copies":9669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11987,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"48.2","version":"1383'12481","reported_seq":12518,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:47.966271+0000","last_change":"2024-09-21T10:37:50.816843+0000","last_active":"2024-09-21T10:40:47.966271+0000","last_peered":"2024-09-21T10:40:47.966271+0000","last_clean":"2024-09-21T10:40:47.966271+0000","last_became_active":"2024-09-21T10:37:50.816490+0000","last_became_peered":"2024-09-21T10:37:50.816490+0000","last_unstale":"2024-09-21T10:40:47.966271+0000","last_undegraded":"2024-09-21T10:40:47.966271+0000","last_fullsized":"2024-09-21T10:40:47.966271+0000","mapping_epoch":1354,"log_start":"1369'6791","ondisk_log_start":"1369'6791","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5690,"log_dups_size":0,"ondisk_log_size":5690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:33.243791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215416832,"num_objects":3287,"num_object_clones":0,"num_object_copies":9861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12481,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.1","version":"1383'12074","reported_seq":18846,"reported_epoch":1383,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:40:48.800469+0000","last_change":"2024-09-21T10:39:49.804470+0000","last_active":"2024-09-21T10:40:48.800469+0000","last_peered":"2024-09-21T10:40:48.800469+0000","last_clean":"2024-09-21T10:38:59.823246+0000","last_became_active":"2024-09-21T10:39:02.073182+0000","last_became_peered":"2024-09-21T10:39:02.073182+0000","last_unstale":"2024-09-21T10:40:48.800469+0000","last_undegraded":"2024-09-21T10:39:49.800872+0000","last_fullsized":"2024-09-21T10:39:01.839371+0000","mapping_epoch":1367,"log_start":"1365'5800","ondisk_log_start":"1365'5800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:06.547860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202899456,"num_objects":3096,"num_object_clones":0,"num_object_copies":9288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1131,"num_objects_misplaced":1440,"num_objects_unfound":0,"num_objects_dirty":3096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12074,"num_write_kb":485440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3362,"num_bytes_recovered":220332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,6],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"5,6,7","objects":1440},{"shards":"6,7","objects":525}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.0","version":"1384'12032","reported_seq":12104,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.426456+0000","last_change":"2024-09-21T10:37:50.812734+0000","last_active":"2024-09-21T10:40:49.426456+0000","last_peered":"2024-09-21T10:40:49.426456+0000","last_clean":"2024-09-21T10:40:49.426456+0000","last_became_active":"2024-09-21T10:37:50.812438+0000","last_became_peered":"2024-09-21T10:37:50.812438+0000","last_unstale":"2024-09-21T10:40:49.426456+0000","last_undegraded":"2024-09-21T10:40:49.426456+0000","last_fullsized":"2024-09-21T10:40:49.426456+0000","mapping_epoch":1354,"log_start":"1362'4300","ondisk_log_start":"1362'4300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7732,"log_dups_size":0,"ondisk_log_size":7732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:25.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204734464,"num_objects":3124,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12032,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":50,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099864+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:40:36.099864+0000","last_peered":"2024-09-21T10:40:36.099864+0000","last_clean":"2024-09-21T10:40:36.099864+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:40:36.099864+0000","last_undegraded":"2024-09-21T10:40:36.099864+0000","last_fullsized":"2024-09-21T10:40:36.099864+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1277,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830224+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:40:20.830224+0000","last_peered":"2024-09-21T10:40:20.830224+0000","last_clean":"2024-09-21T10:40:20.830224+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:40:20.830224+0000","last_undegraded":"2024-09-21T10:40:20.830224+0000","last_fullsized":"2024-09-21T10:40:20.830224+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2087,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106329+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:40:36.106329+0000","last_peered":"2024-09-21T10:40:36.106329+0000","last_clean":"2024-09-21T10:40:36.106329+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:40:36.106329+0000","last_undegraded":"2024-09-21T10:40:36.106329+0000","last_fullsized":"2024-09-21T10:40:36.106329+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2106,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.107181+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:40:36.107181+0000","last_peered":"2024-09-21T10:40:36.107181+0000","last_clean":"2024-09-21T10:40:36.107181+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:40:36.107181+0000","last_undegraded":"2024-09-21T10:40:36.107181+0000","last_fullsized":"2024-09-21T10:40:36.107181+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":151,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823631+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:40:20.823631+0000","last_peered":"2024-09-21T10:40:20.823631+0000","last_clean":"2024-09-21T10:40:20.823631+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:40:20.823631+0000","last_undegraded":"2024-09-21T10:40:20.823631+0000","last_fullsized":"2024-09-21T10:40:20.823631+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1289,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106486+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:40:36.106486+0000","last_peered":"2024-09-21T10:40:36.106486+0000","last_clean":"2024-09-21T10:40:36.106486+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:40:36.106486+0000","last_undegraded":"2024-09-21T10:40:36.106486+0000","last_fullsized":"2024-09-21T10:40:36.106486+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":151,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823414+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:40:20.823414+0000","last_peered":"2024-09-21T10:40:20.823414+0000","last_clean":"2024-09-21T10:40:20.823414+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:40:20.823414+0000","last_undegraded":"2024-09-21T10:40:20.823414+0000","last_fullsized":"2024-09-21T10:40:20.823414+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1376,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.107077+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:40:36.107077+0000","last_peered":"2024-09-21T10:40:36.107077+0000","last_clean":"2024-09-21T10:40:36.107077+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:40:36.107077+0000","last_undegraded":"2024-09-21T10:40:36.107077+0000","last_fullsized":"2024-09-21T10:40:36.107077+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":44,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100043+0000","last_change":"2024-09-21T10:39:00.847895+0000","last_active":"2024-09-21T10:40:36.100043+0000","last_peered":"2024-09-21T10:40:36.100043+0000","last_clean":"2024-09-21T10:40:36.100043+0000","last_became_active":"2024-09-21T10:39:00.847533+0000","last_became_peered":"2024-09-21T10:39:00.847533+0000","last_unstale":"2024-09-21T10:40:36.100043+0000","last_undegraded":"2024-09-21T10:40:36.100043+0000","last_fullsized":"2024-09-21T10:40:36.100043+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:55.080913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":75,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.176964+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:40:49.176964+0000","last_peered":"2024-09-21T10:40:49.176964+0000","last_clean":"2024-09-21T10:40:49.176964+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:40:49.176964+0000","last_undegraded":"2024-09-21T10:40:49.176964+0000","last_fullsized":"2024-09-21T10:40:49.176964+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":151,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.176025+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:40:49.176025+0000","last_peered":"2024-09-21T10:40:49.176025+0000","last_clean":"2024-09-21T10:40:49.176025+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:40:49.176025+0000","last_undegraded":"2024-09-21T10:40:49.176025+0000","last_fullsized":"2024-09-21T10:40:49.176025+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2095,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.108749+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:40:36.108749+0000","last_peered":"2024-09-21T10:40:36.108749+0000","last_clean":"2024-09-21T10:40:36.108749+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:40:36.108749+0000","last_undegraded":"2024-09-21T10:40:36.108749+0000","last_fullsized":"2024-09-21T10:40:36.108749+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2196,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179605+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:40:49.179605+0000","last_peered":"2024-09-21T10:40:49.179605+0000","last_clean":"2024-09-21T10:40:49.179605+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:40:49.179605+0000","last_undegraded":"2024-09-21T10:40:49.179605+0000","last_fullsized":"2024-09-21T10:40:49.179605+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2203,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099771+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:40:36.099771+0000","last_peered":"2024-09-21T10:40:36.099771+0000","last_clean":"2024-09-21T10:40:36.099771+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:40:36.099771+0000","last_undegraded":"2024-09-21T10:40:36.099771+0000","last_fullsized":"2024-09-21T10:40:36.099771+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2121,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106532+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:40:36.106532+0000","last_peered":"2024-09-21T10:40:36.106532+0000","last_clean":"2024-09-21T10:40:36.106532+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:40:36.106532+0000","last_undegraded":"2024-09-21T10:40:36.106532+0000","last_fullsized":"2024-09-21T10:40:36.106532+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1341,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830709+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:40:20.830709+0000","last_peered":"2024-09-21T10:40:20.830709+0000","last_clean":"2024-09-21T10:40:20.830709+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:40:20.830709+0000","last_undegraded":"2024-09-21T10:40:20.830709+0000","last_fullsized":"2024-09-21T10:40:20.830709+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2100,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106946+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:40:36.106946+0000","last_peered":"2024-09-21T10:40:36.106946+0000","last_clean":"2024-09-21T10:40:36.106946+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:40:36.106946+0000","last_undegraded":"2024-09-21T10:40:36.106946+0000","last_fullsized":"2024-09-21T10:40:36.106946+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2213,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100049+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:40:36.100049+0000","last_peered":"2024-09-21T10:40:36.100049+0000","last_clean":"2024-09-21T10:40:36.100049+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:40:36.100049+0000","last_undegraded":"2024-09-21T10:40:36.100049+0000","last_fullsized":"2024-09-21T10:40:36.100049+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":232,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.177048+0000","last_change":"2024-09-21T10:37:47.788643+0000","last_active":"2024-09-21T10:40:49.177048+0000","last_peered":"2024-09-21T10:40:49.177048+0000","last_clean":"2024-09-21T10:40:49.177048+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:40:49.177048+0000","last_undegraded":"2024-09-21T10:40:49.177048+0000","last_fullsized":"2024-09-21T10:40:49.177048+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:39.188916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2083,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.107392+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:40:36.107392+0000","last_peered":"2024-09-21T10:40:36.107392+0000","last_clean":"2024-09-21T10:40:36.107392+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:40:36.107392+0000","last_undegraded":"2024-09-21T10:40:36.107392+0000","last_fullsized":"2024-09-21T10:40:36.107392+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823998+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:40:20.823998+0000","last_peered":"2024-09-21T10:40:20.823998+0000","last_clean":"2024-09-21T10:40:20.823998+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:40:20.823998+0000","last_undegraded":"2024-09-21T10:40:20.823998+0000","last_fullsized":"2024-09-21T10:40:20.823998+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1746,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.178585+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:40:49.178585+0000","last_peered":"2024-09-21T10:40:49.178585+0000","last_clean":"2024-09-21T10:40:49.178585+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:40:49.178585+0000","last_undegraded":"2024-09-21T10:40:49.178585+0000","last_fullsized":"2024-09-21T10:40:49.178585+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2188,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.176186+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:40:49.176186+0000","last_peered":"2024-09-21T10:40:49.176186+0000","last_clean":"2024-09-21T10:40:49.176186+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:40:49.176186+0000","last_undegraded":"2024-09-21T10:40:49.176186+0000","last_fullsized":"2024-09-21T10:40:49.176186+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":398,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824752+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:40:20.824752+0000","last_peered":"2024-09-21T10:40:20.824752+0000","last_clean":"2024-09-21T10:40:20.824752+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:40:20.824752+0000","last_undegraded":"2024-09-21T10:40:20.824752+0000","last_fullsized":"2024-09-21T10:40:20.824752+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1242,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829988+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:40:20.829988+0000","last_peered":"2024-09-21T10:40:20.829988+0000","last_clean":"2024-09-21T10:40:20.829988+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:40:20.829988+0000","last_undegraded":"2024-09-21T10:40:20.829988+0000","last_fullsized":"2024-09-21T10:40:20.829988+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":342,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.109223+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:40:36.109223+0000","last_peered":"2024-09-21T10:40:36.109223+0000","last_clean":"2024-09-21T10:40:36.109223+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:40:36.109223+0000","last_undegraded":"2024-09-21T10:40:36.109223+0000","last_fullsized":"2024-09-21T10:40:36.109223+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1276,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823003+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:40:20.823003+0000","last_peered":"2024-09-21T10:40:20.823003+0000","last_clean":"2024-09-21T10:40:20.823003+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:40:20.823003+0000","last_undegraded":"2024-09-21T10:40:20.823003+0000","last_fullsized":"2024-09-21T10:40:20.823003+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1337,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830640+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:40:20.830640+0000","last_peered":"2024-09-21T10:40:20.830640+0000","last_clean":"2024-09-21T10:40:20.830640+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:40:20.830640+0000","last_undegraded":"2024-09-21T10:40:20.830640+0000","last_fullsized":"2024-09-21T10:40:20.830640+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":77,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099611+0000","last_change":"2024-09-21T10:37:41.926686+0000","last_active":"2024-09-21T10:40:36.099611+0000","last_peered":"2024-09-21T10:40:36.099611+0000","last_clean":"2024-09-21T10:40:36.099611+0000","last_became_active":"2024-09-21T10:37:41.922706+0000","last_became_peered":"2024-09-21T10:37:41.922706+0000","last_unstale":"2024-09-21T10:40:36.099611+0000","last_undegraded":"2024-09-21T10:40:36.099611+0000","last_fullsized":"2024-09-21T10:40:36.099611+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:08.955843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":134,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.175883+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:40:49.175883+0000","last_peered":"2024-09-21T10:40:49.175883+0000","last_clean":"2024-09-21T10:40:49.175883+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:40:49.175883+0000","last_undegraded":"2024-09-21T10:40:49.175883+0000","last_fullsized":"2024-09-21T10:40:49.175883+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":39,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.822763+0000","last_change":"2024-09-21T10:39:00.847173+0000","last_active":"2024-09-21T10:40:20.822763+0000","last_peered":"2024-09-21T10:40:20.822763+0000","last_clean":"2024-09-21T10:40:20.822763+0000","last_became_active":"2024-09-21T10:39:00.846851+0000","last_became_peered":"2024-09-21T10:39:00.846851+0000","last_unstale":"2024-09-21T10:40:20.822763+0000","last_undegraded":"2024-09-21T10:40:20.822763+0000","last_fullsized":"2024-09-21T10:40:20.822763+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:47.133721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2205,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.178579+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:40:49.178579+0000","last_peered":"2024-09-21T10:40:49.178579+0000","last_clean":"2024-09-21T10:40:49.178579+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:40:49.178579+0000","last_undegraded":"2024-09-21T10:40:49.178579+0000","last_fullsized":"2024-09-21T10:40:49.178579+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2082,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.105741+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:40:36.105741+0000","last_peered":"2024-09-21T10:40:36.105741+0000","last_clean":"2024-09-21T10:40:36.105741+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:40:36.105741+0000","last_undegraded":"2024-09-21T10:40:36.105741+0000","last_fullsized":"2024-09-21T10:40:36.105741+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":508,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829790+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:40:20.829790+0000","last_peered":"2024-09-21T10:40:20.829790+0000","last_clean":"2024-09-21T10:40:20.829790+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:40:20.829790+0000","last_undegraded":"2024-09-21T10:40:20.829790+0000","last_fullsized":"2024-09-21T10:40:20.829790+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2205,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.101320+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:40:36.101320+0000","last_peered":"2024-09-21T10:40:36.101320+0000","last_clean":"2024-09-21T10:40:36.101320+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:40:36.101320+0000","last_undegraded":"2024-09-21T10:40:36.101320+0000","last_fullsized":"2024-09-21T10:40:36.101320+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1400,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824847+0000","last_change":"2024-09-21T10:37:41.922996+0000","last_active":"2024-09-21T10:40:20.824847+0000","last_peered":"2024-09-21T10:40:20.824847+0000","last_clean":"2024-09-21T10:40:20.824847+0000","last_became_active":"2024-09-21T10:37:41.922703+0000","last_became_peered":"2024-09-21T10:37:41.922703+0000","last_unstale":"2024-09-21T10:40:20.824847+0000","last_undegraded":"2024-09-21T10:40:20.824847+0000","last_fullsized":"2024-09-21T10:40:20.824847+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:23.614770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2193,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.178978+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:40:49.178978+0000","last_peered":"2024-09-21T10:40:49.178978+0000","last_clean":"2024-09-21T10:40:49.178978+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:40:49.178978+0000","last_undegraded":"2024-09-21T10:40:49.178978+0000","last_fullsized":"2024-09-21T10:40:49.178978+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":508,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830565+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:40:20.830565+0000","last_peered":"2024-09-21T10:40:20.830565+0000","last_clean":"2024-09-21T10:40:20.830565+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:40:20.830565+0000","last_undegraded":"2024-09-21T10:40:20.830565+0000","last_fullsized":"2024-09-21T10:40:20.830565+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2198,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.176307+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:40:49.176307+0000","last_peered":"2024-09-21T10:40:49.176307+0000","last_clean":"2024-09-21T10:40:49.176307+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:40:49.176307+0000","last_undegraded":"2024-09-21T10:40:49.176307+0000","last_fullsized":"2024-09-21T10:40:49.176307+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2193,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179149+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:40:49.179149+0000","last_peered":"2024-09-21T10:40:49.179149+0000","last_clean":"2024-09-21T10:40:49.179149+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:40:49.179149+0000","last_undegraded":"2024-09-21T10:40:49.179149+0000","last_fullsized":"2024-09-21T10:40:49.179149+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823541+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:40:20.823541+0000","last_peered":"2024-09-21T10:40:20.823541+0000","last_clean":"2024-09-21T10:40:20.823541+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:40:20.823541+0000","last_undegraded":"2024-09-21T10:40:20.823541+0000","last_fullsized":"2024-09-21T10:40:20.823541+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":143,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823097+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:40:20.823097+0000","last_peered":"2024-09-21T10:40:20.823097+0000","last_clean":"2024-09-21T10:40:20.823097+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:40:20.823097+0000","last_undegraded":"2024-09-21T10:40:20.823097+0000","last_fullsized":"2024-09-21T10:40:20.823097+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2190,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179067+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:40:49.179067+0000","last_peered":"2024-09-21T10:40:49.179067+0000","last_clean":"2024-09-21T10:40:49.179067+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:40:49.179067+0000","last_undegraded":"2024-09-21T10:40:49.179067+0000","last_fullsized":"2024-09-21T10:40:49.179067+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1308,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100627+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:40:36.100627+0000","last_peered":"2024-09-21T10:40:36.100627+0000","last_clean":"2024-09-21T10:40:36.100627+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:40:36.100627+0000","last_undegraded":"2024-09-21T10:40:36.100627+0000","last_fullsized":"2024-09-21T10:40:36.100627+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":137,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823246+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:40:20.823246+0000","last_peered":"2024-09-21T10:40:20.823246+0000","last_clean":"2024-09-21T10:40:20.823246+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:40:20.823246+0000","last_undegraded":"2024-09-21T10:40:20.823246+0000","last_fullsized":"2024-09-21T10:40:20.823246+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":511,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099455+0000","last_change":"2024-09-21T10:37:38.606346+0000","last_active":"2024-09-21T10:40:36.099455+0000","last_peered":"2024-09-21T10:40:36.099455+0000","last_clean":"2024-09-21T10:40:36.099455+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:40:36.099455+0000","last_undegraded":"2024-09-21T10:40:36.099455+0000","last_fullsized":"2024-09-21T10:40:36.099455+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:16.461221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2208,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179299+0000","last_change":"2024-09-21T10:37:41.924374+0000","last_active":"2024-09-21T10:40:49.179299+0000","last_peered":"2024-09-21T10:40:49.179299+0000","last_clean":"2024-09-21T10:40:49.179299+0000","last_became_active":"2024-09-21T10:37:41.922589+0000","last_became_peered":"2024-09-21T10:37:41.922589+0000","last_unstale":"2024-09-21T10:40:49.179299+0000","last_undegraded":"2024-09-21T10:40:49.179299+0000","last_fullsized":"2024-09-21T10:40:49.179299+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:51.987384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2191,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.175818+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:40:49.175818+0000","last_peered":"2024-09-21T10:40:49.175818+0000","last_clean":"2024-09-21T10:40:49.175818+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:40:49.175818+0000","last_undegraded":"2024-09-21T10:40:49.175818+0000","last_fullsized":"2024-09-21T10:40:49.175818+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2121,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.175438+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:40:49.175438+0000","last_peered":"2024-09-21T10:40:49.175438+0000","last_clean":"2024-09-21T10:40:49.175438+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:40:49.175438+0000","last_undegraded":"2024-09-21T10:40:49.175438+0000","last_fullsized":"2024-09-21T10:40:49.175438+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":376,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179911+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:40:49.179911+0000","last_peered":"2024-09-21T10:40:49.179911+0000","last_clean":"2024-09-21T10:40:49.179911+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:40:49.179911+0000","last_undegraded":"2024-09-21T10:40:49.179911+0000","last_fullsized":"2024-09-21T10:40:49.179911+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":44,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099509+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:40:36.099509+0000","last_peered":"2024-09-21T10:40:36.099509+0000","last_clean":"2024-09-21T10:40:36.099509+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:40:36.099509+0000","last_undegraded":"2024-09-21T10:40:36.099509+0000","last_fullsized":"2024-09-21T10:40:36.099509+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2220,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.110970+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:40:36.110970+0000","last_peered":"2024-09-21T10:40:36.110970+0000","last_clean":"2024-09-21T10:40:36.110970+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:40:36.110970+0000","last_undegraded":"2024-09-21T10:40:36.110970+0000","last_fullsized":"2024-09-21T10:40:36.110970+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1370,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824469+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:40:20.824469+0000","last_peered":"2024-09-21T10:40:20.824469+0000","last_clean":"2024-09-21T10:40:20.824469+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:40:20.824469+0000","last_undegraded":"2024-09-21T10:40:20.824469+0000","last_fullsized":"2024-09-21T10:40:20.824469+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1343,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829812+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:40:20.829812+0000","last_peered":"2024-09-21T10:40:20.829812+0000","last_clean":"2024-09-21T10:40:20.829812+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:40:20.829812+0000","last_undegraded":"2024-09-21T10:40:20.829812+0000","last_fullsized":"2024-09-21T10:40:20.829812+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2105,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106574+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:40:36.106574+0000","last_peered":"2024-09-21T10:40:36.106574+0000","last_clean":"2024-09-21T10:40:36.106574+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:40:36.106574+0000","last_undegraded":"2024-09-21T10:40:36.106574+0000","last_fullsized":"2024-09-21T10:40:36.106574+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":137,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823363+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:40:20.823363+0000","last_peered":"2024-09-21T10:40:20.823363+0000","last_clean":"2024-09-21T10:40:20.823363+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:40:20.823363+0000","last_undegraded":"2024-09-21T10:40:20.823363+0000","last_fullsized":"2024-09-21T10:40:20.823363+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1553,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824368+0000","last_change":"2024-09-21T10:37:41.922049+0000","last_active":"2024-09-21T10:40:20.824368+0000","last_peered":"2024-09-21T10:40:20.824368+0000","last_clean":"2024-09-21T10:40:20.824368+0000","last_became_active":"2024-09-21T10:37:41.921709+0000","last_became_peered":"2024-09-21T10:37:41.921709+0000","last_unstale":"2024-09-21T10:40:20.824368+0000","last_undegraded":"2024-09-21T10:40:20.824368+0000","last_fullsized":"2024-09-21T10:40:20.824368+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:12.446157+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823352+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:40:20.823352+0000","last_peered":"2024-09-21T10:40:20.823352+0000","last_clean":"2024-09-21T10:40:20.823352+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:40:20.823352+0000","last_undegraded":"2024-09-21T10:40:20.823352+0000","last_fullsized":"2024-09-21T10:40:20.823352+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2196,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179211+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:40:49.179211+0000","last_peered":"2024-09-21T10:40:49.179211+0000","last_clean":"2024-09-21T10:40:49.179211+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:40:49.179211+0000","last_undegraded":"2024-09-21T10:40:49.179211+0000","last_fullsized":"2024-09-21T10:40:49.179211+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2091,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106630+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:40:36.106630+0000","last_peered":"2024-09-21T10:40:36.106630+0000","last_clean":"2024-09-21T10:40:36.106630+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:40:36.106630+0000","last_undegraded":"2024-09-21T10:40:36.106630+0000","last_fullsized":"2024-09-21T10:40:36.106630+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":359,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179812+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:40:49.179812+0000","last_peered":"2024-09-21T10:40:49.179812+0000","last_clean":"2024-09-21T10:40:49.179812+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:40:49.179812+0000","last_undegraded":"2024-09-21T10:40:49.179812+0000","last_fullsized":"2024-09-21T10:40:49.179812+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1242,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829869+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:40:20.829869+0000","last_peered":"2024-09-21T10:40:20.829869+0000","last_clean":"2024-09-21T10:40:20.829869+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:40:20.829869+0000","last_undegraded":"2024-09-21T10:40:20.829869+0000","last_fullsized":"2024-09-21T10:40:20.829869+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1249,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.175568+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:40:49.175568+0000","last_peered":"2024-09-21T10:40:49.175568+0000","last_clean":"2024-09-21T10:40:49.175568+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:40:49.175568+0000","last_undegraded":"2024-09-21T10:40:49.175568+0000","last_fullsized":"2024-09-21T10:40:49.175568+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":137,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.822988+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:40:20.822988+0000","last_peered":"2024-09-21T10:40:20.822988+0000","last_clean":"2024-09-21T10:40:20.822988+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:40:20.822988+0000","last_undegraded":"2024-09-21T10:40:20.822988+0000","last_fullsized":"2024-09-21T10:40:20.822988+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1256,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.176039+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:40:49.176039+0000","last_peered":"2024-09-21T10:40:49.176039+0000","last_clean":"2024-09-21T10:40:49.176039+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:40:49.176039+0000","last_undegraded":"2024-09-21T10:40:49.176039+0000","last_fullsized":"2024-09-21T10:40:49.176039+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1349,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830839+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:40:20.830839+0000","last_peered":"2024-09-21T10:40:20.830839+0000","last_clean":"2024-09-21T10:40:20.830839+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:40:20.830839+0000","last_undegraded":"2024-09-21T10:40:20.830839+0000","last_fullsized":"2024-09-21T10:40:20.830839+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1504,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.827301+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:40:20.827301+0000","last_peered":"2024-09-21T10:40:20.827301+0000","last_clean":"2024-09-21T10:40:20.827301+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:40:20.827301+0000","last_undegraded":"2024-09-21T10:40:20.827301+0000","last_fullsized":"2024-09-21T10:40:20.827301+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1306,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.112601+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:40:36.112601+0000","last_peered":"2024-09-21T10:40:36.112601+0000","last_clean":"2024-09-21T10:40:36.112601+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:40:36.112601+0000","last_undegraded":"2024-09-21T10:40:36.112601+0000","last_fullsized":"2024-09-21T10:40:36.112601+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":319,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.106967+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:40:36.106967+0000","last_peered":"2024-09-21T10:40:36.106967+0000","last_clean":"2024-09-21T10:40:36.106967+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:40:36.106967+0000","last_undegraded":"2024-09-21T10:40:36.106967+0000","last_fullsized":"2024-09-21T10:40:36.106967+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":78,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.825814+0000","last_change":"2024-09-21T10:37:37.360857+0000","last_active":"2024-09-21T10:40:20.825814+0000","last_peered":"2024-09-21T10:40:20.825814+0000","last_clean":"2024-09-21T10:40:20.825814+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:40:20.825814+0000","last_undegraded":"2024-09-21T10:40:20.825814+0000","last_fullsized":"2024-09-21T10:40:20.825814+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.360779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.360779+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:47:25.310501+0000","scrub_duration":0.001559979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2219,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.100536+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:40:36.100536+0000","last_peered":"2024-09-21T10:40:36.100536+0000","last_clean":"2024-09-21T10:40:36.100536+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:40:36.100536+0000","last_undegraded":"2024-09-21T10:40:36.100536+0000","last_fullsized":"2024-09-21T10:40:36.100536+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823355+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:40:20.823355+0000","last_peered":"2024-09-21T10:40:20.823355+0000","last_clean":"2024-09-21T10:40:20.823355+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:40:20.823355+0000","last_undegraded":"2024-09-21T10:40:20.823355+0000","last_fullsized":"2024-09-21T10:40:20.823355+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":527,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830683+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:40:20.830683+0000","last_peered":"2024-09-21T10:40:20.830683+0000","last_clean":"2024-09-21T10:40:20.830683+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:40:20.830683+0000","last_undegraded":"2024-09-21T10:40:20.830683+0000","last_fullsized":"2024-09-21T10:40:20.830683+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1321,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830769+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:40:20.830769+0000","last_peered":"2024-09-21T10:40:20.830769+0000","last_clean":"2024-09-21T10:40:20.830769+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:40:20.830769+0000","last_undegraded":"2024-09-21T10:40:20.830769+0000","last_fullsized":"2024-09-21T10:40:20.830769+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1328,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823508+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:40:20.823508+0000","last_peered":"2024-09-21T10:40:20.823508+0000","last_clean":"2024-09-21T10:40:20.823508+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:40:20.823508+0000","last_undegraded":"2024-09-21T10:40:20.823508+0000","last_fullsized":"2024-09-21T10:40:20.823508+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":357,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.109125+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:40:36.109125+0000","last_peered":"2024-09-21T10:40:36.109125+0000","last_clean":"2024-09-21T10:40:36.109125+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:40:36.109125+0000","last_undegraded":"2024-09-21T10:40:36.109125+0000","last_fullsized":"2024-09-21T10:40:36.109125+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1292,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179701+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:40:49.179701+0000","last_peered":"2024-09-21T10:40:49.179701+0000","last_clean":"2024-09-21T10:40:49.179701+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:40:49.179701+0000","last_undegraded":"2024-09-21T10:40:49.179701+0000","last_fullsized":"2024-09-21T10:40:49.179701+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1388,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.826075+0000","last_change":"2024-09-21T10:38:34.901523+0000","last_active":"2024-09-21T10:40:20.826075+0000","last_peered":"2024-09-21T10:40:20.826075+0000","last_clean":"2024-09-21T10:40:20.826075+0000","last_became_active":"2024-09-21T10:38:34.887435+0000","last_became_peered":"2024-09-21T10:38:34.887435+0000","last_unstale":"2024-09-21T10:40:20.826075+0000","last_undegraded":"2024-09-21T10:40:20.826075+0000","last_fullsized":"2024-09-21T10:40:20.826075+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:27.993702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":130,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.824582+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:40:20.824582+0000","last_peered":"2024-09-21T10:40:20.824582+0000","last_clean":"2024-09-21T10:40:20.824582+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:40:20.824582+0000","last_undegraded":"2024-09-21T10:40:20.824582+0000","last_fullsized":"2024-09-21T10:40:20.824582+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":133,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823018+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:40:20.823018+0000","last_peered":"2024-09-21T10:40:20.823018+0000","last_clean":"2024-09-21T10:40:20.823018+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:40:20.823018+0000","last_undegraded":"2024-09-21T10:40:20.823018+0000","last_fullsized":"2024-09-21T10:40:20.823018+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1401,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.825892+0000","last_change":"2024-09-21T10:38:34.891682+0000","last_active":"2024-09-21T10:40:20.825892+0000","last_peered":"2024-09-21T10:40:20.825892+0000","last_clean":"2024-09-21T10:40:20.825892+0000","last_became_active":"2024-09-21T10:38:34.885292+0000","last_became_peered":"2024-09-21T10:38:34.885292+0000","last_unstale":"2024-09-21T10:40:20.825892+0000","last_undegraded":"2024-09-21T10:40:20.825892+0000","last_fullsized":"2024-09-21T10:40:20.825892+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:30.111501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":120,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823048+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:40:20.823048+0000","last_peered":"2024-09-21T10:40:20.823048+0000","last_clean":"2024-09-21T10:40:20.823048+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:40:20.823048+0000","last_undegraded":"2024-09-21T10:40:20.823048+0000","last_fullsized":"2024-09-21T10:40:20.823048+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1418,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.830539+0000","last_change":"2024-09-21T10:39:00.852998+0000","last_active":"2024-09-21T10:40:20.830539+0000","last_peered":"2024-09-21T10:40:20.830539+0000","last_clean":"2024-09-21T10:40:20.830539+0000","last_became_active":"2024-09-21T10:39:00.846811+0000","last_became_peered":"2024-09-21T10:39:00.846811+0000","last_unstale":"2024-09-21T10:40:20.830539+0000","last_undegraded":"2024-09-21T10:40:20.830539+0000","last_fullsized":"2024-09-21T10:40:20.830539+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:14.374741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2081,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.105406+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:40:36.105406+0000","last_peered":"2024-09-21T10:40:36.105406+0000","last_clean":"2024-09-21T10:40:36.105406+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:40:36.105406+0000","last_undegraded":"2024-09-21T10:40:36.105406+0000","last_fullsized":"2024-09-21T10:40:36.105406+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":366,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.179770+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:40:49.179770+0000","last_peered":"2024-09-21T10:40:49.179770+0000","last_clean":"2024-09-21T10:40:49.179770+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:40:49.179770+0000","last_undegraded":"2024-09-21T10:40:49.179770+0000","last_fullsized":"2024-09-21T10:40:49.179770+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1237,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829616+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:40:20.829616+0000","last_peered":"2024-09-21T10:40:20.829616+0000","last_clean":"2024-09-21T10:40:20.829616+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:40:20.829616+0000","last_undegraded":"2024-09-21T10:40:20.829616+0000","last_fullsized":"2024-09-21T10:40:20.829616+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2200,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099765+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:40:36.099765+0000","last_peered":"2024-09-21T10:40:36.099765+0000","last_clean":"2024-09-21T10:40:36.099765+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:40:36.099765+0000","last_undegraded":"2024-09-21T10:40:36.099765+0000","last_fullsized":"2024-09-21T10:40:36.099765+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":120,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.822869+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:40:20.822869+0000","last_peered":"2024-09-21T10:40:20.822869+0000","last_clean":"2024-09-21T10:40:20.822869+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:40:20.822869+0000","last_undegraded":"2024-09-21T10:40:20.822869+0000","last_fullsized":"2024-09-21T10:40:20.822869+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2199,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.099563+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:40:36.099563+0000","last_peered":"2024-09-21T10:40:36.099563+0000","last_clean":"2024-09-21T10:40:36.099563+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:40:36.099563+0000","last_undegraded":"2024-09-21T10:40:36.099563+0000","last_fullsized":"2024-09-21T10:40:36.099563+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":143,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.823579+0000","last_change":"2024-09-21T10:38:38.962357+0000","last_active":"2024-09-21T10:40:20.823579+0000","last_peered":"2024-09-21T10:40:20.823579+0000","last_clean":"2024-09-21T10:40:20.823579+0000","last_became_active":"2024-09-21T10:38:38.962017+0000","last_became_peered":"2024-09-21T10:38:38.962017+0000","last_unstale":"2024-09-21T10:40:20.823579+0000","last_undegraded":"2024-09-21T10:40:20.823579+0000","last_fullsized":"2024-09-21T10:40:20.823579+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.355706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":81,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.110659+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:40:36.110659+0000","last_peered":"2024-09-21T10:40:36.110659+0000","last_clean":"2024-09-21T10:40:36.110659+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:40:36.110659+0000","last_undegraded":"2024-09-21T10:40:36.110659+0000","last_fullsized":"2024-09-21T10:40:36.110659+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1389,"reported_epoch":1382,"state":"active+clean","last_fresh":"2024-09-21T10:40:20.829388+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:40:20.829388+0000","last_peered":"2024-09-21T10:40:20.829388+0000","last_clean":"2024-09-21T10:40:20.829388+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:40:20.829388+0000","last_undegraded":"2024-09-21T10:40:20.829388+0000","last_fullsized":"2024-09-21T10:40:20.829388+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2089,"reported_epoch":1383,"state":"active+clean","last_fresh":"2024-09-21T10:40:36.112327+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:40:36.112327+0000","last_peered":"2024-09-21T10:40:36.112327+0000","last_clean":"2024-09-21T10:40:36.112327+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:40:36.112327+0000","last_undegraded":"2024-09-21T10:40:36.112327+0000","last_fullsized":"2024-09-21T10:40:36.112327+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":48,"num_pg":16,"stat_sum":{"num_bytes":3331260440,"num_objects":50832,"num_object_clones":0,"num_object_copies":152496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1131,"num_objects_misplaced":4614,"num_objects_unfound":0,"num_objects_dirty":50832,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195584,"num_write_kb":7885249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3362,"num_bytes_recovered":220332032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10269634560,"data_stored":10269622344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103309,"ondisk_log_size":103309,"up":48,"acting":47,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":481141,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6268,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398055,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2759140,"kb_used_data":1863516,"kb_used_omap":1,"kb_used_meta":895614,"kb_avail":90986012,"statfs":{"total":95995035648,"available":93169676288,"internally_reserved":0,"allocated":1908240384,"data_stored":1902236065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917109080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049714,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3036964,"kb_used_data":2105500,"kb_used_omap":242,"kb_used_meta":931213,"kb_avail":90708188,"statfs":{"total":95995035648,"available":92885184512,"internally_reserved":0,"allocated":2156032000,"data_stored":2150027686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":953562335},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790513,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1898392,"kb_used_data":1005520,"kb_used_omap":1,"kb_used_meta":892862,"kb_avail":91846760,"statfs":{"total":95995035648,"available":94051082240,"internally_reserved":0,"allocated":1029652480,"data_stored":1023656758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914291032},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395619,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644673,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2808404,"kb_used_data":2061580,"kb_used_omap":67,"kb_used_meta":746748,"kb_avail":90936748,"statfs":{"total":95995035648,"available":93119229952,"internally_reserved":0,"allocated":2111057920,"data_stored":2105065769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":764670813},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527691,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1970464,"kb_used_data":1259416,"kb_used_omap":75,"kb_used_meta":710964,"kb_avail":91774688,"statfs":{"total":95995035648,"available":93977280512,"internally_reserved":0,"allocated":1289641984,"data_stored":1283642803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":728027290},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274844,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2155556,"kb_used_data":1410140,"kb_used_omap":51,"kb_used_meta":745036,"kb_avail":91589596,"statfs":{"total":95995035648,"available":93787746304,"internally_reserved":0,"allocated":1443983360,"data_stored":1437983372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":762917205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354082,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1107220,"kb_used_data":541068,"kb_used_omap":66,"kb_used_meta":566013,"kb_avail":92637932,"statfs":{"total":95995035648,"available":94861242368,"internally_reserved":0,"allocated":554053632,"data_stored":548061521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":579597636},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":533266432,"data_stored":533266432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1416298496,"data_stored":1416298496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1251213312,"data_stored":1251213312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2080182272,"data_stored":2080178200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1005387776,"data_stored":1005387776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2112098304,"data_stored":2112094232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1871187968,"data_stored":1871183896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:40:54.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:40:54.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:54.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:54.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:54.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:54.423 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:54.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:54.451 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:54 smithi119 ceph-mon[105288]: from='client.36864 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:54.452 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:54 smithi119 ceph-mon[101432]: from='client.36864 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:54.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:54 smithi145 ceph-mon[89359]: from='client.36864 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:40:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:54.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:54.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:54.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:54.743 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:40:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:40:54] "GET /metrics HTTP/1.1" 200 36866 "" "Prometheus/2.51.0" 2024-09-21T10:40:54.760 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:54.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:54.826 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:54.834 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:40:54.834 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:40:54.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:54.877 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:40:54.877 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:40:54.910 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:40:54.910 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:40:54.941 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:54.946 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:40:54.947 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:40:54.966 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:54.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:54.982 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:40:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:40:55.015 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:40:55.015 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:40:55.045 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:40:55.045 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:40:55.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:55.077 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:40:55.077 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:40:55.094 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:55.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:55.240 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:55.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:55 smithi119 ceph-mon[101432]: pgmap v2314: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 1183/143388 objects degraded (0.825%); 4340/143388 objects misplaced (3.027%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:55 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1183/143388 objects degraded (0.825%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3014541612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:55 smithi119 ceph-mon[105288]: pgmap v2314: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 1183/143388 objects degraded (0.825%); 4340/143388 objects misplaced (3.027%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:55 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1183/143388 objects degraded (0.825%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3014541612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:55.380 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:55.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:55.518 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:55 smithi145 ceph-mon[89359]: pgmap v2314: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.9 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 886 op/s; 1183/143388 objects degraded (0.825%); 4340/143388 objects misplaced (3.027%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:55 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1183/143388 objects degraded (0.825%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:40:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3014541612' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:40:55.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:55.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:55.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:55.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:55.885 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:55.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:55.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:56.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:56.042 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:56.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:56.183 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:56.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:56.285 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:56.309 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:56.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:56.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:56.444 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:56.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:56.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:56.679 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:56.719 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:56.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:56.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:56.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:56.986 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:56.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:57.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:40:57.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:57.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:40:57.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:57.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:57.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:57.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:57.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:57.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:40:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:57 smithi145 ceph-mon[89359]: pgmap v2315: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 878 op/s; 1183/139254 objects degraded (0.850%); 4044/139254 objects misplaced (2.904%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:57.546 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:40:57.555 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:57 smithi119 ceph-mon[105288]: pgmap v2315: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 878 op/s; 1183/139254 objects degraded (0.850%); 4044/139254 objects misplaced (2.904%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:57.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:57 smithi119 ceph-mon[101432]: pgmap v2315: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 878 op/s; 1183/139254 objects degraded (0.850%); 4044/139254 objects misplaced (2.904%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:57.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:57.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.684 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:57.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:57.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:40:57.816 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:40:57.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:57.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:57.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:58.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:40:58.096 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:40:58.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:58.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:58.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.359 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:58.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:58.460 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:40:58.494 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:40:58.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:58.641 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:58.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:58.786 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:58.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:40:58.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:58.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:40:59.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:59.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:59.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:59.159 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:59.187 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:59.289 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:59.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:59.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:59.448 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:40:59.478 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:59.506 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:40:59 smithi119 ceph-mon[105288]: pgmap v2316: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 1183/138384 objects degraded (0.855%); 4044/138384 objects misplaced (2.922%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:59.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:40:59 smithi119 ceph-mon[101432]: pgmap v2316: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 1183/138384 objects degraded (0.855%); 4044/138384 objects misplaced (2.922%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:40:59 smithi145 ceph-mon[89359]: pgmap v2316: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 882 op/s; 1183/138384 objects degraded (0.855%); 4044/138384 objects misplaced (2.922%); 3.0 MiB/s, 47 objects/s recovering 2024-09-21T10:40:59.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:59.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:59.719 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:40:59.750 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:40:59.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:40:59.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:40:59.888 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:40:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:40:59.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:40:59.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:00.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:00.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:00.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:00.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:00.260 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:00.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:00.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:00.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:00.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:00.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:00.477 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:00 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1183/138384 objects degraded (0.855%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:00.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:00 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1183/138384 objects degraded (0.855%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:00.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:00 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1183/138384 objects degraded (0.855%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:41:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:41:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:41:00.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:00.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:00.687 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:00.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:00.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:00.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:00.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:00.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:01.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:01.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:01.203 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:01.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:01.240 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:41:01.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:01.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:01 smithi119 ceph-mon[101432]: pgmap v2317: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 1202/126033 objects degraded (0.954%); 3759/126033 objects misplaced (2.983%); 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:41:01.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3764764509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:01.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:01 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:01.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:01 smithi119 ceph-mon[105288]: pgmap v2317: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 1202/126033 objects degraded (0.954%); 3759/126033 objects misplaced (2.983%); 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:41:01.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3764764509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:01.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:01 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:01.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:01.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:01.500 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:01.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:41:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:01 smithi145 ceph-mon[89359]: pgmap v2317: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.6 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 960 op/s; 1202/126033 objects degraded (0.954%); 3759/126033 objects misplaced (2.983%); 2.9 MiB/s, 45 objects/s recovering 2024-09-21T10:41:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3764764509' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:01 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:41:01.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:01.629 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:41:01.629 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:41:01.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:01.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:01.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:01.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:01.893 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:01.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:01.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:02.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:02.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:02.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:02.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:02.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:02.407 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:02.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:02.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:41:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:02 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:02 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:41:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:02 smithi145 ceph-mon[89359]: osdmap e1385: 8 total, 8 up, 7 in 2024-09-21T10:41:02.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:02.551 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:02 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:02.551 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:02 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:41:02.552 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:02 smithi119 ceph-mon[105288]: osdmap e1385: 8 total, 8 up, 7 in 2024-09-21T10:41:02.552 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:02 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:02.552 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:02 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:41:02.552 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:02 smithi119 ceph-mon[101432]: osdmap e1385: 8 total, 8 up, 7 in 2024-09-21T10:41:02.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:02.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:02.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:02.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:02.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:02.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:02.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:03.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.003 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:03.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:03.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:03.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:03.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:03.282 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:03.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:03.414 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:03.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:03 smithi145 ceph-mon[89359]: pgmap v2319: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 1202/125085 objects degraded (0.961%); 3443/125085 objects misplaced (2.753%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:41:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:03.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:03 smithi119 ceph-mon[105288]: pgmap v2319: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 1202/125085 objects degraded (0.961%); 3443/125085 objects misplaced (2.753%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:41:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:03 smithi119 ceph-mon[101432]: pgmap v2319: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 915 op/s; 1202/125085 objects degraded (0.961%); 3443/125085 objects misplaced (2.753%); 3.0 MiB/s, 48 objects/s recovering 2024-09-21T10:41:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:03.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:03.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:03.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:03.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:03.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:03.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:03.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:04.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:04.096 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:04.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:04.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:04.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:04.334 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:04.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:04.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:04.513 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:04.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:04.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:04.641 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:04.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.779 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:04.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:41:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:41:04] "GET /metrics HTTP/1.1" 200 36858 "" "Prometheus/2.51.0" 2024-09-21T10:41:04.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:04.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:04.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:05.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:05.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:05.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:05.170 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:05.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:05.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:05.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:05.444 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:05 smithi145 ceph-mon[89359]: pgmap v2320: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 1202/116805 objects degraded (1.029%); 3443/116805 objects misplaced (2.948%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:41:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:05 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1202/116805 objects degraded (1.029%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:05.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:05.579 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:05 smithi119 ceph-mon[105288]: pgmap v2320: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 1202/116805 objects degraded (1.029%); 3443/116805 objects misplaced (2.948%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:41:05.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:05 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1202/116805 objects degraded (1.029%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:05 smithi119 ceph-mon[101432]: pgmap v2320: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 885 op/s; 1202/116805 objects degraded (1.029%); 3443/116805 objects misplaced (2.948%); 1.7 MiB/s, 26 objects/s recovering 2024-09-21T10:41:05.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:05 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1202/116805 objects degraded (1.029%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:05.681 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:05.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:05.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:05.834 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:05.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:05.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:05.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:06.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:06.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:06.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:06.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:06.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:06.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:06.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:06.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:06.642 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:06.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:06.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:06.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:07.016 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:07.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:07.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:07.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:07.275 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:07.303 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:07.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:07.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:07 smithi145 ceph-mon[89359]: pgmap v2321: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 965 op/s; 1234/110280 objects degraded (1.119%); 2819/110280 objects misplaced (2.556%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T10:41:07.552 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:07.562 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:07 smithi119 ceph-mon[101432]: pgmap v2321: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 965 op/s; 1234/110280 objects degraded (1.119%); 2819/110280 objects misplaced (2.556%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T10:41:07.562 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:07 smithi119 ceph-mon[105288]: pgmap v2321: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 965 op/s; 1234/110280 objects degraded (1.119%); 2819/110280 objects misplaced (2.556%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T10:41:07.571 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:07.577 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:07.637 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:07.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:07.705 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:07.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:07.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:07.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:07.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:07.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:08.073 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:08.092 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:41:08.092 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1234/110280 objects degraded (1.119%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":2195,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1385,"num_osds":8,"num_up_osds":8,"osd_up_since":1726914438,"num_in_osds":7,"osd_in_since":1726914980,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":107},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+remapped+backfill_wait","count":1}],"num_pgs":109,"num_pools":5,"num_objects":36760,"data_bytes":2407820602,"bytes_used":13465849856,"bytes_avail":658499399680,"bytes_total":671965249536,"degraded_objects":1234,"degraded_total":110280,"degraded_ratio":0.011189698948132027,"misplaced_objects":2819,"misplaced_total":110280,"misplaced_ratio":0.025562205295611172,"recovering_objects_per_sec":53,"recovering_bytes_per_sec":3530098,"recovering_keys_per_sec":0,"num_objects_recovered":539,"num_bytes_recovered":35323904,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":965},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi119.oikglq","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.119:8443/","prometheus":"http://172.21.15.119:9283/"}},"servicemap":{"epoch":33,"modified":"2024-09-21T10:39:53.783483+0000","services":{}},"progress_events":{"57a74cac-3140-4ecf-9dfd-e4dca5b6c6c4":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (6m)\n [============................] (remaining: 7m)","progress":0.44999998807907104,"add_to_ceph_s":true},"c53079b5-2e29-436f-bd84-f7d6d8532e32":{"message":"Global Recovery Event (2m)\n [===========================.] (remaining: 2s)","progress":0.98165136575698853,"add_to_ceph_s":true}}} 2024-09-21T10:41:08.100 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:08.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:08.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:08.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:08.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:08.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:08.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:08.505 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:08 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/990373859' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:41:08.505 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:08 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/990373859' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:41:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:08 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/990373859' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:41:08.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:08.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:08.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:08.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:08.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:08.923 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:08.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:08.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:41:09.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:09.052 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:09.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:09.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:09.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:09.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:09.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:09.480 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:09.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:09 smithi145 ceph-mon[89359]: pgmap v2322: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 967 op/s; 1234/109311 objects degraded (1.129%); 2819/109311 objects misplaced (2.579%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T10:41:09.582 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:09.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:09 smithi119 ceph-mon[105288]: pgmap v2322: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 967 op/s; 1234/109311 objects degraded (1.129%); 2819/109311 objects misplaced (2.579%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T10:41:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:09 smithi119 ceph-mon[101432]: pgmap v2322: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 967 op/s; 1234/109311 objects degraded (1.129%); 2819/109311 objects misplaced (2.579%); 3.4 MiB/s, 53 objects/s recovering 2024-09-21T10:41:09.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:09.736 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:09.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:09.871 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:09.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:09.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:10.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:10.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:10.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:10.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:10.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:10.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:10.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:10.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:10.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:41:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:41:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:41:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:10 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1234/109311 objects degraded (1.129%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:10.564 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T10:41:10.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:41:10.564 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:10 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1234/109311 objects degraded (1.129%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:10.565 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:10 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1234/109311 objects degraded (1.129%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:10.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.584 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:10.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:10.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:10.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:10.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:10.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:10.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:10.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:10.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:11.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:11.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:11.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:11.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:11.189 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:11.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:11.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:11.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:11.406 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:11.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:11.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:11.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:11.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:11 smithi119 ceph-mon[101432]: pgmap v2323: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 986 op/s; 1236/96405 objects degraded (1.282%); 2489/96405 objects misplaced (2.582%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:41:11.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:11 smithi119 ceph-mon[105288]: pgmap v2323: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 986 op/s; 1236/96405 objects degraded (1.282%); 2489/96405 objects misplaced (2.582%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:41:11.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:11.701 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:11 smithi145 ceph-mon[89359]: pgmap v2323: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 986 op/s; 1236/96405 objects degraded (1.282%); 2489/96405 objects misplaced (2.582%); 3.8 MiB/s, 60 objects/s recovering 2024-09-21T10:41:11.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:11.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:11.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:11.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:11.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:12.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:12.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:12.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:12.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:12.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:12.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:12.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.537 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:12.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:12.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:12.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:12.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:12.808 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:12.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:12.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:12.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:12.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:13.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:13.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:13.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:13.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:13.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:13.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:13.471 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:13.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:13.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:13.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:13 smithi119 ceph-mon[101432]: pgmap v2324: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 1236/95523 objects degraded (1.294%); 2195/95523 objects misplaced (2.298%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:41:13.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:13 smithi119 ceph-mon[105288]: pgmap v2324: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 1236/95523 objects degraded (1.294%); 2195/95523 objects misplaced (2.298%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:41:13.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:13.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:13.753 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:13 smithi145 ceph-mon[89359]: pgmap v2324: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 928 op/s; 1236/95523 objects degraded (1.294%); 2195/95523 objects misplaced (2.298%); 3.5 MiB/s, 56 objects/s recovering 2024-09-21T10:41:13.855 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:13.892 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:13.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:13.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:13.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:14.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:14.040 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:41:14.040 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:41:14.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:14.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:14.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:14.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:14.280 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:14.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:14.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:14.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:14.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:14.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:14.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:14.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:14.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:41:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:41:14] "GET /metrics HTTP/1.1" 200 36858 "" "Prometheus/2.51.0" 2024-09-21T10:41:14.920 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:14.941 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:14.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:15.042 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:15.074 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:15.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:15.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:15.318 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:15.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:15.453 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:15.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:15.584 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:15.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:15 smithi119 ceph-mon[101432]: pgmap v2325: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 1236/87063 objects degraded (1.420%); 2195/87063 objects misplaced (2.521%); 3.1 MiB/s, 50 objects/s recovering 2024-09-21T10:41:15.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:15 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1236/87063 objects degraded (1.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:15 smithi119 ceph-mon[105288]: pgmap v2325: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 1236/87063 objects degraded (1.420%); 2195/87063 objects misplaced (2.521%); 3.1 MiB/s, 50 objects/s recovering 2024-09-21T10:41:15.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:15 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1236/87063 objects degraded (1.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:15.718 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:15.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:15 smithi145 ceph-mon[89359]: pgmap v2325: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 1236/87063 objects degraded (1.420%); 2195/87063 objects misplaced (2.521%); 3.1 MiB/s, 50 objects/s recovering 2024-09-21T10:41:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:15 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1236/87063 objects degraded (1.420%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:15.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:15.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:15.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:15.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:16.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:16.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:16.148 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:16.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:16.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:16.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:16.416 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:16.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:16.518 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:16.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:16.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:16.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:16.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:16.894 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:16.901 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:41:16.901 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:16.933 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:41:16.933 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:16.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:16.961 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:41:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:16.964 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:16.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:17.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:17.104 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:17.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:17.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:17.246 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:17.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:17.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:17.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:17.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:17.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:17 smithi119 ceph-mon[105288]: pgmap v2326: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 1122/80577 objects degraded (1.392%); 1633/80577 objects misplaced (2.027%); 4.4 MiB/s, 80 objects/s recovering 2024-09-21T10:41:17.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1125173126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:17.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:17 smithi119 ceph-mon[101432]: pgmap v2326: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 1122/80577 objects degraded (1.392%); 1633/80577 objects misplaced (2.027%); 4.4 MiB/s, 80 objects/s recovering 2024-09-21T10:41:17.620 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1125173126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:17.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:17.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:41:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:17.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:17 smithi145 ceph-mon[89359]: pgmap v2326: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 1122/80577 objects degraded (1.392%); 1633/80577 objects misplaced (2.027%); 4.4 MiB/s, 80 objects/s recovering 2024-09-21T10:41:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1125173126' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.816 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:17.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:17.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:17.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:17.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:18.073 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.100 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:18.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:18.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:18.234 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:18.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:18.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:18.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:18.467 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:18.500 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:18.592 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:18.593 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:18.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:18.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:18.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:18.762 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:18.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:18.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:18.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:18.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:19.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:19.040 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:19.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:19.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:19.271 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:19.281 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:41:19.282 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:41:19.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:19.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:19.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:19.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:19.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:19 smithi119 ceph-mon[101432]: pgmap v2327: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 1122/79587 objects degraded (1.410%); 1633/79587 objects misplaced (2.052%); 3.0 MiB/s, 57 objects/s recovering 2024-09-21T10:41:19.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:19 smithi119 ceph-mon[105288]: pgmap v2327: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 1122/79587 objects degraded (1.410%); 1633/79587 objects misplaced (2.052%); 3.0 MiB/s, 57 objects/s recovering 2024-09-21T10:41:19.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:19.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:19.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:19.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:19 smithi145 ceph-mon[89359]: pgmap v2327: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 851 op/s; 1122/79587 objects degraded (1.410%); 1633/79587 objects misplaced (2.052%); 3.0 MiB/s, 57 objects/s recovering 2024-09-21T10:41:19.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:19.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:19.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:19.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:20.002 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:20.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.140 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:20.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:20.214 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2327,"stamp":"2024-09-21T10:41:17.845667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1737321786,"num_objects":26529,"num_object_clones":0,"num_object_copies":79587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":1633,"num_objects_unfound":0,"num_objects_dirty":26529,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219976,"num_write_kb":7885280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4867,"num_bytes_recovered":308746927,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105796,"ondisk_log_size":105796,"up":327,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":11227100,"kb_used_data":5706020,"kb_used_omap":507,"kb_used_meta":5520388,"kb_avail":644988964,"statfs":{"total":671965249536,"available":660468699136,"internally_reserved":0,"allocated":5842964480,"data_stored":5800881560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":5652877855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-670498816,"num_objects":-10231,"num_object_clones":0,"num_object_copies":-30693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-112,"num_objects_misplaced":-1186,"num_objects_unfound":0,"num_objects_dirty":-10231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":688,"num_bytes_recovered":37617664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011870"},"pg_stats":[{"pgid":"48.f","version":"1385'13713","reported_seq":13787,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.713836+0000","last_change":"2024-09-21T10:37:50.818738+0000","last_active":"2024-09-21T10:41:13.713836+0000","last_peered":"2024-09-21T10:41:13.713836+0000","last_clean":"2024-09-21T10:41:13.713836+0000","last_became_active":"2024-09-21T10:37:50.818409+0000","last_became_peered":"2024-09-21T10:37:50.818409+0000","last_unstale":"2024-09-21T10:41:13.713836+0000","last_undegraded":"2024-09-21T10:41:13.713836+0000","last_fullsized":"2024-09-21T10:41:13.713836+0000","mapping_epoch":1354,"log_start":"1372'8100","ondisk_log_start":"1372'8100","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5613,"log_dups_size":0,"ondisk_log_size":5613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:24:13.373038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106102784,"num_objects":1619,"num_object_clones":0,"num_object_copies":4857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13713,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.e","version":"1385'13852","reported_seq":13925,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.478859+0000","last_change":"2024-09-21T10:37:50.816783+0000","last_active":"2024-09-21T10:41:14.478859+0000","last_peered":"2024-09-21T10:41:14.478859+0000","last_clean":"2024-09-21T10:41:14.478859+0000","last_became_active":"2024-09-21T10:37:50.816482+0000","last_became_peered":"2024-09-21T10:37:50.816482+0000","last_unstale":"2024-09-21T10:41:14.478859+0000","last_undegraded":"2024-09-21T10:41:14.478859+0000","last_fullsized":"2024-09-21T10:41:14.478859+0000","mapping_epoch":1354,"log_start":"1368'6100","ondisk_log_start":"1368'6100","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7752,"log_dups_size":0,"ondisk_log_size":7752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:57.011771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112328704,"num_objects":1714,"num_object_clones":0,"num_object_copies":5142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13852,"num_write_kb":498112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.d","version":"1385'13751","reported_seq":13791,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.013628+0000","last_change":"2024-09-21T10:37:50.816596+0000","last_active":"2024-09-21T10:41:13.013628+0000","last_peered":"2024-09-21T10:41:13.013628+0000","last_clean":"2024-09-21T10:41:13.013628+0000","last_became_active":"2024-09-21T10:37:50.816060+0000","last_became_peered":"2024-09-21T10:37:50.816060+0000","last_unstale":"2024-09-21T10:41:13.013628+0000","last_undegraded":"2024-09-21T10:41:13.013628+0000","last_fullsized":"2024-09-21T10:41:13.013628+0000","mapping_epoch":1354,"log_start":"1372'8074","ondisk_log_start":"1372'8074","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5677,"log_dups_size":0,"ondisk_log_size":5677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:09.093088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113770520,"num_objects":1737,"num_object_clones":0,"num_object_copies":5211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13751,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.c","version":"1385'13935","reported_seq":14009,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.731252+0000","last_change":"2024-09-21T10:37:50.814694+0000","last_active":"2024-09-21T10:41:13.731252+0000","last_peered":"2024-09-21T10:41:13.731252+0000","last_clean":"2024-09-21T10:41:13.731252+0000","last_became_active":"2024-09-21T10:37:50.814424+0000","last_became_peered":"2024-09-21T10:37:50.814424+0000","last_unstale":"2024-09-21T10:41:13.731252+0000","last_undegraded":"2024-09-21T10:41:13.731252+0000","last_fullsized":"2024-09-21T10:41:13.731252+0000","mapping_epoch":1354,"log_start":"1373'8300","ondisk_log_start":"1373'8300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5635,"log_dups_size":0,"ondisk_log_size":5635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:22:22.559518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107937792,"num_objects":1647,"num_object_clones":0,"num_object_copies":4941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13935,"num_write_kb":498624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.b","version":"1385'13648","reported_seq":13687,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.826660+0000","last_change":"2024-09-21T10:37:50.809808+0000","last_active":"2024-09-21T10:41:13.826660+0000","last_peered":"2024-09-21T10:41:13.826660+0000","last_clean":"2024-09-21T10:41:13.826660+0000","last_became_active":"2024-09-21T10:37:50.809451+0000","last_became_peered":"2024-09-21T10:37:50.809451+0000","last_unstale":"2024-09-21T10:41:13.826660+0000","last_undegraded":"2024-09-21T10:41:13.826660+0000","last_fullsized":"2024-09-21T10:41:13.826660+0000","mapping_epoch":1354,"log_start":"1370'7300","ondisk_log_start":"1370'7300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6348,"log_dups_size":0,"ondisk_log_size":6348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:46.455275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107479040,"num_objects":1640,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13648,"num_write_kb":489216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.a","version":"1385'14093","reported_seq":14167,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.468767+0000","last_change":"2024-09-21T10:37:50.806280+0000","last_active":"2024-09-21T10:41:14.468767+0000","last_peered":"2024-09-21T10:41:14.468767+0000","last_clean":"2024-09-21T10:41:14.468767+0000","last_became_active":"2024-09-21T10:37:50.805997+0000","last_became_peered":"2024-09-21T10:37:50.805997+0000","last_unstale":"2024-09-21T10:41:14.468767+0000","last_undegraded":"2024-09-21T10:41:14.468767+0000","last_fullsized":"2024-09-21T10:41:14.468767+0000","mapping_epoch":1354,"log_start":"1368'6400","ondisk_log_start":"1368'6400","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7693,"log_dups_size":0,"ondisk_log_size":7693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.061122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106496000,"num_objects":1625,"num_object_clones":0,"num_object_copies":4875,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1625,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14093,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.9","version":"1385'13402","reported_seq":13476,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.773183+0000","last_change":"2024-09-21T10:37:50.814434+0000","last_active":"2024-09-21T10:41:13.773183+0000","last_peered":"2024-09-21T10:41:13.773183+0000","last_clean":"2024-09-21T10:41:13.773183+0000","last_became_active":"2024-09-21T10:37:50.814150+0000","last_became_peered":"2024-09-21T10:37:50.814150+0000","last_unstale":"2024-09-21T10:41:13.773183+0000","last_undegraded":"2024-09-21T10:41:13.773183+0000","last_fullsized":"2024-09-21T10:41:13.773183+0000","mapping_epoch":1354,"log_start":"1371'7800","ondisk_log_start":"1371'7800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5602,"log_dups_size":0,"ondisk_log_size":5602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:18.301417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13402,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.8","version":"1385'13584","reported_seq":13623,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.633837+0000","last_change":"2024-09-21T10:37:50.812704+0000","last_active":"2024-09-21T10:41:13.633837+0000","last_peered":"2024-09-21T10:41:13.633837+0000","last_clean":"2024-09-21T10:41:13.633837+0000","last_became_active":"2024-09-21T10:37:50.812368+0000","last_became_peered":"2024-09-21T10:37:50.812368+0000","last_unstale":"2024-09-21T10:41:13.633837+0000","last_undegraded":"2024-09-21T10:41:13.633837+0000","last_fullsized":"2024-09-21T10:41:13.633837+0000","mapping_epoch":1354,"log_start":"1365'5800","ondisk_log_start":"1365'5800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7784,"log_dups_size":0,"ondisk_log_size":7784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:22.704564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13584,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"48.7","version":"1385'14145","reported_seq":14223,"reported_epoch":1385,"state":"active+remapped+backfill_wait","last_fresh":"2024-09-21T10:41:15.637247+0000","last_change":"2024-09-21T10:40:12.260234+0000","last_active":"2024-09-21T10:41:15.637247+0000","last_peered":"2024-09-21T10:41:15.637247+0000","last_clean":"2024-09-21T10:40:09.890233+0000","last_became_active":"2024-09-21T10:40:12.242243+0000","last_became_peered":"2024-09-21T10:40:12.242243+0000","last_unstale":"2024-09-21T10:41:15.637247+0000","last_undegraded":"2024-09-21T10:41:15.637247+0000","last_fullsized":"2024-09-21T10:41:15.637247+0000","mapping_epoch":1379,"log_start":"1372'8400","ondisk_log_start":"1372'8400","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5745,"log_dups_size":0,"ondisk_log_size":5745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107020288,"num_objects":1633,"num_object_clones":0,"num_object_copies":4899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1633,"num_objects_unfound":0,"num_objects_dirty":1633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14145,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":1633}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"48.6","version":"1385'13724","reported_seq":13797,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.799402+0000","last_change":"2024-09-21T10:37:50.811591+0000","last_active":"2024-09-21T10:41:13.799402+0000","last_peered":"2024-09-21T10:41:13.799402+0000","last_clean":"2024-09-21T10:41:13.799402+0000","last_became_active":"2024-09-21T10:37:50.811260+0000","last_became_peered":"2024-09-21T10:37:50.811260+0000","last_unstale":"2024-09-21T10:41:13.799402+0000","last_undegraded":"2024-09-21T10:41:13.799402+0000","last_fullsized":"2024-09-21T10:41:13.799402+0000","mapping_epoch":1354,"log_start":"1372'8100","ondisk_log_start":"1372'8100","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:19.966736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107610112,"num_objects":1642,"num_object_clones":0,"num_object_copies":4926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13724,"num_write_kb":491712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.5","version":"1385'13586","reported_seq":13624,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:13.013516+0000","last_change":"2024-09-21T10:37:50.816067+0000","last_active":"2024-09-21T10:41:13.013516+0000","last_peered":"2024-09-21T10:41:13.013516+0000","last_clean":"2024-09-21T10:41:13.013516+0000","last_became_active":"2024-09-21T10:37:50.815783+0000","last_became_peered":"2024-09-21T10:37:50.815783+0000","last_unstale":"2024-09-21T10:41:13.013516+0000","last_undegraded":"2024-09-21T10:41:13.013516+0000","last_fullsized":"2024-09-21T10:41:13.013516+0000","mapping_epoch":1354,"log_start":"1371'7910","ondisk_log_start":"1371'7910","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5676,"log_dups_size":0,"ondisk_log_size":5676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:03.096851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114819072,"num_objects":1752,"num_object_clones":0,"num_object_copies":5256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13586,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.4","version":"1385'13632","reported_seq":13706,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.465044+0000","last_change":"2024-09-21T10:37:50.810595+0000","last_active":"2024-09-21T10:41:14.465044+0000","last_peered":"2024-09-21T10:41:14.465044+0000","last_clean":"2024-09-21T10:41:14.465044+0000","last_became_active":"2024-09-21T10:37:50.810178+0000","last_became_peered":"2024-09-21T10:37:50.810178+0000","last_unstale":"2024-09-21T10:41:14.465044+0000","last_undegraded":"2024-09-21T10:41:14.465044+0000","last_fullsized":"2024-09-21T10:41:14.465044+0000","mapping_epoch":1354,"log_start":"1365'5900","ondisk_log_start":"1365'5900","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7732,"log_dups_size":0,"ondisk_log_size":7732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:30.334590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109576192,"num_objects":1672,"num_object_clones":0,"num_object_copies":5016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13632,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.3","version":"1385'13796","reported_seq":13870,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:17.556434+0000","last_change":"2024-09-21T10:37:50.808064+0000","last_active":"2024-09-21T10:41:17.556434+0000","last_peered":"2024-09-21T10:41:17.556434+0000","last_clean":"2024-09-21T10:41:17.556434+0000","last_became_active":"2024-09-21T10:37:50.807674+0000","last_became_peered":"2024-09-21T10:37:50.807674+0000","last_unstale":"2024-09-21T10:41:17.556434+0000","last_undegraded":"2024-09-21T10:41:17.556434+0000","last_fullsized":"2024-09-21T10:41:17.556434+0000","mapping_epoch":1354,"log_start":"1369'6800","ondisk_log_start":"1369'6800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6996,"log_dups_size":0,"ondisk_log_size":6996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:21.766427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92667904,"num_objects":1414,"num_object_clones":0,"num_object_copies":4242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13796,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"48.2","version":"1385'14021","reported_seq":14060,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:12.993634+0000","last_change":"2024-09-21T10:37:50.816843+0000","last_active":"2024-09-21T10:41:12.993634+0000","last_peered":"2024-09-21T10:41:12.993634+0000","last_clean":"2024-09-21T10:41:12.993634+0000","last_became_active":"2024-09-21T10:37:50.816490+0000","last_became_peered":"2024-09-21T10:37:50.816490+0000","last_unstale":"2024-09-21T10:41:12.993634+0000","last_undegraded":"2024-09-21T10:41:12.993634+0000","last_fullsized":"2024-09-21T10:41:12.993634+0000","mapping_epoch":1354,"log_start":"1372'8291","ondisk_log_start":"1372'8291","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5730,"log_dups_size":0,"ondisk_log_size":5730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:33.243791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114491392,"num_objects":1747,"num_object_clones":0,"num_object_copies":5241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14021,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.1","version":"1385'13530","reported_seq":23202,"reported_epoch":1385,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-21T10:41:13.844175+0000","last_change":"2024-09-21T10:39:49.804470+0000","last_active":"2024-09-21T10:41:13.844175+0000","last_peered":"2024-09-21T10:41:13.844175+0000","last_clean":"2024-09-21T10:38:59.823246+0000","last_became_active":"2024-09-21T10:39:02.073182+0000","last_became_peered":"2024-09-21T10:39:02.073182+0000","last_unstale":"2024-09-21T10:41:13.844175+0000","last_undegraded":"2024-09-21T10:39:49.800872+0000","last_fullsized":"2024-09-21T10:39:01.839371+0000","mapping_epoch":1367,"log_start":"1370'7200","ondisk_log_start":"1370'7200","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6330,"log_dups_size":0,"ondisk_log_size":6330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:06.547860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107479040,"num_objects":1640,"num_object_clones":0,"num_object_copies":4920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13530,"num_write_kb":485440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4811,"num_bytes_recovered":307822592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,6],"avail_no_missing":["7","5","6"],"object_location_counts":[{"shards":"6,7","objects":518}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.0","version":"1385'13501","reported_seq":13575,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:14.441057+0000","last_change":"2024-09-21T10:37:50.812734+0000","last_active":"2024-09-21T10:41:14.441057+0000","last_peered":"2024-09-21T10:41:14.441057+0000","last_clean":"2024-09-21T10:41:14.441057+0000","last_became_active":"2024-09-21T10:37:50.812438+0000","last_became_peered":"2024-09-21T10:37:50.812438+0000","last_unstale":"2024-09-21T10:41:14.441057+0000","last_undegraded":"2024-09-21T10:41:14.441057+0000","last_fullsized":"2024-09-21T10:41:14.441057+0000","mapping_epoch":1354,"log_start":"1365'5800","ondisk_log_start":"1365'5800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:25.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13501,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":55,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236820+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:41:01.236820+0000","last_peered":"2024-09-21T10:41:01.236820+0000","last_clean":"2024-09-21T10:41:01.236820+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:41:01.236820+0000","last_undegraded":"2024-09-21T10:41:01.236820+0000","last_fullsized":"2024-09-21T10:41:01.236820+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1279,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.167724+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:40:49.167724+0000","last_peered":"2024-09-21T10:40:49.167724+0000","last_clean":"2024-09-21T10:40:49.167724+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:40:49.167724+0000","last_undegraded":"2024-09-21T10:40:49.167724+0000","last_fullsized":"2024-09-21T10:40:49.167724+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2092,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.248978+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:41:01.248978+0000","last_peered":"2024-09-21T10:41:01.248978+0000","last_clean":"2024-09-21T10:41:01.248978+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:41:01.248978+0000","last_undegraded":"2024-09-21T10:41:01.248978+0000","last_fullsized":"2024-09-21T10:41:01.248978+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2110,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.238214+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:41:01.238214+0000","last_peered":"2024-09-21T10:41:01.238214+0000","last_clean":"2024-09-21T10:41:01.238214+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:41:01.238214+0000","last_undegraded":"2024-09-21T10:41:01.238214+0000","last_fullsized":"2024-09-21T10:41:01.238214+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":153,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.160766+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:40:49.160766+0000","last_peered":"2024-09-21T10:40:49.160766+0000","last_clean":"2024-09-21T10:40:49.160766+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:40:49.160766+0000","last_undegraded":"2024-09-21T10:40:49.160766+0000","last_fullsized":"2024-09-21T10:40:49.160766+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1293,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.237664+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:41:01.237664+0000","last_peered":"2024-09-21T10:41:01.237664+0000","last_clean":"2024-09-21T10:41:01.237664+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:41:01.237664+0000","last_undegraded":"2024-09-21T10:41:01.237664+0000","last_fullsized":"2024-09-21T10:41:01.237664+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":153,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.160057+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:40:49.160057+0000","last_peered":"2024-09-21T10:40:49.160057+0000","last_clean":"2024-09-21T10:40:49.160057+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:40:49.160057+0000","last_undegraded":"2024-09-21T10:40:49.160057+0000","last_fullsized":"2024-09-21T10:40:49.160057+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1380,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.235805+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:41:01.235805+0000","last_peered":"2024-09-21T10:41:01.235805+0000","last_clean":"2024-09-21T10:41:01.235805+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:41:01.235805+0000","last_undegraded":"2024-09-21T10:41:01.235805+0000","last_fullsized":"2024-09-21T10:41:01.235805+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":48,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.235896+0000","last_change":"2024-09-21T10:39:00.847895+0000","last_active":"2024-09-21T10:41:01.235896+0000","last_peered":"2024-09-21T10:41:01.235896+0000","last_clean":"2024-09-21T10:41:01.235896+0000","last_became_active":"2024-09-21T10:39:00.847533+0000","last_became_peered":"2024-09-21T10:39:00.847533+0000","last_unstale":"2024-09-21T10:41:01.235896+0000","last_undegraded":"2024-09-21T10:41:01.235896+0000","last_fullsized":"2024-09-21T10:41:01.235896+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:55.080913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":77,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.237473+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:41:01.237473+0000","last_peered":"2024-09-21T10:41:01.237473+0000","last_clean":"2024-09-21T10:41:01.237473+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:41:01.237473+0000","last_undegraded":"2024-09-21T10:41:01.237473+0000","last_fullsized":"2024-09-21T10:41:01.237473+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":153,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233034+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:41:01.233034+0000","last_peered":"2024-09-21T10:41:01.233034+0000","last_clean":"2024-09-21T10:41:01.233034+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:41:01.233034+0000","last_undegraded":"2024-09-21T10:41:01.233034+0000","last_fullsized":"2024-09-21T10:41:01.233034+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2099,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.235613+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:41:01.235613+0000","last_peered":"2024-09-21T10:41:01.235613+0000","last_clean":"2024-09-21T10:41:01.235613+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:41:01.235613+0000","last_undegraded":"2024-09-21T10:41:01.235613+0000","last_fullsized":"2024-09-21T10:41:01.235613+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2198,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232283+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:41:01.232283+0000","last_peered":"2024-09-21T10:41:01.232283+0000","last_clean":"2024-09-21T10:41:01.232283+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:41:01.232283+0000","last_undegraded":"2024-09-21T10:41:01.232283+0000","last_fullsized":"2024-09-21T10:41:01.232283+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2207,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236685+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:41:01.236685+0000","last_peered":"2024-09-21T10:41:01.236685+0000","last_clean":"2024-09-21T10:41:01.236685+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:41:01.236685+0000","last_undegraded":"2024-09-21T10:41:01.236685+0000","last_fullsized":"2024-09-21T10:41:01.236685+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2125,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.229969+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:41:01.229969+0000","last_peered":"2024-09-21T10:41:01.229969+0000","last_clean":"2024-09-21T10:41:01.229969+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:41:01.229969+0000","last_undegraded":"2024-09-21T10:41:01.229969+0000","last_fullsized":"2024-09-21T10:41:01.229969+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1343,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.167052+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:40:49.167052+0000","last_peered":"2024-09-21T10:40:49.167052+0000","last_clean":"2024-09-21T10:40:49.167052+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:40:49.167052+0000","last_undegraded":"2024-09-21T10:40:49.167052+0000","last_fullsized":"2024-09-21T10:40:49.167052+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2104,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.231196+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:41:01.231196+0000","last_peered":"2024-09-21T10:41:01.231196+0000","last_clean":"2024-09-21T10:41:01.231196+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:41:01.231196+0000","last_undegraded":"2024-09-21T10:41:01.231196+0000","last_fullsized":"2024-09-21T10:41:01.231196+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2217,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236979+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:41:01.236979+0000","last_peered":"2024-09-21T10:41:01.236979+0000","last_clean":"2024-09-21T10:41:01.236979+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:41:01.236979+0000","last_undegraded":"2024-09-21T10:41:01.236979+0000","last_fullsized":"2024-09-21T10:41:01.236979+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":234,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.237644+0000","last_change":"2024-09-21T10:37:47.788643+0000","last_active":"2024-09-21T10:41:01.237644+0000","last_peered":"2024-09-21T10:41:01.237644+0000","last_clean":"2024-09-21T10:41:01.237644+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:41:01.237644+0000","last_undegraded":"2024-09-21T10:41:01.237644+0000","last_fullsized":"2024-09-21T10:41:01.237644+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:39.188916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2087,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.238465+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:41:01.238465+0000","last_peered":"2024-09-21T10:41:01.238465+0000","last_clean":"2024-09-21T10:41:01.238465+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:41:01.238465+0000","last_undegraded":"2024-09-21T10:41:01.238465+0000","last_fullsized":"2024-09-21T10:41:01.238465+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":135,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.160922+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:40:49.160922+0000","last_peered":"2024-09-21T10:40:49.160922+0000","last_clean":"2024-09-21T10:40:49.160922+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:40:49.160922+0000","last_undegraded":"2024-09-21T10:40:49.160922+0000","last_fullsized":"2024-09-21T10:40:49.160922+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1748,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232871+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:41:01.232871+0000","last_peered":"2024-09-21T10:41:01.232871+0000","last_clean":"2024-09-21T10:41:01.232871+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:41:01.232871+0000","last_undegraded":"2024-09-21T10:41:01.232871+0000","last_fullsized":"2024-09-21T10:41:01.232871+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2190,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233141+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:41:01.233141+0000","last_peered":"2024-09-21T10:41:01.233141+0000","last_clean":"2024-09-21T10:41:01.233141+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:41:01.233141+0000","last_undegraded":"2024-09-21T10:41:01.233141+0000","last_fullsized":"2024-09-21T10:41:01.233141+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":400,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.161969+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:40:49.161969+0000","last_peered":"2024-09-21T10:40:49.161969+0000","last_clean":"2024-09-21T10:40:49.161969+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:40:49.161969+0000","last_undegraded":"2024-09-21T10:40:49.161969+0000","last_fullsized":"2024-09-21T10:40:49.161969+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1244,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.165958+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:40:49.165958+0000","last_peered":"2024-09-21T10:40:49.165958+0000","last_clean":"2024-09-21T10:40:49.165958+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:40:49.165958+0000","last_undegraded":"2024-09-21T10:40:49.165958+0000","last_fullsized":"2024-09-21T10:40:49.165958+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":347,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.251994+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:41:01.251994+0000","last_peered":"2024-09-21T10:41:01.251994+0000","last_clean":"2024-09-21T10:41:01.251994+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:41:01.251994+0000","last_undegraded":"2024-09-21T10:41:01.251994+0000","last_fullsized":"2024-09-21T10:41:01.251994+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1278,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159735+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:40:49.159735+0000","last_peered":"2024-09-21T10:40:49.159735+0000","last_clean":"2024-09-21T10:40:49.159735+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:40:49.159735+0000","last_undegraded":"2024-09-21T10:40:49.159735+0000","last_fullsized":"2024-09-21T10:40:49.159735+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1339,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.166903+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:40:49.166903+0000","last_peered":"2024-09-21T10:40:49.166903+0000","last_clean":"2024-09-21T10:40:49.166903+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:40:49.166903+0000","last_undegraded":"2024-09-21T10:40:49.166903+0000","last_fullsized":"2024-09-21T10:40:49.166903+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":81,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236456+0000","last_change":"2024-09-21T10:37:41.926686+0000","last_active":"2024-09-21T10:41:01.236456+0000","last_peered":"2024-09-21T10:41:01.236456+0000","last_clean":"2024-09-21T10:41:01.236456+0000","last_became_active":"2024-09-21T10:37:41.922706+0000","last_became_peered":"2024-09-21T10:37:41.922706+0000","last_unstale":"2024-09-21T10:41:01.236456+0000","last_undegraded":"2024-09-21T10:41:01.236456+0000","last_fullsized":"2024-09-21T10:41:01.236456+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:08.955843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":136,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.247667+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:41:01.247667+0000","last_peered":"2024-09-21T10:41:01.247667+0000","last_clean":"2024-09-21T10:41:01.247667+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:41:01.247667+0000","last_undegraded":"2024-09-21T10:41:01.247667+0000","last_fullsized":"2024-09-21T10:41:01.247667+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":41,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159431+0000","last_change":"2024-09-21T10:39:00.847173+0000","last_active":"2024-09-21T10:40:49.159431+0000","last_peered":"2024-09-21T10:40:49.159431+0000","last_clean":"2024-09-21T10:40:49.159431+0000","last_became_active":"2024-09-21T10:39:00.846851+0000","last_became_peered":"2024-09-21T10:39:00.846851+0000","last_unstale":"2024-09-21T10:40:49.159431+0000","last_undegraded":"2024-09-21T10:40:49.159431+0000","last_fullsized":"2024-09-21T10:40:49.159431+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:47.133721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2207,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.240215+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:41:01.240215+0000","last_peered":"2024-09-21T10:41:01.240215+0000","last_clean":"2024-09-21T10:41:01.240215+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:41:01.240215+0000","last_undegraded":"2024-09-21T10:41:01.240215+0000","last_fullsized":"2024-09-21T10:41:01.240215+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2086,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236308+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:41:01.236308+0000","last_peered":"2024-09-21T10:41:01.236308+0000","last_clean":"2024-09-21T10:41:01.236308+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:41:01.236308+0000","last_undegraded":"2024-09-21T10:41:01.236308+0000","last_fullsized":"2024-09-21T10:41:01.236308+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":510,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.165459+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:40:49.165459+0000","last_peered":"2024-09-21T10:40:49.165459+0000","last_clean":"2024-09-21T10:40:49.165459+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:40:49.165459+0000","last_undegraded":"2024-09-21T10:40:49.165459+0000","last_fullsized":"2024-09-21T10:40:49.165459+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2209,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236507+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:41:01.236507+0000","last_peered":"2024-09-21T10:41:01.236507+0000","last_clean":"2024-09-21T10:41:01.236507+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:41:01.236507+0000","last_undegraded":"2024-09-21T10:41:01.236507+0000","last_fullsized":"2024-09-21T10:41:01.236507+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1402,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.164107+0000","last_change":"2024-09-21T10:37:41.922996+0000","last_active":"2024-09-21T10:40:49.164107+0000","last_peered":"2024-09-21T10:40:49.164107+0000","last_clean":"2024-09-21T10:40:49.164107+0000","last_became_active":"2024-09-21T10:37:41.922703+0000","last_became_peered":"2024-09-21T10:37:41.922703+0000","last_unstale":"2024-09-21T10:40:49.164107+0000","last_undegraded":"2024-09-21T10:40:49.164107+0000","last_fullsized":"2024-09-21T10:40:49.164107+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:23.614770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2195,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232645+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:41:01.232645+0000","last_peered":"2024-09-21T10:41:01.232645+0000","last_clean":"2024-09-21T10:41:01.232645+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:41:01.232645+0000","last_undegraded":"2024-09-21T10:41:01.232645+0000","last_fullsized":"2024-09-21T10:41:01.232645+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":510,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.166830+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:40:49.166830+0000","last_peered":"2024-09-21T10:40:49.166830+0000","last_clean":"2024-09-21T10:40:49.166830+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:40:49.166830+0000","last_undegraded":"2024-09-21T10:40:49.166830+0000","last_fullsized":"2024-09-21T10:40:49.166830+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2200,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233281+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:41:01.233281+0000","last_peered":"2024-09-21T10:41:01.233281+0000","last_clean":"2024-09-21T10:41:01.233281+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:41:01.233281+0000","last_undegraded":"2024-09-21T10:41:01.233281+0000","last_fullsized":"2024-09-21T10:41:01.233281+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2195,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233123+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:41:01.233123+0000","last_peered":"2024-09-21T10:41:01.233123+0000","last_clean":"2024-09-21T10:41:01.233123+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:41:01.233123+0000","last_undegraded":"2024-09-21T10:41:01.233123+0000","last_fullsized":"2024-09-21T10:41:01.233123+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":135,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.160641+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:40:49.160641+0000","last_peered":"2024-09-21T10:40:49.160641+0000","last_clean":"2024-09-21T10:40:49.160641+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:40:49.160641+0000","last_undegraded":"2024-09-21T10:40:49.160641+0000","last_fullsized":"2024-09-21T10:40:49.160641+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":145,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159804+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:40:49.159804+0000","last_peered":"2024-09-21T10:40:49.159804+0000","last_clean":"2024-09-21T10:40:49.159804+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:40:49.159804+0000","last_undegraded":"2024-09-21T10:40:49.159804+0000","last_fullsized":"2024-09-21T10:40:49.159804+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2192,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232745+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:41:01.232745+0000","last_peered":"2024-09-21T10:41:01.232745+0000","last_clean":"2024-09-21T10:41:01.232745+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:41:01.232745+0000","last_undegraded":"2024-09-21T10:41:01.232745+0000","last_fullsized":"2024-09-21T10:41:01.232745+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1312,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236414+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:41:01.236414+0000","last_peered":"2024-09-21T10:41:01.236414+0000","last_clean":"2024-09-21T10:41:01.236414+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:41:01.236414+0000","last_undegraded":"2024-09-21T10:41:01.236414+0000","last_fullsized":"2024-09-21T10:41:01.236414+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":139,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159685+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:40:49.159685+0000","last_peered":"2024-09-21T10:40:49.159685+0000","last_clean":"2024-09-21T10:40:49.159685+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:40:49.159685+0000","last_undegraded":"2024-09-21T10:40:49.159685+0000","last_fullsized":"2024-09-21T10:40:49.159685+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":515,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236044+0000","last_change":"2024-09-21T10:37:38.606346+0000","last_active":"2024-09-21T10:41:01.236044+0000","last_peered":"2024-09-21T10:41:01.236044+0000","last_clean":"2024-09-21T10:41:01.236044+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:41:01.236044+0000","last_undegraded":"2024-09-21T10:41:01.236044+0000","last_fullsized":"2024-09-21T10:41:01.236044+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:16.461221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2210,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233447+0000","last_change":"2024-09-21T10:37:41.924374+0000","last_active":"2024-09-21T10:41:01.233447+0000","last_peered":"2024-09-21T10:41:01.233447+0000","last_clean":"2024-09-21T10:41:01.233447+0000","last_became_active":"2024-09-21T10:37:41.922589+0000","last_became_peered":"2024-09-21T10:37:41.922589+0000","last_unstale":"2024-09-21T10:41:01.233447+0000","last_undegraded":"2024-09-21T10:41:01.233447+0000","last_fullsized":"2024-09-21T10:41:01.233447+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:51.987384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2193,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232897+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:41:01.232897+0000","last_peered":"2024-09-21T10:41:01.232897+0000","last_clean":"2024-09-21T10:41:01.232897+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:41:01.232897+0000","last_undegraded":"2024-09-21T10:41:01.232897+0000","last_fullsized":"2024-09-21T10:41:01.232897+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2123,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233702+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:41:01.233702+0000","last_peered":"2024-09-21T10:41:01.233702+0000","last_clean":"2024-09-21T10:41:01.233702+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:41:01.233702+0000","last_undegraded":"2024-09-21T10:41:01.233702+0000","last_fullsized":"2024-09-21T10:41:01.233702+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":378,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232707+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:41:01.232707+0000","last_peered":"2024-09-21T10:41:01.232707+0000","last_clean":"2024-09-21T10:41:01.232707+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:41:01.232707+0000","last_undegraded":"2024-09-21T10:41:01.232707+0000","last_fullsized":"2024-09-21T10:41:01.232707+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":49,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236343+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:41:01.236343+0000","last_peered":"2024-09-21T10:41:01.236343+0000","last_clean":"2024-09-21T10:41:01.236343+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:41:01.236343+0000","last_undegraded":"2024-09-21T10:41:01.236343+0000","last_fullsized":"2024-09-21T10:41:01.236343+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2224,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.239937+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:41:01.239937+0000","last_peered":"2024-09-21T10:41:01.239937+0000","last_clean":"2024-09-21T10:41:01.239937+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:41:01.239937+0000","last_undegraded":"2024-09-21T10:41:01.239937+0000","last_fullsized":"2024-09-21T10:41:01.239937+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1372,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.161573+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:40:49.161573+0000","last_peered":"2024-09-21T10:40:49.161573+0000","last_clean":"2024-09-21T10:40:49.161573+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:40:49.161573+0000","last_undegraded":"2024-09-21T10:40:49.161573+0000","last_fullsized":"2024-09-21T10:40:49.161573+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1345,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.165534+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:40:49.165534+0000","last_peered":"2024-09-21T10:40:49.165534+0000","last_clean":"2024-09-21T10:40:49.165534+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:40:49.165534+0000","last_undegraded":"2024-09-21T10:40:49.165534+0000","last_fullsized":"2024-09-21T10:40:49.165534+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2109,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.237755+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:41:01.237755+0000","last_peered":"2024-09-21T10:41:01.237755+0000","last_clean":"2024-09-21T10:41:01.237755+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:41:01.237755+0000","last_undegraded":"2024-09-21T10:41:01.237755+0000","last_fullsized":"2024-09-21T10:41:01.237755+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":139,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.160245+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:40:49.160245+0000","last_peered":"2024-09-21T10:40:49.160245+0000","last_clean":"2024-09-21T10:40:49.160245+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:40:49.160245+0000","last_undegraded":"2024-09-21T10:40:49.160245+0000","last_fullsized":"2024-09-21T10:40:49.160245+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1555,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.161344+0000","last_change":"2024-09-21T10:37:41.922049+0000","last_active":"2024-09-21T10:40:49.161344+0000","last_peered":"2024-09-21T10:40:49.161344+0000","last_clean":"2024-09-21T10:40:49.161344+0000","last_became_active":"2024-09-21T10:37:41.921709+0000","last_became_peered":"2024-09-21T10:37:41.921709+0000","last_unstale":"2024-09-21T10:40:49.161344+0000","last_undegraded":"2024-09-21T10:40:49.161344+0000","last_fullsized":"2024-09-21T10:40:49.161344+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:12.446157+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":135,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159700+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:40:49.159700+0000","last_peered":"2024-09-21T10:40:49.159700+0000","last_clean":"2024-09-21T10:40:49.159700+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:40:49.159700+0000","last_undegraded":"2024-09-21T10:40:49.159700+0000","last_fullsized":"2024-09-21T10:40:49.159700+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2198,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233362+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:41:01.233362+0000","last_peered":"2024-09-21T10:41:01.233362+0000","last_clean":"2024-09-21T10:41:01.233362+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:41:01.233362+0000","last_undegraded":"2024-09-21T10:41:01.233362+0000","last_fullsized":"2024-09-21T10:41:01.233362+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2095,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.231723+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:41:01.231723+0000","last_peered":"2024-09-21T10:41:01.231723+0000","last_clean":"2024-09-21T10:41:01.231723+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:41:01.231723+0000","last_undegraded":"2024-09-21T10:41:01.231723+0000","last_fullsized":"2024-09-21T10:41:01.231723+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":362,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232654+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:41:01.232654+0000","last_peered":"2024-09-21T10:41:01.232654+0000","last_clean":"2024-09-21T10:41:01.232654+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:41:01.232654+0000","last_undegraded":"2024-09-21T10:41:01.232654+0000","last_fullsized":"2024-09-21T10:41:01.232654+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1244,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.165534+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:40:49.165534+0000","last_peered":"2024-09-21T10:40:49.165534+0000","last_clean":"2024-09-21T10:40:49.165534+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:40:49.165534+0000","last_undegraded":"2024-09-21T10:40:49.165534+0000","last_fullsized":"2024-09-21T10:40:49.165534+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1251,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.233780+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:41:01.233780+0000","last_peered":"2024-09-21T10:41:01.233780+0000","last_clean":"2024-09-21T10:41:01.233780+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:41:01.233780+0000","last_undegraded":"2024-09-21T10:41:01.233780+0000","last_fullsized":"2024-09-21T10:41:01.233780+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":139,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159597+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:40:49.159597+0000","last_peered":"2024-09-21T10:40:49.159597+0000","last_clean":"2024-09-21T10:40:49.159597+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:40:49.159597+0000","last_undegraded":"2024-09-21T10:40:49.159597+0000","last_fullsized":"2024-09-21T10:40:49.159597+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1259,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.248016+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:41:01.248016+0000","last_peered":"2024-09-21T10:41:01.248016+0000","last_clean":"2024-09-21T10:41:01.248016+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:41:01.248016+0000","last_undegraded":"2024-09-21T10:41:01.248016+0000","last_fullsized":"2024-09-21T10:41:01.248016+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1351,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.167744+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:40:49.167744+0000","last_peered":"2024-09-21T10:40:49.167744+0000","last_clean":"2024-09-21T10:40:49.167744+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:40:49.167744+0000","last_undegraded":"2024-09-21T10:40:49.167744+0000","last_fullsized":"2024-09-21T10:40:49.167744+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1506,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.168411+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:40:49.168411+0000","last_peered":"2024-09-21T10:40:49.168411+0000","last_clean":"2024-09-21T10:40:49.168411+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:40:49.168411+0000","last_undegraded":"2024-09-21T10:40:49.168411+0000","last_fullsized":"2024-09-21T10:40:49.168411+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1311,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.252275+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:41:01.252275+0000","last_peered":"2024-09-21T10:41:01.252275+0000","last_clean":"2024-09-21T10:41:01.252275+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:41:01.252275+0000","last_undegraded":"2024-09-21T10:41:01.252275+0000","last_fullsized":"2024-09-21T10:41:01.252275+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":323,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.235694+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:41:01.235694+0000","last_peered":"2024-09-21T10:41:01.235694+0000","last_clean":"2024-09-21T10:41:01.235694+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:41:01.235694+0000","last_undegraded":"2024-09-21T10:41:01.235694+0000","last_fullsized":"2024-09-21T10:41:01.235694+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":80,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.162105+0000","last_change":"2024-09-21T10:37:37.360857+0000","last_active":"2024-09-21T10:40:49.162105+0000","last_peered":"2024-09-21T10:40:49.162105+0000","last_clean":"2024-09-21T10:40:49.162105+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:40:49.162105+0000","last_undegraded":"2024-09-21T10:40:49.162105+0000","last_fullsized":"2024-09-21T10:40:49.162105+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.360779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.360779+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:47:25.310501+0000","scrub_duration":0.001559979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2223,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236319+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:41:01.236319+0000","last_peered":"2024-09-21T10:41:01.236319+0000","last_clean":"2024-09-21T10:41:01.236319+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:41:01.236319+0000","last_undegraded":"2024-09-21T10:41:01.236319+0000","last_fullsized":"2024-09-21T10:41:01.236319+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":135,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159984+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:40:49.159984+0000","last_peered":"2024-09-21T10:40:49.159984+0000","last_clean":"2024-09-21T10:40:49.159984+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:40:49.159984+0000","last_undegraded":"2024-09-21T10:40:49.159984+0000","last_fullsized":"2024-09-21T10:40:49.159984+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":529,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.166975+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:40:49.166975+0000","last_peered":"2024-09-21T10:40:49.166975+0000","last_clean":"2024-09-21T10:40:49.166975+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:40:49.166975+0000","last_undegraded":"2024-09-21T10:40:49.166975+0000","last_fullsized":"2024-09-21T10:40:49.166975+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1323,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.167603+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:40:49.167603+0000","last_peered":"2024-09-21T10:40:49.167603+0000","last_clean":"2024-09-21T10:40:49.167603+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:40:49.167603+0000","last_undegraded":"2024-09-21T10:40:49.167603+0000","last_fullsized":"2024-09-21T10:40:49.167603+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1330,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159952+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:40:49.159952+0000","last_peered":"2024-09-21T10:40:49.159952+0000","last_clean":"2024-09-21T10:40:49.159952+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:40:49.159952+0000","last_undegraded":"2024-09-21T10:40:49.159952+0000","last_fullsized":"2024-09-21T10:40:49.159952+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":362,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.251947+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:41:01.251947+0000","last_peered":"2024-09-21T10:41:01.251947+0000","last_clean":"2024-09-21T10:41:01.251947+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:41:01.251947+0000","last_undegraded":"2024-09-21T10:41:01.251947+0000","last_fullsized":"2024-09-21T10:41:01.251947+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1294,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232382+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:41:01.232382+0000","last_peered":"2024-09-21T10:41:01.232382+0000","last_clean":"2024-09-21T10:41:01.232382+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:41:01.232382+0000","last_undegraded":"2024-09-21T10:41:01.232382+0000","last_fullsized":"2024-09-21T10:41:01.232382+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1390,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.162243+0000","last_change":"2024-09-21T10:38:34.901523+0000","last_active":"2024-09-21T10:40:49.162243+0000","last_peered":"2024-09-21T10:40:49.162243+0000","last_clean":"2024-09-21T10:40:49.162243+0000","last_became_active":"2024-09-21T10:38:34.887435+0000","last_became_peered":"2024-09-21T10:38:34.887435+0000","last_unstale":"2024-09-21T10:40:49.162243+0000","last_undegraded":"2024-09-21T10:40:49.162243+0000","last_fullsized":"2024-09-21T10:40:49.162243+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:27.993702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":132,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.163876+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:40:49.163876+0000","last_peered":"2024-09-21T10:40:49.163876+0000","last_clean":"2024-09-21T10:40:49.163876+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:40:49.163876+0000","last_undegraded":"2024-09-21T10:40:49.163876+0000","last_fullsized":"2024-09-21T10:40:49.163876+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":135,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159622+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:40:49.159622+0000","last_peered":"2024-09-21T10:40:49.159622+0000","last_clean":"2024-09-21T10:40:49.159622+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:40:49.159622+0000","last_undegraded":"2024-09-21T10:40:49.159622+0000","last_fullsized":"2024-09-21T10:40:49.159622+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1403,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.162198+0000","last_change":"2024-09-21T10:38:34.891682+0000","last_active":"2024-09-21T10:40:49.162198+0000","last_peered":"2024-09-21T10:40:49.162198+0000","last_clean":"2024-09-21T10:40:49.162198+0000","last_became_active":"2024-09-21T10:38:34.885292+0000","last_became_peered":"2024-09-21T10:38:34.885292+0000","last_unstale":"2024-09-21T10:40:49.162198+0000","last_undegraded":"2024-09-21T10:40:49.162198+0000","last_fullsized":"2024-09-21T10:40:49.162198+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:30.111501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":122,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159596+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:40:49.159596+0000","last_peered":"2024-09-21T10:40:49.159596+0000","last_clean":"2024-09-21T10:40:49.159596+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:40:49.159596+0000","last_undegraded":"2024-09-21T10:40:49.159596+0000","last_fullsized":"2024-09-21T10:40:49.159596+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1420,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.167916+0000","last_change":"2024-09-21T10:39:00.852998+0000","last_active":"2024-09-21T10:40:49.167916+0000","last_peered":"2024-09-21T10:40:49.167916+0000","last_clean":"2024-09-21T10:40:49.167916+0000","last_became_active":"2024-09-21T10:39:00.846811+0000","last_became_peered":"2024-09-21T10:39:00.846811+0000","last_unstale":"2024-09-21T10:40:49.167916+0000","last_undegraded":"2024-09-21T10:40:49.167916+0000","last_fullsized":"2024-09-21T10:40:49.167916+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:14.374741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2085,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236021+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:41:01.236021+0000","last_peered":"2024-09-21T10:41:01.236021+0000","last_clean":"2024-09-21T10:41:01.236021+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:41:01.236021+0000","last_undegraded":"2024-09-21T10:41:01.236021+0000","last_fullsized":"2024-09-21T10:41:01.236021+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":368,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.232519+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:41:01.232519+0000","last_peered":"2024-09-21T10:41:01.232519+0000","last_clean":"2024-09-21T10:41:01.232519+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:41:01.232519+0000","last_undegraded":"2024-09-21T10:41:01.232519+0000","last_fullsized":"2024-09-21T10:41:01.232519+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1239,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.165184+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:40:49.165184+0000","last_peered":"2024-09-21T10:40:49.165184+0000","last_clean":"2024-09-21T10:40:49.165184+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:40:49.165184+0000","last_undegraded":"2024-09-21T10:40:49.165184+0000","last_fullsized":"2024-09-21T10:40:49.165184+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2204,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236609+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:41:01.236609+0000","last_peered":"2024-09-21T10:41:01.236609+0000","last_clean":"2024-09-21T10:41:01.236609+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:41:01.236609+0000","last_undegraded":"2024-09-21T10:41:01.236609+0000","last_fullsized":"2024-09-21T10:41:01.236609+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":122,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.159517+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:40:49.159517+0000","last_peered":"2024-09-21T10:40:49.159517+0000","last_clean":"2024-09-21T10:40:49.159517+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:40:49.159517+0000","last_undegraded":"2024-09-21T10:40:49.159517+0000","last_fullsized":"2024-09-21T10:40:49.159517+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2204,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.236187+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:41:01.236187+0000","last_peered":"2024-09-21T10:41:01.236187+0000","last_clean":"2024-09-21T10:41:01.236187+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:41:01.236187+0000","last_undegraded":"2024-09-21T10:41:01.236187+0000","last_fullsized":"2024-09-21T10:41:01.236187+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":145,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.160701+0000","last_change":"2024-09-21T10:38:38.962357+0000","last_active":"2024-09-21T10:40:49.160701+0000","last_peered":"2024-09-21T10:40:49.160701+0000","last_clean":"2024-09-21T10:40:49.160701+0000","last_became_active":"2024-09-21T10:38:38.962017+0000","last_became_peered":"2024-09-21T10:38:38.962017+0000","last_unstale":"2024-09-21T10:40:49.160701+0000","last_undegraded":"2024-09-21T10:40:49.160701+0000","last_fullsized":"2024-09-21T10:40:49.160701+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.355706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":85,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.239646+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:41:01.239646+0000","last_peered":"2024-09-21T10:41:01.239646+0000","last_clean":"2024-09-21T10:41:01.239646+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:41:01.239646+0000","last_undegraded":"2024-09-21T10:41:01.239646+0000","last_fullsized":"2024-09-21T10:41:01.239646+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1391,"reported_epoch":1384,"state":"active+clean","last_fresh":"2024-09-21T10:40:49.163330+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:40:49.163330+0000","last_peered":"2024-09-21T10:40:49.163330+0000","last_clean":"2024-09-21T10:40:49.163330+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:40:49.163330+0000","last_undegraded":"2024-09-21T10:40:49.163330+0000","last_fullsized":"2024-09-21T10:40:49.163330+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2093,"reported_epoch":1385,"state":"active+clean","last_fresh":"2024-09-21T10:41:01.252073+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:41:01.252073+0000","last_peered":"2024-09-21T10:41:01.252073+0000","last_clean":"2024-09-21T10:41:01.252073+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:41:01.252073+0000","last_undegraded":"2024-09-21T10:41:01.252073+0000","last_fullsized":"2024-09-21T10:41:01.252073+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":48,"num_pg":16,"stat_sum":{"num_bytes":1736835096,"num_objects":26503,"num_object_clones":0,"num_object_copies":79509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1122,"num_objects_misplaced":1633,"num_objects_unfound":0,"num_objects_dirty":26503,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":219913,"num_write_kb":7885249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4811,"num_bytes_recovered":307822592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5590429696,"data_stored":5590417480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103638,"ondisk_log_size":103638,"up":48,"acting":47,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":481141,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6268,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398060,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1907396,"kb_used_data":1011772,"kb_used_omap":1,"kb_used_meta":895614,"kb_avail":91837756,"statfs":{"total":95995035648,"available":94041862144,"internally_reserved":0,"allocated":1036054528,"data_stored":1030035573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917109080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049719,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2085572,"kb_used_data":1158268,"kb_used_omap":242,"kb_used_meta":927053,"kb_avail":91659580,"statfs":{"total":95995035648,"available":93859409920,"internally_reserved":0,"allocated":1186066432,"data_stored":1180047482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":949302495},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790518,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1620792,"kb_used_data":727920,"kb_used_omap":1,"kb_used_meta":892862,"kb_avail":92124360,"statfs":{"total":95995035648,"available":94335344640,"internally_reserved":0,"allocated":745390080,"data_stored":739379722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":914291032},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644678,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1842100,"kb_used_data":1095212,"kb_used_omap":67,"kb_used_meta":746748,"kb_avail":91903052,"statfs":{"total":95995035648,"available":94108725248,"internally_reserved":0,"allocated":1121497088,"data_stored":1115490301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":764670813},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527696,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1411392,"kb_used_data":682296,"kb_used_omap":75,"kb_used_meta":729012,"kb_avail":92333760,"statfs":{"total":95995035648,"available":94549770240,"internally_reserved":0,"allocated":698671104,"data_stored":692657287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":746508442},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274849,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1500468,"kb_used_data":733100,"kb_used_omap":51,"kb_used_meta":767244,"kb_avail":92244684,"statfs":{"total":95995035648,"available":94458556416,"internally_reserved":0,"allocated":750694400,"data_stored":744687094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":785658197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354088,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859380,"kb_used_data":297452,"kb_used_omap":66,"kb_used_meta":561853,"kb_avail":92885772,"statfs":{"total":95995035648,"available":95115030528,"internally_reserved":0,"allocated":304590848,"data_stored":298584101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":575337796},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":281018368,"data_stored":281018368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":720764928,"data_stored":720764928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":657850368,"data_stored":657850368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1094914048,"data_stored":1094909976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":718077952,"data_stored":718077952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1132400640,"data_stored":1132396568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":985403392,"data_stored":985399320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:41:20.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:20.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:20.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:20.416 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:20.450 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:20 smithi119 ceph-mon[105288]: from='client.46787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:20.450 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:20 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 1122/79587 objects degraded (1.410%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:20.450 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:20 smithi119 ceph-mon[101432]: from='client.46787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:20.450 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:20 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 1122/79587 objects degraded (1.410%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:20.517 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:20 smithi145 ceph-mon[89359]: from='client.46787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:20.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:20 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 1122/79587 objects degraded (1.410%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:41:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:41:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:41:20.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:20.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:20.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:41:20.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:20.681 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:20.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:20.742 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:20.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:20.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:20.932 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:20.987 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:20.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:21.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:21.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:21.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:21.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:21.269 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:21.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:21.374 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:41:21.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:21.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:21.543 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:21 smithi119 ceph-mon[101432]: pgmap v2328: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 762/66369 objects degraded (1.148%); 1633/66369 objects misplaced (2.460%); 3.0 MiB/s, 87 objects/s recovering 2024-09-21T10:41:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2925006823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:21 smithi119 ceph-mon[105288]: pgmap v2328: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 762/66369 objects degraded (1.148%); 1633/66369 objects misplaced (2.460%); 3.0 MiB/s, 87 objects/s recovering 2024-09-21T10:41:21.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2925006823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:21.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:21.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:21.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:21 smithi145 ceph-mon[89359]: pgmap v2328: 109 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait, 107 active+clean; 1.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.19k op/s; 762/66369 objects degraded (1.148%); 1633/66369 objects misplaced (2.460%); 3.0 MiB/s, 87 objects/s recovering 2024-09-21T10:41:21.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2925006823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:21.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:21.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:21.914 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:21.946 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:21.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:22.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:22.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:22.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:22.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:22.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:22.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:22.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:22.494 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:22.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:22 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2925006823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:41:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:22 smithi119 ceph-mon[101432]: osdmap e1386: 8 total, 8 up, 7 in 2024-09-21T10:41:22.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:22 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:22.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2925006823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:41:22.619 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:22 smithi119 ceph-mon[105288]: osdmap e1386: 8 total, 8 up, 7 in 2024-09-21T10:41:22.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:22.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:22.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:22 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2925006823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:41:22.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:22 smithi145 ceph-mon[89359]: osdmap e1386: 8 total, 8 up, 7 in 2024-09-21T10:41:22.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:22.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:22.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:23.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:23.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:23.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:23.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:23.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:23.219 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:41:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:23.276 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:23.300 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:23.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:23.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:23.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:23.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:23 smithi119 ceph-mon[101432]: pgmap v2330: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 762/65421 objects degraded (1.165%); 1317/65421 objects misplaced (2.013%); 1.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:23.525 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:23 smithi119 ceph-mon[105288]: pgmap v2330: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 762/65421 objects degraded (1.165%); 1317/65421 objects misplaced (2.013%); 1.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:23.579 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:23.606 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:23.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:23.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:23 smithi145 ceph-mon[89359]: pgmap v2330: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.00k op/s; 762/65421 objects degraded (1.165%); 1317/65421 objects misplaced (2.013%); 1.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:23.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:23.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:23.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:23.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:24.016 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:24.101 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:163978 46.b 46:dffc492e:::benchmark_data_smithi119_206743_object58877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1268)", 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:33.262711+0000", 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 590.76863789599997, 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.241971462, 2024-09-21T10:41:24.102 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 163978 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262711+0000", 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262711+0000", 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262713+0000", 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8819999999999999e-06 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262718+0000", 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8139999999999997e-06 2024-09-21T10:41:24.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262730+0000", 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1805e-05 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262738+0000", 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5179999999999998e-06 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262804+0000", 2024-09-21T10:41:24.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6328000000000006e-05 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.262937+0000", 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013334800000000001 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.268645+0000", 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057077730000000002 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.268703+0000", 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7797000000000002e-05 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.268867+0000", 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000164442 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281516+0000", 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012648313 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281555+0000", 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9184999999999998e-05 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.501427+0000", 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.219872346 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.501523+0000", 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6273999999999993e-05 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.504682+0000", 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031586370000000002 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:163991 46.b 46:d214d743:::benchmark_data_smithi119_206743_object58890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1268)", 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:31:33.277884+0000", 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 590.75346483999999, 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.240157019, 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.109 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 163991 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277884+0000", 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277884+0000", 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277885+0000", 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5009999999999999e-06 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277890+0000", 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5480000000000003e-06 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277901+0000", 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1134999999999999e-05 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.277908+0000", 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2099999999999996e-06 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281580+0000", 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036715089999999999 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for rw locks", 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.281645+0000", 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5312000000000001e-05 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.288724+0000", 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007079381 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.288793+0000", 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8491999999999994e-05 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.289471+0000", 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00067867700000000003 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.299520+0000", 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010048443000000001 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.299536+0000", 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6291999999999999e-05 2024-09-21T10:41:24.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.515106+0000", 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21557004900000001 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.515164+0000", 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7834999999999998e-05 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:31:33.518041+0000", 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028766350000000002 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36246.0:192766 46.9 46:9d2e794c:::benchmark_data_smithi119_206743_object87665:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1269)", 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:32:12.232572+0000", 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 551.79877691399997, 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.248148233, 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.116 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36246", 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/3152735415", 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192766 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232572+0000", 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232572+0000", 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232573+0000", 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4839999999999999e-06 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232577+0000", 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.33e-06 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232588+0000", 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0647e-05 2024-09-21T10:41:24.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232594+0000", 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6149999999999996e-06 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232641+0000", 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6866999999999997e-05 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232724+0000", 2024-09-21T10:41:24.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3366000000000001e-05 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.232801+0000", 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6679000000000002e-05 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.236637+0000", 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038365209999999999 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.236882+0000", 2024-09-21T10:41:24.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024469100000000001 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.480552+0000", 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24366981200000001 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.480684+0000", 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000132667 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:32:12.480720+0000", 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5553999999999999e-05 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54809 47.6 47:62b03f81:::benchmark_data_smithi119_238957_object54808:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.645023+0000", 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.386325457, 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.2786305, 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.122 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54809 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645023+0000", 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645023+0000", 2024-09-21T10:41:24.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645025+0000", 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9719999999999999e-06 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645112+0000", 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6680000000000004e-05 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645129+0000", 2024-09-21T10:41:24.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6762e-05 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645137+0000", 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0989999999999999e-06 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645217+0000", 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0133e-05 2024-09-21T10:41:24.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645358+0000", 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000141018 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645527+0000", 2024-09-21T10:41:24.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016966999999999999 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647754+0000", 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022260779999999998 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.649544+0000", 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017901670000000001 2024-09-21T10:41:24.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923550+0000", 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27400582499999998 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923610+0000", 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0229000000000002e-05 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923654+0000", 2024-09-21T10:41:24.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3866999999999999e-05 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54810 47.3 47:c9045a95:::benchmark_data_smithi119_238957_object54809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.645164+0000", 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.38618418900001, 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27704334800000002, 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.129 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54810 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645164+0000", 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645164+0000", 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645165+0000", 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6899999999999997e-07 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645213+0000", 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7490000000000001e-05 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645221+0000", 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.585e-06 2024-09-21T10:41:24.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645227+0000", 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0090000000000004e-06 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645279+0000", 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2379000000000001e-05 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645404+0000", 2024-09-21T10:41:24.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012490799999999999 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.645532+0000", 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012801699999999999 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.649757+0000", 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042247789999999997 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.650723+0000", 2024-09-21T10:41:24.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000966258 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.922047+0000", 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27132334600000002 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.922103+0000", 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5900999999999999e-05 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.922208+0000", 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010500700000000001 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54814 47.3 47:c491dd3a:::benchmark_data_smithi119_238957_object54813:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.647488+0000", 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.38386017699997, 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27654822899999998, 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.135 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54814 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647488+0000", 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647488+0000", 2024-09-21T10:41:24.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647490+0000", 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9470000000000002e-06 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647521+0000", 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0806999999999998e-05 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647536+0000", 2024-09-21T10:41:24.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5344000000000001e-05 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647545+0000", 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1680000000000004e-06 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647598+0000", 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3816000000000001e-05 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647764+0000", 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016596800000000001 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.647898+0000", 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013345100000000001 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.649906+0000", 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020083200000000001 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651295+0000", 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0013887680000000001 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923936+0000", 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27264077599999997 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923984+0000", 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8647999999999999e-05 2024-09-21T10:41:24.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924037+0000", 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2216000000000003e-05 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54822 47.6 47:676ee123:::benchmark_data_smithi119_238957_object54821:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.651507+0000", 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.37984120800002, 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27229856499999999, 2024-09-21T10:41:24.141 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54822 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651507+0000", 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651507+0000", 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651509+0000", 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.395e-06 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651530+0000", 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1491000000000001e-05 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651543+0000", 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3192e-05 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651550+0000", 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2400000000000004e-06 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651614+0000", 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4203999999999997e-05 2024-09-21T10:41:24.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651763+0000", 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000149603 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.651859+0000", 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5863e-05 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653689+0000", 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00182942 2024-09-21T10:41:24.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654168+0000", 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047968099999999998 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923704+0000", 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26953568999999999 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923768+0000", 2024-09-21T10:41:24.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4109999999999994e-05 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923806+0000", 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7676000000000003e-05 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54824 47.3 47:c6adad3d:::benchmark_data_smithi119_238957_object54823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.147 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.653983+0000", 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.37736550699998, 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27022529699999998, 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54824 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653983+0000", 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653983+0000", 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.653985+0000", 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7230000000000001e-06 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654035+0000", 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0497000000000001e-05 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654048+0000", 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2682e-05 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654056+0000", 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9999999999999996e-06 2024-09-21T10:41:24.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654115+0000", 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9545e-05 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654238+0000", 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012221700000000001 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654361+0000", 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123657 2024-09-21T10:41:24.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656784+0000", 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024229429999999999 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656899+0000", 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114821 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924076+0000", 2024-09-21T10:41:24.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.267176529 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924151+0000", 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5177000000000002e-05 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.924208+0000", 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7506000000000001e-05 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54826 47.6 47:6297b7c8:::benchmark_data_smithi119_238957_object54825:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.654495+0000", 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.376853848, 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26946603200000002, 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54826 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654495+0000", 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654495+0000", 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654496+0000", 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0410000000000001e-06 2024-09-21T10:41:24.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654549+0000", 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3205999999999998e-05 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654563+0000", 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3793e-05 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654570+0000", 2024-09-21T10:41:24.156 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5820000000000004e-06 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654626+0000", 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5566000000000003e-05 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654759+0000", 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013260999999999999 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.654910+0000", 2024-09-21T10:41:24.157 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015190000000000001 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656822+0000", 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001912003 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.656868+0000", 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6035999999999998e-05 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.158 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923863+0000", 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26699409200000002 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923922+0000", 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9947000000000001e-05 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.923961+0000", 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8256000000000001e-05 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.159 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54829 47.6 47:6dbd0281:::benchmark_data_smithi119_238957_object54828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.658574+0000", 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.37277453899998, 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26858875599999998, 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54829 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.160 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658574+0000", 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658574+0000", 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658576+0000", 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7460000000000001e-06 2024-09-21T10:41:24.161 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658629+0000", 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3380000000000001e-05 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658646+0000", 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7034999999999999e-05 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658654+0000", 2024-09-21T10:41:24.162 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2379999999999997e-06 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658714+0000", 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0053000000000003e-05 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658841+0000", 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012661899999999999 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.658968+0000", 2024-09-21T10:41:24.163 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012709200000000001 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.660925+0000", 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019565519999999999 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.661023+0000", 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8535000000000004e-05 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927013+0000", 2024-09-21T10:41:24.164 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26598982599999998 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927115+0000", 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000102339 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927163+0000", 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7341e-05 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.165 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54830 47.3 47:ccba9b51:::benchmark_data_smithi119_238957_object54829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.659013+0000", 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.37233508999998, 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26840792899999999, 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54830 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.166 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659013+0000", 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659013+0000", 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659014+0000", 2024-09-21T10:41:24.167 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0200000000000001e-07 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659051+0000", 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7007e-05 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659059+0000", 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1319999999999994e-06 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659064+0000", 2024-09-21T10:41:24.168 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4240000000000003e-06 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659120+0000", 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6322e-05 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659247+0000", 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000127179 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:41:24.169 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.659374+0000", 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012619199999999999 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.661230+0000", 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018564409999999999 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.661469+0000", 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00023875200000000001 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.170 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927285+0000", 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26581638400000002 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927381+0000", 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6274999999999996e-05 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.927421+0000", 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0018999999999997e-05 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.171 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:54833 47.3 47:c8abb190:::benchmark_data_smithi119_238957_object54832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1292)", 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:01.664934+0000", 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 442.36641446700003, 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26395025999999999, 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 54833 2024-09-21T10:41:24.172 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664934+0000", 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664934+0000", 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664935+0000", 2024-09-21T10:41:24.173 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2839999999999999e-06 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664974+0000", 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8825999999999997e-05 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664985+0000", 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0944999999999999e-05 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.174 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.664990+0000", 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8690000000000003e-06 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.665030+0000", 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0249e-05 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.665120+0000", 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9876999999999993e-05 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,7", 2024-09-21T10:41:24.175 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.665210+0000", 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9479000000000002e-05 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.666610+0000", 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014004950000000001 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.667132+0000", 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00052222300000000004 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.176 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.928787+0000", 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26165429600000001 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.928836+0000", 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9849999999999999e-05 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:01.928884+0000", 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7867000000000001e-05 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.177 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55562 47.13 47:ce06e73b:::benchmark_data_smithi119_238957_object55561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406134+0000", 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 441.62521479399999, 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.028150865, 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55562 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.178 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406134+0000", 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406134+0000", 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.179 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406136+0000", 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2859999999999998e-06 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406165+0000", 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9339000000000001e-05 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406183+0000", 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7816e-05 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.180 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406193+0000", 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.842e-06 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.482248+0000", 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.076055306000000003 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.482258+0000", 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9490000000000008e-06 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.181 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.434220+0000", 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.95196215299999998 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.434285+0000", 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4173999999999999e-05 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55574 47.16 47:68107290:::benchmark_data_smithi119_238957_object55573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:41:24.182 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.411282+0000", 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 441.62006666600001, 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0768573640000001, 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55574 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.183 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411282+0000", 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411282+0000", 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411285+0000", 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8949999999999998e-06 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.184 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411316+0000", 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1191e-05 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411341+0000", 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.472e-05 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411352+0000", 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.131e-05 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.185 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.500144+0000", 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.088791963000000002 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.500155+0000", 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0655999999999999e-05 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.479289+0000", 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.97913472499999998 2024-09-21T10:41:24.186 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.479451+0000", 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016205599999999999 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,2", 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.479692+0000", 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000240583 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481857+0000", 2024-09-21T10:41:24.187 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021645470000000002 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.482008+0000", 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000151068 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.488040+0000", 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060320979999999996 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.488095+0000", 2024-09-21T10:41:24.188 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5127000000000003e-05 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.488139+0000", 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4425000000000002e-05 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:25233 48.3 48:cb9f643a:::benchmark_data_smithi119_267254_object25232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1358)", 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:08.768702+0000", 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 195.26264675600001, 2024-09-21T10:41:24.189 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.37026426699999998, 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 25233 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768702+0000", 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.190 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768702+0000", 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768705+0000", 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0900000000000001e-06 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768733+0000", 2024-09-21T10:41:24.191 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7912999999999999e-05 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768751+0000", 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8212000000000001e-05 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768761+0000", 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7510000000000003e-06 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768828+0000", 2024-09-21T10:41:24.192 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7180000000000004e-05 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136028+0000", 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.36719971499999998 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136181+0000", 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015294199999999999 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138840+0000", 2024-09-21T10:41:24.193 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002659509 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138860+0000", 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9865e-05 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138903+0000", 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2698999999999999e-05 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.194 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138936+0000", 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3741000000000001e-05 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138966+0000", 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9649999999999999e-05 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:25232 48.4 e1358/1354 48:22eca9e3:::benchmark_data_smithi119_267254_object25231:head v 1358'1560, mlcod=1358'1558)", 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:08.769205+0000", 2024-09-21T10:41:24.195 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 195.26214361000001, 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.37107258300000001, 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769205+0000", 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769205+0000", 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769206+0000", 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.08e-06 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769250+0000", 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4456999999999998e-05 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136045+0000", 2024-09-21T10:41:24.197 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.36679430699999999 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136053+0000", 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.909e-06 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136107+0000", 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3874e-05 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.198 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136129+0000", 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2670000000000001e-05 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.140235+0000", 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004105427 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.140278+0000", 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2858999999999997e-05 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.199 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:192769 48.3 48:ce25c197:::benchmark_data_smithi119_267254_object69559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:45.556416+0000", 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 38.474932955, 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23024958800000001, 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192769 2024-09-21T10:41:24.200 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556416+0000", 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556416+0000", 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.201 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556417+0000", 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.68e-06 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556421+0000", 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7299999999999999e-06 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556434+0000", 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3057000000000001e-05 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556441+0000", 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0809999999999996e-06 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556592+0000", 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000151337 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556881+0000", 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028865100000000001 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.557117+0000", 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000235999 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.561134+0000", 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040171859999999998 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.563527+0000", 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023929369999999999 2024-09-21T10:41:24.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.786554+0000", 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.223026643 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.786614+0000", 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9735000000000003e-05 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.786665+0000", 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1551999999999997e-05 2024-09-21T10:41:24.205 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:224592 48.3 48:c8cc153e:::benchmark_data_smithi119_267254_object101382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1385)", 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:41:18.834271+0000", 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 5.1970779050000004, 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.037353219, 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:41:24.206 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 224592 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.834271+0000", 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.834271+0000", 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.834272+0000", 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6729999999999999e-06 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.834278+0000", 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0390000000000001e-06 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.834300+0000", 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1325999999999999e-05 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.209 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.834308+0000", 2024-09-21T10:41:24.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9910000000000002e-06 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.855021+0000", 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.020712864000000001 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.210 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.855142+0000", 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012139299999999999 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.855239+0000", 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7547999999999995e-05 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.863184+0000", 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079447300000000005 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.211 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.867278+0000", 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040932870000000001 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.871486+0000", 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042081419999999998 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.871566+0000", 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0483999999999997e-05 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.212 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:18.871624+0000", 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7741999999999997e-05 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:227653 48.3 48:c77e81f4:::benchmark_data_smithi119_267254_object104443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:41:22.002629+0000", 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.0287196070000002, 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.035790124, 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.213 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 227653 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.002629+0000", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.002629+0000", 2024-09-21T10:41:24.214 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.002631+0000", 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0470000000000001e-06 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.002637+0000", 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4130000000000003e-06 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.215 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.002668+0000", 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0148999999999999e-05 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.002676+0000", 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4919999999999993e-06 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.017833+0000", 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015156945 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.216 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.020574+0000", 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027407579999999998 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.020712+0000", 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013828500000000001 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.025124+0000", 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044121330000000004 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.217 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.038268+0000", 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013144296 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.038310+0000", 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1325000000000002e-05 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.038374+0000", 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4338000000000006e-05 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.218 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.038419+0000", 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4943e-05 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:228416 48.3 48:cb12f2d0:::benchmark_data_smithi119_267254_object105206:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1386)", 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:41:22.750710+0000", 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.280638532, 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.026942796000000001, 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:41:24.219 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 228416 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.750710+0000", 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.750710+0000", 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:41:24.220 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.750712+0000", 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.635e-06 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.750718+0000", 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8690000000000001e-06 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.750736+0000", 2024-09-21T10:41:24.221 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8023999999999999e-05 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.750744+0000", 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2179999999999999e-06 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.765089+0000", 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014345488 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.765235+0000", 2024-09-21T10:41:24.222 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000145687 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.765380+0000", 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000144924 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.772214+0000", 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068342519999999999 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.223 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.772499+0000", 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000284663 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.777318+0000", 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00481891 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.777538+0000", 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000219915 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:41:24.224 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:41:22.777653+0000", 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011521100000000001 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:41:24.225 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:41:24.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:24.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:24.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:24.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:24.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:24.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:24.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:24.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:24.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:24.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:24.797 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:24.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:41:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:41:24] "GET /metrics HTTP/1.1" 200 36857 "" "Prometheus/2.51.0" 2024-09-21T10:41:24.898 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:24.925 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:24.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:25.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:25.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:25.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:25.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:25.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:41:25.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:25.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:25.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:25.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:25.576 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:25.602 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:25.662 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:25 smithi119 ceph-mon[101432]: pgmap v2331: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 998 op/s; 762/57105 objects degraded (1.334%); 1317/57105 objects misplaced (2.306%); 1.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:25.662 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:25 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 762/57105 objects degraded (1.334%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:25.662 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:25 smithi119 ceph-mon[105288]: pgmap v2331: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 998 op/s; 762/57105 objects degraded (1.334%); 1317/57105 objects misplaced (2.306%); 1.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:25.662 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:25 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 762/57105 objects degraded (1.334%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:25.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:25.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:25.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:25 smithi145 ceph-mon[89359]: pgmap v2331: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.2 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 998 op/s; 762/57105 objects degraded (1.334%); 1317/57105 objects misplaced (2.306%); 1.5 MiB/s, 71 objects/s recovering 2024-09-21T10:41:25.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:25 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 762/57105 objects degraded (1.334%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:25.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:25.871 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:25.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:25.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:25.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:26.101 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:26.154 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:26.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:26.286 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:26.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:26.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:26.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:26.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:26.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:26.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:26.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:26.704 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:26.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:41:26.773 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:26.807 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:26.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:26.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:26.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:26.981 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:27.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:27.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:27.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:27.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:27.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:27.298 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:27.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:27.399 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:27.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:27.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:27.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:27.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:27.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:27.720 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:27 smithi119 ceph-mon[105288]: pgmap v2332: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.0 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 994 op/s; 348/50736 objects degraded (0.686%); 1014/50736 objects misplaced (1.999%); 0 B/s, 77 objects/s recovering 2024-09-21T10:41:27.721 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:27 smithi119 ceph-mon[101432]: pgmap v2332: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.0 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 994 op/s; 348/50736 objects degraded (0.686%); 1014/50736 objects misplaced (1.999%); 0 B/s, 77 objects/s recovering 2024-09-21T10:41:27.754 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:41:27.754 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:27.787 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:41:27.787 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:27 smithi145 ceph-mon[89359]: pgmap v2332: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.0 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 994 op/s; 348/50736 objects degraded (0.686%); 1014/50736 objects misplaced (1.999%); 0 B/s, 77 objects/s recovering 2024-09-21T10:41:27.816 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:41:27.816 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:27.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:27.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:27.843 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:41:27.844 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:27.871 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:41:27.871 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:27.892 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:41:27.892 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:27.913 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:41:27.914 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:27.935 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:41:27.935 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:27.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:27.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:27.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:27.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:28.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:28.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:28.199 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:28.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:28.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:28.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:28.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:28.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:28.569 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:28.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:28 smithi119 ceph-mon[105288]: osdmap e1387: 8 total, 8 up, 7 in 2024-09-21T10:41:28.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:28 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1694501108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:28.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:28 smithi119 ceph-mon[101432]: osdmap e1387: 8 total, 8 up, 7 in 2024-09-21T10:41:28.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:28 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1694501108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:28.597 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:28.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:28.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:28 smithi145 ceph-mon[89359]: osdmap e1387: 8 total, 8 up, 7 in 2024-09-21T10:41:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:28 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1694501108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:28.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:28.872 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:28.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:28.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:29.010 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:29.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:29.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:29.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:29.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:29.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:29.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:29.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:29.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:29.561 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:29.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:29.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:29.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:29 smithi119 ceph-mon[101432]: pgmap v2334: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 348/49878 objects degraded (0.698%); 1014/49878 objects misplaced (2.033%); 0 B/s, 51 objects/s recovering 2024-09-21T10:41:29.710 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:29 smithi119 ceph-mon[101432]: osdmap e1388: 8 total, 8 up, 7 in 2024-09-21T10:41:29.711 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:29 smithi119 ceph-mon[105288]: pgmap v2334: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 348/49878 objects degraded (0.698%); 1014/49878 objects misplaced (2.033%); 0 B/s, 51 objects/s recovering 2024-09-21T10:41:29.711 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:29 smithi119 ceph-mon[105288]: osdmap e1388: 8 total, 8 up, 7 in 2024-09-21T10:41:29.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:29 smithi145 ceph-mon[89359]: pgmap v2334: 109 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+undersized+degraded+remapped, 107 active+clean; 1.0 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 686 op/s; 348/49878 objects degraded (0.698%); 1014/49878 objects misplaced (2.033%); 0 B/s, 51 objects/s recovering 2024-09-21T10:41:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:29 smithi145 ceph-mon[89359]: osdmap e1388: 8 total, 8 up, 7 in 2024-09-21T10:41:29.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:29.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:29.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:30.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:30.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:30.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:30.162 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:30.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:30.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:30.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:30.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:30.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:41:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:41:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:41:30.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:30 smithi145 ceph-mon[89359]: Health check update: Degraded data redundancy: 348/49878 objects degraded (0.698%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:30.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:30.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:30.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:30.682 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:30.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:30.813 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:30.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:30 smithi119 ceph-mon[101432]: Health check update: Degraded data redundancy: 348/49878 objects degraded (0.698%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:30.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:30 smithi119 ceph-mon[105288]: Health check update: Degraded data redundancy: 348/49878 objects degraded (0.698%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-21T10:41:30.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:30.936 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:30.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:31.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:31.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:31.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:31.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:31.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:31.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:31.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:31.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:31.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:31.593 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:31.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:41:31.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:31.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:31.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:31.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:31 smithi119 ceph-mon[105288]: pgmap v2336: 109 pgs: 1 active+remapped+backfill_wait, 108 active+clean; 813 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 1014/39063 objects misplaced (2.596%); 0 B/s, 95 objects/s recovering 2024-09-21T10:41:31.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:31 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 348/49878 objects degraded (0.698%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:41:31.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:31 smithi119 ceph-mon[101432]: pgmap v2336: 109 pgs: 1 active+remapped+backfill_wait, 108 active+clean; 813 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 1014/39063 objects misplaced (2.596%); 0 B/s, 95 objects/s recovering 2024-09-21T10:41:31.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:31 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 348/49878 objects degraded (0.698%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:41:31.727 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:31 smithi145 ceph-mon[89359]: pgmap v2336: 109 pgs: 1 active+remapped+backfill_wait, 108 active+clean; 813 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 1.10k op/s; 1014/39063 objects misplaced (2.596%); 0 B/s, 95 objects/s recovering 2024-09-21T10:41:31.727 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:31 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 348/49878 objects degraded (0.698%), 1 pg degraded, 1 pg undersized) 2024-09-21T10:41:31.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:31.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:31.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:31.961 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:31.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:32.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:32.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:32.185 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:32.206 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:32.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:32.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:32.329 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:32.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:32.451 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:32.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:32.577 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:32.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:32.710 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:32.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:32.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:32 smithi145 ceph-mon[89359]: osdmap e1389: 8 total, 8 up, 7 in 2024-09-21T10:41:32.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:32.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:32.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:32 smithi119 ceph-mon[105288]: osdmap e1389: 8 total, 8 up, 7 in 2024-09-21T10:41:32.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:32 smithi119 ceph-mon[101432]: osdmap e1389: 8 total, 8 up, 7 in 2024-09-21T10:41:32.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:32.964 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:32.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:33.066 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:33.072 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:41:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:41:33.077 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:33.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:33.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:33.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:33.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:33.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:33.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:33.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:33.644 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:33.745 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:33 smithi145 ceph-mon[89359]: pgmap v2338: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 795 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 70/38205 objects misplaced (0.183%); 8.0 MiB/s, 185 objects/s recovering 2024-09-21T10:41:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:33 smithi145 ceph-mon[89359]: from='client.36891 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:33 smithi145 ceph-mon[89359]: osdmap e1390: 8 total, 8 up, 7 in 2024-09-21T10:41:33.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:33.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:33.796 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:41:33.803 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[101432]: pgmap v2338: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 795 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 70/38205 objects misplaced (0.183%); 8.0 MiB/s, 185 objects/s recovering 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[101432]: from='client.36891 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[101432]: osdmap e1390: 8 total, 8 up, 7 in 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:41:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[105288]: pgmap v2338: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 795 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 70/38205 objects misplaced (0.183%); 8.0 MiB/s, 185 objects/s recovering 2024-09-21T10:41:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[105288]: from='client.36891 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:41:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[105288]: osdmap e1390: 8 total, 8 up, 7 in 2024-09-21T10:41:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:33.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:41:33.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:33.940 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:33.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:34.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:34.089 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:34.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:34.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:34.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:34.328 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:34.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:34.456 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:34.483 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:34.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:34.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:34.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:34.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:34.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:34.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:34.869 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:41:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:41:34] "GET /metrics HTTP/1.1" 200 36855 "" "Prometheus/2.51.0" 2024-09-21T10:41:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:34.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:34.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:35.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:35.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:35.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:35.224 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:35.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:35.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:35.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:35.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:35.517 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:35.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:35.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:35.755 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:35.775 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:35 smithi119 ceph-mon[101432]: pgmap v2340: 109 pgs: 109 active+clean; 671 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 978 op/s; 8.7 MiB/s, 196 objects/s recovering 2024-09-21T10:41:35.775 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:35.775 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:35 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:35.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:35 smithi119 ceph-mon[105288]: pgmap v2340: 109 pgs: 109 active+clean; 671 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 978 op/s; 8.7 MiB/s, 196 objects/s recovering 2024-09-21T10:41:35.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:35.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:35 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:35 smithi145 ceph-mon[89359]: pgmap v2340: 109 pgs: 109 active+clean; 671 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 978 op/s; 8.7 MiB/s, 196 objects/s recovering 2024-09-21T10:41:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:35 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:35.797 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:35.899 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:35.928 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:35.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:36.029 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:36.069 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:36.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:36.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.205 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:36.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:36.290 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2338,"stamp":"2024-09-21T10:41:31.852706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":833318202,"num_objects":12735,"num_object_clones":0,"num_object_copies":38205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":70,"num_objects_unfound":0,"num_objects_dirty":12735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233770,"num_write_kb":7885280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6754,"num_bytes_recovered":358881967,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105890,"ondisk_log_size":105890,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":329,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8551972,"kb_used_data":3012820,"kb_used_omap":507,"kb_used_meta":5538436,"kb_avail":647664092,"statfs":{"total":671965249536,"available":663208030208,"internally_reserved":0,"allocated":3085127680,"data_stored":3042934927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":5671359007},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,1,9,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-273743872,"num_objects":-4177,"num_object_clones":0,"num_object_copies":-12531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-348,"num_objects_misplaced":-944,"num_objects_unfound":0,"num_objects_dirty":-4177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4177,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1113,"num_bytes_recovered":50135040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003156"},"pg_stats":[{"pgid":"48.f","version":"1388'14574","reported_seq":14654,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.808225+0000","last_change":"2024-09-21T10:37:50.818738+0000","last_active":"2024-09-21T10:41:28.808225+0000","last_peered":"2024-09-21T10:41:28.808225+0000","last_clean":"2024-09-21T10:41:28.808225+0000","last_became_active":"2024-09-21T10:37:50.818409+0000","last_became_peered":"2024-09-21T10:37:50.818409+0000","last_unstale":"2024-09-21T10:41:28.808225+0000","last_undegraded":"2024-09-21T10:41:28.808225+0000","last_fullsized":"2024-09-21T10:41:28.808225+0000","mapping_epoch":1354,"log_start":"1375'9000","ondisk_log_start":"1375'9000","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5574,"log_dups_size":0,"ondisk_log_size":5574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:24:13.373038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14574,"num_write_kb":490624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.e","version":"1388'14774","reported_seq":14850,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:29.456254+0000","last_change":"2024-09-21T10:37:50.816783+0000","last_active":"2024-09-21T10:41:29.456254+0000","last_peered":"2024-09-21T10:41:29.456254+0000","last_clean":"2024-09-21T10:41:29.456254+0000","last_became_active":"2024-09-21T10:37:50.816482+0000","last_became_peered":"2024-09-21T10:37:50.816482+0000","last_unstale":"2024-09-21T10:41:29.456254+0000","last_undegraded":"2024-09-21T10:41:29.456254+0000","last_fullsized":"2024-09-21T10:41:29.456254+0000","mapping_epoch":1354,"log_start":"1369'7000","ondisk_log_start":"1369'7000","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7774,"log_dups_size":0,"ondisk_log_size":7774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:57.011771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51904512,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14774,"num_write_kb":498112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.d","version":"1387'14688","reported_seq":14730,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.858456+0000","last_change":"2024-09-21T10:37:50.816596+0000","last_active":"2024-09-21T10:41:27.858456+0000","last_peered":"2024-09-21T10:41:27.858456+0000","last_clean":"2024-09-21T10:41:27.858456+0000","last_became_active":"2024-09-21T10:37:50.816060+0000","last_became_peered":"2024-09-21T10:37:50.816060+0000","last_unstale":"2024-09-21T10:41:27.858456+0000","last_undegraded":"2024-09-21T10:41:27.858456+0000","last_fullsized":"2024-09-21T10:41:27.858456+0000","mapping_epoch":1354,"log_start":"1374'8974","ondisk_log_start":"1374'8974","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:31:09.093088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52363288,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":800,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14688,"num_write_kb":495553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.c","version":"1388'14798","reported_seq":14876,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.807815+0000","last_change":"2024-09-21T10:37:50.814694+0000","last_active":"2024-09-21T10:41:28.807815+0000","last_peered":"2024-09-21T10:41:28.807815+0000","last_clean":"2024-09-21T10:41:28.807815+0000","last_became_active":"2024-09-21T10:37:50.814424+0000","last_became_peered":"2024-09-21T10:37:50.814424+0000","last_unstale":"2024-09-21T10:41:28.807815+0000","last_undegraded":"2024-09-21T10:41:28.807815+0000","last_fullsized":"2024-09-21T10:41:28.807815+0000","mapping_epoch":1354,"log_start":"1375'9200","ondisk_log_start":"1375'9200","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5598,"log_dups_size":0,"ondisk_log_size":5598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:22:22.559518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51380224,"num_objects":784,"num_object_clones":0,"num_object_copies":2352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14798,"num_write_kb":498624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.b","version":"1388'14473","reported_seq":14515,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.821481+0000","last_change":"2024-09-21T10:37:50.809808+0000","last_active":"2024-09-21T10:41:28.821481+0000","last_peered":"2024-09-21T10:41:28.821481+0000","last_clean":"2024-09-21T10:41:28.821481+0000","last_became_active":"2024-09-21T10:37:50.809451+0000","last_became_peered":"2024-09-21T10:37:50.809451+0000","last_unstale":"2024-09-21T10:41:28.821481+0000","last_undegraded":"2024-09-21T10:41:28.821481+0000","last_fullsized":"2024-09-21T10:41:28.821481+0000","mapping_epoch":1354,"log_start":"1373'8200","ondisk_log_start":"1373'8200","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6273,"log_dups_size":0,"ondisk_log_size":6273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:32:46.455275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14473,"num_write_kb":489216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.a","version":"1388'14940","reported_seq":15020,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:29.484112+0000","last_change":"2024-09-21T10:37:50.806280+0000","last_active":"2024-09-21T10:41:29.484112+0000","last_peered":"2024-09-21T10:41:29.484112+0000","last_clean":"2024-09-21T10:41:29.484112+0000","last_became_active":"2024-09-21T10:37:50.805997+0000","last_became_peered":"2024-09-21T10:37:50.805997+0000","last_unstale":"2024-09-21T10:41:29.484112+0000","last_undegraded":"2024-09-21T10:41:29.484112+0000","last_fullsized":"2024-09-21T10:41:29.484112+0000","mapping_epoch":1354,"log_start":"1370'7200","ondisk_log_start":"1370'7200","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7740,"log_dups_size":0,"ondisk_log_size":7740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:09.061122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14940,"num_write_kb":502976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.9","version":"1388'14273","reported_seq":14353,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.797198+0000","last_change":"2024-09-21T10:37:50.814434+0000","last_active":"2024-09-21T10:41:28.797198+0000","last_peered":"2024-09-21T10:41:28.797198+0000","last_clean":"2024-09-21T10:41:28.797198+0000","last_became_active":"2024-09-21T10:37:50.814150+0000","last_became_peered":"2024-09-21T10:37:50.814150+0000","last_unstale":"2024-09-21T10:41:28.797198+0000","last_undegraded":"2024-09-21T10:41:28.797198+0000","last_fullsized":"2024-09-21T10:41:28.797198+0000","mapping_epoch":1354,"log_start":"1374'8700","ondisk_log_start":"1374'8700","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5573,"log_dups_size":0,"ondisk_log_size":5573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:18.301417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14273,"num_write_kb":480448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.8","version":"1388'14486","reported_seq":14528,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.643727+0000","last_change":"2024-09-21T10:37:50.812704+0000","last_active":"2024-09-21T10:41:28.643727+0000","last_peered":"2024-09-21T10:41:28.643727+0000","last_clean":"2024-09-21T10:41:28.643727+0000","last_became_active":"2024-09-21T10:37:50.812368+0000","last_became_peered":"2024-09-21T10:37:50.812368+0000","last_unstale":"2024-09-21T10:41:28.643727+0000","last_undegraded":"2024-09-21T10:41:28.643727+0000","last_fullsized":"2024-09-21T10:41:28.643727+0000","mapping_epoch":1354,"log_start":"1369'6700","ondisk_log_start":"1369'6700","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7786,"log_dups_size":0,"ondisk_log_size":7786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:28:22.704564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55836672,"num_objects":852,"num_object_clones":0,"num_object_copies":2556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14486,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"48.7","version":"1388'15050","reported_seq":16665,"reported_epoch":1388,"state":"active+remapped+backfilling","last_fresh":"2024-09-21T10:41:30.678556+0000","last_change":"2024-09-21T10:41:27.234121+0000","last_active":"2024-09-21T10:41:30.678556+0000","last_peered":"2024-09-21T10:41:30.678556+0000","last_clean":"2024-09-21T10:40:09.890233+0000","last_became_active":"2024-09-21T10:40:12.242243+0000","last_became_peered":"2024-09-21T10:40:12.242243+0000","last_unstale":"2024-09-21T10:41:30.678556+0000","last_undegraded":"2024-09-21T10:41:30.678556+0000","last_fullsized":"2024-09-21T10:41:30.678556+0000","mapping_epoch":1379,"log_start":"1375'9300","ondisk_log_start":"1375'9300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5750,"log_dups_size":0,"ondisk_log_size":5750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":70,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15050,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":765,"num_bytes_recovered":50135040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":728}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[]},{"pgid":"48.6","version":"1388'14599","reported_seq":14679,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.800618+0000","last_change":"2024-09-21T10:37:50.811591+0000","last_active":"2024-09-21T10:41:28.800618+0000","last_peered":"2024-09-21T10:41:28.800618+0000","last_clean":"2024-09-21T10:41:28.800618+0000","last_became_active":"2024-09-21T10:37:50.811260+0000","last_became_peered":"2024-09-21T10:37:50.811260+0000","last_unstale":"2024-09-21T10:41:28.800618+0000","last_undegraded":"2024-09-21T10:41:28.800618+0000","last_fullsized":"2024-09-21T10:41:28.800618+0000","mapping_epoch":1354,"log_start":"1375'9000","ondisk_log_start":"1375'9000","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:59:19.966736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":767,"num_object_clones":0,"num_object_copies":2301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":491712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"48.5","version":"1387'14461","reported_seq":14502,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.923396+0000","last_change":"2024-09-21T10:37:50.816067+0000","last_active":"2024-09-21T10:41:27.923396+0000","last_peered":"2024-09-21T10:41:27.923396+0000","last_clean":"2024-09-21T10:41:27.923396+0000","last_became_active":"2024-09-21T10:37:50.815783+0000","last_became_peered":"2024-09-21T10:37:50.815783+0000","last_unstale":"2024-09-21T10:41:27.923396+0000","last_undegraded":"2024-09-21T10:41:27.923396+0000","last_fullsized":"2024-09-21T10:41:27.923396+0000","mapping_epoch":1354,"log_start":"1373'8710","ondisk_log_start":"1373'8710","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5751,"log_dups_size":0,"ondisk_log_size":5751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:46:03.096851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57475072,"num_objects":877,"num_object_clones":0,"num_object_copies":2631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":877,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14461,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.4","version":"1388'14538","reported_seq":14618,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:29.505121+0000","last_change":"2024-09-21T10:37:50.810595+0000","last_active":"2024-09-21T10:41:29.505121+0000","last_peered":"2024-09-21T10:41:29.505121+0000","last_clean":"2024-09-21T10:41:29.505121+0000","last_became_active":"2024-09-21T10:37:50.810178+0000","last_became_peered":"2024-09-21T10:37:50.810178+0000","last_unstale":"2024-09-21T10:41:29.505121+0000","last_undegraded":"2024-09-21T10:41:29.505121+0000","last_fullsized":"2024-09-21T10:41:29.505121+0000","mapping_epoch":1354,"log_start":"1369'6800","ondisk_log_start":"1369'6800","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7738,"log_dups_size":0,"ondisk_log_size":7738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:30.334590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50200576,"num_objects":766,"num_object_clones":0,"num_object_copies":2298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14538,"num_write_kb":489728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"48.3","version":"1387'14364","reported_seq":14442,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.590015+0000","last_change":"2024-09-21T10:37:50.808064+0000","last_active":"2024-09-21T10:41:27.590015+0000","last_peered":"2024-09-21T10:41:27.590015+0000","last_clean":"2024-09-21T10:41:27.590015+0000","last_became_active":"2024-09-21T10:37:50.807674+0000","last_became_peered":"2024-09-21T10:37:50.807674+0000","last_unstale":"2024-09-21T10:41:27.590015+0000","last_undegraded":"2024-09-21T10:41:27.590015+0000","last_fullsized":"2024-09-21T10:41:27.590015+0000","mapping_epoch":1354,"log_start":"1370'7300","ondisk_log_start":"1370'7300","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7064,"log_dups_size":0,"ondisk_log_size":7064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:21.766427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14364,"num_write_kb":486720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"48.2","version":"1387'14922","reported_seq":14963,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.887587+0000","last_change":"2024-09-21T10:37:50.816843+0000","last_active":"2024-09-21T10:41:27.887587+0000","last_peered":"2024-09-21T10:41:27.887587+0000","last_clean":"2024-09-21T10:41:27.887587+0000","last_became_active":"2024-09-21T10:37:50.816490+0000","last_became_peered":"2024-09-21T10:37:50.816490+0000","last_unstale":"2024-09-21T10:41:27.887587+0000","last_undegraded":"2024-09-21T10:41:27.887587+0000","last_fullsized":"2024-09-21T10:41:27.887587+0000","mapping_epoch":1354,"log_start":"1375'9191","ondisk_log_start":"1375'9191","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":5731,"log_dups_size":0,"ondisk_log_size":5731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:01:33.243791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14922,"num_write_kb":504576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"48.1","version":"1388'14380","reported_seq":26310,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.839348+0000","last_change":"2024-09-21T10:41:28.498598+0000","last_active":"2024-09-21T10:41:28.839348+0000","last_peered":"2024-09-21T10:41:28.839348+0000","last_clean":"2024-09-21T10:41:28.839348+0000","last_became_active":"2024-09-21T10:41:28.430059+0000","last_became_peered":"2024-09-21T10:41:28.430059+0000","last_unstale":"2024-09-21T10:41:28.839348+0000","last_undegraded":"2024-09-21T10:41:28.839348+0000","last_fullsized":"2024-09-21T10:41:28.839348+0000","mapping_epoch":1387,"log_start":"1372'8100","ondisk_log_start":"1372'8100","created":1354,"last_epoch_clean":1388,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":6280,"log_dups_size":0,"ondisk_log_size":6280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:13:26.602594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14380,"num_write_kb":485440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5933,"num_bytes_recovered":307822592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"48.0","version":"1388'14387","reported_seq":14467,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:29.515921+0000","last_change":"2024-09-21T10:37:50.812734+0000","last_active":"2024-09-21T10:41:29.515921+0000","last_peered":"2024-09-21T10:41:29.515921+0000","last_clean":"2024-09-21T10:41:29.515921+0000","last_became_active":"2024-09-21T10:37:50.812438+0000","last_became_peered":"2024-09-21T10:37:50.812438+0000","last_unstale":"2024-09-21T10:41:29.515921+0000","last_undegraded":"2024-09-21T10:41:29.515921+0000","last_fullsized":"2024-09-21T10:41:29.515921+0000","mapping_epoch":1354,"log_start":"1369'6600","ondisk_log_start":"1369'6600","created":1354,"last_epoch_clean":1355,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:37:49.780560+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.780560+0000","objects_scrubbed":0,"log_size":7787,"log_dups_size":0,"ondisk_log_size":7787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:25.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50397184,"num_objects":769,"num_object_clones":0,"num_object_copies":2307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14387,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":59,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.435696+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:41:27.435696+0000","last_peered":"2024-09-21T10:41:27.435696+0000","last_clean":"2024-09-21T10:41:27.435696+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:41:27.435696+0000","last_undegraded":"2024-09-21T10:41:27.435696+0000","last_fullsized":"2024-09-21T10:41:27.435696+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1281,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.373579+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:41:21.373579+0000","last_peered":"2024-09-21T10:41:21.373579+0000","last_clean":"2024-09-21T10:41:21.373579+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:41:21.373579+0000","last_undegraded":"2024-09-21T10:41:21.373579+0000","last_fullsized":"2024-09-21T10:41:21.373579+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2098,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.504687+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:41:28.504687+0000","last_peered":"2024-09-21T10:41:28.504687+0000","last_clean":"2024-09-21T10:41:28.504687+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:41:28.504687+0000","last_undegraded":"2024-09-21T10:41:28.504687+0000","last_fullsized":"2024-09-21T10:41:28.504687+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2116,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.500829+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:41:28.500829+0000","last_peered":"2024-09-21T10:41:28.500829+0000","last_clean":"2024-09-21T10:41:28.500829+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:41:28.500829+0000","last_undegraded":"2024-09-21T10:41:28.500829+0000","last_fullsized":"2024-09-21T10:41:28.500829+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":156,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432144+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:41:27.432144+0000","last_peered":"2024-09-21T10:41:27.432144+0000","last_clean":"2024-09-21T10:41:27.432144+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:41:27.432144+0000","last_undegraded":"2024-09-21T10:41:27.432144+0000","last_fullsized":"2024-09-21T10:41:27.432144+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1300,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.500768+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:41:28.500768+0000","last_peered":"2024-09-21T10:41:28.500768+0000","last_clean":"2024-09-21T10:41:28.500768+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:41:28.500768+0000","last_undegraded":"2024-09-21T10:41:28.500768+0000","last_fullsized":"2024-09-21T10:41:28.500768+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":156,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431954+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:41:27.431954+0000","last_peered":"2024-09-21T10:41:27.431954+0000","last_clean":"2024-09-21T10:41:27.431954+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:41:27.431954+0000","last_undegraded":"2024-09-21T10:41:27.431954+0000","last_fullsized":"2024-09-21T10:41:27.431954+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1387,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.504324+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:41:28.504324+0000","last_peered":"2024-09-21T10:41:28.504324+0000","last_clean":"2024-09-21T10:41:28.504324+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:41:28.504324+0000","last_undegraded":"2024-09-21T10:41:28.504324+0000","last_fullsized":"2024-09-21T10:41:28.504324+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":52,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.436102+0000","last_change":"2024-09-21T10:39:00.847895+0000","last_active":"2024-09-21T10:41:27.436102+0000","last_peered":"2024-09-21T10:41:27.436102+0000","last_clean":"2024-09-21T10:41:27.436102+0000","last_became_active":"2024-09-21T10:39:00.847533+0000","last_became_peered":"2024-09-21T10:39:00.847533+0000","last_unstale":"2024-09-21T10:41:27.436102+0000","last_undegraded":"2024-09-21T10:41:27.436102+0000","last_fullsized":"2024-09-21T10:41:27.436102+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:55.080913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":83,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.422572+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:41:28.422572+0000","last_peered":"2024-09-21T10:41:28.422572+0000","last_clean":"2024-09-21T10:41:28.422572+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:41:28.422572+0000","last_undegraded":"2024-09-21T10:41:28.422572+0000","last_fullsized":"2024-09-21T10:41:28.422572+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":160,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431118+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:41:28.431118+0000","last_peered":"2024-09-21T10:41:28.431118+0000","last_clean":"2024-09-21T10:41:28.431118+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:41:28.431118+0000","last_undegraded":"2024-09-21T10:41:28.431118+0000","last_fullsized":"2024-09-21T10:41:28.431118+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2105,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501551+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:41:28.501551+0000","last_peered":"2024-09-21T10:41:28.501551+0000","last_clean":"2024-09-21T10:41:28.501551+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:41:28.501551+0000","last_undegraded":"2024-09-21T10:41:28.501551+0000","last_fullsized":"2024-09-21T10:41:28.501551+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2204,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.430919+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:41:28.430919+0000","last_peered":"2024-09-21T10:41:28.430919+0000","last_clean":"2024-09-21T10:41:28.430919+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:41:28.430919+0000","last_undegraded":"2024-09-21T10:41:28.430919+0000","last_fullsized":"2024-09-21T10:41:28.430919+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2211,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.435607+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:41:27.435607+0000","last_peered":"2024-09-21T10:41:27.435607+0000","last_clean":"2024-09-21T10:41:27.435607+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:41:27.435607+0000","last_undegraded":"2024-09-21T10:41:27.435607+0000","last_fullsized":"2024-09-21T10:41:27.435607+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2131,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.500722+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:41:28.500722+0000","last_peered":"2024-09-21T10:41:28.500722+0000","last_clean":"2024-09-21T10:41:28.500722+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:41:28.500722+0000","last_undegraded":"2024-09-21T10:41:28.500722+0000","last_fullsized":"2024-09-21T10:41:28.500722+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1345,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374829+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:41:21.374829+0000","last_peered":"2024-09-21T10:41:21.374829+0000","last_clean":"2024-09-21T10:41:21.374829+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:41:21.374829+0000","last_undegraded":"2024-09-21T10:41:21.374829+0000","last_fullsized":"2024-09-21T10:41:21.374829+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2110,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.500653+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:41:28.500653+0000","last_peered":"2024-09-21T10:41:28.500653+0000","last_clean":"2024-09-21T10:41:28.500653+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:41:28.500653+0000","last_undegraded":"2024-09-21T10:41:28.500653+0000","last_fullsized":"2024-09-21T10:41:28.500653+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2221,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.435898+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:41:27.435898+0000","last_peered":"2024-09-21T10:41:27.435898+0000","last_clean":"2024-09-21T10:41:27.435898+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:41:27.435898+0000","last_undegraded":"2024-09-21T10:41:27.435898+0000","last_fullsized":"2024-09-21T10:41:27.435898+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":240,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.422671+0000","last_change":"2024-09-21T10:37:47.788643+0000","last_active":"2024-09-21T10:41:28.422671+0000","last_peered":"2024-09-21T10:41:28.422671+0000","last_clean":"2024-09-21T10:41:28.422671+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:41:28.422671+0000","last_undegraded":"2024-09-21T10:41:28.422671+0000","last_fullsized":"2024-09-21T10:41:28.422671+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:39.188916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2093,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501056+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:41:28.501056+0000","last_peered":"2024-09-21T10:41:28.501056+0000","last_clean":"2024-09-21T10:41:28.501056+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:41:28.501056+0000","last_undegraded":"2024-09-21T10:41:28.501056+0000","last_fullsized":"2024-09-21T10:41:28.501056+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":138,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431649+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:41:27.431649+0000","last_peered":"2024-09-21T10:41:27.431649+0000","last_clean":"2024-09-21T10:41:27.431649+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:41:27.431649+0000","last_undegraded":"2024-09-21T10:41:27.431649+0000","last_fullsized":"2024-09-21T10:41:27.431649+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1754,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431456+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:41:28.431456+0000","last_peered":"2024-09-21T10:41:28.431456+0000","last_clean":"2024-09-21T10:41:28.431456+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:41:28.431456+0000","last_undegraded":"2024-09-21T10:41:28.431456+0000","last_fullsized":"2024-09-21T10:41:28.431456+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2196,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431313+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:41:28.431313+0000","last_peered":"2024-09-21T10:41:28.431313+0000","last_clean":"2024-09-21T10:41:28.431313+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:41:28.431313+0000","last_undegraded":"2024-09-21T10:41:28.431313+0000","last_fullsized":"2024-09-21T10:41:28.431313+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":403,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.423579+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:41:27.423579+0000","last_peered":"2024-09-21T10:41:27.423579+0000","last_clean":"2024-09-21T10:41:27.423579+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:41:27.423579+0000","last_undegraded":"2024-09-21T10:41:27.423579+0000","last_fullsized":"2024-09-21T10:41:27.423579+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1246,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374163+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:41:21.374163+0000","last_peered":"2024-09-21T10:41:21.374163+0000","last_clean":"2024-09-21T10:41:21.374163+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:41:21.374163+0000","last_undegraded":"2024-09-21T10:41:21.374163+0000","last_fullsized":"2024-09-21T10:41:21.374163+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":353,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501438+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:41:28.501438+0000","last_peered":"2024-09-21T10:41:28.501438+0000","last_clean":"2024-09-21T10:41:28.501438+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:41:28.501438+0000","last_undegraded":"2024-09-21T10:41:28.501438+0000","last_fullsized":"2024-09-21T10:41:28.501438+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1281,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432605+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:41:27.432605+0000","last_peered":"2024-09-21T10:41:27.432605+0000","last_clean":"2024-09-21T10:41:27.432605+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:41:27.432605+0000","last_undegraded":"2024-09-21T10:41:27.432605+0000","last_fullsized":"2024-09-21T10:41:27.432605+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1341,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374682+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:41:21.374682+0000","last_peered":"2024-09-21T10:41:21.374682+0000","last_clean":"2024-09-21T10:41:21.374682+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:41:21.374682+0000","last_undegraded":"2024-09-21T10:41:21.374682+0000","last_fullsized":"2024-09-21T10:41:21.374682+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":85,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.435402+0000","last_change":"2024-09-21T10:37:41.926686+0000","last_active":"2024-09-21T10:41:27.435402+0000","last_peered":"2024-09-21T10:41:27.435402+0000","last_clean":"2024-09-21T10:41:27.435402+0000","last_became_active":"2024-09-21T10:37:41.922706+0000","last_became_peered":"2024-09-21T10:37:41.922706+0000","last_unstale":"2024-09-21T10:41:27.435402+0000","last_undegraded":"2024-09-21T10:41:27.435402+0000","last_fullsized":"2024-09-21T10:41:27.435402+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:08.955843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":142,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.422655+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:41:28.422655+0000","last_peered":"2024-09-21T10:41:28.422655+0000","last_clean":"2024-09-21T10:41:28.422655+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:41:28.422655+0000","last_undegraded":"2024-09-21T10:41:28.422655+0000","last_fullsized":"2024-09-21T10:41:28.422655+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":44,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431346+0000","last_change":"2024-09-21T10:39:00.847173+0000","last_active":"2024-09-21T10:41:27.431346+0000","last_peered":"2024-09-21T10:41:27.431346+0000","last_clean":"2024-09-21T10:41:27.431346+0000","last_became_active":"2024-09-21T10:39:00.846851+0000","last_became_peered":"2024-09-21T10:39:00.846851+0000","last_unstale":"2024-09-21T10:41:27.431346+0000","last_undegraded":"2024-09-21T10:41:27.431346+0000","last_fullsized":"2024-09-21T10:41:27.431346+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:47.133721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2214,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431599+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:41:28.431599+0000","last_peered":"2024-09-21T10:41:28.431599+0000","last_clean":"2024-09-21T10:41:28.431599+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:41:28.431599+0000","last_undegraded":"2024-09-21T10:41:28.431599+0000","last_fullsized":"2024-09-21T10:41:28.431599+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2093,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501977+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:41:28.501977+0000","last_peered":"2024-09-21T10:41:28.501977+0000","last_clean":"2024-09-21T10:41:28.501977+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:41:28.501977+0000","last_undegraded":"2024-09-21T10:41:28.501977+0000","last_fullsized":"2024-09-21T10:41:28.501977+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":512,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.373958+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:41:21.373958+0000","last_peered":"2024-09-21T10:41:21.373958+0000","last_clean":"2024-09-21T10:41:21.373958+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:41:21.373958+0000","last_undegraded":"2024-09-21T10:41:21.373958+0000","last_fullsized":"2024-09-21T10:41:21.373958+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2213,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.437054+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:41:27.437054+0000","last_peered":"2024-09-21T10:41:27.437054+0000","last_clean":"2024-09-21T10:41:27.437054+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:41:27.437054+0000","last_undegraded":"2024-09-21T10:41:27.437054+0000","last_fullsized":"2024-09-21T10:41:27.437054+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1405,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.424259+0000","last_change":"2024-09-21T10:37:41.922996+0000","last_active":"2024-09-21T10:41:27.424259+0000","last_peered":"2024-09-21T10:41:27.424259+0000","last_clean":"2024-09-21T10:41:27.424259+0000","last_became_active":"2024-09-21T10:37:41.922703+0000","last_became_peered":"2024-09-21T10:37:41.922703+0000","last_unstale":"2024-09-21T10:41:27.424259+0000","last_undegraded":"2024-09-21T10:41:27.424259+0000","last_fullsized":"2024-09-21T10:41:27.424259+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:23.614770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2201,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431680+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:41:28.431680+0000","last_peered":"2024-09-21T10:41:28.431680+0000","last_clean":"2024-09-21T10:41:28.431680+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:41:28.431680+0000","last_undegraded":"2024-09-21T10:41:28.431680+0000","last_fullsized":"2024-09-21T10:41:28.431680+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":512,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374582+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:41:21.374582+0000","last_peered":"2024-09-21T10:41:21.374582+0000","last_clean":"2024-09-21T10:41:21.374582+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:41:21.374582+0000","last_undegraded":"2024-09-21T10:41:21.374582+0000","last_fullsized":"2024-09-21T10:41:21.374582+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2206,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431455+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:41:28.431455+0000","last_peered":"2024-09-21T10:41:28.431455+0000","last_clean":"2024-09-21T10:41:28.431455+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:41:28.431455+0000","last_undegraded":"2024-09-21T10:41:28.431455+0000","last_fullsized":"2024-09-21T10:41:28.431455+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2201,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431763+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:41:28.431763+0000","last_peered":"2024-09-21T10:41:28.431763+0000","last_clean":"2024-09-21T10:41:28.431763+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:41:28.431763+0000","last_undegraded":"2024-09-21T10:41:28.431763+0000","last_fullsized":"2024-09-21T10:41:28.431763+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":138,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432001+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:41:27.432001+0000","last_peered":"2024-09-21T10:41:27.432001+0000","last_clean":"2024-09-21T10:41:27.432001+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:41:27.432001+0000","last_undegraded":"2024-09-21T10:41:27.432001+0000","last_fullsized":"2024-09-21T10:41:27.432001+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":148,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432679+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:41:27.432679+0000","last_peered":"2024-09-21T10:41:27.432679+0000","last_clean":"2024-09-21T10:41:27.432679+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:41:27.432679+0000","last_undegraded":"2024-09-21T10:41:27.432679+0000","last_fullsized":"2024-09-21T10:41:27.432679+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2198,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431743+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:41:28.431743+0000","last_peered":"2024-09-21T10:41:28.431743+0000","last_clean":"2024-09-21T10:41:28.431743+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:41:28.431743+0000","last_undegraded":"2024-09-21T10:41:28.431743+0000","last_fullsized":"2024-09-21T10:41:28.431743+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1316,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.436965+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:41:27.436965+0000","last_peered":"2024-09-21T10:41:27.436965+0000","last_clean":"2024-09-21T10:41:27.436965+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:41:27.436965+0000","last_undegraded":"2024-09-21T10:41:27.436965+0000","last_fullsized":"2024-09-21T10:41:27.436965+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":142,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432038+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:41:27.432038+0000","last_peered":"2024-09-21T10:41:27.432038+0000","last_clean":"2024-09-21T10:41:27.432038+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:41:27.432038+0000","last_undegraded":"2024-09-21T10:41:27.432038+0000","last_fullsized":"2024-09-21T10:41:27.432038+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":519,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.436065+0000","last_change":"2024-09-21T10:37:38.606346+0000","last_active":"2024-09-21T10:41:27.436065+0000","last_peered":"2024-09-21T10:41:27.436065+0000","last_clean":"2024-09-21T10:41:27.436065+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:41:27.436065+0000","last_undegraded":"2024-09-21T10:41:27.436065+0000","last_fullsized":"2024-09-21T10:41:27.436065+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:16.461221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2217,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431858+0000","last_change":"2024-09-21T10:37:41.924374+0000","last_active":"2024-09-21T10:41:28.431858+0000","last_peered":"2024-09-21T10:41:28.431858+0000","last_clean":"2024-09-21T10:41:28.431858+0000","last_became_active":"2024-09-21T10:37:41.922589+0000","last_became_peered":"2024-09-21T10:37:41.922589+0000","last_unstale":"2024-09-21T10:41:28.431858+0000","last_undegraded":"2024-09-21T10:41:28.431858+0000","last_fullsized":"2024-09-21T10:41:28.431858+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:51.987384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2199,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.430916+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:41:28.430916+0000","last_peered":"2024-09-21T10:41:28.430916+0000","last_clean":"2024-09-21T10:41:28.430916+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:41:28.430916+0000","last_undegraded":"2024-09-21T10:41:28.430916+0000","last_fullsized":"2024-09-21T10:41:28.430916+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2130,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.422420+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:41:28.422420+0000","last_peered":"2024-09-21T10:41:28.422420+0000","last_clean":"2024-09-21T10:41:28.422420+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:41:28.422420+0000","last_undegraded":"2024-09-21T10:41:28.422420+0000","last_fullsized":"2024-09-21T10:41:28.422420+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":384,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431363+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:41:28.431363+0000","last_peered":"2024-09-21T10:41:28.431363+0000","last_clean":"2024-09-21T10:41:28.431363+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:41:28.431363+0000","last_undegraded":"2024-09-21T10:41:28.431363+0000","last_fullsized":"2024-09-21T10:41:28.431363+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":53,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.435264+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:41:27.435264+0000","last_peered":"2024-09-21T10:41:27.435264+0000","last_clean":"2024-09-21T10:41:27.435264+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:41:27.435264+0000","last_undegraded":"2024-09-21T10:41:27.435264+0000","last_fullsized":"2024-09-21T10:41:27.435264+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2228,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.439633+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:41:27.439633+0000","last_peered":"2024-09-21T10:41:27.439633+0000","last_clean":"2024-09-21T10:41:27.439633+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:41:27.439633+0000","last_undegraded":"2024-09-21T10:41:27.439633+0000","last_fullsized":"2024-09-21T10:41:27.439633+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1375,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.423124+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:41:27.423124+0000","last_peered":"2024-09-21T10:41:27.423124+0000","last_clean":"2024-09-21T10:41:27.423124+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:41:27.423124+0000","last_undegraded":"2024-09-21T10:41:27.423124+0000","last_fullsized":"2024-09-21T10:41:27.423124+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1347,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374672+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:41:21.374672+0000","last_peered":"2024-09-21T10:41:21.374672+0000","last_clean":"2024-09-21T10:41:21.374672+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:41:21.374672+0000","last_undegraded":"2024-09-21T10:41:21.374672+0000","last_fullsized":"2024-09-21T10:41:21.374672+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2115,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.500862+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:41:28.500862+0000","last_peered":"2024-09-21T10:41:28.500862+0000","last_clean":"2024-09-21T10:41:28.500862+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:41:28.500862+0000","last_undegraded":"2024-09-21T10:41:28.500862+0000","last_fullsized":"2024-09-21T10:41:28.500862+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":142,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431755+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:41:27.431755+0000","last_peered":"2024-09-21T10:41:27.431755+0000","last_clean":"2024-09-21T10:41:27.431755+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:41:27.431755+0000","last_undegraded":"2024-09-21T10:41:27.431755+0000","last_fullsized":"2024-09-21T10:41:27.431755+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1558,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.422947+0000","last_change":"2024-09-21T10:37:41.922049+0000","last_active":"2024-09-21T10:41:27.422947+0000","last_peered":"2024-09-21T10:41:27.422947+0000","last_clean":"2024-09-21T10:41:27.422947+0000","last_became_active":"2024-09-21T10:37:41.921709+0000","last_became_peered":"2024-09-21T10:37:41.921709+0000","last_unstale":"2024-09-21T10:41:27.422947+0000","last_undegraded":"2024-09-21T10:41:27.422947+0000","last_fullsized":"2024-09-21T10:41:27.422947+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:12.446157+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":138,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432114+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:41:27.432114+0000","last_peered":"2024-09-21T10:41:27.432114+0000","last_clean":"2024-09-21T10:41:27.432114+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:41:27.432114+0000","last_undegraded":"2024-09-21T10:41:27.432114+0000","last_fullsized":"2024-09-21T10:41:27.432114+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2204,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431839+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:41:28.431839+0000","last_peered":"2024-09-21T10:41:28.431839+0000","last_clean":"2024-09-21T10:41:28.431839+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:41:28.431839+0000","last_undegraded":"2024-09-21T10:41:28.431839+0000","last_fullsized":"2024-09-21T10:41:28.431839+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2102,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.500600+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:41:28.500600+0000","last_peered":"2024-09-21T10:41:28.500600+0000","last_clean":"2024-09-21T10:41:28.500600+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:41:28.500600+0000","last_undegraded":"2024-09-21T10:41:28.500600+0000","last_fullsized":"2024-09-21T10:41:28.500600+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":368,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431242+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:41:28.431242+0000","last_peered":"2024-09-21T10:41:28.431242+0000","last_clean":"2024-09-21T10:41:28.431242+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:41:28.431242+0000","last_undegraded":"2024-09-21T10:41:28.431242+0000","last_fullsized":"2024-09-21T10:41:28.431242+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1246,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374038+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:41:21.374038+0000","last_peered":"2024-09-21T10:41:21.374038+0000","last_clean":"2024-09-21T10:41:21.374038+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:41:21.374038+0000","last_undegraded":"2024-09-21T10:41:21.374038+0000","last_fullsized":"2024-09-21T10:41:21.374038+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1257,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.422435+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:41:28.422435+0000","last_peered":"2024-09-21T10:41:28.422435+0000","last_clean":"2024-09-21T10:41:28.422435+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:41:28.422435+0000","last_undegraded":"2024-09-21T10:41:28.422435+0000","last_fullsized":"2024-09-21T10:41:28.422435+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":142,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431608+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:41:27.431608+0000","last_peered":"2024-09-21T10:41:27.431608+0000","last_clean":"2024-09-21T10:41:27.431608+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:41:27.431608+0000","last_undegraded":"2024-09-21T10:41:27.431608+0000","last_fullsized":"2024-09-21T10:41:27.431608+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1265,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.422815+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:41:28.422815+0000","last_peered":"2024-09-21T10:41:28.422815+0000","last_clean":"2024-09-21T10:41:28.422815+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:41:28.422815+0000","last_undegraded":"2024-09-21T10:41:28.422815+0000","last_fullsized":"2024-09-21T10:41:28.422815+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1353,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374209+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:41:21.374209+0000","last_peered":"2024-09-21T10:41:21.374209+0000","last_clean":"2024-09-21T10:41:21.374209+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:41:21.374209+0000","last_undegraded":"2024-09-21T10:41:21.374209+0000","last_fullsized":"2024-09-21T10:41:21.374209+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1509,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.424618+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:41:27.424618+0000","last_peered":"2024-09-21T10:41:27.424618+0000","last_clean":"2024-09-21T10:41:27.424618+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:41:27.424618+0000","last_undegraded":"2024-09-21T10:41:27.424618+0000","last_fullsized":"2024-09-21T10:41:27.424618+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1317,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501820+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:41:28.501820+0000","last_peered":"2024-09-21T10:41:28.501820+0000","last_clean":"2024-09-21T10:41:28.501820+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:41:28.501820+0000","last_undegraded":"2024-09-21T10:41:28.501820+0000","last_fullsized":"2024-09-21T10:41:28.501820+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":329,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501590+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:41:28.501590+0000","last_peered":"2024-09-21T10:41:28.501590+0000","last_clean":"2024-09-21T10:41:28.501590+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:41:28.501590+0000","last_undegraded":"2024-09-21T10:41:28.501590+0000","last_fullsized":"2024-09-21T10:41:28.501590+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":83,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.423102+0000","last_change":"2024-09-21T10:37:37.360857+0000","last_active":"2024-09-21T10:41:27.423102+0000","last_peered":"2024-09-21T10:41:27.423102+0000","last_clean":"2024-09-21T10:41:27.423102+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:41:27.423102+0000","last_undegraded":"2024-09-21T10:41:27.423102+0000","last_fullsized":"2024-09-21T10:41:27.423102+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.360779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.360779+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:47:25.310501+0000","scrub_duration":0.001559979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2227,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.436870+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:41:27.436870+0000","last_peered":"2024-09-21T10:41:27.436870+0000","last_clean":"2024-09-21T10:41:27.436870+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:41:27.436870+0000","last_undegraded":"2024-09-21T10:41:27.436870+0000","last_fullsized":"2024-09-21T10:41:27.436870+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":138,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431859+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:41:27.431859+0000","last_peered":"2024-09-21T10:41:27.431859+0000","last_clean":"2024-09-21T10:41:27.431859+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:41:27.431859+0000","last_undegraded":"2024-09-21T10:41:27.431859+0000","last_fullsized":"2024-09-21T10:41:27.431859+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":531,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374753+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:41:21.374753+0000","last_peered":"2024-09-21T10:41:21.374753+0000","last_clean":"2024-09-21T10:41:21.374753+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:41:21.374753+0000","last_undegraded":"2024-09-21T10:41:21.374753+0000","last_fullsized":"2024-09-21T10:41:21.374753+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1325,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.373972+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:41:21.373972+0000","last_peered":"2024-09-21T10:41:21.373972+0000","last_clean":"2024-09-21T10:41:21.373972+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:41:21.373972+0000","last_undegraded":"2024-09-21T10:41:21.373972+0000","last_fullsized":"2024-09-21T10:41:21.373972+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1333,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432174+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:41:27.432174+0000","last_peered":"2024-09-21T10:41:27.432174+0000","last_clean":"2024-09-21T10:41:27.432174+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:41:27.432174+0000","last_undegraded":"2024-09-21T10:41:27.432174+0000","last_fullsized":"2024-09-21T10:41:27.432174+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":368,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501336+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:41:28.501336+0000","last_peered":"2024-09-21T10:41:28.501336+0000","last_clean":"2024-09-21T10:41:28.501336+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:41:28.501336+0000","last_undegraded":"2024-09-21T10:41:28.501336+0000","last_fullsized":"2024-09-21T10:41:28.501336+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1300,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431111+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:41:28.431111+0000","last_peered":"2024-09-21T10:41:28.431111+0000","last_clean":"2024-09-21T10:41:28.431111+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:41:28.431111+0000","last_undegraded":"2024-09-21T10:41:28.431111+0000","last_fullsized":"2024-09-21T10:41:28.431111+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1393,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.423364+0000","last_change":"2024-09-21T10:38:34.901523+0000","last_active":"2024-09-21T10:41:27.423364+0000","last_peered":"2024-09-21T10:41:27.423364+0000","last_clean":"2024-09-21T10:41:27.423364+0000","last_became_active":"2024-09-21T10:38:34.887435+0000","last_became_peered":"2024-09-21T10:38:34.887435+0000","last_unstale":"2024-09-21T10:41:27.423364+0000","last_undegraded":"2024-09-21T10:41:27.423364+0000","last_fullsized":"2024-09-21T10:41:27.423364+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:27.993702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":135,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.423996+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:41:27.423996+0000","last_peered":"2024-09-21T10:41:27.423996+0000","last_clean":"2024-09-21T10:41:27.423996+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:41:27.423996+0000","last_undegraded":"2024-09-21T10:41:27.423996+0000","last_fullsized":"2024-09-21T10:41:27.423996+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":138,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431696+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:41:27.431696+0000","last_peered":"2024-09-21T10:41:27.431696+0000","last_clean":"2024-09-21T10:41:27.431696+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:41:27.431696+0000","last_undegraded":"2024-09-21T10:41:27.431696+0000","last_fullsized":"2024-09-21T10:41:27.431696+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1406,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.423662+0000","last_change":"2024-09-21T10:38:34.891682+0000","last_active":"2024-09-21T10:41:27.423662+0000","last_peered":"2024-09-21T10:41:27.423662+0000","last_clean":"2024-09-21T10:41:27.423662+0000","last_became_active":"2024-09-21T10:38:34.885292+0000","last_became_peered":"2024-09-21T10:38:34.885292+0000","last_unstale":"2024-09-21T10:41:27.423662+0000","last_undegraded":"2024-09-21T10:41:27.423662+0000","last_fullsized":"2024-09-21T10:41:27.423662+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:30.111501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":125,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431532+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:41:27.431532+0000","last_peered":"2024-09-21T10:41:27.431532+0000","last_clean":"2024-09-21T10:41:27.431532+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:41:27.431532+0000","last_undegraded":"2024-09-21T10:41:27.431532+0000","last_fullsized":"2024-09-21T10:41:27.431532+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1422,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.374272+0000","last_change":"2024-09-21T10:39:00.852998+0000","last_active":"2024-09-21T10:41:21.374272+0000","last_peered":"2024-09-21T10:41:21.374272+0000","last_clean":"2024-09-21T10:41:21.374272+0000","last_became_active":"2024-09-21T10:39:00.846811+0000","last_became_peered":"2024-09-21T10:39:00.846811+0000","last_unstale":"2024-09-21T10:41:21.374272+0000","last_undegraded":"2024-09-21T10:41:21.374272+0000","last_fullsized":"2024-09-21T10:41:21.374272+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:14.374741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2091,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501512+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:41:28.501512+0000","last_peered":"2024-09-21T10:41:28.501512+0000","last_clean":"2024-09-21T10:41:28.501512+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:41:28.501512+0000","last_undegraded":"2024-09-21T10:41:28.501512+0000","last_fullsized":"2024-09-21T10:41:28.501512+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":374,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.431579+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:41:28.431579+0000","last_peered":"2024-09-21T10:41:28.431579+0000","last_clean":"2024-09-21T10:41:28.431579+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:41:28.431579+0000","last_undegraded":"2024-09-21T10:41:28.431579+0000","last_fullsized":"2024-09-21T10:41:28.431579+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1241,"reported_epoch":1386,"state":"active+clean","last_fresh":"2024-09-21T10:41:21.373877+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:41:21.373877+0000","last_peered":"2024-09-21T10:41:21.373877+0000","last_clean":"2024-09-21T10:41:21.373877+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:41:21.373877+0000","last_undegraded":"2024-09-21T10:41:21.373877+0000","last_fullsized":"2024-09-21T10:41:21.373877+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2208,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.435536+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:41:27.435536+0000","last_peered":"2024-09-21T10:41:27.435536+0000","last_clean":"2024-09-21T10:41:27.435536+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:41:27.435536+0000","last_undegraded":"2024-09-21T10:41:27.435536+0000","last_fullsized":"2024-09-21T10:41:27.435536+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":125,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.431551+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:41:27.431551+0000","last_peered":"2024-09-21T10:41:27.431551+0000","last_clean":"2024-09-21T10:41:27.431551+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:41:27.431551+0000","last_undegraded":"2024-09-21T10:41:27.431551+0000","last_fullsized":"2024-09-21T10:41:27.431551+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2208,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.436156+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:41:27.436156+0000","last_peered":"2024-09-21T10:41:27.436156+0000","last_clean":"2024-09-21T10:41:27.436156+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:41:27.436156+0000","last_undegraded":"2024-09-21T10:41:27.436156+0000","last_fullsized":"2024-09-21T10:41:27.436156+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":148,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.432069+0000","last_change":"2024-09-21T10:38:38.962357+0000","last_active":"2024-09-21T10:41:27.432069+0000","last_peered":"2024-09-21T10:41:27.432069+0000","last_clean":"2024-09-21T10:41:27.432069+0000","last_became_active":"2024-09-21T10:38:38.962017+0000","last_became_peered":"2024-09-21T10:38:38.962017+0000","last_unstale":"2024-09-21T10:41:27.432069+0000","last_undegraded":"2024-09-21T10:41:27.432069+0000","last_fullsized":"2024-09-21T10:41:27.432069+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.355706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":89,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.438031+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:41:27.438031+0000","last_peered":"2024-09-21T10:41:27.438031+0000","last_clean":"2024-09-21T10:41:27.438031+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:41:27.438031+0000","last_undegraded":"2024-09-21T10:41:27.438031+0000","last_fullsized":"2024-09-21T10:41:27.438031+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1394,"reported_epoch":1387,"state":"active+clean","last_fresh":"2024-09-21T10:41:27.427634+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:41:27.427634+0000","last_peered":"2024-09-21T10:41:27.427634+0000","last_clean":"2024-09-21T10:41:27.427634+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:41:27.427634+0000","last_undegraded":"2024-09-21T10:41:27.427634+0000","last_fullsized":"2024-09-21T10:41:27.427634+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2099,"reported_epoch":1388,"state":"active+clean","last_fresh":"2024-09-21T10:41:28.501523+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:41:28.501523+0000","last_peered":"2024-09-21T10:41:28.501523+0000","last_clean":"2024-09-21T10:41:28.501523+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:41:28.501523+0000","last_undegraded":"2024-09-21T10:41:28.501523+0000","last_fullsized":"2024-09-21T10:41:28.501523+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":48,"num_pg":16,"stat_sum":{"num_bytes":832831512,"num_objects":12709,"num_object_clones":0,"num_object_copies":38127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":70,"num_objects_unfound":0,"num_objects_dirty":12709,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":233707,"num_write_kb":7885249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6698,"num_bytes_recovered":357957632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2886873088,"data_stored":2886860872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103732,"ondisk_log_size":103732,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":481141,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6268,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398063,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1388840,"kb_used_data":493212,"kb_used_omap":1,"kb_used_meta":895614,"kb_avail":92356312,"statfs":{"total":95995035648,"available":94572863488,"internally_reserved":0,"allocated":505049088,"data_stored":499015497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917109080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049722,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506260,"kb_used_data":570508,"kb_used_omap":242,"kb_used_meta":935373,"kb_avail":92238892,"statfs":{"total":95995035648,"available":94452625408,"internally_reserved":0,"allocated":584200192,"data_stored":578173924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":957822175},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790521,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1439452,"kb_used_data":550736,"kb_used_omap":1,"kb_used_meta":888702,"kb_avail":92305700,"statfs":{"total":95995035648,"available":94521036800,"internally_reserved":0,"allocated":563953664,"data_stored":557928670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910031192},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644681,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1284268,"kb_used_data":541596,"kb_used_omap":67,"kb_used_meta":742588,"kb_avail":92460884,"statfs":{"total":95995035648,"available":94679945216,"internally_reserved":0,"allocated":554594304,"data_stored":548565542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":760410973},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527699,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1070200,"kb_used_data":341096,"kb_used_omap":75,"kb_used_meta":729012,"kb_avail":92674952,"statfs":{"total":95995035648,"available":94899150848,"internally_reserved":0,"allocated":349282304,"data_stored":343246512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":746508442},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274852,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1121572,"kb_used_data":332060,"kb_used_omap":51,"kb_used_meta":789452,"kb_avail":92623580,"statfs":{"total":95995035648,"available":94846545920,"internally_reserved":0,"allocated":340029440,"data_stored":334000159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":808399189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354090,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":741380,"kb_used_data":183612,"kb_used_omap":66,"kb_used_meta":557693,"kb_avail":93003772,"statfs":{"total":95995035648,"available":95235862528,"internally_reserved":0,"allocated":188018688,"data_stored":182004623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":571077956},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":165871616,"data_stored":165871616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":317390848,"data_stored":317390848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":309788672,"data_stored":309788672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":523440128,"data_stored":523436056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":542375936,"data_stored":542375936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":556077056,"data_stored":556072984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":48,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":471928832,"data_stored":471924760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:41:36.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T10:41:36.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:36.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:36.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:36.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:36.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:36.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:36.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:36.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:36.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:36.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:36.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:36.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:37.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:37.038 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:37.060 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:37.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:37.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:37.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:37.328 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:37.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:37.472 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:37.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:37.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:37.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:37.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:37.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:37 smithi145 ceph-mon[89359]: pgmap v2341: 109 pgs: 109 active+clean; 586 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 7.0 MiB/s, 158 objects/s recovering 2024-09-21T10:41:37.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:37.839 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:37.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:37 smithi119 ceph-mon[105288]: pgmap v2341: 109 pgs: 109 active+clean; 586 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 7.0 MiB/s, 158 objects/s recovering 2024-09-21T10:41:37.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:37 smithi119 ceph-mon[101432]: pgmap v2341: 109 pgs: 109 active+clean; 586 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 855 op/s; 7.0 MiB/s, 158 objects/s recovering 2024-09-21T10:41:37.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:37.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:37.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:37.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:38.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:38.099 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:38.200 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:38.230 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:38.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:38.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:38.374 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:38.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:38.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:38.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:38.614 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:38.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:38.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:38.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:38.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:38.924 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:38.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:39.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:39.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:39.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:39.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:39.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:39.363 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:39.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:39.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:39.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:39.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.648 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:39.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:39.740 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:39 smithi119 ceph-mon[101432]: pgmap v2342: 109 pgs: 109 active+clean; 567 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 490 op/s; 6.5 MiB/s, 104 objects/s recovering 2024-09-21T10:41:39.740 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:39.740 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:39.741 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:39 smithi119 ceph-mon[105288]: pgmap v2342: 109 pgs: 109 active+clean; 567 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 490 op/s; 6.5 MiB/s, 104 objects/s recovering 2024-09-21T10:41:39.741 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:39.741 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:39.748 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.780 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:39.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:39 smithi145 ceph-mon[89359]: pgmap v2342: 109 pgs: 109 active+clean; 567 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 490 op/s; 6.5 MiB/s, 104 objects/s recovering 2024-09-21T10:41:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:39.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:39.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:39.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:40.010 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:40.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:40.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:40.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:40.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.314 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:40.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:40.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:40.445 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:40.486 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:41:40.488 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:41:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:41:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:41:40.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:40.573 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:40.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.704 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:40.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:40.792 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2060201785' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:41:40.792 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2060201785' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:41:40.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2060201785' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:41:40.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:40.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:40.926 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:41:40.926 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:41:40.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:40.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:41.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:41.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:41.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:41.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.351 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:41.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:41.451 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.483 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:41.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:41.551 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[101432]: pgmap v2343: 109 pgs: 109 active+clean; 291 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 517 KiB/s, 8 objects/s recovering 2024-09-21T10:41:41.551 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:41.551 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2060201785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:41:41.551 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[101432]: osdmap e1391: 8 total, 8 up, 7 in 2024-09-21T10:41:41.552 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[105288]: pgmap v2343: 109 pgs: 109 active+clean; 291 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 517 KiB/s, 8 objects/s recovering 2024-09-21T10:41:41.552 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:41.552 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2060201785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:41:41.552 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:41 smithi119 ceph-mon[105288]: osdmap e1391: 8 total, 8 up, 7 in 2024-09-21T10:41:41.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:41.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:41.754 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:41.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:41 smithi145 ceph-mon[89359]: pgmap v2343: 109 pgs: 109 active+clean; 291 MiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 958 op/s; 517 KiB/s, 8 objects/s recovering 2024-09-21T10:41:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:41 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:41:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2060201785' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:41:41.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:41 smithi145 ceph-mon[89359]: osdmap e1391: 8 total, 8 up, 7 in 2024-09-21T10:41:41.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:41.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:41.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:42.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:42.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:41:42.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:42.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:42.340 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:42.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:42.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:42.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:42.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:42.706 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:42.738 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:42.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:42.892 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:42.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:42.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:43.029 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:43.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:43.164 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Removed 123207 objects 2024-09-21T10:41:43.164 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Clean up completed and total clean up time :133.01 2024-09-21T10:41:43.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:43.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:43.267 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:43.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:43.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:43.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:43.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:43.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:43.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:43.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:43.722 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:41:43.722 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:43.729 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:41:43.729 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:43.743 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:43.748 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:41:43.748 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:43.749 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:43 smithi119 ceph-mon[101432]: pgmap v2345: 109 pgs: 109 active+clean; 291 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 463 KiB/s, 7 objects/s recovering 2024-09-21T10:41:43.749 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:43 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:43.749 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:43 smithi119 ceph-mon[105288]: pgmap v2345: 109 pgs: 109 active+clean; 291 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 463 KiB/s, 7 objects/s recovering 2024-09-21T10:41:43.749 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:43 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:43.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:43.768 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:41:43.768 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:43.780 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:43.785 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:43.786 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:41:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:43 smithi145 ceph-mon[89359]: pgmap v2345: 109 pgs: 109 active+clean; 291 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 857 op/s; 463 KiB/s, 7 objects/s recovering 2024-09-21T10:41:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:43 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:41:43.810 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:41:43.810 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:43.827 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:41:43.827 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:43.843 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:41:43.843 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:43.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:43.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:43.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:44.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:44.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:44.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:44.153 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:44.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:44.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:44.276 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:44.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:44.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:44.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:44.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:44.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:41:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.529 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:44.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:44.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1972346790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:44.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1972346790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:44.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:41:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:41:44] "GET /metrics HTTP/1.1" 200 36855 "" "Prometheus/2.51.0" 2024-09-21T10:41:44.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:44.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:44.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:44.774 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1972346790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:44.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:44.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:44.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:45.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:45.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:45.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:45.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:45.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:45.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:45.274 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:45.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:45.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:45.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:45.522 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:45.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:45.648 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:45.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:45.773 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:45.773 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-21T10:41:45.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-21T10:41:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:45 smithi145 ceph-mon[89359]: pgmap v2346: 109 pgs: 109 active+clean; 101 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 911 op/s 2024-09-21T10:41:45.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:45 smithi119 ceph-mon[105288]: pgmap v2346: 109 pgs: 109 active+clean; 101 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 911 op/s 2024-09-21T10:41:45.796 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:45 smithi119 ceph-mon[101432]: pgmap v2346: 109 pgs: 109 active+clean; 101 MiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 911 op/s 2024-09-21T10:41:45.875 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:45.896 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:45.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:45.997 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:46.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:46.124 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:46.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:46.251 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:46.276 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:46.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:46.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:46.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:46.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:46.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:46.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:46.652 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:46.665 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:41:46.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:46.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:46.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:46.903 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:46.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:47.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:47.026 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:47.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:47.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:47.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:47.264 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:41:47.264 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:41:47.264 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:41:47.264 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:41:47.264 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:41:47.271 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:47.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:47.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:47.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:47.517 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:47.522 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:47 smithi119 ceph-mon[101432]: pgmap v2347: 109 pgs: 109 active+clean; 5.3 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-09-21T10:41:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:47 smithi119 ceph-mon[105288]: pgmap v2347: 109 pgs: 109 active+clean; 5.3 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-09-21T10:41:47.619 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:47.645 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:47.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:47.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:47 smithi145 ceph-mon[89359]: pgmap v2347: 109 pgs: 109 active+clean; 5.3 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 928 op/s 2024-09-21T10:41:47.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:47.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:47.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:47.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:48.021 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:48.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:48.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:48.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:41:48.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:48.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:48.309 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:48.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:48.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:48.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:48.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:48.558 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:48.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:48.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:48.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:48.689 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:48.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:48.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:48.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:41:48.811 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:48.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:48.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:48.933 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:48.938 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:48.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:48.945 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:41:48.945 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":2236,"monmap":{"epoch":4,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":1391,"num_osds":8,"num_up_osds":8,"osd_up_since":1726914438,"num_in_osds":7,"osd_in_since":1726914980,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":109}],"num_pgs":109,"num_pools":5,"num_objects":72,"data_bytes":3435834,"bytes_used":6263136256,"bytes_avail":665702113280,"bytes_total":671965249536,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":902},"fsmap":{"epoch":9,"btime":"1970-01-01T00:00:00:000000+0000","id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"foo.smithi119.oikglq","status":"up:active","gid":24557}],"up:standby":1},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.119:8443/","prometheus":"http://172.21.15.119:9283/"}},"servicemap":{"epoch":33,"modified":"2024-09-21T10:39:53.783483+0000","services":{}},"progress_events":{"57a74cac-3140-4ecf-9dfd-e4dca5b6c6c4":{"message":"Upgrade to 19.3.0-5092-g62da54b4 (6m)\n [============................] (remaining: 7m)","progress":0.44999998807907104,"add_to_ceph_s":true}}} 2024-09-21T10:41:49.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:49.065 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:49.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:49.192 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:49.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:49.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:49.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:49.443 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:49.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:49.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:49.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:49.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:49.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.699 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:49 smithi145 ceph-mon[89359]: pgmap v2348: 109 pgs: 109 active+clean; 3.3 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:41:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1231885164' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:41:49.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:49.824 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:49.852 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:49.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:49 smithi119 ceph-mon[101432]: pgmap v2348: 109 pgs: 109 active+clean; 3.3 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:41:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1231885164' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:41:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:49 smithi119 ceph-mon[105288]: pgmap v2348: 109 pgs: 109 active+clean; 3.3 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:41:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1231885164' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-21T10:41:49.926 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:49.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:49.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:50.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:50.073 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:50.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:50.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:50.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:50.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:50.331 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-21T10:41:50.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:41:50.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:50.443 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:50.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:41:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:41:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:41:50.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:50.568 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:50.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:50.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:50.690 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:50.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:50.815 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:50.916 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:50.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:50.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:51.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:51.061 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:51.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:51.185 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:51.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:51.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:51.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:51.430 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:51.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:51.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:51.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:51.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:51 smithi119 ceph-mon[105288]: pgmap v2349: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 463 op/s 2024-09-21T10:41:51.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:51 smithi119 ceph-mon[101432]: pgmap v2349: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 463 op/s 2024-09-21T10:41:51.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:51.675 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:51.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:51.787 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:51 smithi145 ceph-mon[89359]: pgmap v2349: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 463 op/s 2024-09-21T10:41:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:51.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:51.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:51.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:52.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:52.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:52.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:52.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:52.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:52.289 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:52.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:52.413 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:52.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:52.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:52.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:52.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:52.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:52.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:52.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:52.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:52.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:52.835 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:41:52.835 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:41:52.851 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:41:52.851 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:41:52.867 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:41:52.867 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:41:52.883 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:41:52.883 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:41:52.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:52.899 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:41:52.899 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:41:52.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:52.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:52.915 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:41:52.915 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:41:52.930 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:41:52.931 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:41:52.946 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:41:52.946 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:41:53.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.030 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:53.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:53.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:53.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:53.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:53.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:53.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:53.399 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:53.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:53.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:53.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:53.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:53.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:53.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:53 smithi145 ceph-mon[89359]: pgmap v2350: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 408 op/s 2024-09-21T10:41:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4034066271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:53.811 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:53 smithi119 ceph-mon[105288]: pgmap v2350: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 408 op/s 2024-09-21T10:41:53.812 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4034066271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:53.812 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:53 smithi119 ceph-mon[101432]: pgmap v2350: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 408 op/s 2024-09-21T10:41:53.812 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4034066271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:41:53.870 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:53.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:53.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:53.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:54.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:54.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:54.136 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:54.237 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:54.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:54.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:54.387 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:54.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:54.513 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:54.611 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:41:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:41:54] "GET /metrics HTTP/1.1" 200 36828 "" "Prometheus/2.51.0" 2024-09-21T10:41:54.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:54.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:54.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:54.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:54.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:54.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:54.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:54.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:55.010 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:55.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:55.137 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:55.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:55.263 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:55.365 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:55.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:55.489 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:55.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:55.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:55.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:55.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:55 smithi119 ceph-mon[101432]: pgmap v2351: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 386 op/s 2024-09-21T10:41:55.735 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:55 smithi119 ceph-mon[105288]: pgmap v2351: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 386 op/s 2024-09-21T10:41:55.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:55.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:55.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:55 smithi145 ceph-mon[89359]: pgmap v2351: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 386 op/s 2024-09-21T10:41:55.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:55.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:55.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:55.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:56.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.013 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:56.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:56.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:56.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:56.240 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:56.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:56.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:56.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:56.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:56.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:56.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:56.616 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:56.640 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:56.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:56.764 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:56.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:56.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:56.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:56.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:57.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:57.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:57.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:57.200 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:41:57.200 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:41:57.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:57.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:57.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:57.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:57.490 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:57.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:41:57.513 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:57.525 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_3' removed 2024-09-21T10:41:57.614 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:57.637 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:57.739 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:57.761 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:57.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:57.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:57 smithi145 ceph-mon[89359]: pgmap v2352: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 134 op/s 2024-09-21T10:41:57.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1084410432' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:41:57.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:57 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:41:57.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:41:57.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:57 smithi119 ceph-mon[105288]: pgmap v2352: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 134 op/s 2024-09-21T10:41:57.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1084410432' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:41:57.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:57 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:41:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:57 smithi119 ceph-mon[101432]: pgmap v2352: 109 pgs: 109 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 0 B/s wr, 134 op/s 2024-09-21T10:41:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1084410432' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:41:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:57 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:41:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:57.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:41:57.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:58.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:58.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.131 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:58.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:58.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:41:58.252 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:41:58.252 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:41:58.252 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-21T10:41:58.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_4 16 2024-09-21T10:41:58.277 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:41:58.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:58.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:58.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:58.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:58.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:58.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:58.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:58.767 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:58 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:41:58.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:58 smithi145 ceph-mon[89359]: osdmap e1392: 8 total, 8 up, 7 in 2024-09-21T10:41:58.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:58 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:41:58.804 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:58 smithi119 ceph-mon[101432]: osdmap e1392: 8 total, 8 up, 7 in 2024-09-21T10:41:58.805 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:58 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:41:58.805 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:58 smithi119 ceph-mon[105288]: osdmap e1392: 8 total, 8 up, 7 in 2024-09-21T10:41:58.868 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:58.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:58.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:58.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:58.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:41:59.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:41:59.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:41:59.122 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:41:59.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:41:59.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:41:59.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:59.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:59.359 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:59.379 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:59.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:41:59.502 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:41:59.528 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:41:59.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:59.625 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:41:59.658 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:59 smithi119 ceph-mon[101432]: pgmap v2354: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:41:59.658 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:41:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/261195575' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:59.658 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:59 smithi119 ceph-mon[105288]: pgmap v2354: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:41:59.658 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:41:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/261195575' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:59.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:41:59.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:41:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:59 smithi145 ceph-mon[89359]: pgmap v2354: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:41:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:41:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/261195575' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:41:59.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:41:59.871 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:41:59.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:41:59.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:41:59.994 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:41:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:41:59.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:00.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.118 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:00.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:00.218 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:00.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:00.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:00.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:00.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:00.486 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:00.536 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:42:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:42:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:42:00.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:00.609 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:00.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:00.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:00.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:00 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:42:00.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:00 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:42:00.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/261195575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:00.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:00 smithi145 ceph-mon[89359]: osdmap e1393: 8 total, 8 up, 7 in 2024-09-21T10:42:00.837 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:00.858 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/261195575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[105288]: osdmap e1393: 8 total, 8 up, 7 in 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/261195575' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:00 smithi119 ceph-mon[101432]: osdmap e1393: 8 total, 8 up, 7 in 2024-09-21T10:42:00.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:00.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:00.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:01.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:01.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:01.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:01.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:01.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:01.331 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:01.331 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:01.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:01.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:01.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:01.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:01.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.611 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:01.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:01.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:01 smithi119 ceph-mon[105288]: pgmap v2356: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:01.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:01 smithi119 ceph-mon[101432]: pgmap v2356: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:01.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:01.732 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:01 smithi145 ceph-mon[89359]: pgmap v2356: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:01.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:01.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:01.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:01.978 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:01.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:01.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:02.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:02.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:02.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:02.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:02.223 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:02.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:02.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:02.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:02.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:02.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:02.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:02.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:02.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:02.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:02.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:02.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:02.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:02.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:03.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:03.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:42:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:03.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:03.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:03.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:03.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:03.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:03.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:03.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:03.584 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:03.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:03.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:03.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:03 smithi145 ceph-mon[89359]: pgmap v2357: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:03.806 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:03 smithi119 ceph-mon[105288]: pgmap v2357: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:03.806 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:03.806 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:03.806 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:03 smithi119 ceph-mon[101432]: pgmap v2357: 93 pgs: 93 active+clean; 475 KiB data, 5.8 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:03.806 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:03.806 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:03.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:03.834 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:03.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:03.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:03.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:04.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:04.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:04.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:04.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:04.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:04.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:04.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:04.456 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:04.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:04.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:04.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:04.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:04.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:04.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:42:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:42:04] "GET /metrics HTTP/1.1" 200 34428 "" "Prometheus/2.51.0" 2024-09-21T10:42:04.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:04.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:04.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:05.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:05.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.076 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:05.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:05.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:05.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:05.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:05.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:05.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:05.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:05.572 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:05.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:05.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:05.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.818 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:05 smithi119 ceph-mon[101432]: pgmap v2358: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:05 smithi119 ceph-mon[105288]: pgmap v2358: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:05.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:05.941 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:42:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:05.943 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:42:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:05.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:06.020 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:42:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:06.033 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:42:06.033 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:05 smithi145 ceph-mon[89359]: pgmap v2358: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:06.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:06.049 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:42:06.049 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:06.066 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:42:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:06.071 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:06.081 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:42:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:06.097 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:42:06.097 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:06.112 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:42:06.112 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:06.128 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:42:06.128 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:06.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:06.193 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:06.295 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:06.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:06.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:06.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:06.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:06.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:06.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:06.697 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:06.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:06.805 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/386827398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:06.806 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/386827398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:06.824 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:06.926 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:06.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/386827398' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:07.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:07.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:07.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:07.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:07.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:07.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:07.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:07.555 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:07.577 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:07.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:07.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:07.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:07.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[101432]: pgmap v2359: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[101432]: from='client.46844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[105288]: pgmap v2359: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[105288]: from='client.46844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:07 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:07.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:07.954 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:07.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:07 smithi145 ceph-mon[89359]: pgmap v2359: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:07 smithi145 ceph-mon[89359]: from='client.46844 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:07 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:08.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:08.075 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2359,"stamp":"2024-09-21T10:42:05.866380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486690,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":924335,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2158,"ondisk_log_size":2158,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":280,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6022808,"kb_used_data":425208,"kb_used_omap":507,"kb_used_meta":5595332,"kb_avail":650193256,"statfs":{"total":671965249536,"available":665797894144,"internally_reserved":0,"allocated":435412992,"data_stored":393647930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":5729620511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.345666"},"pg_stats":[{"pgid":"3.1f","version":"64'4","reported_seq":71,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895490+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:41:59.895490+0000","last_peered":"2024-09-21T10:41:59.895490+0000","last_clean":"2024-09-21T10:41:59.895490+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:41:59.895490+0000","last_undegraded":"2024-09-21T10:41:59.895490+0000","last_fullsized":"2024-09-21T10:41:59.895490+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1287,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.214345+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:41:58.214345+0000","last_peered":"2024-09-21T10:41:58.214345+0000","last_clean":"2024-09-21T10:41:58.214345+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:41:58.214345+0000","last_undegraded":"2024-09-21T10:41:58.214345+0000","last_fullsized":"2024-09-21T10:41:58.214345+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2108,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.529173+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:41:59.529173+0000","last_peered":"2024-09-21T10:41:59.529173+0000","last_clean":"2024-09-21T10:41:59.529173+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:41:59.529173+0000","last_undegraded":"2024-09-21T10:41:59.529173+0000","last_fullsized":"2024-09-21T10:41:59.529173+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2126,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528458+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:41:59.528458+0000","last_peered":"2024-09-21T10:41:59.528458+0000","last_clean":"2024-09-21T10:41:59.528458+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:41:59.528458+0000","last_undegraded":"2024-09-21T10:41:59.528458+0000","last_fullsized":"2024-09-21T10:41:59.528458+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":161,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934577+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:41:57.934577+0000","last_peered":"2024-09-21T10:41:57.934577+0000","last_clean":"2024-09-21T10:41:57.934577+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:41:57.934577+0000","last_undegraded":"2024-09-21T10:41:57.934577+0000","last_fullsized":"2024-09-21T10:41:57.934577+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1310,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528509+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:41:59.528509+0000","last_peered":"2024-09-21T10:41:59.528509+0000","last_clean":"2024-09-21T10:41:59.528509+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:41:59.528509+0000","last_undegraded":"2024-09-21T10:41:59.528509+0000","last_fullsized":"2024-09-21T10:41:59.528509+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":161,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934354+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:41:57.934354+0000","last_peered":"2024-09-21T10:41:57.934354+0000","last_clean":"2024-09-21T10:41:57.934354+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:41:57.934354+0000","last_undegraded":"2024-09-21T10:41:57.934354+0000","last_fullsized":"2024-09-21T10:41:57.934354+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1397,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528788+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:41:59.528788+0000","last_peered":"2024-09-21T10:41:59.528788+0000","last_clean":"2024-09-21T10:41:59.528788+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:41:59.528788+0000","last_undegraded":"2024-09-21T10:41:59.528788+0000","last_fullsized":"2024-09-21T10:41:59.528788+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":65,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895232+0000","last_change":"2024-09-21T10:39:00.847895+0000","last_active":"2024-09-21T10:41:59.895232+0000","last_peered":"2024-09-21T10:41:59.895232+0000","last_clean":"2024-09-21T10:41:59.895232+0000","last_became_active":"2024-09-21T10:39:00.847533+0000","last_became_peered":"2024-09-21T10:39:00.847533+0000","last_unstale":"2024-09-21T10:41:59.895232+0000","last_undegraded":"2024-09-21T10:41:59.895232+0000","last_fullsized":"2024-09-21T10:41:59.895232+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:55.080913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":93,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.914988+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:41:59.914988+0000","last_peered":"2024-09-21T10:41:59.914988+0000","last_clean":"2024-09-21T10:41:59.914988+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:41:59.914988+0000","last_undegraded":"2024-09-21T10:41:59.914988+0000","last_fullsized":"2024-09-21T10:41:59.914988+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":170,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915292+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:41:59.915292+0000","last_peered":"2024-09-21T10:41:59.915292+0000","last_clean":"2024-09-21T10:41:59.915292+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:41:59.915292+0000","last_undegraded":"2024-09-21T10:41:59.915292+0000","last_fullsized":"2024-09-21T10:41:59.915292+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2116,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.529172+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:41:59.529172+0000","last_peered":"2024-09-21T10:41:59.529172+0000","last_clean":"2024-09-21T10:41:59.529172+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:41:59.529172+0000","last_undegraded":"2024-09-21T10:41:59.529172+0000","last_fullsized":"2024-09-21T10:41:59.529172+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2214,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.914746+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:41:59.914746+0000","last_peered":"2024-09-21T10:41:59.914746+0000","last_clean":"2024-09-21T10:41:59.914746+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:41:59.914746+0000","last_undegraded":"2024-09-21T10:41:59.914746+0000","last_fullsized":"2024-09-21T10:41:59.914746+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2223,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895619+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:41:59.895619+0000","last_peered":"2024-09-21T10:41:59.895619+0000","last_clean":"2024-09-21T10:41:59.895619+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:41:59.895619+0000","last_undegraded":"2024-09-21T10:41:59.895619+0000","last_fullsized":"2024-09-21T10:41:59.895619+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2141,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528380+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:41:59.528380+0000","last_peered":"2024-09-21T10:41:59.528380+0000","last_clean":"2024-09-21T10:41:59.528380+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:41:59.528380+0000","last_undegraded":"2024-09-21T10:41:59.528380+0000","last_fullsized":"2024-09-21T10:41:59.528380+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1351,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.217014+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:41:58.217014+0000","last_peered":"2024-09-21T10:41:58.217014+0000","last_clean":"2024-09-21T10:41:58.217014+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:41:58.217014+0000","last_undegraded":"2024-09-21T10:41:58.217014+0000","last_fullsized":"2024-09-21T10:41:58.217014+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2121,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528504+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:41:59.528504+0000","last_peered":"2024-09-21T10:41:59.528504+0000","last_clean":"2024-09-21T10:41:59.528504+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:41:59.528504+0000","last_undegraded":"2024-09-21T10:41:59.528504+0000","last_fullsized":"2024-09-21T10:41:59.528504+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2233,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895601+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:41:59.895601+0000","last_peered":"2024-09-21T10:41:59.895601+0000","last_clean":"2024-09-21T10:41:59.895601+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:41:59.895601+0000","last_undegraded":"2024-09-21T10:41:59.895601+0000","last_fullsized":"2024-09-21T10:41:59.895601+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":250,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915092+0000","last_change":"2024-09-21T10:37:47.788643+0000","last_active":"2024-09-21T10:41:59.915092+0000","last_peered":"2024-09-21T10:41:59.915092+0000","last_clean":"2024-09-21T10:41:59.915092+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:41:59.915092+0000","last_undegraded":"2024-09-21T10:41:59.915092+0000","last_fullsized":"2024-09-21T10:41:59.915092+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:39.188916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2104,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528584+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:41:59.528584+0000","last_peered":"2024-09-21T10:41:59.528584+0000","last_clean":"2024-09-21T10:41:59.528584+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:41:59.528584+0000","last_undegraded":"2024-09-21T10:41:59.528584+0000","last_fullsized":"2024-09-21T10:41:59.528584+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":143,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934713+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:41:57.934713+0000","last_peered":"2024-09-21T10:41:57.934713+0000","last_clean":"2024-09-21T10:41:57.934713+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:41:57.934713+0000","last_undegraded":"2024-09-21T10:41:57.934713+0000","last_fullsized":"2024-09-21T10:41:57.934713+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1764,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915570+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:41:59.915570+0000","last_peered":"2024-09-21T10:41:59.915570+0000","last_clean":"2024-09-21T10:41:59.915570+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:41:59.915570+0000","last_undegraded":"2024-09-21T10:41:59.915570+0000","last_fullsized":"2024-09-21T10:41:59.915570+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2207,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915474+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:41:59.915474+0000","last_peered":"2024-09-21T10:41:59.915474+0000","last_clean":"2024-09-21T10:41:59.915474+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:41:59.915474+0000","last_undegraded":"2024-09-21T10:41:59.915474+0000","last_fullsized":"2024-09-21T10:41:59.915474+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":408,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.214491+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:41:58.214491+0000","last_peered":"2024-09-21T10:41:58.214491+0000","last_clean":"2024-09-21T10:41:58.214491+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:41:58.214491+0000","last_undegraded":"2024-09-21T10:41:58.214491+0000","last_fullsized":"2024-09-21T10:41:58.214491+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1252,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.217070+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:41:58.217070+0000","last_peered":"2024-09-21T10:41:58.217070+0000","last_clean":"2024-09-21T10:41:58.217070+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:41:58.217070+0000","last_undegraded":"2024-09-21T10:41:58.217070+0000","last_fullsized":"2024-09-21T10:41:58.217070+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":363,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528797+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:41:59.528797+0000","last_peered":"2024-09-21T10:41:59.528797+0000","last_clean":"2024-09-21T10:41:59.528797+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:41:59.528797+0000","last_undegraded":"2024-09-21T10:41:59.528797+0000","last_fullsized":"2024-09-21T10:41:59.528797+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1286,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934672+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:41:57.934672+0000","last_peered":"2024-09-21T10:41:57.934672+0000","last_clean":"2024-09-21T10:41:57.934672+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:41:57.934672+0000","last_undegraded":"2024-09-21T10:41:57.934672+0000","last_fullsized":"2024-09-21T10:41:57.934672+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1347,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.216883+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:41:58.216883+0000","last_peered":"2024-09-21T10:41:58.216883+0000","last_clean":"2024-09-21T10:41:58.216883+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:41:58.216883+0000","last_undegraded":"2024-09-21T10:41:58.216883+0000","last_fullsized":"2024-09-21T10:41:58.216883+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":98,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895547+0000","last_change":"2024-09-21T10:37:41.926686+0000","last_active":"2024-09-21T10:41:59.895547+0000","last_peered":"2024-09-21T10:41:59.895547+0000","last_clean":"2024-09-21T10:41:59.895547+0000","last_became_active":"2024-09-21T10:37:41.922706+0000","last_became_peered":"2024-09-21T10:37:41.922706+0000","last_unstale":"2024-09-21T10:41:59.895547+0000","last_undegraded":"2024-09-21T10:41:59.895547+0000","last_fullsized":"2024-09-21T10:41:59.895547+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:08.955843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":152,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915183+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:41:59.915183+0000","last_peered":"2024-09-21T10:41:59.915183+0000","last_clean":"2024-09-21T10:41:59.915183+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:41:59.915183+0000","last_undegraded":"2024-09-21T10:41:59.915183+0000","last_fullsized":"2024-09-21T10:41:59.915183+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":49,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.933863+0000","last_change":"2024-09-21T10:39:00.847173+0000","last_active":"2024-09-21T10:41:57.933863+0000","last_peered":"2024-09-21T10:41:57.933863+0000","last_clean":"2024-09-21T10:41:57.933863+0000","last_became_active":"2024-09-21T10:39:00.846851+0000","last_became_peered":"2024-09-21T10:39:00.846851+0000","last_unstale":"2024-09-21T10:41:57.933863+0000","last_undegraded":"2024-09-21T10:41:57.933863+0000","last_fullsized":"2024-09-21T10:41:57.933863+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:47.133721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2224,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915759+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:41:59.915759+0000","last_peered":"2024-09-21T10:41:59.915759+0000","last_clean":"2024-09-21T10:41:59.915759+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:41:59.915759+0000","last_undegraded":"2024-09-21T10:41:59.915759+0000","last_fullsized":"2024-09-21T10:41:59.915759+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2103,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.529139+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:41:59.529139+0000","last_peered":"2024-09-21T10:41:59.529139+0000","last_clean":"2024-09-21T10:41:59.529139+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:41:59.529139+0000","last_undegraded":"2024-09-21T10:41:59.529139+0000","last_fullsized":"2024-09-21T10:41:59.529139+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":518,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215901+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:41:58.215901+0000","last_peered":"2024-09-21T10:41:58.215901+0000","last_clean":"2024-09-21T10:41:58.215901+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:41:58.215901+0000","last_undegraded":"2024-09-21T10:41:58.215901+0000","last_fullsized":"2024-09-21T10:41:58.215901+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2226,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895610+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:41:59.895610+0000","last_peered":"2024-09-21T10:41:59.895610+0000","last_clean":"2024-09-21T10:41:59.895610+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:41:59.895610+0000","last_undegraded":"2024-09-21T10:41:59.895610+0000","last_fullsized":"2024-09-21T10:41:59.895610+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1410,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215421+0000","last_change":"2024-09-21T10:37:41.922996+0000","last_active":"2024-09-21T10:41:58.215421+0000","last_peered":"2024-09-21T10:41:58.215421+0000","last_clean":"2024-09-21T10:41:58.215421+0000","last_became_active":"2024-09-21T10:37:41.922703+0000","last_became_peered":"2024-09-21T10:37:41.922703+0000","last_unstale":"2024-09-21T10:41:58.215421+0000","last_undegraded":"2024-09-21T10:41:58.215421+0000","last_fullsized":"2024-09-21T10:41:58.215421+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:23.614770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2212,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915588+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:41:59.915588+0000","last_peered":"2024-09-21T10:41:59.915588+0000","last_clean":"2024-09-21T10:41:59.915588+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:41:59.915588+0000","last_undegraded":"2024-09-21T10:41:59.915588+0000","last_fullsized":"2024-09-21T10:41:59.915588+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":518,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.216803+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:41:58.216803+0000","last_peered":"2024-09-21T10:41:58.216803+0000","last_clean":"2024-09-21T10:41:58.216803+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:41:58.216803+0000","last_undegraded":"2024-09-21T10:41:58.216803+0000","last_fullsized":"2024-09-21T10:41:58.216803+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2217,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915865+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:41:59.915865+0000","last_peered":"2024-09-21T10:41:59.915865+0000","last_clean":"2024-09-21T10:41:59.915865+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:41:59.915865+0000","last_undegraded":"2024-09-21T10:41:59.915865+0000","last_fullsized":"2024-09-21T10:41:59.915865+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2212,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915685+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:41:59.915685+0000","last_peered":"2024-09-21T10:41:59.915685+0000","last_clean":"2024-09-21T10:41:59.915685+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:41:59.915685+0000","last_undegraded":"2024-09-21T10:41:59.915685+0000","last_fullsized":"2024-09-21T10:41:59.915685+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":143,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934408+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:41:57.934408+0000","last_peered":"2024-09-21T10:41:57.934408+0000","last_clean":"2024-09-21T10:41:57.934408+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:41:57.934408+0000","last_undegraded":"2024-09-21T10:41:57.934408+0000","last_fullsized":"2024-09-21T10:41:57.934408+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":153,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934751+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:41:57.934751+0000","last_peered":"2024-09-21T10:41:57.934751+0000","last_clean":"2024-09-21T10:41:57.934751+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:41:57.934751+0000","last_undegraded":"2024-09-21T10:41:57.934751+0000","last_fullsized":"2024-09-21T10:41:57.934751+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2209,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915626+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:41:59.915626+0000","last_peered":"2024-09-21T10:41:59.915626+0000","last_clean":"2024-09-21T10:41:59.915626+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:41:59.915626+0000","last_undegraded":"2024-09-21T10:41:59.915626+0000","last_fullsized":"2024-09-21T10:41:59.915626+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1328,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895565+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:41:59.895565+0000","last_peered":"2024-09-21T10:41:59.895565+0000","last_clean":"2024-09-21T10:41:59.895565+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:41:59.895565+0000","last_undegraded":"2024-09-21T10:41:59.895565+0000","last_fullsized":"2024-09-21T10:41:59.895565+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":147,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934127+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:41:57.934127+0000","last_peered":"2024-09-21T10:41:57.934127+0000","last_clean":"2024-09-21T10:41:57.934127+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:41:57.934127+0000","last_undegraded":"2024-09-21T10:41:57.934127+0000","last_fullsized":"2024-09-21T10:41:57.934127+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":531,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895196+0000","last_change":"2024-09-21T10:37:38.606346+0000","last_active":"2024-09-21T10:41:59.895196+0000","last_peered":"2024-09-21T10:41:59.895196+0000","last_clean":"2024-09-21T10:41:59.895196+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:41:59.895196+0000","last_undegraded":"2024-09-21T10:41:59.895196+0000","last_fullsized":"2024-09-21T10:41:59.895196+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:16.461221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2227,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915593+0000","last_change":"2024-09-21T10:37:41.924374+0000","last_active":"2024-09-21T10:41:59.915593+0000","last_peered":"2024-09-21T10:41:59.915593+0000","last_clean":"2024-09-21T10:41:59.915593+0000","last_became_active":"2024-09-21T10:37:41.922589+0000","last_became_peered":"2024-09-21T10:37:41.922589+0000","last_unstale":"2024-09-21T10:41:59.915593+0000","last_undegraded":"2024-09-21T10:41:59.915593+0000","last_fullsized":"2024-09-21T10:41:59.915593+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:51.987384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2209,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.914686+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:41:59.914686+0000","last_peered":"2024-09-21T10:41:59.914686+0000","last_clean":"2024-09-21T10:41:59.914686+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:41:59.914686+0000","last_undegraded":"2024-09-21T10:41:59.914686+0000","last_fullsized":"2024-09-21T10:41:59.914686+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2140,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.914842+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:41:59.914842+0000","last_peered":"2024-09-21T10:41:59.914842+0000","last_clean":"2024-09-21T10:41:59.914842+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:41:59.914842+0000","last_undegraded":"2024-09-21T10:41:59.914842+0000","last_fullsized":"2024-09-21T10:41:59.914842+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":394,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915485+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:41:59.915485+0000","last_peered":"2024-09-21T10:41:59.915485+0000","last_clean":"2024-09-21T10:41:59.915485+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:41:59.915485+0000","last_undegraded":"2024-09-21T10:41:59.915485+0000","last_fullsized":"2024-09-21T10:41:59.915485+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":65,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895387+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:41:59.895387+0000","last_peered":"2024-09-21T10:41:59.895387+0000","last_clean":"2024-09-21T10:41:59.895387+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:41:59.895387+0000","last_undegraded":"2024-09-21T10:41:59.895387+0000","last_fullsized":"2024-09-21T10:41:59.895387+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2241,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895269+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:41:59.895269+0000","last_peered":"2024-09-21T10:41:59.895269+0000","last_clean":"2024-09-21T10:41:59.895269+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:41:59.895269+0000","last_undegraded":"2024-09-21T10:41:59.895269+0000","last_fullsized":"2024-09-21T10:41:59.895269+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1380,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215214+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:41:58.215214+0000","last_peered":"2024-09-21T10:41:58.215214+0000","last_clean":"2024-09-21T10:41:58.215214+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:41:58.215214+0000","last_undegraded":"2024-09-21T10:41:58.215214+0000","last_fullsized":"2024-09-21T10:41:58.215214+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1353,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.214549+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:41:58.214549+0000","last_peered":"2024-09-21T10:41:58.214549+0000","last_clean":"2024-09-21T10:41:58.214549+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:41:58.214549+0000","last_undegraded":"2024-09-21T10:41:58.214549+0000","last_fullsized":"2024-09-21T10:41:58.214549+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2125,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528599+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:41:59.528599+0000","last_peered":"2024-09-21T10:41:59.528599+0000","last_clean":"2024-09-21T10:41:59.528599+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:41:59.528599+0000","last_undegraded":"2024-09-21T10:41:59.528599+0000","last_fullsized":"2024-09-21T10:41:59.528599+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":147,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934343+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:41:57.934343+0000","last_peered":"2024-09-21T10:41:57.934343+0000","last_clean":"2024-09-21T10:41:57.934343+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:41:57.934343+0000","last_undegraded":"2024-09-21T10:41:57.934343+0000","last_fullsized":"2024-09-21T10:41:57.934343+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1563,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.214733+0000","last_change":"2024-09-21T10:37:41.922049+0000","last_active":"2024-09-21T10:41:58.214733+0000","last_peered":"2024-09-21T10:41:58.214733+0000","last_clean":"2024-09-21T10:41:58.214733+0000","last_became_active":"2024-09-21T10:37:41.921709+0000","last_became_peered":"2024-09-21T10:37:41.921709+0000","last_unstale":"2024-09-21T10:41:58.214733+0000","last_undegraded":"2024-09-21T10:41:58.214733+0000","last_fullsized":"2024-09-21T10:41:58.214733+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:12.446157+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":143,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934169+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:41:57.934169+0000","last_peered":"2024-09-21T10:41:57.934169+0000","last_clean":"2024-09-21T10:41:57.934169+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:41:57.934169+0000","last_undegraded":"2024-09-21T10:41:57.934169+0000","last_fullsized":"2024-09-21T10:41:57.934169+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2214,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915501+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:41:59.915501+0000","last_peered":"2024-09-21T10:41:59.915501+0000","last_clean":"2024-09-21T10:41:59.915501+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:41:59.915501+0000","last_undegraded":"2024-09-21T10:41:59.915501+0000","last_fullsized":"2024-09-21T10:41:59.915501+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2112,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528384+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:41:59.528384+0000","last_peered":"2024-09-21T10:41:59.528384+0000","last_clean":"2024-09-21T10:41:59.528384+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:41:59.528384+0000","last_undegraded":"2024-09-21T10:41:59.528384+0000","last_fullsized":"2024-09-21T10:41:59.528384+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":378,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915379+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:41:59.915379+0000","last_peered":"2024-09-21T10:41:59.915379+0000","last_clean":"2024-09-21T10:41:59.915379+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:41:59.915379+0000","last_undegraded":"2024-09-21T10:41:59.915379+0000","last_fullsized":"2024-09-21T10:41:59.915379+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1252,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215973+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:41:58.215973+0000","last_peered":"2024-09-21T10:41:58.215973+0000","last_clean":"2024-09-21T10:41:58.215973+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:41:58.215973+0000","last_undegraded":"2024-09-21T10:41:58.215973+0000","last_fullsized":"2024-09-21T10:41:58.215973+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1267,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.914881+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:41:59.914881+0000","last_peered":"2024-09-21T10:41:59.914881+0000","last_clean":"2024-09-21T10:41:59.914881+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:41:59.914881+0000","last_undegraded":"2024-09-21T10:41:59.914881+0000","last_fullsized":"2024-09-21T10:41:59.914881+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":147,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934709+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:41:57.934709+0000","last_peered":"2024-09-21T10:41:57.934709+0000","last_clean":"2024-09-21T10:41:57.934709+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:41:57.934709+0000","last_undegraded":"2024-09-21T10:41:57.934709+0000","last_fullsized":"2024-09-21T10:41:57.934709+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1275,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915283+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:41:59.915283+0000","last_peered":"2024-09-21T10:41:59.915283+0000","last_clean":"2024-09-21T10:41:59.915283+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:41:59.915283+0000","last_undegraded":"2024-09-21T10:41:59.915283+0000","last_fullsized":"2024-09-21T10:41:59.915283+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1359,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.217303+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:41:58.217303+0000","last_peered":"2024-09-21T10:41:58.217303+0000","last_clean":"2024-09-21T10:41:58.217303+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:41:58.217303+0000","last_undegraded":"2024-09-21T10:41:58.217303+0000","last_fullsized":"2024-09-21T10:41:58.217303+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1514,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.217958+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:41:58.217958+0000","last_peered":"2024-09-21T10:41:58.217958+0000","last_clean":"2024-09-21T10:41:58.217958+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:41:58.217958+0000","last_undegraded":"2024-09-21T10:41:58.217958+0000","last_fullsized":"2024-09-21T10:41:58.217958+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1327,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.529146+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:41:59.529146+0000","last_peered":"2024-09-21T10:41:59.529146+0000","last_clean":"2024-09-21T10:41:59.529146+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:41:59.529146+0000","last_undegraded":"2024-09-21T10:41:59.529146+0000","last_fullsized":"2024-09-21T10:41:59.529146+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":339,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528785+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:41:59.528785+0000","last_peered":"2024-09-21T10:41:59.528785+0000","last_clean":"2024-09-21T10:41:59.528785+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:41:59.528785+0000","last_undegraded":"2024-09-21T10:41:59.528785+0000","last_fullsized":"2024-09-21T10:41:59.528785+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":88,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.214706+0000","last_change":"2024-09-21T10:37:37.360857+0000","last_active":"2024-09-21T10:41:58.214706+0000","last_peered":"2024-09-21T10:41:58.214706+0000","last_clean":"2024-09-21T10:41:58.214706+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:41:58.214706+0000","last_undegraded":"2024-09-21T10:41:58.214706+0000","last_fullsized":"2024-09-21T10:41:58.214706+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.360779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.360779+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:47:25.310501+0000","scrub_duration":0.001559979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2239,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895553+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:41:59.895553+0000","last_peered":"2024-09-21T10:41:59.895553+0000","last_clean":"2024-09-21T10:41:59.895553+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:41:59.895553+0000","last_undegraded":"2024-09-21T10:41:59.895553+0000","last_fullsized":"2024-09-21T10:41:59.895553+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":143,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934300+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:41:57.934300+0000","last_peered":"2024-09-21T10:41:57.934300+0000","last_clean":"2024-09-21T10:41:57.934300+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:41:57.934300+0000","last_undegraded":"2024-09-21T10:41:57.934300+0000","last_fullsized":"2024-09-21T10:41:57.934300+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":537,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.216951+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:41:58.216951+0000","last_peered":"2024-09-21T10:41:58.216951+0000","last_clean":"2024-09-21T10:41:58.216951+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:41:58.216951+0000","last_undegraded":"2024-09-21T10:41:58.216951+0000","last_fullsized":"2024-09-21T10:41:58.216951+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1331,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.217141+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:41:58.217141+0000","last_peered":"2024-09-21T10:41:58.217141+0000","last_clean":"2024-09-21T10:41:58.217141+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:41:58.217141+0000","last_undegraded":"2024-09-21T10:41:58.217141+0000","last_fullsized":"2024-09-21T10:41:58.217141+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1338,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934611+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:41:57.934611+0000","last_peered":"2024-09-21T10:41:57.934611+0000","last_clean":"2024-09-21T10:41:57.934611+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:41:57.934611+0000","last_undegraded":"2024-09-21T10:41:57.934611+0000","last_fullsized":"2024-09-21T10:41:57.934611+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":378,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528780+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:41:59.528780+0000","last_peered":"2024-09-21T10:41:59.528780+0000","last_clean":"2024-09-21T10:41:59.528780+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:41:59.528780+0000","last_undegraded":"2024-09-21T10:41:59.528780+0000","last_fullsized":"2024-09-21T10:41:59.528780+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1310,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915074+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:41:59.915074+0000","last_peered":"2024-09-21T10:41:59.915074+0000","last_clean":"2024-09-21T10:41:59.915074+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:41:59.915074+0000","last_undegraded":"2024-09-21T10:41:59.915074+0000","last_fullsized":"2024-09-21T10:41:59.915074+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1398,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.218849+0000","last_change":"2024-09-21T10:38:34.901523+0000","last_active":"2024-09-21T10:41:58.218849+0000","last_peered":"2024-09-21T10:41:58.218849+0000","last_clean":"2024-09-21T10:41:58.218849+0000","last_became_active":"2024-09-21T10:38:34.887435+0000","last_became_peered":"2024-09-21T10:38:34.887435+0000","last_unstale":"2024-09-21T10:41:58.218849+0000","last_undegraded":"2024-09-21T10:41:58.218849+0000","last_fullsized":"2024-09-21T10:41:58.218849+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:27.993702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":140,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215525+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:41:58.215525+0000","last_peered":"2024-09-21T10:41:58.215525+0000","last_clean":"2024-09-21T10:41:58.215525+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:41:58.215525+0000","last_undegraded":"2024-09-21T10:41:58.215525+0000","last_fullsized":"2024-09-21T10:41:58.215525+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":143,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934791+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:41:57.934791+0000","last_peered":"2024-09-21T10:41:57.934791+0000","last_clean":"2024-09-21T10:41:57.934791+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:41:57.934791+0000","last_undegraded":"2024-09-21T10:41:57.934791+0000","last_fullsized":"2024-09-21T10:41:57.934791+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1411,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.218396+0000","last_change":"2024-09-21T10:38:34.891682+0000","last_active":"2024-09-21T10:41:58.218396+0000","last_peered":"2024-09-21T10:41:58.218396+0000","last_clean":"2024-09-21T10:41:58.218396+0000","last_became_active":"2024-09-21T10:38:34.885292+0000","last_became_peered":"2024-09-21T10:38:34.885292+0000","last_unstale":"2024-09-21T10:41:58.218396+0000","last_undegraded":"2024-09-21T10:41:58.218396+0000","last_fullsized":"2024-09-21T10:41:58.218396+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:30.111501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":130,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934038+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:41:57.934038+0000","last_peered":"2024-09-21T10:41:57.934038+0000","last_clean":"2024-09-21T10:41:57.934038+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:41:57.934038+0000","last_undegraded":"2024-09-21T10:41:57.934038+0000","last_fullsized":"2024-09-21T10:41:57.934038+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1428,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215514+0000","last_change":"2024-09-21T10:39:00.852998+0000","last_active":"2024-09-21T10:41:58.215514+0000","last_peered":"2024-09-21T10:41:58.215514+0000","last_clean":"2024-09-21T10:41:58.215514+0000","last_became_active":"2024-09-21T10:39:00.846811+0000","last_became_peered":"2024-09-21T10:39:00.846811+0000","last_unstale":"2024-09-21T10:41:58.215514+0000","last_undegraded":"2024-09-21T10:41:58.215514+0000","last_fullsized":"2024-09-21T10:41:58.215514+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:14.374741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2101,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528992+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:41:59.528992+0000","last_peered":"2024-09-21T10:41:59.528992+0000","last_clean":"2024-09-21T10:41:59.528992+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:41:59.528992+0000","last_undegraded":"2024-09-21T10:41:59.528992+0000","last_fullsized":"2024-09-21T10:41:59.528992+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":385,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.915287+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:41:59.915287+0000","last_peered":"2024-09-21T10:41:59.915287+0000","last_clean":"2024-09-21T10:41:59.915287+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:41:59.915287+0000","last_undegraded":"2024-09-21T10:41:59.915287+0000","last_fullsized":"2024-09-21T10:41:59.915287+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1247,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.214352+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:41:58.214352+0000","last_peered":"2024-09-21T10:41:58.214352+0000","last_clean":"2024-09-21T10:41:58.214352+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:41:58.214352+0000","last_undegraded":"2024-09-21T10:41:58.214352+0000","last_fullsized":"2024-09-21T10:41:58.214352+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2220,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895556+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:41:59.895556+0000","last_peered":"2024-09-21T10:41:59.895556+0000","last_clean":"2024-09-21T10:41:59.895556+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:41:59.895556+0000","last_undegraded":"2024-09-21T10:41:59.895556+0000","last_fullsized":"2024-09-21T10:41:59.895556+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":130,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.933959+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:41:57.933959+0000","last_peered":"2024-09-21T10:41:57.933959+0000","last_clean":"2024-09-21T10:41:57.933959+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:41:57.933959+0000","last_undegraded":"2024-09-21T10:41:57.933959+0000","last_fullsized":"2024-09-21T10:41:57.933959+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2220,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895252+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:41:59.895252+0000","last_peered":"2024-09-21T10:41:59.895252+0000","last_clean":"2024-09-21T10:41:59.895252+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:41:59.895252+0000","last_undegraded":"2024-09-21T10:41:59.895252+0000","last_fullsized":"2024-09-21T10:41:59.895252+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":153,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:57.934488+0000","last_change":"2024-09-21T10:38:38.962357+0000","last_active":"2024-09-21T10:41:57.934488+0000","last_peered":"2024-09-21T10:41:57.934488+0000","last_clean":"2024-09-21T10:41:57.934488+0000","last_became_active":"2024-09-21T10:38:38.962017+0000","last_became_peered":"2024-09-21T10:38:38.962017+0000","last_unstale":"2024-09-21T10:41:57.934488+0000","last_undegraded":"2024-09-21T10:41:57.934488+0000","last_fullsized":"2024-09-21T10:41:57.934488+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.355706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":101,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.895067+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:41:59.895067+0000","last_peered":"2024-09-21T10:41:59.895067+0000","last_clean":"2024-09-21T10:41:59.895067+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:41:59.895067+0000","last_undegraded":"2024-09-21T10:41:59.895067+0000","last_fullsized":"2024-09-21T10:41:59.895067+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1399,"reported_epoch":1392,"state":"active+clean","last_fresh":"2024-09-21T10:41:58.215331+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:41:58.215331+0000","last_peered":"2024-09-21T10:41:58.215331+0000","last_clean":"2024-09-21T10:41:58.215331+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:41:58.215331+0000","last_undegraded":"2024-09-21T10:41:58.215331+0000","last_fullsized":"2024-09-21T10:41:58.215331+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2110,"reported_epoch":1393,"state":"active+clean","last_fresh":"2024-09-21T10:41:59.528915+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:41:59.528915+0000","last_peered":"2024-09-21T10:41:59.528915+0000","last_clean":"2024-09-21T10:41:59.528915+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:41:59.528915+0000","last_undegraded":"2024-09-21T10:41:59.528915+0000","last_fullsized":"2024-09-21T10:41:59.528915+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":481141,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6268,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398070,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919756,"kb_used_data":15804,"kb_used_omap":1,"kb_used_meta":903934,"kb_avail":92825396,"statfs":{"total":95995035648,"available":95053205504,"internally_reserved":0,"allocated":16183296,"data_stored":10105122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925628760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049729,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943436,"kb_used_data":15648,"kb_used_omap":242,"kb_used_meta":927053,"kb_avail":92801716,"statfs":{"total":95995035648,"available":95028957184,"internally_reserved":0,"allocated":16023552,"data_stored":10161172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":949302495},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790528,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1227648,"kb_used_data":330136,"kb_used_omap":1,"kb_used_meta":897022,"kb_avail":92517504,"statfs":{"total":95995035648,"available":94737924096,"internally_reserved":0,"allocated":338059264,"data_stored":332201404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918550872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395634,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644688,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785404,"kb_used_data":16364,"kb_used_omap":67,"kb_used_meta":768956,"kb_avail":92959748,"statfs":{"total":95995035648,"available":95190781952,"internally_reserved":0,"allocated":16756736,"data_stored":10690730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":787411805},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527706,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749324,"kb_used_data":15824,"kb_used_omap":75,"kb_used_meta":733172,"kb_avail":92995828,"statfs":{"total":95995035648,"available":95227727872,"internally_reserved":0,"allocated":16203776,"data_stored":10232371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":750768282},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274859,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":792828,"kb_used_data":15556,"kb_used_omap":51,"kb_used_meta":776972,"kb_avail":92952324,"statfs":{"total":95995035648,"available":95183179776,"internally_reserved":0,"allocated":15929344,"data_stored":9964450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":795619669},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354097,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":604412,"kb_used_data":15876,"kb_used_omap":66,"kb_used_meta":588221,"kb_avail":93140740,"statfs":{"total":95995035648,"available":95376117760,"internally_reserved":0,"allocated":16257024,"data_stored":10292681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":602338628},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:42:08.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:08.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:08.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:08.199 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:08.204 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:08.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:08.331 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:08.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:08.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:08.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:08.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:08.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:08.565 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_4' created 2024-09-21T10:42:08.585 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:08.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:08.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:08.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:08.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:08.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:08.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:08 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1917873631' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:42:08.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:08 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1917873631' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:42:08.935 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:08.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:08.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:08 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1917873631' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-21T10:42:09.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:09.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:09.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:09.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:42:09.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:09.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:09.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:09.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:09.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:09.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:09.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:09 smithi119 ceph-mon[101432]: pgmap v2360: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:09.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1917873631' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:42:09.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:09 smithi119 ceph-mon[101432]: osdmap e1394: 8 total, 8 up, 7 in 2024-09-21T10:42:09.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:09.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:09.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:09.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-21T10:42:09.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:09.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:09.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:09 smithi119 ceph-mon[105288]: pgmap v2360: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1917873631' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:42:09.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:09 smithi119 ceph-mon[105288]: osdmap e1394: 8 total, 8 up, 7 in 2024-09-21T10:42:09.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:09.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:09 smithi145 ceph-mon[89359]: pgmap v2360: 93 pgs: 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1917873631' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-21T10:42:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:09 smithi145 ceph-mon[89359]: osdmap e1394: 8 total, 8 up, 7 in 2024-09-21T10:42:10.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:10.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:10.189 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:10.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:10.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:10.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:10.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:10.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:10.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:42:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:42:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:42:10.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:10.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:10.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:10.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:10.818 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:10.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[105288]: osdmap e1395: 8 total, 8 up, 7 in 2024-09-21T10:42:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[105288]: pgmap v2363: 109 pgs: 9 creating+peering, 7 unknown, 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:42:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:42:10.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:10.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[101432]: osdmap e1395: 8 total, 8 up, 7 in 2024-09-21T10:42:10.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[101432]: pgmap v2363: 109 pgs: 9 creating+peering, 7 unknown, 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:10.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:42:10.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:42:10.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:10.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:10.970 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:10.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:10 smithi145 ceph-mon[89359]: osdmap e1395: 8 total, 8 up, 7 in 2024-09-21T10:42:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:10 smithi145 ceph-mon[89359]: pgmap v2363: 109 pgs: 9 creating+peering, 7 unknown, 93 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:42:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:42:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:11.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:11.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:42:11.096 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:11.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:11.218 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:11.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:11.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:11.341 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:42:11.342 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:42:11.342 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 0 2024-09-21T10:42:11.342 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:42:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:11.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:11.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:11.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:11.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:11 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:42:11.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:11 smithi119 ceph-mon[105288]: osdmap e1396: 8 total, 8 up, 7 in 2024-09-21T10:42:11.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:11 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:42:11.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:11 smithi119 ceph-mon[101432]: osdmap e1396: 8 total, 8 up, 7 in 2024-09-21T10:42:11.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:11.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:11.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:11.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:11.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:11.818 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:11.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:11.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:11.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:11 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:42:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:11 smithi145 ceph-mon[89359]: osdmap e1396: 8 total, 8 up, 7 in 2024-09-21T10:42:12.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:12.098 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:12.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:12.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:12.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:12.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:12.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:12.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:12.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:12.593 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:12.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:12.715 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:12.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:12.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:42:12.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:12.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:12.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:12.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:12 smithi119 ceph-mon[101432]: pgmap v2365: 109 pgs: 9 creating+peering, 3 unknown, 97 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:12.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:12 smithi119 ceph-mon[105288]: pgmap v2365: 109 pgs: 9 creating+peering, 3 unknown, 97 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:12.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:12.965 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:12.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:13.039 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:13.041 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:12 smithi145 ceph-mon[89359]: pgmap v2365: 109 pgs: 9 creating+peering, 3 unknown, 97 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:13.067 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:13.082 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:13.093 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:13.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:13.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:13.313 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:13.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:13.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:13.470 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:13.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:13.594 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:42:13.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:13.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:13.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:13.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:13.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:13 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3417079374' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:42:13.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:13 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3417079374' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:42:13.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:13.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:13.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:13.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:13.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:13.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:13 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3417079374' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:42:14.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:14.091 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:14.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:14.214 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:14.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:14.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:14.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:14.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:14.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:14.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:14.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:14.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:14.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:14.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:14.842 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:42:14] "GET /metrics HTTP/1.1" 200 34428 "" "Prometheus/2.51.0" 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3417079374' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[105288]: osdmap e1397: 8 total, 8 up, 7 in 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[105288]: pgmap v2367: 109 pgs: 5 creating+peering, 104 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3417079374' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[101432]: osdmap e1397: 8 total, 8 up, 7 in 2024-09-21T10:42:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:14 smithi119 ceph-mon[101432]: pgmap v2367: 109 pgs: 5 creating+peering, 104 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:14.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:14.966 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:14.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:14.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:14 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3417079374' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:42:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:14 smithi145 ceph-mon[89359]: osdmap e1397: 8 total, 8 up, 7 in 2024-09-21T10:42:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:14 smithi145 ceph-mon[89359]: pgmap v2367: 109 pgs: 5 creating+peering, 104 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:15.068 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:15.093 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:15.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:15.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:15.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:15.344 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:15.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:15.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:15.470 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:15.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:15.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:15.721 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:15.823 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:15.848 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:15.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:15.972 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:15.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:16.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.098 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:16.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:16.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:16.222 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:16.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:16.324 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:16.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:16.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:16.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:16.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:16.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:16.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:16.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:16.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:16.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.846 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:16.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:16.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:16.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:16.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:16.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:17.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:17.096 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:16 smithi119 ceph-mon[105288]: pgmap v2368: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:17.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:16 smithi119 ceph-mon[101432]: pgmap v2368: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:17.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:17.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:16 smithi145 ceph-mon[89359]: pgmap v2368: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:17.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:17.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:17.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:42:17.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:17.470 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:17.572 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:17.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:17.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:17.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:17.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:17.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:17.844 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:17.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:17.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:17.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:18.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:18.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:18.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:18.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:18.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:18.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:18.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:18.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:18.591 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:18.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:18.648 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:42:18.649 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:42:18.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:18.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:18.761 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:42:18.761 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:42:18.761 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:42:18.761 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:42:18.761 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:42:18.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:18.845 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:18.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:18.970 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:18.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:18 smithi145 ceph-mon[89359]: pgmap v2369: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:19.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:19.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:19.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:18 smithi119 ceph-mon[101432]: pgmap v2369: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:18 smithi119 ceph-mon[105288]: pgmap v2369: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:19.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:19.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:19.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:19.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:19.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:19.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:19.562 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:19.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:42:19.587 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:19.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:19.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:19.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:19.793 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:19 smithi119 ceph-mon[101432]: from='client.36957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:19.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:19.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:19.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:19.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:19.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:19 smithi145 ceph-mon[89359]: from='client.36957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:20.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:20.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:19 smithi119 ceph-mon[105288]: from='client.36957 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:42:20.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:42:20.170 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:20.184 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:42:20.184 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:20.189 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:20.200 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:42:20.200 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:20.214 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:20.216 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:42:20.216 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:20.231 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:42:20.231 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:20.247 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:42:20.247 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:20.263 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:42:20.263 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:20.279 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:42:20.279 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:20.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:20.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:20.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:20.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:42:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:42:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:42:20.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:20.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:20.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:20.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:20.708 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:20.810 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:20.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:20.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:20.957 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:20.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:21.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:21.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:20 smithi119 ceph-mon[105288]: pgmap v2370: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2008425293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:20 smithi119 ceph-mon[101432]: pgmap v2370: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2008425293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:21.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:21.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:20 smithi145 ceph-mon[89359]: pgmap v2370: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2008425293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:21.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:21.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:21.433 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:21.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:21.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:21.494 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:21.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:21.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:21.683 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:21.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:21.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:21.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:21.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:21.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:21.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:22.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:22.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:22.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:22.202 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:22.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:22.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:22.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:22.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:22.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:22.451 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:22.553 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:22.578 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:22.679 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:22.689 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:22.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:22.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:22.833 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:22.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-21T10:42:22.935 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:22.958 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:22.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:23.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:23.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:23.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:22 smithi119 ceph-mon[105288]: pgmap v2371: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:23.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3382685280' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:42:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:22 smithi119 ceph-mon[101432]: pgmap v2371: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:23.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3382685280' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:42:23.147 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2369,"stamp":"2024-09-21T10:42:17.869389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":486690,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":924335,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2158,"ondisk_log_size":2158,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5997068,"kb_used_data":398016,"kb_used_omap":507,"kb_used_meta":5595332,"kb_avail":650218996,"statfs":{"total":671965249536,"available":665824251904,"internally_reserved":0,"allocated":407568384,"data_stored":369014121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":519649,"internal_metadata":5729620511},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002228"},"pg_stats":[{"pgid":"49.f","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662346+0000","last_change":"2024-09-21T10:42:09.581683+0000","last_active":"2024-09-21T10:42:13.662346+0000","last_peered":"2024-09-21T10:42:13.662346+0000","last_clean":"2024-09-21T10:42:13.662346+0000","last_became_active":"2024-09-21T10:42:09.581391+0000","last_became_peered":"2024-09-21T10:42:09.581391+0000","last_unstale":"2024-09-21T10:42:13.662346+0000","last_undegraded":"2024-09-21T10:42:13.662346+0000","last_fullsized":"2024-09-21T10:42:13.662346+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:02.046573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.e","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662080+0000","last_change":"2024-09-21T10:42:09.582776+0000","last_active":"2024-09-21T10:42:13.662080+0000","last_peered":"2024-09-21T10:42:13.662080+0000","last_clean":"2024-09-21T10:42:13.662080+0000","last_became_active":"2024-09-21T10:42:09.582392+0000","last_became_peered":"2024-09-21T10:42:09.582392+0000","last_unstale":"2024-09-21T10:42:13.662080+0000","last_undegraded":"2024-09-21T10:42:13.662080+0000","last_fullsized":"2024-09-21T10:42:13.662080+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:49:48.719554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.d","version":"0'0","reported_seq":11,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517697+0000","last_change":"2024-09-21T10:42:09.579350+0000","last_active":"2024-09-21T10:42:11.517697+0000","last_peered":"2024-09-21T10:42:11.517697+0000","last_clean":"2024-09-21T10:42:11.517697+0000","last_became_active":"2024-09-21T10:42:09.579050+0000","last_became_peered":"2024-09-21T10:42:09.579050+0000","last_unstale":"2024-09-21T10:42:11.517697+0000","last_undegraded":"2024-09-21T10:42:11.517697+0000","last_fullsized":"2024-09-21T10:42:11.517697+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:01:49.319549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.c","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597627+0000","last_change":"2024-09-21T10:42:09.581463+0000","last_active":"2024-09-21T10:42:13.597627+0000","last_peered":"2024-09-21T10:42:13.597627+0000","last_clean":"2024-09-21T10:42:13.597627+0000","last_became_active":"2024-09-21T10:42:09.581139+0000","last_became_peered":"2024-09-21T10:42:09.581139+0000","last_unstale":"2024-09-21T10:42:13.597627+0000","last_undegraded":"2024-09-21T10:42:13.597627+0000","last_fullsized":"2024-09-21T10:42:13.597627+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:19.680248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.b","version":"0'0","reported_seq":11,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516986+0000","last_change":"2024-09-21T10:42:09.578384+0000","last_active":"2024-09-21T10:42:11.516986+0000","last_peered":"2024-09-21T10:42:11.516986+0000","last_clean":"2024-09-21T10:42:11.516986+0000","last_became_active":"2024-09-21T10:42:09.578049+0000","last_became_peered":"2024-09-21T10:42:09.578049+0000","last_unstale":"2024-09-21T10:42:11.516986+0000","last_undegraded":"2024-09-21T10:42:11.516986+0000","last_fullsized":"2024-09-21T10:42:11.516986+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:30:05.248763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"49.a","version":"0'0","reported_seq":11,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517186+0000","last_change":"2024-09-21T10:42:09.581384+0000","last_active":"2024-09-21T10:42:11.517186+0000","last_peered":"2024-09-21T10:42:11.517186+0000","last_clean":"2024-09-21T10:42:11.517186+0000","last_became_active":"2024-09-21T10:42:09.581063+0000","last_became_peered":"2024-09-21T10:42:09.581063+0000","last_unstale":"2024-09-21T10:42:11.517186+0000","last_undegraded":"2024-09-21T10:42:11.517186+0000","last_fullsized":"2024-09-21T10:42:11.517186+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:46:53.038287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.9","version":"0'0","reported_seq":11,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516955+0000","last_change":"2024-09-21T10:42:09.579857+0000","last_active":"2024-09-21T10:42:11.516955+0000","last_peered":"2024-09-21T10:42:11.516955+0000","last_clean":"2024-09-21T10:42:11.516955+0000","last_became_active":"2024-09-21T10:42:09.579527+0000","last_became_peered":"2024-09-21T10:42:09.579527+0000","last_unstale":"2024-09-21T10:42:11.516955+0000","last_undegraded":"2024-09-21T10:42:11.516955+0000","last_fullsized":"2024-09-21T10:42:11.516955+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:24:18.217758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.8","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661550+0000","last_change":"2024-09-21T10:42:09.575454+0000","last_active":"2024-09-21T10:42:13.661550+0000","last_peered":"2024-09-21T10:42:13.661550+0000","last_clean":"2024-09-21T10:42:13.661550+0000","last_became_active":"2024-09-21T10:42:09.575180+0000","last_became_peered":"2024-09-21T10:42:09.575180+0000","last_unstale":"2024-09-21T10:42:13.661550+0000","last_undegraded":"2024-09-21T10:42:13.661550+0000","last_fullsized":"2024-09-21T10:42:13.661550+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:25:05.580839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"49.7","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662432+0000","last_change":"2024-09-21T10:42:09.583740+0000","last_active":"2024-09-21T10:42:13.662432+0000","last_peered":"2024-09-21T10:42:13.662432+0000","last_clean":"2024-09-21T10:42:13.662432+0000","last_became_active":"2024-09-21T10:42:09.582836+0000","last_became_peered":"2024-09-21T10:42:09.582836+0000","last_unstale":"2024-09-21T10:42:13.662432+0000","last_undegraded":"2024-09-21T10:42:13.662432+0000","last_fullsized":"2024-09-21T10:42:13.662432+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:57:11.245222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.6","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597323+0000","last_change":"2024-09-21T10:42:09.579241+0000","last_active":"2024-09-21T10:42:13.597323+0000","last_peered":"2024-09-21T10:42:13.597323+0000","last_clean":"2024-09-21T10:42:13.597323+0000","last_became_active":"2024-09-21T10:42:09.578975+0000","last_became_peered":"2024-09-21T10:42:09.578975+0000","last_unstale":"2024-09-21T10:42:13.597323+0000","last_undegraded":"2024-09-21T10:42:13.597323+0000","last_fullsized":"2024-09-21T10:42:13.597323+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:10:45.201602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"49.5","version":"0'0","reported_seq":11,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517618+0000","last_change":"2024-09-21T10:42:09.582200+0000","last_active":"2024-09-21T10:42:11.517618+0000","last_peered":"2024-09-21T10:42:11.517618+0000","last_clean":"2024-09-21T10:42:11.517618+0000","last_became_active":"2024-09-21T10:42:09.581861+0000","last_became_peered":"2024-09-21T10:42:09.581861+0000","last_unstale":"2024-09-21T10:42:11.517618+0000","last_undegraded":"2024-09-21T10:42:11.517618+0000","last_fullsized":"2024-09-21T10:42:11.517618+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:06:49.307459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"49.4","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661607+0000","last_change":"2024-09-21T10:42:09.580771+0000","last_active":"2024-09-21T10:42:13.661607+0000","last_peered":"2024-09-21T10:42:13.661607+0000","last_clean":"2024-09-21T10:42:13.661607+0000","last_became_active":"2024-09-21T10:42:09.580510+0000","last_became_peered":"2024-09-21T10:42:09.580510+0000","last_unstale":"2024-09-21T10:42:13.661607+0000","last_undegraded":"2024-09-21T10:42:13.661607+0000","last_fullsized":"2024-09-21T10:42:13.661607+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"49.3","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662947+0000","last_change":"2024-09-21T10:42:09.584021+0000","last_active":"2024-09-21T10:42:13.662947+0000","last_peered":"2024-09-21T10:42:13.662947+0000","last_clean":"2024-09-21T10:42:13.662947+0000","last_became_active":"2024-09-21T10:42:09.583513+0000","last_became_peered":"2024-09-21T10:42:09.583513+0000","last_unstale":"2024-09-21T10:42:13.662947+0000","last_undegraded":"2024-09-21T10:42:13.662947+0000","last_fullsized":"2024-09-21T10:42:13.662947+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:30:51.926519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"49.2","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662085+0000","last_change":"2024-09-21T10:42:09.580603+0000","last_active":"2024-09-21T10:42:13.662085+0000","last_peered":"2024-09-21T10:42:13.662085+0000","last_clean":"2024-09-21T10:42:13.662085+0000","last_became_active":"2024-09-21T10:42:09.580324+0000","last_became_peered":"2024-09-21T10:42:09.580324+0000","last_unstale":"2024-09-21T10:42:13.662085+0000","last_undegraded":"2024-09-21T10:42:13.662085+0000","last_fullsized":"2024-09-21T10:42:13.662085+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:32:09.613627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"49.1","version":"0'0","reported_seq":16,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662601+0000","last_change":"2024-09-21T10:42:09.578522+0000","last_active":"2024-09-21T10:42:13.662601+0000","last_peered":"2024-09-21T10:42:13.662601+0000","last_clean":"2024-09-21T10:42:13.662601+0000","last_became_active":"2024-09-21T10:42:09.578258+0000","last_became_peered":"2024-09-21T10:42:09.578258+0000","last_unstale":"2024-09-21T10:42:13.662601+0000","last_undegraded":"2024-09-21T10:42:13.662601+0000","last_fullsized":"2024-09-21T10:42:13.662601+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:13:59.124703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"49.0","version":"0'0","reported_seq":10,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.580510+0000","last_change":"2024-09-21T10:42:09.580192+0000","last_active":"2024-09-21T10:42:09.580510+0000","last_peered":"2024-09-21T10:42:09.580510+0000","last_clean":"2024-09-21T10:42:09.580510+0000","last_became_active":"2024-09-21T10:42:09.579862+0000","last_became_peered":"2024-09-21T10:42:09.579862+0000","last_unstale":"2024-09-21T10:42:09.580510+0000","last_undegraded":"2024-09-21T10:42:09.580510+0000","last_fullsized":"2024-09-21T10:42:09.580510+0000","mapping_epoch":1394,"log_start":"0'0","ondisk_log_start":"0'0","created":1394,"last_epoch_clean":1395,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:42:08.553581+0000","last_clean_scrub_stamp":"2024-09-21T10:42:08.553581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:56:38.481273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":79,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663032+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:42:13.663032+0000","last_peered":"2024-09-21T10:42:13.663032+0000","last_clean":"2024-09-21T10:42:13.663032+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:42:13.663032+0000","last_undegraded":"2024-09-21T10:42:13.663032+0000","last_fullsized":"2024-09-21T10:42:13.663032+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1290,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.576892+0000","last_change":"2024-09-21T10:36:21.845536+0000","last_active":"2024-09-21T10:42:09.576892+0000","last_peered":"2024-09-21T10:42:09.576892+0000","last_clean":"2024-09-21T10:42:09.576892+0000","last_became_active":"2024-09-21T10:36:21.845226+0000","last_became_peered":"2024-09-21T10:36:21.845226+0000","last_unstale":"2024-09-21T10:42:09.576892+0000","last_undegraded":"2024-09-21T10:42:09.576892+0000","last_fullsized":"2024-09-21T10:42:09.576892+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:00:36.638139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2116,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597977+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:42:13.597977+0000","last_peered":"2024-09-21T10:42:13.597977+0000","last_clean":"2024-09-21T10:42:13.597977+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:42:13.597977+0000","last_undegraded":"2024-09-21T10:42:13.597977+0000","last_fullsized":"2024-09-21T10:42:13.597977+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2134,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597309+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:42:13.597309+0000","last_peered":"2024-09-21T10:42:13.597309+0000","last_clean":"2024-09-21T10:42:13.597309+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:42:13.597309+0000","last_undegraded":"2024-09-21T10:42:13.597309+0000","last_fullsized":"2024-09-21T10:42:13.597309+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":165,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517987+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:42:11.517987+0000","last_peered":"2024-09-21T10:42:11.517987+0000","last_clean":"2024-09-21T10:42:11.517987+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:42:11.517987+0000","last_undegraded":"2024-09-21T10:42:11.517987+0000","last_fullsized":"2024-09-21T10:42:11.517987+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1318,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597568+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:42:13.597568+0000","last_peered":"2024-09-21T10:42:13.597568+0000","last_clean":"2024-09-21T10:42:13.597568+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:42:13.597568+0000","last_undegraded":"2024-09-21T10:42:13.597568+0000","last_fullsized":"2024-09-21T10:42:13.597568+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":165,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517740+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:42:11.517740+0000","last_peered":"2024-09-21T10:42:11.517740+0000","last_clean":"2024-09-21T10:42:11.517740+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:42:11.517740+0000","last_undegraded":"2024-09-21T10:42:11.517740+0000","last_fullsized":"2024-09-21T10:42:11.517740+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1405,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597433+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:42:13.597433+0000","last_peered":"2024-09-21T10:42:13.597433+0000","last_clean":"2024-09-21T10:42:13.597433+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:42:13.597433+0000","last_undegraded":"2024-09-21T10:42:13.597433+0000","last_fullsized":"2024-09-21T10:42:13.597433+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":73,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663031+0000","last_change":"2024-09-21T10:39:00.847895+0000","last_active":"2024-09-21T10:42:13.663031+0000","last_peered":"2024-09-21T10:42:13.663031+0000","last_clean":"2024-09-21T10:42:13.663031+0000","last_became_active":"2024-09-21T10:39:00.847533+0000","last_became_peered":"2024-09-21T10:39:00.847533+0000","last_unstale":"2024-09-21T10:42:13.663031+0000","last_undegraded":"2024-09-21T10:42:13.663031+0000","last_fullsized":"2024-09-21T10:42:13.663031+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:55.080913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":101,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661726+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:42:13.661726+0000","last_peered":"2024-09-21T10:42:13.661726+0000","last_clean":"2024-09-21T10:42:13.661726+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:42:13.661726+0000","last_undegraded":"2024-09-21T10:42:13.661726+0000","last_fullsized":"2024-09-21T10:42:13.661726+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":178,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661465+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:42:13.661465+0000","last_peered":"2024-09-21T10:42:13.661465+0000","last_clean":"2024-09-21T10:42:13.661465+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:42:13.661465+0000","last_undegraded":"2024-09-21T10:42:13.661465+0000","last_fullsized":"2024-09-21T10:42:13.661465+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2124,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597686+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:42:13.597686+0000","last_peered":"2024-09-21T10:42:13.597686+0000","last_clean":"2024-09-21T10:42:13.597686+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:42:13.597686+0000","last_undegraded":"2024-09-21T10:42:13.597686+0000","last_fullsized":"2024-09-21T10:42:13.597686+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":2222,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661353+0000","last_change":"2024-09-21T10:36:21.855474+0000","last_active":"2024-09-21T10:42:13.661353+0000","last_peered":"2024-09-21T10:42:13.661353+0000","last_clean":"2024-09-21T10:42:13.661353+0000","last_became_active":"2024-09-21T10:36:21.854904+0000","last_became_peered":"2024-09-21T10:36:21.854904+0000","last_unstale":"2024-09-21T10:42:13.661353+0000","last_undegraded":"2024-09-21T10:42:13.661353+0000","last_fullsized":"2024-09-21T10:42:13.661353+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:23:53.202978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2232,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663279+0000","last_change":"2024-09-21T10:08:55.100435+0000","last_active":"2024-09-21T10:42:13.663279+0000","last_peered":"2024-09-21T10:42:13.663279+0000","last_clean":"2024-09-21T10:42:13.663279+0000","last_became_active":"2024-09-21T10:08:55.100176+0000","last_became_peered":"2024-09-21T10:08:55.100176+0000","last_unstale":"2024-09-21T10:42:13.663279+0000","last_undegraded":"2024-09-21T10:42:13.663279+0000","last_fullsized":"2024-09-21T10:42:13.663279+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:41.317566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2149,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.598055+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:42:13.598055+0000","last_peered":"2024-09-21T10:42:13.598055+0000","last_clean":"2024-09-21T10:42:13.598055+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:42:13.598055+0000","last_undegraded":"2024-09-21T10:42:13.598055+0000","last_fullsized":"2024-09-21T10:42:13.598055+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1354,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.578305+0000","last_change":"2024-09-21T10:30:09.414957+0000","last_active":"2024-09-21T10:42:09.578305+0000","last_peered":"2024-09-21T10:42:09.578305+0000","last_clean":"2024-09-21T10:42:09.578305+0000","last_became_active":"2024-09-21T10:30:09.414656+0000","last_became_peered":"2024-09-21T10:30:09.414656+0000","last_unstale":"2024-09-21T10:42:09.578305+0000","last_undegraded":"2024-09-21T10:42:09.578305+0000","last_fullsized":"2024-09-21T10:42:09.578305+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:35:55.523839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2129,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.598249+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:42:13.598249+0000","last_peered":"2024-09-21T10:42:13.598249+0000","last_clean":"2024-09-21T10:42:13.598249+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:42:13.598249+0000","last_undegraded":"2024-09-21T10:42:13.598249+0000","last_fullsized":"2024-09-21T10:42:13.598249+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2241,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663373+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:42:13.663373+0000","last_peered":"2024-09-21T10:42:13.663373+0000","last_clean":"2024-09-21T10:42:13.663373+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:42:13.663373+0000","last_undegraded":"2024-09-21T10:42:13.663373+0000","last_fullsized":"2024-09-21T10:42:13.663373+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":258,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661889+0000","last_change":"2024-09-21T10:37:47.788643+0000","last_active":"2024-09-21T10:42:13.661889+0000","last_peered":"2024-09-21T10:42:13.661889+0000","last_clean":"2024-09-21T10:42:13.661889+0000","last_became_active":"2024-09-21T10:36:21.851489+0000","last_became_peered":"2024-09-21T10:36:21.851489+0000","last_unstale":"2024-09-21T10:42:13.661889+0000","last_undegraded":"2024-09-21T10:42:13.661889+0000","last_fullsized":"2024-09-21T10:42:13.661889+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:39.188916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2112,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597606+0000","last_change":"2024-09-21T10:26:04.193094+0000","last_active":"2024-09-21T10:42:13.597606+0000","last_peered":"2024-09-21T10:42:13.597606+0000","last_clean":"2024-09-21T10:42:13.597606+0000","last_became_active":"2024-09-21T10:26:04.192779+0000","last_became_peered":"2024-09-21T10:26:04.192779+0000","last_unstale":"2024-09-21T10:42:13.597606+0000","last_undegraded":"2024-09-21T10:42:13.597606+0000","last_fullsized":"2024-09-21T10:42:13.597606+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:54.472748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.518102+0000","last_change":"2024-09-21T10:30:09.412429+0000","last_active":"2024-09-21T10:42:11.518102+0000","last_peered":"2024-09-21T10:42:11.518102+0000","last_clean":"2024-09-21T10:42:11.518102+0000","last_became_active":"2024-09-21T10:30:09.412124+0000","last_became_peered":"2024-09-21T10:30:09.412124+0000","last_unstale":"2024-09-21T10:42:11.518102+0000","last_undegraded":"2024-09-21T10:42:11.518102+0000","last_fullsized":"2024-09-21T10:42:11.518102+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:26.335041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":1772,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661856+0000","last_change":"2024-09-21T10:36:21.871814+0000","last_active":"2024-09-21T10:42:13.661856+0000","last_peered":"2024-09-21T10:42:13.661856+0000","last_clean":"2024-09-21T10:42:13.661856+0000","last_became_active":"2024-09-21T10:36:21.858187+0000","last_became_peered":"2024-09-21T10:36:21.858187+0000","last_unstale":"2024-09-21T10:42:13.661856+0000","last_undegraded":"2024-09-21T10:42:13.661856+0000","last_fullsized":"2024-09-21T10:42:13.661856+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:20:35.490730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":2215,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661573+0000","last_change":"2024-09-21T10:26:04.185932+0000","last_active":"2024-09-21T10:42:13.661573+0000","last_peered":"2024-09-21T10:42:13.661573+0000","last_clean":"2024-09-21T10:42:13.661573+0000","last_became_active":"2024-09-21T10:26:04.185522+0000","last_became_peered":"2024-09-21T10:26:04.185522+0000","last_unstale":"2024-09-21T10:42:13.661573+0000","last_undegraded":"2024-09-21T10:42:13.661573+0000","last_fullsized":"2024-09-21T10:42:13.661573+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:47:08.108540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":412,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517650+0000","last_change":"2024-09-21T10:36:21.870261+0000","last_active":"2024-09-21T10:42:11.517650+0000","last_peered":"2024-09-21T10:42:11.517650+0000","last_clean":"2024-09-21T10:42:11.517650+0000","last_became_active":"2024-09-21T10:36:21.850193+0000","last_became_peered":"2024-09-21T10:36:21.850193+0000","last_unstale":"2024-09-21T10:42:11.517650+0000","last_undegraded":"2024-09-21T10:42:11.517650+0000","last_fullsized":"2024-09-21T10:42:11.517650+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:20:30.077264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1255,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577463+0000","last_change":"2024-09-21T10:36:21.857798+0000","last_active":"2024-09-21T10:42:09.577463+0000","last_peered":"2024-09-21T10:42:09.577463+0000","last_clean":"2024-09-21T10:42:09.577463+0000","last_became_active":"2024-09-21T10:36:21.857440+0000","last_became_peered":"2024-09-21T10:36:21.857440+0000","last_unstale":"2024-09-21T10:42:09.577463+0000","last_undegraded":"2024-09-21T10:42:09.577463+0000","last_fullsized":"2024-09-21T10:42:09.577463+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:42:38.072068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":371,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597753+0000","last_change":"2024-09-21T10:34:27.532184+0000","last_active":"2024-09-21T10:42:13.597753+0000","last_peered":"2024-09-21T10:42:13.597753+0000","last_clean":"2024-09-21T10:42:13.597753+0000","last_became_active":"2024-09-21T10:30:09.413356+0000","last_became_peered":"2024-09-21T10:30:09.413356+0000","last_unstale":"2024-09-21T10:42:13.597753+0000","last_undegraded":"2024-09-21T10:42:13.597753+0000","last_fullsized":"2024-09-21T10:42:13.597753+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-30T08:51:42.538339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109226,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1290,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517203+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:42:11.517203+0000","last_peered":"2024-09-21T10:42:11.517203+0000","last_clean":"2024-09-21T10:42:11.517203+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:42:11.517203+0000","last_undegraded":"2024-09-21T10:42:11.517203+0000","last_fullsized":"2024-09-21T10:42:11.517203+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1350,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.578464+0000","last_change":"2024-09-21T10:26:04.188931+0000","last_active":"2024-09-21T10:42:09.578464+0000","last_peered":"2024-09-21T10:42:09.578464+0000","last_clean":"2024-09-21T10:42:09.578464+0000","last_became_active":"2024-09-21T10:26:04.188653+0000","last_became_peered":"2024-09-21T10:26:04.188653+0000","last_unstale":"2024-09-21T10:42:09.578464+0000","last_undegraded":"2024-09-21T10:42:09.578464+0000","last_fullsized":"2024-09-21T10:42:09.578464+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:46.670051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":106,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663146+0000","last_change":"2024-09-21T10:37:41.926686+0000","last_active":"2024-09-21T10:42:13.663146+0000","last_peered":"2024-09-21T10:42:13.663146+0000","last_clean":"2024-09-21T10:42:13.663146+0000","last_became_active":"2024-09-21T10:37:41.922706+0000","last_became_peered":"2024-09-21T10:37:41.922706+0000","last_unstale":"2024-09-21T10:42:13.663146+0000","last_undegraded":"2024-09-21T10:42:13.663146+0000","last_fullsized":"2024-09-21T10:42:13.663146+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:32:08.955843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":160,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662423+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:42:13.662423+0000","last_peered":"2024-09-21T10:42:13.662423+0000","last_clean":"2024-09-21T10:42:13.662423+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:42:13.662423+0000","last_undegraded":"2024-09-21T10:42:13.662423+0000","last_fullsized":"2024-09-21T10:42:13.662423+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":53,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516844+0000","last_change":"2024-09-21T10:39:00.847173+0000","last_active":"2024-09-21T10:42:11.516844+0000","last_peered":"2024-09-21T10:42:11.516844+0000","last_clean":"2024-09-21T10:42:11.516844+0000","last_became_active":"2024-09-21T10:39:00.846851+0000","last_became_peered":"2024-09-21T10:39:00.846851+0000","last_unstale":"2024-09-21T10:42:11.516844+0000","last_undegraded":"2024-09-21T10:42:11.516844+0000","last_fullsized":"2024-09-21T10:42:11.516844+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:11:47.133721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2232,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662160+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:42:13.662160+0000","last_peered":"2024-09-21T10:42:13.662160+0000","last_clean":"2024-09-21T10:42:13.662160+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:42:13.662160+0000","last_undegraded":"2024-09-21T10:42:13.662160+0000","last_fullsized":"2024-09-21T10:42:13.662160+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":2111,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.598965+0000","last_change":"2024-09-21T10:08:55.106680+0000","last_active":"2024-09-21T10:42:13.598965+0000","last_peered":"2024-09-21T10:42:13.598965+0000","last_clean":"2024-09-21T10:42:13.598965+0000","last_became_active":"2024-09-21T10:08:55.106234+0000","last_became_peered":"2024-09-21T10:08:55.106234+0000","last_unstale":"2024-09-21T10:42:13.598965+0000","last_undegraded":"2024-09-21T10:42:13.598965+0000","last_fullsized":"2024-09-21T10:42:13.598965+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.387611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":521,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577796+0000","last_change":"2024-09-21T10:33:24.801854+0000","last_active":"2024-09-21T10:42:09.577796+0000","last_peered":"2024-09-21T10:42:09.577796+0000","last_clean":"2024-09-21T10:42:09.577796+0000","last_became_active":"2024-09-21T10:29:57.001924+0000","last_became_peered":"2024-09-21T10:29:57.001924+0000","last_unstale":"2024-09-21T10:42:09.577796+0000","last_undegraded":"2024-09-21T10:42:09.577796+0000","last_fullsized":"2024-09-21T10:42:09.577796+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:24.801786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:24.801786+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:30:56.332166+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2234,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663361+0000","last_change":"2024-09-21T10:30:09.405909+0000","last_active":"2024-09-21T10:42:13.663361+0000","last_peered":"2024-09-21T10:42:13.663361+0000","last_clean":"2024-09-21T10:42:13.663361+0000","last_became_active":"2024-09-21T10:30:09.405506+0000","last_became_peered":"2024-09-21T10:30:09.405506+0000","last_unstale":"2024-09-21T10:42:13.663361+0000","last_undegraded":"2024-09-21T10:42:13.663361+0000","last_fullsized":"2024-09-21T10:42:13.663361+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:11.673342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1414,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517757+0000","last_change":"2024-09-21T10:37:41.922996+0000","last_active":"2024-09-21T10:42:11.517757+0000","last_peered":"2024-09-21T10:42:11.517757+0000","last_clean":"2024-09-21T10:42:11.517757+0000","last_became_active":"2024-09-21T10:37:41.922703+0000","last_became_peered":"2024-09-21T10:37:41.922703+0000","last_unstale":"2024-09-21T10:42:11.517757+0000","last_undegraded":"2024-09-21T10:42:11.517757+0000","last_fullsized":"2024-09-21T10:42:11.517757+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:23.614770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2220,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661969+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:42:13.661969+0000","last_peered":"2024-09-21T10:42:13.661969+0000","last_clean":"2024-09-21T10:42:13.661969+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:42:13.661969+0000","last_undegraded":"2024-09-21T10:42:13.661969+0000","last_fullsized":"2024-09-21T10:42:13.661969+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":521,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.578139+0000","last_change":"2024-09-21T10:33:26.846529+0000","last_active":"2024-09-21T10:42:09.578139+0000","last_peered":"2024-09-21T10:42:09.578139+0000","last_clean":"2024-09-21T10:42:09.578139+0000","last_became_active":"2024-09-21T10:29:57.002165+0000","last_became_peered":"2024-09-21T10:29:57.002165+0000","last_unstale":"2024-09-21T10:42:09.578139+0000","last_undegraded":"2024-09-21T10:42:09.578139+0000","last_fullsized":"2024-09-21T10:42:09.578139+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1258,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:26.846473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:26.846473+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:48:27.604513+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2225,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661888+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:42:13.661888+0000","last_peered":"2024-09-21T10:42:13.661888+0000","last_clean":"2024-09-21T10:42:13.661888+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:42:13.661888+0000","last_undegraded":"2024-09-21T10:42:13.661888+0000","last_fullsized":"2024-09-21T10:42:13.661888+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2220,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662137+0000","last_change":"2024-09-21T10:06:42.629339+0000","last_active":"2024-09-21T10:42:13.662137+0000","last_peered":"2024-09-21T10:42:13.662137+0000","last_clean":"2024-09-21T10:42:13.662137+0000","last_became_active":"2024-09-21T10:06:42.627246+0000","last_became_peered":"2024-09-21T10:06:42.627246+0000","last_unstale":"2024-09-21T10:42:13.662137+0000","last_undegraded":"2024-09-21T10:42:13.662137+0000","last_fullsized":"2024-09-21T10:42:13.662137+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:41:39.862777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":147,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517876+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:42:11.517876+0000","last_peered":"2024-09-21T10:42:11.517876+0000","last_clean":"2024-09-21T10:42:11.517876+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:42:11.517876+0000","last_undegraded":"2024-09-21T10:42:11.517876+0000","last_fullsized":"2024-09-21T10:42:11.517876+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":157,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517577+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:42:11.517577+0000","last_peered":"2024-09-21T10:42:11.517577+0000","last_clean":"2024-09-21T10:42:11.517577+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:42:11.517577+0000","last_undegraded":"2024-09-21T10:42:11.517577+0000","last_fullsized":"2024-09-21T10:42:11.517577+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2217,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662056+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:42:13.662056+0000","last_peered":"2024-09-21T10:42:13.662056+0000","last_clean":"2024-09-21T10:42:13.662056+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:42:13.662056+0000","last_undegraded":"2024-09-21T10:42:13.662056+0000","last_fullsized":"2024-09-21T10:42:13.662056+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1336,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663276+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:42:13.663276+0000","last_peered":"2024-09-21T10:42:13.663276+0000","last_clean":"2024-09-21T10:42:13.663276+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:42:13.663276+0000","last_undegraded":"2024-09-21T10:42:13.663276+0000","last_fullsized":"2024-09-21T10:42:13.663276+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":151,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517311+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:42:11.517311+0000","last_peered":"2024-09-21T10:42:11.517311+0000","last_clean":"2024-09-21T10:42:11.517311+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:42:11.517311+0000","last_undegraded":"2024-09-21T10:42:11.517311+0000","last_fullsized":"2024-09-21T10:42:11.517311+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":539,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662864+0000","last_change":"2024-09-21T10:37:38.606346+0000","last_active":"2024-09-21T10:42:13.662864+0000","last_peered":"2024-09-21T10:42:13.662864+0000","last_clean":"2024-09-21T10:42:13.662864+0000","last_became_active":"2024-09-21T10:36:21.852296+0000","last_became_peered":"2024-09-21T10:36:21.852296+0000","last_unstale":"2024-09-21T10:42:13.662864+0000","last_undegraded":"2024-09-21T10:42:13.662864+0000","last_fullsized":"2024-09-21T10:42:13.662864+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:39:16.461221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":2235,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662561+0000","last_change":"2024-09-21T10:37:41.924374+0000","last_active":"2024-09-21T10:42:13.662561+0000","last_peered":"2024-09-21T10:42:13.662561+0000","last_clean":"2024-09-21T10:42:13.662561+0000","last_became_active":"2024-09-21T10:37:41.922589+0000","last_became_peered":"2024-09-21T10:37:41.922589+0000","last_unstale":"2024-09-21T10:42:13.662561+0000","last_undegraded":"2024-09-21T10:42:13.662561+0000","last_fullsized":"2024-09-21T10:42:13.662561+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:33:51.987384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2217,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661194+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:42:13.661194+0000","last_peered":"2024-09-21T10:42:13.661194+0000","last_clean":"2024-09-21T10:42:13.661194+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:42:13.661194+0000","last_undegraded":"2024-09-21T10:42:13.661194+0000","last_fullsized":"2024-09-21T10:42:13.661194+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2148,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661991+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:42:13.661991+0000","last_peered":"2024-09-21T10:42:13.661991+0000","last_clean":"2024-09-21T10:42:13.661991+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:42:13.661991+0000","last_undegraded":"2024-09-21T10:42:13.661991+0000","last_fullsized":"2024-09-21T10:42:13.661991+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":402,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661856+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:42:13.661856+0000","last_peered":"2024-09-21T10:42:13.661856+0000","last_clean":"2024-09-21T10:42:13.661856+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:42:13.661856+0000","last_undegraded":"2024-09-21T10:42:13.661856+0000","last_fullsized":"2024-09-21T10:42:13.661856+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":73,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663037+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:42:13.663037+0000","last_peered":"2024-09-21T10:42:13.663037+0000","last_clean":"2024-09-21T10:42:13.663037+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:42:13.663037+0000","last_undegraded":"2024-09-21T10:42:13.663037+0000","last_fullsized":"2024-09-21T10:42:13.663037+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2249,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662824+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:42:13.662824+0000","last_peered":"2024-09-21T10:42:13.662824+0000","last_clean":"2024-09-21T10:42:13.662824+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:42:13.662824+0000","last_undegraded":"2024-09-21T10:42:13.662824+0000","last_fullsized":"2024-09-21T10:42:13.662824+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1384,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516744+0000","last_change":"2024-09-21T10:06:42.628962+0000","last_active":"2024-09-21T10:42:11.516744+0000","last_peered":"2024-09-21T10:42:11.516744+0000","last_clean":"2024-09-21T10:42:11.516744+0000","last_became_active":"2024-09-21T10:06:42.628681+0000","last_became_peered":"2024-09-21T10:06:42.628681+0000","last_unstale":"2024-09-21T10:42:11.516744+0000","last_undegraded":"2024-09-21T10:42:11.516744+0000","last_fullsized":"2024-09-21T10:42:11.516744+0000","mapping_epoch":542,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":543,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:12:19.557017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1356,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.578534+0000","last_change":"2024-09-21T10:36:21.875392+0000","last_active":"2024-09-21T10:42:09.578534+0000","last_peered":"2024-09-21T10:42:09.578534+0000","last_clean":"2024-09-21T10:42:09.578534+0000","last_became_active":"2024-09-21T10:36:21.875022+0000","last_became_peered":"2024-09-21T10:36:21.875022+0000","last_unstale":"2024-09-21T10:42:09.578534+0000","last_undegraded":"2024-09-21T10:42:09.578534+0000","last_fullsized":"2024-09-21T10:42:09.578534+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:43:38.598732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2133,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597662+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:42:13.597662+0000","last_peered":"2024-09-21T10:42:13.597662+0000","last_clean":"2024-09-21T10:42:13.597662+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:42:13.597662+0000","last_undegraded":"2024-09-21T10:42:13.597662+0000","last_fullsized":"2024-09-21T10:42:13.597662+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":151,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517497+0000","last_change":"2024-09-21T10:30:09.434731+0000","last_active":"2024-09-21T10:42:11.517497+0000","last_peered":"2024-09-21T10:42:11.517497+0000","last_clean":"2024-09-21T10:42:11.517497+0000","last_became_active":"2024-09-21T10:30:09.410046+0000","last_became_peered":"2024-09-21T10:30:09.410046+0000","last_unstale":"2024-09-21T10:42:11.517497+0000","last_undegraded":"2024-09-21T10:42:11.517497+0000","last_fullsized":"2024-09-21T10:42:11.517497+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:41.009293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1567,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517108+0000","last_change":"2024-09-21T10:37:41.922049+0000","last_active":"2024-09-21T10:42:11.517108+0000","last_peered":"2024-09-21T10:42:11.517108+0000","last_clean":"2024-09-21T10:42:11.517108+0000","last_became_active":"2024-09-21T10:37:41.921709+0000","last_became_peered":"2024-09-21T10:37:41.921709+0000","last_unstale":"2024-09-21T10:42:11.517108+0000","last_undegraded":"2024-09-21T10:42:11.517108+0000","last_fullsized":"2024-09-21T10:42:11.517108+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:12.446157+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":147,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517392+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:42:11.517392+0000","last_peered":"2024-09-21T10:42:11.517392+0000","last_clean":"2024-09-21T10:42:11.517392+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:42:11.517392+0000","last_undegraded":"2024-09-21T10:42:11.517392+0000","last_fullsized":"2024-09-21T10:42:11.517392+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2222,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662519+0000","last_change":"2024-09-21T10:36:21.856389+0000","last_active":"2024-09-21T10:42:13.662519+0000","last_peered":"2024-09-21T10:42:13.662519+0000","last_clean":"2024-09-21T10:42:13.662519+0000","last_became_active":"2024-09-21T10:36:21.853888+0000","last_became_peered":"2024-09-21T10:36:21.853888+0000","last_unstale":"2024-09-21T10:42:13.662519+0000","last_undegraded":"2024-09-21T10:42:13.662519+0000","last_fullsized":"2024-09-21T10:42:13.662519+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:53:32.595835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2120,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.598157+0000","last_change":"2024-09-21T10:08:55.116462+0000","last_active":"2024-09-21T10:42:13.598157+0000","last_peered":"2024-09-21T10:42:13.598157+0000","last_clean":"2024-09-21T10:42:13.598157+0000","last_became_active":"2024-09-21T10:08:55.116125+0000","last_became_peered":"2024-09-21T10:08:55.116125+0000","last_unstale":"2024-09-21T10:42:13.598157+0000","last_undegraded":"2024-09-21T10:42:13.598157+0000","last_fullsized":"2024-09-21T10:42:13.598157+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:05.329122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":386,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661757+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:42:13.661757+0000","last_peered":"2024-09-21T10:42:13.661757+0000","last_clean":"2024-09-21T10:42:13.661757+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:42:13.661757+0000","last_undegraded":"2024-09-21T10:42:13.661757+0000","last_fullsized":"2024-09-21T10:42:13.661757+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1255,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577893+0000","last_change":"2024-09-21T10:36:21.862410+0000","last_active":"2024-09-21T10:42:09.577893+0000","last_peered":"2024-09-21T10:42:09.577893+0000","last_clean":"2024-09-21T10:42:09.577893+0000","last_became_active":"2024-09-21T10:36:21.861678+0000","last_became_peered":"2024-09-21T10:36:21.861678+0000","last_unstale":"2024-09-21T10:42:09.577893+0000","last_undegraded":"2024-09-21T10:42:09.577893+0000","last_fullsized":"2024-09-21T10:42:09.577893+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:37:14.365970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1275,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662090+0000","last_change":"2024-09-21T10:36:21.855781+0000","last_active":"2024-09-21T10:42:13.662090+0000","last_peered":"2024-09-21T10:42:13.662090+0000","last_clean":"2024-09-21T10:42:13.662090+0000","last_became_active":"2024-09-21T10:36:21.855442+0000","last_became_peered":"2024-09-21T10:36:21.855442+0000","last_unstale":"2024-09-21T10:42:13.662090+0000","last_undegraded":"2024-09-21T10:42:13.662090+0000","last_fullsized":"2024-09-21T10:42:13.662090+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:15:54.569126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":151,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517036+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:42:11.517036+0000","last_peered":"2024-09-21T10:42:11.517036+0000","last_clean":"2024-09-21T10:42:11.517036+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:42:11.517036+0000","last_undegraded":"2024-09-21T10:42:11.517036+0000","last_fullsized":"2024-09-21T10:42:11.517036+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1283,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662482+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:42:13.662482+0000","last_peered":"2024-09-21T10:42:13.662482+0000","last_clean":"2024-09-21T10:42:13.662482+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:42:13.662482+0000","last_undegraded":"2024-09-21T10:42:13.662482+0000","last_fullsized":"2024-09-21T10:42:13.662482+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1362,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577433+0000","last_change":"2024-09-21T10:36:21.851207+0000","last_active":"2024-09-21T10:42:09.577433+0000","last_peered":"2024-09-21T10:42:09.577433+0000","last_clean":"2024-09-21T10:42:09.577433+0000","last_became_active":"2024-09-21T10:36:21.849996+0000","last_became_peered":"2024-09-21T10:36:21.849996+0000","last_unstale":"2024-09-21T10:42:09.577433+0000","last_undegraded":"2024-09-21T10:42:09.577433+0000","last_fullsized":"2024-09-21T10:42:09.577433+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:08:12.041847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1518,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517053+0000","last_change":"2024-09-21T10:28:22.186748+0000","last_active":"2024-09-21T10:42:11.517053+0000","last_peered":"2024-09-21T10:42:11.517053+0000","last_clean":"2024-09-21T10:42:11.517053+0000","last_became_active":"2024-09-21T10:28:17.260626+0000","last_became_peered":"2024-09-21T10:28:17.260626+0000","last_unstale":"2024-09-21T10:42:11.517053+0000","last_undegraded":"2024-09-21T10:42:11.517053+0000","last_fullsized":"2024-09-21T10:42:11.517053+0000","mapping_epoch":1234,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1235,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:22.186688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:22.186688+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:46.725006+0000","scrub_duration":0.0019925889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1335,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.598137+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:42:13.598137+0000","last_peered":"2024-09-21T10:42:13.598137+0000","last_clean":"2024-09-21T10:42:13.598137+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:42:13.598137+0000","last_undegraded":"2024-09-21T10:42:13.598137+0000","last_fullsized":"2024-09-21T10:42:13.598137+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":347,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597319+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:42:13.597319+0000","last_peered":"2024-09-21T10:42:13.597319+0000","last_clean":"2024-09-21T10:42:13.597319+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:42:13.597319+0000","last_undegraded":"2024-09-21T10:42:13.597319+0000","last_fullsized":"2024-09-21T10:42:13.597319+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":92,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517832+0000","last_change":"2024-09-21T10:37:37.360857+0000","last_active":"2024-09-21T10:42:11.517832+0000","last_peered":"2024-09-21T10:42:11.517832+0000","last_clean":"2024-09-21T10:42:11.517832+0000","last_became_active":"2024-09-21T10:36:21.849339+0000","last_became_peered":"2024-09-21T10:36:21.849339+0000","last_unstale":"2024-09-21T10:42:11.517832+0000","last_undegraded":"2024-09-21T10:42:11.517832+0000","last_fullsized":"2024-09-21T10:42:11.517832+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.360779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.360779+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:47:25.310501+0000","scrub_duration":0.001559979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2248,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663285+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:42:13.663285+0000","last_peered":"2024-09-21T10:42:13.663285+0000","last_clean":"2024-09-21T10:42:13.663285+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:42:13.663285+0000","last_undegraded":"2024-09-21T10:42:13.663285+0000","last_fullsized":"2024-09-21T10:42:13.663285+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":147,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517696+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:42:11.517696+0000","last_peered":"2024-09-21T10:42:11.517696+0000","last_clean":"2024-09-21T10:42:11.517696+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:42:11.517696+0000","last_undegraded":"2024-09-21T10:42:11.517696+0000","last_fullsized":"2024-09-21T10:42:11.517696+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":540,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.578226+0000","last_change":"2024-09-21T10:36:21.859383+0000","last_active":"2024-09-21T10:42:09.578226+0000","last_peered":"2024-09-21T10:42:09.578226+0000","last_clean":"2024-09-21T10:42:09.578226+0000","last_became_active":"2024-09-21T10:36:21.858363+0000","last_became_peered":"2024-09-21T10:36:21.858363+0000","last_unstale":"2024-09-21T10:42:09.578226+0000","last_undegraded":"2024-09-21T10:42:09.578226+0000","last_fullsized":"2024-09-21T10:42:09.578226+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:39.741139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:39.741139+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:11.365379+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1334,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577238+0000","last_change":"2024-09-21T10:26:04.200797+0000","last_active":"2024-09-21T10:42:09.577238+0000","last_peered":"2024-09-21T10:42:09.577238+0000","last_clean":"2024-09-21T10:42:09.577238+0000","last_became_active":"2024-09-21T10:26:04.200105+0000","last_became_peered":"2024-09-21T10:26:04.200105+0000","last_unstale":"2024-09-21T10:42:09.577238+0000","last_undegraded":"2024-09-21T10:42:09.577238+0000","last_fullsized":"2024-09-21T10:42:09.577238+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:54.174940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1342,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517868+0000","last_change":"2024-09-21T10:34:18.518245+0000","last_active":"2024-09-21T10:42:11.517868+0000","last_peered":"2024-09-21T10:42:11.517868+0000","last_clean":"2024-09-21T10:42:11.517868+0000","last_became_active":"2024-09-21T10:34:18.517911+0000","last_became_peered":"2024-09-21T10:34:18.517911+0000","last_unstale":"2024-09-21T10:42:11.517868+0000","last_undegraded":"2024-09-21T10:42:11.517868+0000","last_fullsized":"2024-09-21T10:42:11.517868+0000","mapping_epoch":1301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:24:33.032132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":386,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597713+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:42:13.597713+0000","last_peered":"2024-09-21T10:42:13.597713+0000","last_clean":"2024-09-21T10:42:13.597713+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:42:13.597713+0000","last_undegraded":"2024-09-21T10:42:13.597713+0000","last_fullsized":"2024-09-21T10:42:13.597713+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1318,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661401+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:42:13.661401+0000","last_peered":"2024-09-21T10:42:13.661401+0000","last_clean":"2024-09-21T10:42:13.661401+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:42:13.661401+0000","last_undegraded":"2024-09-21T10:42:13.661401+0000","last_fullsized":"2024-09-21T10:42:13.661401+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":1402,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516807+0000","last_change":"2024-09-21T10:38:34.901523+0000","last_active":"2024-09-21T10:42:11.516807+0000","last_peered":"2024-09-21T10:42:11.516807+0000","last_clean":"2024-09-21T10:42:11.516807+0000","last_became_active":"2024-09-21T10:38:34.887435+0000","last_became_peered":"2024-09-21T10:38:34.887435+0000","last_unstale":"2024-09-21T10:42:11.516807+0000","last_undegraded":"2024-09-21T10:42:11.516807+0000","last_fullsized":"2024-09-21T10:42:11.516807+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:27.993702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":144,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517524+0000","last_change":"2024-09-21T10:36:21.853839+0000","last_active":"2024-09-21T10:42:11.517524+0000","last_peered":"2024-09-21T10:42:11.517524+0000","last_clean":"2024-09-21T10:42:11.517524+0000","last_became_active":"2024-09-21T10:36:21.850944+0000","last_became_peered":"2024-09-21T10:36:21.850944+0000","last_unstale":"2024-09-21T10:42:11.517524+0000","last_undegraded":"2024-09-21T10:42:11.517524+0000","last_fullsized":"2024-09-21T10:42:11.517524+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:27.552710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:27.552710+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:25:43.222653+0000","scrub_duration":0.001292316,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":147,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517112+0000","last_change":"2024-09-21T10:30:09.406216+0000","last_active":"2024-09-21T10:42:11.517112+0000","last_peered":"2024-09-21T10:42:11.517112+0000","last_clean":"2024-09-21T10:42:11.517112+0000","last_became_active":"2024-09-21T10:30:09.405145+0000","last_became_peered":"2024-09-21T10:30:09.405145+0000","last_unstale":"2024-09-21T10:42:11.517112+0000","last_undegraded":"2024-09-21T10:42:11.517112+0000","last_fullsized":"2024-09-21T10:42:11.517112+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:38:09.582077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1415,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516972+0000","last_change":"2024-09-21T10:38:34.891682+0000","last_active":"2024-09-21T10:42:11.516972+0000","last_peered":"2024-09-21T10:42:11.516972+0000","last_clean":"2024-09-21T10:42:11.516972+0000","last_became_active":"2024-09-21T10:38:34.885292+0000","last_became_peered":"2024-09-21T10:38:34.885292+0000","last_unstale":"2024-09-21T10:42:11.516972+0000","last_undegraded":"2024-09-21T10:42:11.516972+0000","last_fullsized":"2024-09-21T10:42:11.516972+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:30.111501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":134,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517082+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:42:11.517082+0000","last_peered":"2024-09-21T10:42:11.517082+0000","last_clean":"2024-09-21T10:42:11.517082+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:42:11.517082+0000","last_undegraded":"2024-09-21T10:42:11.517082+0000","last_fullsized":"2024-09-21T10:42:11.517082+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1431,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577220+0000","last_change":"2024-09-21T10:39:00.852998+0000","last_active":"2024-09-21T10:42:09.577220+0000","last_peered":"2024-09-21T10:42:09.577220+0000","last_clean":"2024-09-21T10:42:09.577220+0000","last_became_active":"2024-09-21T10:39:00.846811+0000","last_became_peered":"2024-09-21T10:39:00.846811+0000","last_unstale":"2024-09-21T10:42:09.577220+0000","last_undegraded":"2024-09-21T10:42:09.577220+0000","last_fullsized":"2024-09-21T10:42:09.577220+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:20:14.374741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2109,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.598864+0000","last_change":"2024-09-21T10:08:55.139043+0000","last_active":"2024-09-21T10:42:13.598864+0000","last_peered":"2024-09-21T10:42:13.598864+0000","last_clean":"2024-09-21T10:42:13.598864+0000","last_became_active":"2024-09-21T10:08:55.138596+0000","last_became_peered":"2024-09-21T10:08:55.138596+0000","last_unstale":"2024-09-21T10:42:13.598864+0000","last_undegraded":"2024-09-21T10:42:13.598864+0000","last_fullsized":"2024-09-21T10:42:13.598864+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:26:35.662928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":393,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.661655+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:42:13.661655+0000","last_peered":"2024-09-21T10:42:13.661655+0000","last_clean":"2024-09-21T10:42:13.661655+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:42:13.661655+0000","last_undegraded":"2024-09-21T10:42:13.661655+0000","last_fullsized":"2024-09-21T10:42:13.661655+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1250,"reported_epoch":1395,"state":"active+clean","last_fresh":"2024-09-21T10:42:09.577128+0000","last_change":"2024-09-21T10:36:21.858915+0000","last_active":"2024-09-21T10:42:09.577128+0000","last_peered":"2024-09-21T10:42:09.577128+0000","last_clean":"2024-09-21T10:42:09.577128+0000","last_became_active":"2024-09-21T10:36:21.858613+0000","last_became_peered":"2024-09-21T10:36:21.858613+0000","last_unstale":"2024-09-21T10:42:09.577128+0000","last_undegraded":"2024-09-21T10:42:09.577128+0000","last_fullsized":"2024-09-21T10:42:09.577128+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:24.329558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2228,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.663250+0000","last_change":"2024-09-21T10:08:55.100290+0000","last_active":"2024-09-21T10:42:13.663250+0000","last_peered":"2024-09-21T10:42:13.663250+0000","last_clean":"2024-09-21T10:42:13.663250+0000","last_became_active":"2024-09-21T10:08:55.100032+0000","last_became_peered":"2024-09-21T10:08:55.100032+0000","last_unstale":"2024-09-21T10:42:13.663250+0000","last_undegraded":"2024-09-21T10:42:13.663250+0000","last_fullsized":"2024-09-21T10:42:13.663250+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:09:50.645494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":134,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.516945+0000","last_change":"2024-09-21T10:33:29.210884+0000","last_active":"2024-09-21T10:42:11.516945+0000","last_peered":"2024-09-21T10:42:11.516945+0000","last_clean":"2024-09-21T10:42:11.516945+0000","last_became_active":"2024-09-21T10:33:20.896951+0000","last_became_peered":"2024-09-21T10:33:20.896951+0000","last_unstale":"2024-09-21T10:42:11.516945+0000","last_undegraded":"2024-09-21T10:42:11.516945+0000","last_fullsized":"2024-09-21T10:42:11.516945+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:44.366383+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2228,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662904+0000","last_change":"2024-09-21T10:08:55.108305+0000","last_active":"2024-09-21T10:42:13.662904+0000","last_peered":"2024-09-21T10:42:13.662904+0000","last_clean":"2024-09-21T10:42:13.662904+0000","last_became_active":"2024-09-21T10:08:55.103250+0000","last_became_peered":"2024-09-21T10:08:55.103250+0000","last_unstale":"2024-09-21T10:42:13.662904+0000","last_undegraded":"2024-09-21T10:42:13.662904+0000","last_fullsized":"2024-09-21T10:42:13.662904+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:56.339476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":157,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517967+0000","last_change":"2024-09-21T10:38:38.962357+0000","last_active":"2024-09-21T10:42:11.517967+0000","last_peered":"2024-09-21T10:42:11.517967+0000","last_clean":"2024-09-21T10:42:11.517967+0000","last_became_active":"2024-09-21T10:38:38.962017+0000","last_became_peered":"2024-09-21T10:38:38.962017+0000","last_unstale":"2024-09-21T10:42:11.517967+0000","last_undegraded":"2024-09-21T10:42:11.517967+0000","last_fullsized":"2024-09-21T10:42:11.517967+0000","mapping_epoch":1363,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1364,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:05.355706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":109,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.662688+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:42:13.662688+0000","last_peered":"2024-09-21T10:42:13.662688+0000","last_clean":"2024-09-21T10:42:13.662688+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:42:13.662688+0000","last_undegraded":"2024-09-21T10:42:13.662688+0000","last_fullsized":"2024-09-21T10:42:13.662688+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":1403,"reported_epoch":1396,"state":"active+clean","last_fresh":"2024-09-21T10:42:11.517215+0000","last_change":"2024-09-21T10:36:21.877151+0000","last_active":"2024-09-21T10:42:11.517215+0000","last_peered":"2024-09-21T10:42:11.517215+0000","last_clean":"2024-09-21T10:42:11.517215+0000","last_became_active":"2024-09-21T10:36:21.850224+0000","last_became_peered":"2024-09-21T10:36:21.850224+0000","last_unstale":"2024-09-21T10:42:11.517215+0000","last_undegraded":"2024-09-21T10:42:11.517215+0000","last_fullsized":"2024-09-21T10:42:11.517215+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:44:54.347741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2118,"reported_epoch":1397,"state":"active+clean","last_fresh":"2024-09-21T10:42:13.597897+0000","last_change":"2024-09-21T10:08:55.126114+0000","last_active":"2024-09-21T10:42:13.597897+0000","last_peered":"2024-09-21T10:42:13.597897+0000","last_clean":"2024-09-21T10:42:13.597897+0000","last_became_active":"2024-09-21T10:08:55.125876+0000","last_became_peered":"2024-09-21T10:08:55.125876+0000","last_unstale":"2024-09-21T10:42:13.597897+0000","last_undegraded":"2024-09-21T10:42:13.597897+0000","last_fullsized":"2024-09-21T10:42:13.597897+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:08:25.006433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":49,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":481141,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6268,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":21446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398072,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":918380,"kb_used_data":13952,"kb_used_omap":1,"kb_used_meta":903934,"kb_avail":92826772,"statfs":{"total":95995035648,"available":95054614528,"internally_reserved":0,"allocated":14286848,"data_stored":8925647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":925628760},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049731,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":942528,"kb_used_data":14256,"kb_used_omap":242,"kb_used_meta":927053,"kb_avail":92802624,"statfs":{"total":95995035648,"available":95029886976,"internally_reserved":0,"allocated":14598144,"data_stored":9244341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":949302495},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790530,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1207060,"kb_used_data":309544,"kb_used_omap":1,"kb_used_meta":897022,"kb_avail":92538092,"statfs":{"total":95995035648,"available":94759006208,"internally_reserved":0,"allocated":316973056,"data_stored":311623773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918550872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644690,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785216,"kb_used_data":15696,"kb_used_omap":67,"kb_used_meta":768956,"kb_avail":92959936,"statfs":{"total":95995035648,"available":95190974464,"internally_reserved":0,"allocated":16072704,"data_stored":10305158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":787411805},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527708,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":748428,"kb_used_data":14928,"kb_used_omap":75,"kb_used_meta":733172,"kb_avail":92996724,"statfs":{"total":95995035648,"available":95228645376,"internally_reserved":0,"allocated":15286272,"data_stored":9707607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":750768282},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274861,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":791936,"kb_used_data":14660,"kb_used_omap":51,"kb_used_meta":776972,"kb_avail":92953216,"statfs":{"total":95995035648,"available":95184093184,"internally_reserved":0,"allocated":15011840,"data_stored":9439678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":795619669},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354100,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603520,"kb_used_data":14980,"kb_used_omap":66,"kb_used_meta":588221,"kb_avail":93141632,"statfs":{"total":95995035648,"available":95377031168,"internally_reserved":0,"allocated":15339520,"data_stored":9767917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":602338628},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":49,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:42:23.152 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:42:23.152 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:42:23.153 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:42:23.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-21T10:42:23.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:23.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:22 smithi145 ceph-mon[89359]: pgmap v2371: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3382685280' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:42:23.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:23.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:23.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:23.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:23.522 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:42:23.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:23.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:23.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:23.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:23.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:23.935 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:23.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:23.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:23.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:24.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:24.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:24.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3382685280' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:42:24.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:23 smithi119 ceph-mon[105288]: osdmap e1398: 8 total, 8 up, 7 in 2024-09-21T10:42:24.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:24.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3382685280' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:42:24.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:23 smithi119 ceph-mon[101432]: osdmap e1398: 8 total, 8 up, 7 in 2024-09-21T10:42:24.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:24.186 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:24.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3382685280' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:42:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:23 smithi145 ceph-mon[89359]: osdmap e1398: 8 total, 8 up, 7 in 2024-09-21T10:42:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:24.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:24.339 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:24.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:24.464 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:24.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:24.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:24.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:24.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:24.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:24.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:24.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:24.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:42:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:42:24] "GET /metrics HTTP/1.1" 200 36771 "" "Prometheus/2.51.0" 2024-09-21T10:42:24.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:24.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:25.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:25.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:25.130 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-21T10:42:25.132 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:42:25.132 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:42:25.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:hints = 1 2024-09-21T10:42:25.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:42:25.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object prefix: benchmark_data_smithi119_301770 2024-09-21T10:42:25.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:42:25.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:42:25.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:25.196 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:24 smithi119 ceph-mon[105288]: pgmap v2373: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:25.197 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:24 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:42:25.197 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:24 smithi119 ceph-mon[101432]: pgmap v2373: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:25.197 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:24 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:42:25.223 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:24 smithi145 ceph-mon[89359]: pgmap v2373: 109 pgs: 109 active+clean; 475 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail 2024-09-21T10:42:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:24 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:42:25.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:25.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:25.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:25.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:25.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:25.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:25.625 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:25.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:25.767 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:25.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:25.907 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:25.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:25.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:26.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:26.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:26.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:26.147 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:26.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:26.177 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:42:26.177 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:42:26.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 1 16 3381 3365 210.301 210.312 0.00281906 0.00473679 2024-09-21T10:42:26.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:26.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:26.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:26.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:26.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.590 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:26.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:26.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:26.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:26.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:26.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:26.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:26.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:26.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:26.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:27.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:27.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:27.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 2 16 6065 6049 189.018 167.75 0.00461675 0.00527562 2024-09-21T10:42:27.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:27.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:26 smithi145 ceph-mon[89359]: pgmap v2374: 109 pgs: 109 active+clean; 29 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 2.9 MiB/s wr, 45 op/s 2024-09-21T10:42:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:27.354 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:26 smithi119 ceph-mon[105288]: pgmap v2374: 109 pgs: 109 active+clean; 29 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 2.9 MiB/s wr, 45 op/s 2024-09-21T10:42:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:26 smithi119 ceph-mon[101432]: pgmap v2374: 109 pgs: 109 active+clean; 29 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 2.9 MiB/s wr, 45 op/s 2024-09-21T10:42:27.380 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:27.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:27.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:27.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:27.654 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:27.756 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:27.782 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:27.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:27.788 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:42:27.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:27.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:27.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:27.963 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:42:28.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:28.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:28.094 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2200818774' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:28.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2200818774' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:28.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:28.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:28.177 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 3 16 8537 8521 177.507 154.5 0.00307594 0.00558914 2024-09-21T10:42:28.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:28.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2200818774' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:42:28.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:28.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:28.442 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:28.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:28.582 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:28.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:28.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:28.724 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:28.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:28.855 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:28.957 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:28.984 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:28.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:29.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:29.180 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 4 16 11123 11107 173.518 161.625 0.00881633 0.0057512 2024-09-21T10:42:29.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:29.251 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:28 smithi145 ceph-mon[89359]: pgmap v2375: 109 pgs: 109 active+clean; 86 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 8.5 MiB/s wr, 136 op/s 2024-09-21T10:42:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:28 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:28 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2200818774' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:28 smithi145 ceph-mon[89359]: osdmap e1399: 8 total, 8 up, 7 in 2024-09-21T10:42:29.353 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[105288]: pgmap v2375: 109 pgs: 109 active+clean; 86 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 8.5 MiB/s wr, 136 op/s 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2200818774' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[105288]: osdmap e1399: 8 total, 8 up, 7 in 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[101432]: pgmap v2375: 109 pgs: 109 active+clean; 86 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 8.5 MiB/s wr, 136 op/s 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2200818774' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:42:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:28 smithi119 ceph-mon[101432]: osdmap e1399: 8 total, 8 up, 7 in 2024-09-21T10:42:29.380 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:29.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:29.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:29.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:29.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:29.636 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:29.738 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:29.764 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:29.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:29.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:29.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:29.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:30.032 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:30.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:30.178 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 5 16 13896 13880 173.469 173.312 0.00630692 0.00575744 2024-09-21T10:42:30.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:30.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:30.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:30.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:30.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:30.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:42:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:42:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:42:30.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:30.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:30.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:30.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:30.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:30.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:30.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:30.961 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:30.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:30.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:31.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:30 smithi119 ceph-mon[105288]: pgmap v2377: 109 pgs: 109 active+clean; 491 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-21T10:42:31.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:30 smithi119 ceph-mon[101432]: pgmap v2377: 109 pgs: 109 active+clean; 491 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-21T10:42:31.091 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:31.119 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:31.178 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 6 16 16527 16511 171.961 164.438 0.00382534 0.00580822 2024-09-21T10:42:31.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:31.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:30 smithi145 ceph-mon[89359]: pgmap v2377: 109 pgs: 109 active+clean; 491 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 61 MiB/s wr, 981 op/s 2024-09-21T10:42:31.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:31.376 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:31.477 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:31.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:31.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:31.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:42:31.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:31.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:31.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:31.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:31.782 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:31.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:31.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:31.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:32.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:32.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:32.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:32.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:32.178 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 7 16 18459 18443 164.644 120.75 0.00690784 0.00606663 2024-09-21T10:42:32.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:32.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:31 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2127089004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:32.299 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:32.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:31 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2127089004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:32.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:31 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2127089004' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:42:32.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:32.426 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:32.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:32.560 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:42:32.560 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:42:32.560 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:32.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:32.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:32.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:32.793 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:32.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:32.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:32.942 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:32.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:33.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.072 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:33.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:33.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:33.178 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 8 16 20452 20436 159.633 124.562 0.00850186 0.00625806 2024-09-21T10:42:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.206 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:33.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:32 smithi145 ceph-mon[89359]: pgmap v2378: 109 pgs: 109 active+clean; 700 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:42:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:32 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:33.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:33.334 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:32 smithi119 ceph-mon[101432]: pgmap v2378: 109 pgs: 109 active+clean; 700 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:42:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:32 smithi119 ceph-mon[105288]: pgmap v2378: 109 pgs: 109 active+clean; 700 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:42:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:42:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:33.436 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:33.458 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:33.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:33.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:33.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:33.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:33.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:33.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:33.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:33.963 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:33.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:33.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:34.091 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.116 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:34.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:34.178 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 9 16 22394 22378 155.381 121.375 0.00750847 0.00642943 2024-09-21T10:42:34.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:34.244 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:34.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:34.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:34.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:34.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:34.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:34.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:34.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:34.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:34.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:42:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:42:34] "GET /metrics HTTP/1.1" 200 36827 "" "Prometheus/2.51.0" 2024-09-21T10:42:34.881 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:34.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:34.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:35.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:35.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:35.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:35.178 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 10 16 24206 24190 151.167 113.25 0.0118223 0.00660892 2024-09-21T10:42:35.250 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:35.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:34 smithi145 ceph-mon[89359]: pgmap v2379: 109 pgs: 109 active+clean; 1.0 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:42:35.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:34 smithi119 ceph-mon[105288]: pgmap v2379: 109 pgs: 109 active+clean; 1.0 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:42:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:34 smithi119 ceph-mon[101432]: pgmap v2379: 109 pgs: 109 active+clean; 1.0 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:42:35.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:35.399 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:35.501 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:35.530 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:35.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:35.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:35.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:35.781 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:35.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:35.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:35.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:35.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:36.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:36.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:36.141 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:36.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:36.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 11 16 25752 25736 146.207 96.625 0.00232656 0.00681795 2024-09-21T10:42:36.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:36.293 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:36.395 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:36.417 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:36.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:36.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:36.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:36.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:36.782 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:36.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:36.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:36.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:36.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:37.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:37.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:37.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:37.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 12 16 27391 27375 142.559 102.438 0.0066546 0.0070074 2024-09-21T10:42:37.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:37.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:36 smithi145 ceph-mon[89359]: pgmap v2380: 109 pgs: 109 active+clean; 1.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:42:37.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:37.341 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:37.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:36 smithi119 ceph-mon[101432]: pgmap v2380: 109 pgs: 109 active+clean; 1.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:42:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:36 smithi119 ceph-mon[105288]: pgmap v2380: 109 pgs: 109 active+clean; 1.4 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 137 MiB/s wr, 2.19k op/s 2024-09-21T10:42:37.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:37.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:37.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:37.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:37.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:37.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:37.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:37.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:37.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:38.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:38.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:38.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 13 15 29060 29045 139.621 104.375 0.00955434 0.00715628 2024-09-21T10:42:38.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:38.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:38.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:38.376 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:38.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:38.477 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:38.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:38.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.658 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:38.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:38.661 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:38.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:38.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:38.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:38.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:38.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:39.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:39.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:39.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:38 smithi119 ceph-mon[105288]: pgmap v2381: 109 pgs: 109 active+clean; 1.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-21T10:42:39.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:38 smithi119 ceph-mon[101432]: pgmap v2381: 109 pgs: 109 active+clean; 1.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-21T10:42:39.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:39.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:39.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 14 16 31026 31010 138.42 122.812 0.00395773 0.0072185 2024-09-21T10:42:39.267 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:39.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:38 smithi145 ceph-mon[89359]: pgmap v2381: 109 pgs: 109 active+clean; 1.4 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 138 MiB/s wr, 2.21k op/s 2024-09-21T10:42:39.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:39.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:39.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:39.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:39.537 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:39.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:39.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:39.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:39.802 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:39.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:39.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:39.982 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:39.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:40.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:40.108 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:40.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 15 16 32943 32927 137.179 119.812 0.00460589 0.00728387 2024-09-21T10:42:40.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:40.250 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:40.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:40.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:40.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:40.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:40.543 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:42:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:42:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:42:40.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:40.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:40.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:40.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:40.805 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:40.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:40.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:40.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:40.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:41.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:41.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:41.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:41.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 16 16 34831 34815 135.98 118 0.00561381 0.00734755 2024-09-21T10:42:41.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:41.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:41.295 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:40 smithi145 ceph-mon[89359]: pgmap v2382: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-21T10:42:41.299 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:40 smithi119 ceph-mon[105288]: pgmap v2382: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-21T10:42:41.299 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:40 smithi119 ceph-mon[101432]: pgmap v2382: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 144 MiB/s wr, 2.31k op/s 2024-09-21T10:42:41.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:41.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:41.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:41.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:41.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:41.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:42:41.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:41.714 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:41.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:41.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:41.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:41.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:42.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:42.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:42.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:42.133 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:42.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:42.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 17 16 36344 36328 133.543 94.5625 0.00332655 0.00748098 2024-09-21T10:42:42.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:42.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:42.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:42.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:42.473 INFO:teuthology.orchestra.run.smithi119.stdout:moved 0 / 327 (0%) 2024-09-21T10:42:42.473 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 5.00612 -> 5.00612 (expected baseline 6.32778) 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 40 -> 40 pgs (0.856269 -> 0.856269 * mean) 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 54 -> 54 pgs (1.15596 -> 1.15596 * mean) 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0175 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.1752 2024-09-21T10:42:42.474 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:42:42.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:42.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:42.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:42.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:42.708 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:42.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:42.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:42.948 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:42.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:43.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:43.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:43 smithi119 ceph-mon[105288]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:42:43.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:43 smithi119 ceph-mon[105288]: pgmap v2383: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-21T10:42:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:42 smithi119 ceph-mon[101432]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:42:43.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:42 smithi119 ceph-mon[101432]: pgmap v2383: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-21T10:42:43.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:43.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 18 16 38010 37994 131.908 104.125 0.00758285 0.00757552 2024-09-21T10:42:43.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:42 smithi145 ceph-mon[89359]: from='client.46907 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:42:43.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:42 smithi145 ceph-mon[89359]: pgmap v2383: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 121 MiB/s wr, 1.93k op/s 2024-09-21T10:42:43.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:43.310 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T10:42:43.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T10:42:43.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:43.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:43.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:43.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:43.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:43.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:43.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:43.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:43.916 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:43.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:44.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:44.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:44.142 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:44.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:44.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 19 16 39453 39437 129.712 90.1875 0.0082597 0.00770357 2024-09-21T10:42:44.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:44.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:44.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:44.444 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:44.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:44.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:44.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:44.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:44.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.839 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:44.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:44.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:42:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:42:44] "GET /metrics HTTP/1.1" 200 36827 "" "Prometheus/2.51.0" 2024-09-21T10:42:44.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:44.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:44.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:45.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:45.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:45.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:42:45.178643+0000 min lat: 0.00121152 max lat: 0.179065 avg lat: 0.00779594 2024-09-21T10:42:45.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:42:45.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 20 16 41037 41021 128.176 99 0.00861844 0.00779594 2024-09-21T10:42:45.199 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:45.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:45 smithi145 ceph-mon[89359]: pgmap v2384: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-21T10:42:45.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:45.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:45 smithi119 ceph-mon[101432]: pgmap v2384: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-21T10:42:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:45 smithi119 ceph-mon[105288]: pgmap v2384: 109 pgs: 109 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 124 MiB/s wr, 1.98k op/s 2024-09-21T10:42:45.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:45.493 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:45.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:45.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:45.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:45.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:45.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:45.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:45.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:45.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:45.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:46.015 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:46.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:46.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:46.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:46.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 21 16 42538 42522 126.539 93.8125 0.0109132 0.00789696 2024-09-21T10:42:46.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:46.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:46.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:46.394 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:46.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:46.496 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:46.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:46.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:46.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:46.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:46.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:46.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:46.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:46.910 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:46.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:47.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:47.043 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:47.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:47.179 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 22 16 43702 43686 124.094 72.75 0.158616 0.00802502 2024-09-21T10:42:47.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:47.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:47.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:47.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:47 smithi145 ceph-mon[89359]: pgmap v2385: 109 pgs: 109 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 120 MiB/s wr, 1.93k op/s 2024-09-21T10:42:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:47 smithi119 ceph-mon[105288]: pgmap v2385: 109 pgs: 109 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 120 MiB/s wr, 1.93k op/s 2024-09-21T10:42:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:47 smithi119 ceph-mon[101432]: pgmap v2385: 109 pgs: 109 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 120 MiB/s wr, 1.93k op/s 2024-09-21T10:42:47.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:47.430 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:47.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:47.565 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:47.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:47.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:47.693 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:47.794 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:47.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:47.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:47.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:47.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:48.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:48.073 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:48.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:48.180 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 23 16 45114 45098 122.533 88.25 0.00710174 0.0081549 2024-09-21T10:42:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:48.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:48.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:48.317 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:48.338 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:48.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:42:48.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:48.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:48.481 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:48.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:48.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:48.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:48.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:48.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:48.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:48.869 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:48.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:48.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:49.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:49.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:49 smithi119 ceph-mon[101432]: pgmap v2386: 109 pgs: 109 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:42:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:49 smithi119 ceph-mon[105288]: pgmap v2386: 109 pgs: 109 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:42:49.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:49.180 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 24 16 46612 46596 121.328 93.625 0.00522253 0.00823687 2024-09-21T10:42:49.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:49.283 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:49.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:49 smithi145 ceph-mon[89359]: pgmap v2386: 109 pgs: 109 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:42:49.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:49.425 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:49.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:49.490 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:42:49.490 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:42:49.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:47040 49.6 49:6f6a4667:::benchmark_data_smithi119_301770_object47039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:49.472573+0000", 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.014621917999999999, 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014656181000000001, 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 47040 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.491 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472573+0000", 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472573+0000", 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472575+0000", 2024-09-21T10:42:49.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6670000000000001e-06 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472596+0000", 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1914999999999999e-05 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472611+0000", 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4246999999999999e-05 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:42:49.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472619+0000", 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8669999999999994e-06 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.472799+0000", 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018026000000000001 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.473136+0000", 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00033696799999999999 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.473511+0000", 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037480899999999998 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.476409+0000", 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028982299999999999 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.478873+0000", 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002464248 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481855+0000", 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029820200000000002 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36978.0:47040 49.6 e1399/1394 ondisk, result = 0)", 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:49.481726+0000", 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0054688480000000001, 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055950829999999998, 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:42:49.496 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481726+0000", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481726+0000", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481727+0000", 2024-09-21T10:42:49.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7000000000000004e-07 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481731+0000", 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4059999999999998e-06 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481744+0000", 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3295999999999999e-05 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481752+0000", 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8469999999999996e-06 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481837+0000", 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4822999999999997e-05 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481844+0000", 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0260000000000002e-06 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.499 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:47052 49.b e1399/1394 49:da7efd66:::benchmark_data_smithi119_301770_object47051:head v 1399'2997, mlcod=1399'2995)", 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:49.481990+0000", 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0052051780000000004, 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0054013960000000001, 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481990+0000", 2024-09-21T10:42:49.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481990+0000", 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.481990+0000", 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4300000000000002e-07 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:42:49.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.482024+0000", 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4051999999999999e-05 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.482036+0000", 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1654000000000001e-05 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.482042+0000", 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8390000000000003e-06 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.482102+0000", 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9586999999999997e-05 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.482118+0000", 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6858999999999999e-05 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:47054 49.e e1399/1394)", 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:49.483290+0000", 2024-09-21T10:42:49.503 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0039050230000000001, 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041461639999999999, 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483290+0000", 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:42:49.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483290+0000", 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483291+0000", 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2900000000000002e-07 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483320+0000", 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9400999999999999e-05 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:42:49.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483330+0000", 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7119999999999995e-06 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483336+0000", 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5200000000000003e-06 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:47057 49.d e1399/1394 49:bc003e91:::benchmark_data_smithi119_301770_object47056:head v 1399'2991, mlcod=1399'2989)", 2024-09-21T10:42:49.506 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:49.483925+0000", 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0032699669999999999, 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0035529939999999999, 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483925+0000", 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:42:49.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483925+0000", 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483926+0000", 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8899999999999998e-07 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483973+0000", 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7571999999999998e-05 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:42:49.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483983+0000", 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8320000000000001e-06 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.483990+0000", 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7490000000000001e-06 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.484063+0000", 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2757999999999997e-05 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:42:49.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.484084+0000", 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1209999999999999e-05 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:47053 49.f e1399/1394 49:f7158f1d:::benchmark_data_smithi119_301770_object47052:head v 1399'2817, mlcod=1399'2815)", 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:49.485786+0000", 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.001409294, 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017417660000000001, 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:42:49.510 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.485786+0000", 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.485786+0000", 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:42:49.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.485787+0000", 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.850000000000001e-07 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.485815+0000", 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8228000000000001e-05 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.485826+0000", 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1031e-05 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:42:49.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.485844+0000", 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8584e-05 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.486028+0000", 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000183234 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:49.486046+0000", 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7906000000000001e-05 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:42:49.513 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.514 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:42:49.514 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:42:49.514 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 6 2024-09-21T10:42:49.514 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:42:49.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:49.569 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:49.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:49.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:49.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:49.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:49.846 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:49.849 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:42:49.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:49.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:49.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:49.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:50.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:50.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:50.180 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 25 16 48089 48073 120.168 92.3125 0.00493985 0.0083162 2024-09-21T10:42:50.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:50.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:50.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:50.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:50.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:50.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:50.532 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:50.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:42:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:42:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:42:50.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:50.676 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:50.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:50.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:50.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:50.904 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:50.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:51.034 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:42:51.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:51.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:51.180 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 26 16 49471 49455 118.868 86.375 0.00492519 0.0084067 2024-09-21T10:42:51.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:51.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:42:51.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:51.264 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:51 smithi119 ceph-mon[105288]: pgmap v2387: 109 pgs: 109 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-21T10:42:51.264 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:51 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2662733742' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:51.264 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:51 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:51.265 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:51 smithi119 ceph-mon[101432]: pgmap v2387: 109 pgs: 109 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-21T10:42:51.265 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:51 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2662733742' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:51.266 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:51 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:51 smithi145 ceph-mon[89359]: pgmap v2387: 109 pgs: 109 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 111 MiB/s wr, 1.78k op/s 2024-09-21T10:42:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:51 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2662733742' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:51 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:42:51.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:51.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:51.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:51.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:51.638 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:51.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:51.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:51.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:51.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:51.916 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:51.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:52.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:52.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:52.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:52.145 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:52.182 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 27 16 50910 50894 117.788 89.9375 0.00468806 0.00848312 2024-09-21T10:42:52.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:52.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:52 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:52.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:52 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:42:52.274 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:52 smithi119 ceph-mon[105288]: osdmap e1400: 8 total, 8 up, 7 in 2024-09-21T10:42:52.275 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:52 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:52.275 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:52 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:42:52.275 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:52 smithi119 ceph-mon[101432]: osdmap e1400: 8 total, 8 up, 7 in 2024-09-21T10:42:52.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:52.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:52.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:52.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:52 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:42:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:52 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:42:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:52 smithi145 ceph-mon[89359]: osdmap e1400: 8 total, 8 up, 7 in 2024-09-21T10:42:52.562 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:52.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:52.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:52.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:52.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:52.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:52.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:52.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:52.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:53.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:53.133 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:53.186 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 28 16 52343 52327 116.763 89.5625 0.00979571 0.00855888 2024-09-21T10:42:53.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:53.275 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:53.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:53 smithi119 ceph-mon[105288]: pgmap v2389: 109 pgs: 109 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T10:42:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:53 smithi119 ceph-mon[101432]: pgmap v2389: 109 pgs: 109 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T10:42:53.377 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:53.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:53.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:53.529 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:53 smithi145 ceph-mon[89359]: pgmap v2389: 109 pgs: 109 active+clean; 2.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T10:42:53.631 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:53.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:53.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:53.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:53.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:53.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:53.926 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:53.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:53.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:54.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:54.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:54.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:54.186 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 29 16 53808 53792 115.894 91.5625 0.0157319 0.00862346 2024-09-21T10:42:54.187 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:54.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:54.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:54.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:54.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:54.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:54.554 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:54.580 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:54.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:54.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:54.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:54.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:54.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:42:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:42:54] "GET /metrics HTTP/1.1" 200 36833 "" "Prometheus/2.51.0" 2024-09-21T10:42:54.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:54.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:54.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:55.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:55.113 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:55.186 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 30 16 55098 55082 114.719 80.625 0.00866364 0.00871176 2024-09-21T10:42:55.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:55.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:55.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:55.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:55.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:55 smithi119 ceph-mon[101432]: pgmap v2390: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:42:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:55 smithi119 ceph-mon[105288]: pgmap v2390: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:42:55.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:55.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:55.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:55.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:55 smithi145 ceph-mon[89359]: pgmap v2390: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-21T10:42:55.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:55.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:55.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:55.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:42:55.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:55.775 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:55.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:55.897 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:55.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:55.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:55.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:56.025 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:56.127 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:56.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:56.186 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 31 16 56138 56122 113.115 65 0.00264467 0.00882038 2024-09-21T10:42:56.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:56.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:56.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:56.376 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:56.409 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:56.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:56.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:56.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:56.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:56.796 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:56.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:56.944 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:56.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:56.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:57.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:57.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:57.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 32 16 57330 57314 111.908 74.5 0.00396994 0.00892983 2024-09-21T10:42:57.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:57.233 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:57.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:42:57.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:42:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:57 smithi119 ceph-mon[105288]: pgmap v2391: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:42:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:57 smithi119 ceph-mon[101432]: pgmap v2391: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:42:57.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:57.482 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:57 smithi145 ceph-mon[89359]: pgmap v2391: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:42:57.584 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:57.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:57.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:57.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:57.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:57.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:57.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:57.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:58.001 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:58.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:58.138 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:58.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 33 16 58075 58059 109.928 46.5625 0.00300496 0.00908473 2024-09-21T10:42:58.240 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:42:58.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:58.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:42:58.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:42:58.408 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:42:58.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:58.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:58.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:58.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:58.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:58.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:58.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:42:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:58.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:42:58.860 INFO:teuthology.orchestra.run.smithi119.stdout:moved 0 / 327 (0%) 2024-09-21T10:42:58.860 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:42:58.860 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 5.00612 -> 5.00612 (expected baseline 6.32778) 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 40 -> 40 pgs (0.856269 -> 0.856269 * mean) 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 54 -> 54 pgs (1.15596 -> 1.15596 * mean) 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0251 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.2509 2024-09-21T10:42:58.861 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:42:58.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:58.939 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:58.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:59.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:59.067 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:42:59 smithi119 ceph-mon[101432]: pgmap v2392: 109 pgs: 109 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:42:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:42:59 smithi119 ceph-mon[105288]: pgmap v2392: 109 pgs: 109 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:42:59.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:59.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 34 16 59174 59158 108.716 68.6875 0.0108118 0.00919282 2024-09-21T10:42:59.195 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:59.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:59.334 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:59.401 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:42:59.402 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:42:59.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:42:59.441 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:42:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:42:59.473 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:42:59.473 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:42:59.477 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:42:59.511 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:42:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:42:59.539 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:42:59.539 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:42:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:42:59 smithi145 ceph-mon[89359]: pgmap v2392: 109 pgs: 109 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-21T10:42:59.570 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:42:59.571 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:42:59.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:42:59.600 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:42:59.600 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:42:59.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:42:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:42:59.631 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:42:59.631 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:42:59.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:42:59.753 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:42:59.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:42:59.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:42:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:42:59.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:42:59.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:00.007 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:00.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:00.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:00.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:00.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 35 16 60098 60082 107.259 57.75 0.00231588 0.00927987 2024-09-21T10:43:00.231 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:00.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:00.355 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:00 smithi119 ceph-mon[101432]: from='client.56038 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2947095455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:00 smithi119 ceph-mon[105288]: from='client.56038 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2947095455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:00.376 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:00.478 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:00.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:00 smithi145 ceph-mon[89359]: from='client.56038 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2947095455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:43:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:43:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:43:00.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:00.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:00.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:00.725 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T10:43:00.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 110 0.05 2024-09-21T10:43:00.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:00.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:00.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:00.961 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:00.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:01.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:01.122 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:01.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 36 16 61095 61079 106.011 62.3125 0.00876916 0.00942772 2024-09-21T10:43:01.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:01.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:01.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:01.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:01 smithi119 ceph-mon[105288]: pgmap v2393: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:43:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:01 smithi119 ceph-mon[101432]: pgmap v2393: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:43:01.383 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:01.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:01.507 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:01.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:01 smithi145 ceph-mon[89359]: pgmap v2393: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-21T10:43:01.554 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:01.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:01.631 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:01.632 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:01.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:01.759 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:01.861 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:01.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:01.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:01.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:02.034 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:02.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:02.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:02.177 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:02.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:02.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 37 16 62292 62276 105.168 74.8125 0.0142456 0.00950296 2024-09-21T10:43:02.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:02.302 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:02.404 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:02.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:02.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:02.576 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:02.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:02.703 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:02.805 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:02.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:02.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:02.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:02.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:02.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:03.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.110 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:03.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:03.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:03 smithi119 ceph-mon[101432]: pgmap v2394: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-21T10:43:03.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:03 smithi119 ceph-mon[105288]: pgmap v2394: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-21T10:43:03.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:03.187 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 38 16 63624 63608 104.591 83.25 0.00369799 0.00955519 2024-09-21T10:43:03.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:03.255 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:03.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:03.400 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:03.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:03.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:03 smithi145 ceph-mon[89359]: pgmap v2394: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-21T10:43:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:03.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:43:03.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:03.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:03.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:03.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:03.781 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:03.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:03.909 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:03.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:03.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:04.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:04.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:04.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:04.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:04.188 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 39 16 64833 64817 103.844 75.5625 0.0122012 0.00962503 2024-09-21T10:43:04.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:04.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:04.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:04.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:04.555 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:04.581 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:04.683 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:04.724 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:04.825 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:43:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:43:04] "GET /metrics HTTP/1.1" 200 36836 "" "Prometheus/2.51.0" 2024-09-21T10:43:04.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:04.833 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:04.833 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:04.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:04.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:04.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:04.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:05.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:05.129 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:05.188 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:43:05.187491+0000 min lat: 0.00121152 max lat: 0.319481 avg lat: 0.00968804 2024-09-21T10:43:05.188 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:43:05.188 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 40 16 65994 65978 103.062 72.5625 0.00525832 0.00968804 2024-09-21T10:43:05.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:05.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:05.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:05 smithi119 ceph-mon[101432]: pgmap v2395: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T10:43:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:05 smithi119 ceph-mon[105288]: pgmap v2395: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T10:43:05.380 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:05.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:05.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:05.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:05 smithi145 ceph-mon[89359]: pgmap v2395: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 80 MiB/s wr, 1.27k op/s 2024-09-21T10:43:05.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.637 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:05.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:05.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:05.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:05.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:05.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:05.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:05.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:06.007 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:06.046 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:06.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:06.188 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:06.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 41 16 67161 67145 102.327 72.9375 0.00681674 0.00976677 2024-09-21T10:43:06.290 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:06.315 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:06.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:06.442 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:06.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:06.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:06.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:06.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:06.712 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:06.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:06.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:06.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:06.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:06.965 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:06.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:07.067 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:07.188 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 42 16 68032 68016 101.187 54.4375 0.00735168 0.00987611 2024-09-21T10:43:07.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:07.236 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:07.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:07.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:07 smithi119 ceph-mon[101432]: pgmap v2396: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:43:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/288318823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:07 smithi119 ceph-mon[105288]: pgmap v2396: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:43:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/288318823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:07.387 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:43:07.467 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:07 smithi145 ceph-mon[89359]: pgmap v2396: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.10k op/s 2024-09-21T10:43:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/288318823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:07.603 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:07.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:07.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:07.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:07.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:07.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:08.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.046 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:08.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:08.147 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:08.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:08.188 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 43 16 69169 69153 100.487 71.0625 0.0157712 0.00994645 2024-09-21T10:43:08.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[105288]: osdmap e1401: 8 total, 8 up, 7 in 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:08.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:08 smithi119 ceph-mon[101432]: osdmap e1401: 8 total, 8 up, 7 in 2024-09-21T10:43:08.416 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:08.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:08.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:08 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:08 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:43:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:08 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:08 smithi145 ceph-mon[89359]: osdmap e1401: 8 total, 8 up, 7 in 2024-09-21T10:43:08.566 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:08.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:08.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:43:08.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:08.793 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:08.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:08.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:08.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:08.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:09.046 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:09.071 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:09.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:09.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 44 16 70351 70335 99.8812 73.875 0.0076072 0.0100071 2024-09-21T10:43:09.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:09.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:09.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:09.346 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:09 smithi119 ceph-mon[105288]: pgmap v2398: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:43:09.347 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:09 smithi119 ceph-mon[101432]: pgmap v2398: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:43:09.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:09.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:09.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:09 smithi145 ceph-mon[89359]: pgmap v2398: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 69 MiB/s wr, 1.11k op/s 2024-09-21T10:43:09.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:09.582 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:09.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:09.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:09.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:09.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:09.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:09.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:09.861 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:09.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:09.962 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:10.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:10.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:10.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 45 16 71521 71505 99.2866 73.125 0.0235959 0.0100658 2024-09-21T10:43:10.192 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:10.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:10.311 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.320 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:10.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:10.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:43:10.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:43:10.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:10.476 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:43:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:43:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:43:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:43:10.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:10.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:10.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:10.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:10.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:10.891 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:10.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:10.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:11.018 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:11.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:43:11.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:11.190 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 46 16 72649 72633 98.6608 70.5 0.0104748 0.0101304 2024-09-21T10:43:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:11.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:11.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.367 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:11.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:11.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:11.487 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:11 smithi119 ceph-mon[105288]: pgmap v2399: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:43:11.487 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:11.487 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:11.488 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:11 smithi119 ceph-mon[101432]: pgmap v2399: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:43:11.489 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:11.489 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:11.493 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:11.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:11 smithi145 ceph-mon[89359]: pgmap v2399: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:43:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:11.595 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:11.624 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:11.726 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:11.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:11.865 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:11.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:12.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:12.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:12.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:12.163 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:12.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 47 16 73743 73727 98.0163 68.375 0.0193036 0.0101959 2024-09-21T10:43:12.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:12.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:12.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:12.441 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:12.543 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:12.576 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:12.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:12.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:12.718 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:12.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:12.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:12.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:12.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:12.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:13.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:13.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:13.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:13.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 48 16 74852 74836 97.4182 69.3125 0.00945492 0.0102593 2024-09-21T10:43:13.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:13.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:13.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:13.403 INFO:teuthology.orchestra.run.smithi119.stdout:moved 4 / 327 (1.22324%) 2024-09-21T10:43:13.403 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:43:13.403 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 5.00612 -> 4.36592 (expected baseline 6.32778) 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 40 -> 40 pgs (0.856269 -> 0.856269 * mean) 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.1 with 54 -> 50 pgs (1.15596 -> 1.07034 * mean) 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:oload 110 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 0.05 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0298 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.0327 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-21T10:43:13.404 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1 weight 1.0000 -> 0.9500 2024-09-21T10:43:13.461 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:13 smithi119 ceph-mon[101432]: pgmap v2400: 109 pgs: 109 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:43:13.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:13 smithi119 ceph-mon[101432]: from='client.46931 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:13.462 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:13 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260'}"} : dispatch 2024-09-21T10:43:13.462 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:13 smithi119 ceph-mon[105288]: pgmap v2400: 109 pgs: 109 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:43:13.462 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:13 smithi119 ceph-mon[105288]: from='client.46931 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:13.462 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:13 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260'}"} : dispatch 2024-09-21T10:43:13.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:13.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:13.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:13.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:13.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:13.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:13 smithi145 ceph-mon[89359]: pgmap v2400: 109 pgs: 109 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-21T10:43:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:13 smithi145 ceph-mon[89359]: from='client.46931 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:13 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260'}"} : dispatch 2024-09-21T10:43:13.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:13.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:13.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:13.920 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:13.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:13.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:13.994 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:43:13.994 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:14.036 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:43:14.037 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:14.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:14.080 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:43:14.080 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:14.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:14.112 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:43:14.112 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:14.141 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:43:14.142 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:14.171 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:43:14.171 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:14.184 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:14.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 49 16 75919 75903 96.791 66.6875 0.0133698 0.0103255 2024-09-21T10:43:14.209 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:43:14.209 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:14.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:14.250 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:43:14.250 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:14.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:14.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:14.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:14.494 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260'}"}]': finished 2024-09-21T10:43:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:14 smithi145 ceph-mon[89359]: osdmap e1402: 8 total, 8 up, 7 in 2024-09-21T10:43:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1971257857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:14.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:14.607 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260'}"}]': finished 2024-09-21T10:43:14.607 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-mon[101432]: osdmap e1402: 8 total, 8 up, 7 in 2024-09-21T10:43:14.607 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1971257857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:14.607 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:43:14] "GET /metrics HTTP/1.1" 200 36836 "" "Prometheus/2.51.0" 2024-09-21T10:43:14.607 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'62260'}"}]': finished 2024-09-21T10:43:14.608 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-mon[105288]: osdmap e1402: 8 total, 8 up, 7 in 2024-09-21T10:43:14.608 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1971257857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:14.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:14.637 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:43:14.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T10:43:14.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:14.749 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:14.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:14.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:14.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:14.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:14.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:15.022 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:15.124 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:15.149 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:15.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:15.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 50 16 76985 76969 96.1877 66.625 0.00798687 0.0103917 2024-09-21T10:43:15.251 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:15.276 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:15.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:15.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:15.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:15.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:15.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:15.546 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:15.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:15 smithi119 ceph-mon[105288]: pgmap v2402: 109 pgs: 1 peering, 108 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:43:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:15 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:43:15.554 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:15 smithi119 ceph-mon[105288]: osdmap e1403: 8 total, 8 up, 7 in 2024-09-21T10:43:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:15 smithi119 ceph-mon[101432]: pgmap v2402: 109 pgs: 1 peering, 108 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:43:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:15 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:43:15.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:15 smithi119 ceph-mon[101432]: osdmap e1403: 8 total, 8 up, 7 in 2024-09-21T10:43:15.630 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:15 smithi145 ceph-mon[89359]: pgmap v2402: 109 pgs: 1 peering, 108 active+clean; 4.5 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-21T10:43:15.630 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:15 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T10:43:15.630 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:15 smithi145 ceph-mon[89359]: osdmap e1403: 8 total, 8 up, 7 in 2024-09-21T10:43:15.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:15.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:15.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:15.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:15.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:15.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:15.943 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:15.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:15.973 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:43:15.973 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55556 47.4 47:20ae005e:::benchmark_data_smithi119_238957_object55555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406246+0000", 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 553.55802293399995, 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.049347604, 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:15.974 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55556 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406246+0000", 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406246+0000", 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406248+0000", 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3640000000000002e-06 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406272+0000", 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3833999999999999e-05 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406292+0000", 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9598000000000001e-05 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406300+0000", 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5469999999999997e-06 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406522+0000", 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00022146899999999999 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op must wait for map", 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406528+0000", 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1809999999999998e-06 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.977 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589630+0000", 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.183101813 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589635+0000", 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.604e-06 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.455474+0000", 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.86583890500000005 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.455594+0000", 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119289 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55563 47.14 47:2837fb2c:::benchmark_data_smithi119_238957_object55562:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406329+0000", 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 553.55793982299997, 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0804854319999999, 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:15.979 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55563 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406329+0000", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406329+0000", 2024-09-21T10:43:15.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406330+0000", 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7899999999999998e-07 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406351+0000", 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1458999999999999e-05 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406362+0000", 2024-09-21T10:43:15.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1031e-05 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406368+0000", 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6409999999999999e-06 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589708+0000", 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.183339892 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589716+0000", 2024-09-21T10:43:15.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9189999999999999e-06 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.477620+0000", 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.88790453000000003 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for degraded object", 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.480345+0000", 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002724831 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.983 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.486756+0000", 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064112860000000004 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.486815+0000", 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8264000000000003e-05 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55569 47.4 47:24811331:::benchmark_data_smithi119_238957_object55568:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:43:15.984 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 553.55789295600005, 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.061292364, 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55569 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406376+0000", 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0699999999999998e-07 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406395+0000", 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9015999999999999e-05 2024-09-21T10:43:15.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406405+0000", 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1989999999999995e-06 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406410+0000", 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8450000000000002e-06 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406663+0000", 2024-09-21T10:43:15.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025364300000000001 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting_for_map not empty", 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.406668+0000", 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6809999999999997e-06 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589689+0000", 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.18302143100000001 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.589693+0000", 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7129999999999999e-06 2024-09-21T10:43:15.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.455625+0000", 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.86593157700000001 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for degraded object", 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.458580+0000", 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0029557210000000001 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.467600+0000", 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090200220000000008 2024-09-21T10:43:15.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.467668+0000", 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8108999999999995e-05 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:55575 47.15 47:afa8fef1:::benchmark_data_smithi119_238957_object55574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:34:02.411557+0000", 2024-09-21T10:43:15.990 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 553.55271180199998, 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0780909969999999, 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 55575 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411557+0000", 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411557+0000", 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411559+0000", 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1749999999999999e-06 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411587+0000", 2024-09-21T10:43:15.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7732000000000001e-05 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411612+0000", 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5049999999999999e-05 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.411622+0000", 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4900000000000006e-06 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.571507+0000", 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.15988506099999999 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:02.571519+0000", 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2184000000000001e-05 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481556+0000", 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.91003687 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481709+0000", 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015347599999999999 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,1", 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.481865+0000", 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000155339 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.483549+0000", 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00168486 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.485261+0000", 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017114330000000001 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.489542+0000", 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0042811730000000001 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.489605+0000", 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3213999999999995e-05 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:34:03.489648+0000", 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2939999999999999e-05 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188940 47.14 47:2b2878f2:::benchmark_data_smithi119_238957_object64111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.337463+0000", 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.62680606999999, 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0420609350000001, 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:15.997 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188940 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337463+0000", 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:15.998 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337463+0000", 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337465+0000", 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7710000000000001e-06 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337471+0000", 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0900000000000001e-06 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:15.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337487+0000", 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5948e-05 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337495+0000", 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3529999999999995e-06 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337561+0000", 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5837e-05 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.337705+0000", 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014370100000000001 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338300+0000", 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00059517599999999999 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.342262+0000", 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039625090000000003 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352397+0000", 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010134513 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352407+0000", 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0586e-05 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.379449+0000", 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0270420339999999 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.379524+0000", 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4417000000000003e-05 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188944 47.14 47:28ba1a9b:::benchmark_data_smithi119_238957_object64115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1325)", 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.338123+0000", 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.62614577300002, 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.057266204, 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.003 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188944 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338123+0000", 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338123+0000", 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338124+0000", 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.004e-06 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338128+0000", 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1649999999999999e-06 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338141+0000", 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2663999999999999e-05 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.338149+0000", 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8849999999999999e-06 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.342528+0000", 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043793360000000002 2024-09-21T10:43:16.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.342735+0000", 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020680300000000001 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.343233+0000", 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049773700000000005 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352434+0000", 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0092011950000000006 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.352441+0000", 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1470000000000004e-06 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387576+0000", 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0351347230000001 2024-09-21T10:43:16.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for missing object", 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387697+0000", 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121187 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395338+0000", 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076414100000000004 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395389+0000", 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0948000000000001e-05 2024-09-21T10:43:16.009 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188958 47.14 47:28b3f8ea:::benchmark_data_smithi119_238957_object64129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.358429+0000", 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.60583962999999, 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0371616340000001, 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.010 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188958 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358429+0000", 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358429+0000", 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358431+0000", 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4479999999999999e-06 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358436+0000", 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7099999999999998e-06 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358452+0000", 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6008999999999999e-05 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358459+0000", 2024-09-21T10:43:16.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3420000000000002e-06 2024-09-21T10:43:16.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358518+0000", 2024-09-21T10:43:16.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9611000000000002e-05 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.358529+0000", 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0013000000000001e-05 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387780+0000", 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.029251371 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.387899+0000", 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119255 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.388392+0000", 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00049286499999999999 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395300+0000", 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006907631 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395441+0000", 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014154599999999999 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395491+0000", 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9666000000000002e-05 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395522+0000", 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1401e-05 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395591+0000", 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8765999999999999e-05 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188969 47.14 47:2833d4a7:::benchmark_data_smithi119_238957_object64140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.369661+0000", 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.59460763700002, 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0431977539999999, 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188969 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369661+0000", 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369661+0000", 2024-09-21T10:43:16.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369663+0000", 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4100000000000001e-06 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369667+0000", 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9890000000000003e-06 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369677+0000", 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9340000000000001e-06 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.369683+0000", 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.947e-06 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378389+0000", 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0087062170000000005 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378398+0000", 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8729999999999996e-06 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.390476+0000", 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.012078727 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.390552+0000", 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5896999999999995e-05 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.390766+0000", 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021314899999999999 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395307+0000", 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0045415730000000001 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.403326+0000", 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080186179999999999 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412690+0000", 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.009364341 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412746+0000", 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6301999999999997e-05 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412859+0000", 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011277699999999999 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.023 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188973 47.14 47:2ce0866e:::benchmark_data_smithi119_238957_object64144:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.375700+0000", 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.588569076, 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.03734028, 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188973 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.024 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375700+0000", 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375700+0000", 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375702+0000", 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7659999999999999e-06 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375707+0000", 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3190000000000002e-06 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375724+0000", 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6906e-05 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.375732+0000", 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5900000000000002e-06 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378545+0000", 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028136979999999999 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.378551+0000", 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1399999999999997e-06 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.392161+0000", 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0136094010000001 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.392229+0000", 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7960000000000007e-05 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.392419+0000", 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019038200000000001 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395456+0000", 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003036969 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.403365+0000", 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079085119999999995 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412904+0000", 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095398970000000003 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.029 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.412952+0000", 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7236e-05 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413040+0000", 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8504000000000005e-05 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:188993 47.14 47:2a03ebbf:::benchmark_data_smithi119_238957_object64164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.393087+0000", 2024-09-21T10:43:16.030 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.57118197400001, 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0201839669999999, 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 188993 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393087+0000", 2024-09-21T10:43:16.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393087+0000", 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393088+0000", 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2810000000000001e-06 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393093+0000", 2024-09-21T10:43:16.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8849999999999998e-06 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393108+0000", 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4514e-05 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393116+0000", 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4810000000000006e-06 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393325+0000", 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00020838399999999999 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.393334+0000", 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7089999999999997e-06 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.393584+0000", 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0002499899999999 2024-09-21T10:43:16.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.393648+0000", 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4220999999999994e-05 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.393901+0000", 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025227499999999999 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395461+0000", 2024-09-21T10:43:16.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015599240000000001 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.407039+0000", 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011578128 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413113+0000", 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060744789999999998 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413169+0000", 2024-09-21T10:43:16.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5862000000000001e-05 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.413271+0000", 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101834 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:189012 47.14 47:2d28305b:::benchmark_data_smithi119_238957_object64183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1326)", 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:10.410915+0000", 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 425.55335360100003, 2024-09-21T10:43:16.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0085174750000001, 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 189012 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410915+0000", 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410915+0000", 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410917+0000", 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.916e-06 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410921+0000", 2024-09-21T10:43:16.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0960000000000003e-06 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410931+0000", 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.91e-06 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.410938+0000", 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8009999999999998e-06 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.411001+0000", 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2686000000000003e-05 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:10.411010+0000", 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4700000000000008e-06 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395623+0000", 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.98461289399999996 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395693+0000", 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9830000000000001e-05 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 4,5", 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.395949+0000", 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025582600000000001 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.405124+0000", 2024-09-21T10:43:16.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0091752299999999995 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.407085+0000", 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019611939999999999 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.419297+0000", 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012211539 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.419335+0000", 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8654e-05 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:11.419433+0000", 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7429000000000005e-05 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46418.0:215054 47.4 47:2419daf5:::benchmark_data_smithi119_238957_object90225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1335)", 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:36:42.661029+0000", 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 393.30323977699999, 2024-09-21T10:43:16.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0238612810000001, 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "reached pg", 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46418", 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 215054 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661029+0000", 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661029+0000", 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661031+0000", 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.454e-06 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661034+0000", 2024-09-21T10:43:16.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6490000000000001e-06 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661043+0000", 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7250000000000003e-06 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661051+0000", 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0439999999999996e-06 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661624+0000", 2024-09-21T10:43:16.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057317900000000005 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661740+0000", 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116045 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,4,6", 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.661886+0000", 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014524899999999999 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.665341+0000", 2024-09-21T10:43:16.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0034553710000000001 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.671135+0000", 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057939430000000002 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for peered", 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:42.671150+0000", 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4844000000000001e-05 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:43.684799+0000", 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0136492100000001 2024-09-21T10:43:16.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:36:43.684891+0000", 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1568000000000001e-05 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:170880 48.6 48:64193ea4:::benchmark_data_smithi119_267254_object47670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:21.698536+0000", 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 174.265732886, 2024-09-21T10:43:16.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29018087799999998, 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 170880 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.698536+0000", 2024-09-21T10:43:16.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.698536+0000", 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.698537+0000", 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.305e-06 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.698542+0000", 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1729999999999997e-06 2024-09-21T10:43:16.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.698555+0000", 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3016e-05 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.698562+0000", 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.418e-06 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.053 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.699375+0000", 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00081303199999999999 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.699479+0000", 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104207 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:43:16.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.699743+0000", 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026412400000000002 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.707823+0000", 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0080795859999999997 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.718487+0000", 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010664219000000001 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.988647+0000", 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27015953199999998 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.988683+0000", 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6077e-05 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.988717+0000", 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4189000000000001e-05 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.057 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:170887 48.6 48:6248d39f:::benchmark_data_smithi119_267254_object47677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:21.703149+0000", 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 174.26111957099999, 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.288657987, 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 170887 2024-09-21T10:43:16.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.703149+0000", 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.703149+0000", 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.703151+0000", 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7099999999999999e-06 2024-09-21T10:43:16.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.703157+0000", 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6529999999999999e-06 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.703173+0000", 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6646e-05 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.703182+0000", 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.772e-06 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.707858+0000", 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004675378 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.707979+0000", 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012129300000000001 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.708097+0000", 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000118239 2024-09-21T10:43:16.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.714323+0000", 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006226171 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.718492+0000", 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041690349999999998 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.991574+0000", 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27308178500000002 2024-09-21T10:43:16.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.991735+0000", 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000160951 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.991807+0000", 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2354000000000006e-05 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:170897 48.6 48:664d34fb:::benchmark_data_smithi119_267254_object47687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:21.713760+0000", 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 174.250508669, 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29370562, 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 170897 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.064 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.713760+0000", 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.713760+0000", 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.713762+0000", 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.347e-06 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.713766+0000", 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6890000000000003e-06 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.713775+0000", 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7199999999999995e-06 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.713783+0000", 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8949999999999998e-06 2024-09-21T10:43:16.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.714361+0000", 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057767400000000003 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.714479+0000", 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011807 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.714565+0000", 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5982000000000004e-05 2024-09-21T10:43:16.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.719628+0000", 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005063746 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:21.719639+0000", 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0492000000000001e-05 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:22.007318+0000", 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.28767864199999998 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:22.007399+0000", 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1800999999999998e-05 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:22.007466+0000", 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6562000000000004e-05 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57587 49.e e1400/1394 49:79512dcc:::benchmark_data_smithi119_301770_object57586:head v 1400'3729, mlcod=1400'3723)", 2024-09-21T10:43:16.069 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 18.588828361000001, 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29749656200000002, 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3000000000000002e-07 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375459+0000", 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8499999999999999e-05 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375466+0000", 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375470+0000", 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5229999999999997e-06 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378768+0000", 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003297763 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.072 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378786+0000", 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8192000000000001e-05 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380891+0000", 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021050470000000001 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672937+0000", 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29204598900000001 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57594 49.8 e1400/1394 49:127af930:::benchmark_data_smithi119_301770_object57593:head v 1400'3519, mlcod=1400'3517)", 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.378651+0000", 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 18.585617557999999, 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29406905999999999, 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378651+0000", 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.074 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378651+0000", 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378652+0000", 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0400000000000005e-07 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378674+0000", 2024-09-21T10:43:16.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1175999999999999e-05 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378685+0000", 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1063000000000001e-05 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378691+0000", 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.618e-06 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378813+0000", 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121687 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378831+0000", 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8059000000000001e-05 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380832+0000", 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020010470000000002 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672721+0000", 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29188850599999999 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57596 49.6 49:6ba8cc7d:::benchmark_data_smithi119_301770_object57595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.380340+0000", 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 18.583928823000001, 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30705738599999999, 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.078 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57596 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380340+0000", 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.079 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380340+0000", 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380342+0000", 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.747e-06 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380365+0000", 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3220000000000001e-05 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380379+0000", 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3472000000000001e-05 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380386+0000", 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1160000000000001e-06 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380444+0000", 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8150999999999999e-05 2024-09-21T10:43:16.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380593+0000", 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014878600000000001 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380735+0000", 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014229100000000001 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673023+0000", 2024-09-21T10:43:16.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29228787099999998 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673778+0000", 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075495100000000002 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687296+0000", 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013517972 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687352+0000", 2024-09-21T10:43:16.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6647000000000002e-05 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687398+0000", 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5161999999999999e-05 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57598 49.c 49:3fc93c6d:::benchmark_data_smithi119_301770_object57597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.380860+0000", 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 18.583409177, 2024-09-21T10:43:16.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.303667727, 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57598 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380860+0000", 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380860+0000", 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380861+0000", 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1600000000000001e-07 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380879+0000", 2024-09-21T10:43:16.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7966999999999999e-05 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380893+0000", 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4806e-05 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380903+0000", 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4390000000000006e-06 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673605+0000", 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29270185599999998 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673994+0000", 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038972699999999998 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.674177+0000", 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018300900000000001 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.683774+0000", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095964210000000008 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684421+0000", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00064724400000000005 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684431+0000", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0114e-05 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684481+0000", 2024-09-21T10:43:16.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0355000000000002e-05 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684528+0000", 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6072999999999997e-05 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57597 49.0 e1400/1394 49:0fe7e42d:::benchmark_data_smithi119_301770_object57596:head v 1400'3700, mlcod=1400'3698)", 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.380894+0000", 2024-09-21T10:43:16.090 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 18.583375166, 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29711208099999997, 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380894+0000", 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380894+0000", 2024-09-21T10:43:16.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380895+0000", 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3900000000000004e-07 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380965+0000", 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0871000000000006e-05 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672802+0000", 2024-09-21T10:43:16.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29183669200000001 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672811+0000", 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2969999999999992e-06 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673118+0000", 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000306726 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673142+0000", 2024-09-21T10:43:16.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3787000000000001e-05 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677964+0000", 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048218999999999996 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678006+0000", 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2069e-05 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:43:16.094 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:43:16.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:16.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 51 16 78023 78007 95.5737 64.875 0.0129 0.0104577 2024-09-21T10:43:16.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:16.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:16.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:16.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:16.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:16.527 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:16.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:16.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:16.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:16.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:16.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:16.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:16.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:17.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:17.055 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:17.157 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:17.183 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:17.190 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 52 16 79162 79146 95.1047 71.1875 0.00811213 0.0105098 2024-09-21T10:43:17.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:17 smithi119 ceph-mon[105288]: pgmap v2404: 109 pgs: 1 peering, 108 active+clean; 4.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:43:17.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:17.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:17.282 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:17 smithi119 ceph-mon[101432]: pgmap v2404: 109 pgs: 1 peering, 108 active+clean; 4.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:43:17.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:17.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:17.283 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:43:17.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:17 smithi145 ceph-mon[89359]: pgmap v2404: 109 pgs: 1 peering, 108 active+clean; 4.6 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-21T10:43:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:17.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:17.312 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:17.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:43:17.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:17.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:17.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:17.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.596 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:17.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:17.607 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:17.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:17.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:17.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:17.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:17.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:17.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:18.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:18.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:18.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:18.189 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 53 16 80163 80147 94.4908 62.5625 0.0147467 0.010578 2024-09-21T10:43:18.248 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:18.288 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:18.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:18.442 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:18.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:18.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:18.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:18.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:18.718 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:18.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:18.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:18.842 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:18.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:18.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:18.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:19.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:19.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:19.011 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:18 smithi145 ceph-mon[89359]: pgmap v2405: 109 pgs: 1 peering, 108 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-21T10:43:19.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:19.098 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:19.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:19.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:18 smithi119 ceph-mon[101432]: pgmap v2405: 109 pgs: 1 peering, 108 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-21T10:43:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:19.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:19.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:18 smithi119 ceph-mon[105288]: pgmap v2405: 109 pgs: 1 peering, 108 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 59 MiB/s wr, 950 op/s 2024-09-21T10:43:19.191 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 54 16 81208 81192 93.9473 65.3125 0.0124644 0.0106386 2024-09-21T10:43:19.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:19.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:19.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:19.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:19.455 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:19.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:19.520 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:19.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:19.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:19.629 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:43:19.629 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:43:19.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.629 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:81576 49.d 49:b802d60f:::benchmark_data_smithi119_301770_object81575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1403)", 2024-09-21T10:43:19.629 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:43:19.620960+0000", 2024-09-21T10:43:19.629 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0052099709999999999, 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052452009999999997, 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 81576 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.620960+0000", 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.620960+0000", 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.620958+0000", 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621011+0000", 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2911999999999998e-05 2024-09-21T10:43:19.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621029+0000", 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7657999999999999e-05 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621036+0000", 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1600000000000001e-06 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621076+0000", 2024-09-21T10:43:19.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0624000000000003e-05 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621172+0000", 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5246000000000003e-05 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621268+0000", 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6026000000000005e-05 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:19.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624860+0000", 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0035923840000000001 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:81578 49.b 49:d128c4cd:::benchmark_data_smithi119_301770_object81577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1403)", 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:43:19.621048+0000", 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0051218009999999996, 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0052505529999999998, 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:19.634 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 81578 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621048+0000", 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621048+0000", 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621047+0000", 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621077+0000", 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9912000000000001e-05 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621084+0000", 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9809999999999997e-06 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621088+0000", 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8319999999999999e-06 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621158+0000", 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0090000000000001e-05 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621284+0000", 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012611299999999999 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.621416+0000", 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013148399999999999 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:81579 49.b 49:da929416:::benchmark_data_smithi119_301770_object81578:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1403)", 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:43:19.622765+0000", 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.0034051989999999998, 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036083629999999998, 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:19.638 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 81579 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.622765+0000", 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:19.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.622765+0000", 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.622763+0000", 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.622793+0000", 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9468e-05 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:19.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.622801+0000", 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2090000000000005e-06 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.622805+0000", 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1579999999999998e-06 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624181+0000", 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0013757070000000001 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:19.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624337+0000", 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000156446 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624529+0000", 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019187800000000001 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.36978.0:81571 49.4 e1403/1394 49:2c292441:::benchmark_data_smithi119_301770_object81570:head v 1403'5077, mlcod=1403'5075)", 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:43:19.624284+0000", 2024-09-21T10:43:19.642 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 0.00188554, 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021564219999999999, 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624284+0000", 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624284+0000", 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624253+0000", 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999969 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624323+0000", 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0219e-05 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624337+0000", 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3756e-05 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624343+0000", 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7919999999999998e-06 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624399+0000", 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6076999999999999e-05 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:43:19.624418+0000", 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8712e-05 2024-09-21T10:43:19.645 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.646 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:19.646 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.646 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:19.646 INFO:teuthology.orchestra.run.smithi145.stdout: ], 2024-09-21T10:43:19.646 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 4 2024-09-21T10:43:19.646 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:43:19.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:19.697 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:19.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:19.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:19.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:19.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:19.938 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:43:19.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:20.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:20.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1036669731' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:20.037 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1036669731' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:20.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:20.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:20.191 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 55 16 82095 82079 93.2472 55.4375 0.0142494 0.010719 2024-09-21T10:43:20.215 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1036669731' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:43:20.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:20.273 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:20.375 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:20.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:20.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:43:20.494 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:43:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:43:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:43:20.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:20.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:20.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:20.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:20.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:20.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:20.804 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:20.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:20.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:20.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:21.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:21.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:21.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:21.191 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 56 16 83183 83167 92.7963 68 0.0197477 0.0107712 2024-09-21T10:43:21.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:21.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:21.260 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:20 smithi145 ceph-mon[89359]: pgmap v2406: 109 pgs: 109 active+clean; 4.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T10:43:21.260 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:20 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:21.260 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1036669731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:43:21.260 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:20 smithi145 ceph-mon[89359]: osdmap e1404: 8 total, 8 up, 7 in 2024-09-21T10:43:21.260 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:20 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:43:21.276 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[101432]: pgmap v2406: 109 pgs: 109 active+clean; 4.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1036669731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[101432]: osdmap e1404: 8 total, 8 up, 7 in 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[105288]: pgmap v2406: 109 pgs: 109 active+clean; 4.9 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1036669731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[105288]: osdmap e1404: 8 total, 8 up, 7 in 2024-09-21T10:43:21.277 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:20 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T10:43:21.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.371 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:21.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:21.472 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:21.495 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:21.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:21.625 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:21.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:21.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:21.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:21.892 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:21.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:21.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:22.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:22.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:22.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:22.175 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:22.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:22.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 57 16 84205 84189 92.2889 63.875 0.0222918 0.0108301 2024-09-21T10:43:22.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:22.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:22.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:22.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:22.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:22.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:22.594 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:22.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:22.736 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:22.745 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:43:22.745 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:43:22.745 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:43:22.745 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:43:22.745 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:43:22.839 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:22.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:22.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:22.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:22.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:23.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:23.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:23.141 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:23.192 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 58 16 85199 85183 91.7689 62.125 0.0220894 0.0108915 2024-09-21T10:43:23.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:22 smithi119 ceph-mon[101432]: pgmap v2408: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s 2024-09-21T10:43:23.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:22 smithi119 ceph-mon[105288]: pgmap v2408: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s 2024-09-21T10:43:23.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:23.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:22 smithi145 ceph-mon[89359]: pgmap v2408: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 62 MiB/s wr, 995 op/s 2024-09-21T10:43:23.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:43:23.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:23.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:23.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:23.535 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:23.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:23.682 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:23.783 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:23.805 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:23.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:23.956 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:23.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:24.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:24.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:24.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:24.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 59 16 86159 86143 91.2271 60 0.0188066 0.010956 2024-09-21T10:43:24.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:24.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:24.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:moved 0 / 327 (0%) 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 4.36592 -> 4.36592 (expected baseline 6.32778) 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 40 -> 40 pgs (0.856269 -> 0.856269 * mean) 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.3 with 54 -> 54 pgs (1.15596 -> 1.15596 * mean) 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 1 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0336 2024-09-21T10:43:24.322 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.3365 2024-09-21T10:43:24.323 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:43:24.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:24.341 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:24.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:24.467 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:24.498 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:24.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:24.610 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:24.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:24.774 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:24.844 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:43:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:43:24] "GET /metrics HTTP/1.1" 200 36857 "" "Prometheus/2.51.0" 2024-09-21T10:43:24.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:24.916 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:24.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:25.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:25.041 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:24 smithi145 ceph-mon[89359]: pgmap v2409: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:43:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:24 smithi145 ceph-mon[89359]: from='client.37035 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:24 smithi119 ceph-mon[105288]: pgmap v2409: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:43:25.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:24 smithi119 ceph-mon[105288]: from='client.37035 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:24 smithi119 ceph-mon[101432]: pgmap v2409: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:43:25.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:24 smithi119 ceph-mon[101432]: from='client.37035 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:25.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:25.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:25.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:25.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:43:25.193314+0000 min lat: 0.00121152 max lat: 0.319481 avg lat: 0.0110218 2024-09-21T10:43:25.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:43:25.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 60 16 87101 87085 90.688 58.875 0.00592615 0.0110218 2024-09-21T10:43:25.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:25.303 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:25.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:25.430 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:25.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:25.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:25.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:25.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:25.805 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:25.831 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:25.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:25.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:25.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:25.984 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T10:43:25.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-21T10:43:26.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:26.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:26.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:26.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 61 16 88111 88095 90.2361 63.125 0.00912202 0.0110759 2024-09-21T10:43:26.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:26.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:26.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:26.359 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:26.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:26.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:26.507 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:26.510 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:43:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:43:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:43:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:43:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.510 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55565 47.11 47:8d2e24a3:::benchmark_data_smithi119_238957_object55564:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.408320+0000", 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 564.097799184, 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0753061399999999, 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55565 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408320+0000", 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408320+0000", 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408316+0000", 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999962 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.512 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408370+0000", 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3699000000000002e-05 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408390+0000", 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9998999999999999e-05 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408399+0000", 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6969999999999999e-06 2024-09-21T10:43:26.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.463739+0000", 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.055340093999999999 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.463760+0000", 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1586e-05 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.473493+0000", 2024-09-21T10:43:26.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0097327549999999 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.473569+0000", 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6061000000000002e-05 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.473678+0000", 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108972 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.474612+0000", 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00093417999999999995 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.475720+0000", 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011081470000000001 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.483517+0000", 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077964180000000003 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.483585+0000", 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8565000000000005e-05 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.483626+0000", 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0572999999999999e-05 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55573 47.10 47:0ef07b0d:::benchmark_data_smithi119_238957_object55572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.408412+0000", 2024-09-21T10:43:26.517 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 564.09770713499995, 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0028219249999999, 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55573 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408412+0000", 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408412+0000", 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408411+0000", 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408448+0000", 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7005999999999998e-05 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408455+0000", 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.092e-06 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.408460+0000", 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1609999999999996e-06 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.478472+0000", 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.070012106000000005 2024-09-21T10:43:26.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.478482+0000", 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0696000000000001e-05 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.408993+0000", 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93051024100000002 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.409084+0000", 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1686000000000003e-05 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.409181+0000", 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6817999999999994e-05 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.410864+0000", 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016828469999999999 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411115+0000", 2024-09-21T10:43:26.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000251108 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411145+0000", 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9635999999999999e-05 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411186+0000", 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1270000000000003e-05 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411234+0000", 2024-09-21T10:43:26.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7766000000000003e-05 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:55581 47.10 47:0cc25ec3:::benchmark_data_smithi119_238957_object55580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1293)", 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:34:02.568526+0000", 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 563.93759272499994, 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.84286496399999999, 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55581 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568526+0000", 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568526+0000", 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568525+0000", 2024-09-21T10:43:26.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568586+0000", 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1499000000000002e-05 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568596+0000", 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6930000000000002e-06 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568602+0000", 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5069999999999997e-06 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568643+0000", 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0488999999999998e-05 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:02.568650+0000", 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3509999999999996e-06 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.409915+0000", 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.84126443799999995 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.410032+0000", 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000117043 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 4,6", 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.410172+0000", 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000140075 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411246+0000", 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001074336 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411278+0000", 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2325e-05 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411341+0000", 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2892999999999996e-05 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411371+0000", 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9804000000000001e-05 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:34:03.411391+0000", 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0091000000000001e-05 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227710 47.11 47:8e5c4614:::benchmark_data_smithi119_238957_object102881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.485865+0000", 2024-09-21T10:43:26.530 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 388.02025444200001, 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96921146700000005, 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227710 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485865+0000", 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485865+0000", 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485863+0000", 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485870+0000", 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.81e-06 2024-09-21T10:43:26.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485883+0000", 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2697999999999999e-05 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485888+0000", 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7239999999999997e-06 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485930+0000", 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1780000000000003e-05 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485938+0000", 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0949999999999996e-06 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.417081+0000", 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93114243699999999 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.417189+0000", 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108144 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.417700+0000", 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051170399999999998 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.429700+0000", 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011999434999999999 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454935+0000", 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.025234910999999999 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454980+0000", 2024-09-21T10:43:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4999000000000001e-05 2024-09-21T10:43:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455026+0000", 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6554999999999997e-05 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455076+0000", 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9663999999999997e-05 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227718 47.11 47:8d67ec24:::benchmark_data_smithi119_238957_object102889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.537 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.495418+0000", 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 388.01070066, 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.95983292899999995, 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227718 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495418+0000", 2024-09-21T10:43:26.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495418+0000", 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495417+0000", 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495424+0000", 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7700000000000001e-06 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495439+0000", 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4962e-05 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495446+0000", 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495497+0000", 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1347000000000002e-05 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495509+0000", 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1649e-05 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.420588+0000", 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92507857999999998 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.420678+0000", 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0593000000000005e-05 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.420994+0000", 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031541799999999998 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434783+0000", 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013789321 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455131+0000", 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020347547000000001 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455175+0000", 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4443999999999997e-05 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455216+0000", 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1389e-05 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455251+0000", 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4854999999999997e-05 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227823 47.11 47:88c12679:::benchmark_data_smithi119_238957_object102994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.643965+0000", 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.86215359599998, 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.811510175, 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.544 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227823 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643965+0000", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643965+0000", 2024-09-21T10:43:26.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643964+0000", 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643971+0000", 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7649999999999996e-06 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643984+0000", 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3444e-05 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643989+0000", 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6720000000000003e-06 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644035+0000", 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5924999999999998e-05 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644042+0000", 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2840000000000001e-06 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.423268+0000", 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.77922614199999995 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.423378+0000", 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109663 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.423717+0000", 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033897799999999998 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434788+0000", 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011071009999999999 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455365+0000", 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020577357000000001 2024-09-21T10:43:26.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455418+0000", 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2346999999999999e-05 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455453+0000", 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5759000000000001e-05 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455476+0000", 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2135999999999999e-05 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227826 47.11 47:8d7f136b:::benchmark_data_smithi119_238957_object102997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.644137+0000", 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.86198212699998, 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.81139437000000003, 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.551 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227826 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644137+0000", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644137+0000", 2024-09-21T10:43:26.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644136+0000", 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644140+0000", 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9110000000000003e-06 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644149+0000", 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2650000000000002e-06 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644154+0000", 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8570000000000003e-06 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644186+0000", 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1538000000000002e-05 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644190+0000", 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4839999999999998e-06 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.426298+0000", 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.78210749899999998 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.426402+0000", 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103837 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.426777+0000", 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037539600000000001 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437304+0000", 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010526598999999999 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455402+0000", 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018098521999999999 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455495+0000", 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3066000000000006e-05 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455514+0000", 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9185999999999999e-05 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455531+0000", 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6874000000000001e-05 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227980 47.11 47:8fdf9918:::benchmark_data_smithi119_238957_object103151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.865973+0000", 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.64014642799998, 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.589663194, 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227980 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865973+0000", 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865973+0000", 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865971+0000", 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865980+0000", 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.901e-06 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866001+0000", 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1821999999999999e-05 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866009+0000", 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4359999999999996e-06 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866059+0000", 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0068000000000002e-05 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866067+0000", 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.429784+0000", 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.56371681500000004 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.429914+0000", 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012982599999999999 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.430295+0000", 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00038109299999999999 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437313+0000", 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0070180449999999997 2024-09-21T10:43:26.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455552+0000", 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018239317000000001 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455568+0000", 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5642999999999999e-05 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455603+0000", 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5519999999999999e-05 2024-09-21T10:43:26.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455636+0000", 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2622e-05 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228046 47.11 47:8c47e8df:::benchmark_data_smithi119_238957_object103217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.945725+0000", 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.56039357899999, 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.510127424, 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.564 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228046 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945725+0000", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945725+0000", 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945724+0000", 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945732+0000", 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5360000000000004e-06 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945745+0000", 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3373000000000001e-05 2024-09-21T10:43:26.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945751+0000", 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1330000000000004e-06 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945800+0000", 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8693999999999998e-05 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945807+0000", 2024-09-21T10:43:26.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.221e-06 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.432313+0000", 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.48650629699999998 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.432392+0000", 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8780000000000001e-05 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.432698+0000", 2024-09-21T10:43:26.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030580400000000002 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444325+0000", 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011627513000000001 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455683+0000", 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011357544000000001 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455767+0000", 2024-09-21T10:43:26.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4588999999999999e-05 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455819+0000", 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1533999999999998e-05 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455853+0000", 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3862000000000002e-05 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228055 47.11 47:8bc14e8b:::benchmark_data_smithi119_238957_object103226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.955945+0000", 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.55017399100001, 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.50006493799999996, 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228055 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955945+0000", 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955945+0000", 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955944+0000", 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:26.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955949+0000", 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7140000000000001e-06 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955961+0000", 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1867e-05 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955965+0000", 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955997+0000", 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2515999999999998e-05 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.956002+0000", 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8590000000000004e-06 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434830+0000", 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.47882809500000001 2024-09-21T10:43:26.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434944+0000", 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011323999999999999 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.435227+0000", 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028319100000000002 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.448939+0000", 2024-09-21T10:43:26.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013712330999999999 2024-09-21T10:43:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455729+0000", 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067898419999999999 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455924+0000", 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019455000000000001 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455976+0000", 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2771e-05 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456010+0000", 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3640999999999999e-05 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228076 47.11 47:8c7d5a4c:::benchmark_data_smithi119_238957_object103247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.978403+0000", 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.52771632899999, 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.477718429, 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.578 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228076 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978403+0000", 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978403+0000", 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978401+0000", 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978409+0000", 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.678e-06 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978427+0000", 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8317000000000001e-05 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978433+0000", 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9540000000000001e-06 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978484+0000", 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0674000000000003e-05 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978493+0000", 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.645e-06 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437343+0000", 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.45884937799999997 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437420+0000", 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6736000000000005e-05 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437662+0000", 2024-09-21T10:43:26.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000242773 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.448944+0000", 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011282034999999999 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455894+0000", 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069497150000000004 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456051+0000", 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015661000000000001 2024-09-21T10:43:26.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456092+0000", 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1384000000000002e-05 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456121+0000", 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9037999999999999e-05 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228090 47.11 47:88c3266c:::benchmark_data_smithi119_238957_object103261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.993068+0000", 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.51305051899999, 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46330631799999999, 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228090 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993068+0000", 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993068+0000", 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993066+0000", 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993076+0000", 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.004e-05 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993095+0000", 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9156000000000001e-05 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993103+0000", 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8639999999999996e-06 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993163+0000", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0134999999999999e-05 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993172+0000", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5050000000000007e-06 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.439707+0000", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.44653536300000002 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.439820+0000", 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011309 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.440102+0000", 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028139300000000001 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.448949+0000", 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0088475849999999998 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456156+0000", 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072068439999999996 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456186+0000", 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9742000000000001e-05 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456227+0000", 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0849000000000001e-05 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456375+0000", 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014811699999999999 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228092 47.11 47:8fbd4fb1:::benchmark_data_smithi119_238957_object103263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.994568+0000", 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.51155143900002, 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46230310600000002, 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.591 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228092 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994568+0000", 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994568+0000", 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994567+0000", 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994573+0000", 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2539999999999998e-06 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994591+0000", 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7785e-05 2024-09-21T10:43:26.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994597+0000", 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3090000000000001e-06 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994669+0000", 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2006000000000001e-05 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994677+0000", 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1610000000000003e-06 2024-09-21T10:43:26.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.441995+0000", 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.44731786099999998 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.442061+0000", 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6523e-05 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.442289+0000", 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000227982 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454345+0000", 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012055203 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456567+0000", 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022221900000000002 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456700+0000", 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013318800000000001 2024-09-21T10:43:26.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456839+0000", 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013906500000000001 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456871+0000", 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1646000000000002e-05 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.597 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228108 47.11 47:8bc9ff00:::benchmark_data_smithi119_238957_object103279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:59.020543+0000", 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 387.48557579499999, 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.43645614300000002, 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228108 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020543+0000", 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020543+0000", 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020542+0000", 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020549+0000", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5179999999999998e-06 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020565+0000", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6081000000000001e-05 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020573+0000", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4379999999999998e-06 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020619+0000", 2024-09-21T10:43:26.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5859999999999998e-05 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020625+0000", 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7880000000000001e-06 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444368+0000", 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.42374235599999999 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444472+0000", 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010404300000000001 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444786+0000", 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000313849 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454370+0000", 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095846319999999992 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456646+0000", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002276255 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456906+0000", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025980699999999997 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456969+0000", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2584999999999998e-05 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456999+0000", 2024-09-21T10:43:26.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0480999999999998e-05 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:83857 48.8 48:19427d55:::benchmark_data_smithi119_267254_object83856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:48.367161+0000", 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 278.138957839, 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24199303899999999, 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:26.604 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 83857 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367161+0000", 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367161+0000", 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367160+0000", 2024-09-21T10:43:26.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367213+0000", 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2580000000000001e-05 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367225+0000", 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1969e-05 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367229+0000", 2024-09-21T10:43:26.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5070000000000002e-06 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367265+0000", 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6173000000000001e-05 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367364+0000", 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8857999999999994e-05 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:43:26.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367452+0000", 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8385000000000002e-05 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369785+0000", 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023325249999999998 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.373459+0000", 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036735930000000002 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.609060+0000", 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.235601171 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.609114+0000", 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4073000000000002e-05 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.609154+0000", 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0379000000000001e-05 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:83863 48.8 48:1204cd08:::benchmark_data_smithi119_267254_object83862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:48.369716+0000", 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 278.136402997, 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.246415941, 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.611 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 83863 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369716+0000", 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369716+0000", 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369715+0000", 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369752+0000", 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7326000000000001e-05 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369767+0000", 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5194000000000001e-05 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369773+0000", 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2689999999999996e-06 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369811+0000", 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7265999999999999e-05 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369932+0000", 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012138000000000001 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.370074+0000", 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141395 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.372772+0000", 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026980929999999999 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.373561+0000", 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078955099999999999 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.616041+0000", 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24247988100000001 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.616093+0000", 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2329e-05 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.616132+0000", 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8593000000000001e-05 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:170883 48.9 e1382/1354 48:978671ab:::benchmark_data_smithi119_267254_object47673:head v 1382'10478, mlcod=1382'10474)", 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.699619+0000", 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 184.80650016600001, 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.27863969300000002, 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699619+0000", 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699619+0000", 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699618+0000", 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:26.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699625+0000", 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.63e-06 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699636+0000", 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.19e-05 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699642+0000", 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8209999999999998e-06 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699696+0000", 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3321e-05 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699724+0000", 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8535999999999999e-05 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978130+0000", 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.27840601500000001 2024-09-21T10:43:26.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978259+0000", 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012841999999999999 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:170880 48.6 e1382/1354 48:64193ea4:::benchmark_data_smithi119_267254_object47670:head v 1382'10699, mlcod=1382'10696)", 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.701982+0000", 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 184.804136578, 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.28656793400000002, 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.621 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701982+0000", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701982+0000", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701981+0000", 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:26.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701990+0000", 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5099999999999998e-06 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.702004+0000", 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4238e-05 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.702011+0000", 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7889999999999997e-06 2024-09-21T10:43:26.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978180+0000", 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.27616892900000001 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978205+0000", 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5301999999999999e-05 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.988479+0000", 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010273801000000001 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.988550+0000", 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1526999999999995e-05 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:170887 48.6 e1382/1354 48:6248d39f:::benchmark_data_smithi119_267254_object47677:head v 1382'10700, mlcod=1382'10696)", 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.708468+0000", 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 184.797650554, 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.28266307899999998, 2024-09-21T10:43:26.625 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708468+0000", 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708468+0000", 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708467+0000", 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708477+0000", 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3999999999999998e-06 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978116+0000", 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.26963935999999999 2024-09-21T10:43:26.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978131+0000", 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4849e-05 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.981054+0000", 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002923316 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.981070+0000", 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5781e-05 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.991078+0000", 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010008217999999999 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.991132+0000", 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3386e-05 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.629 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:170930 48.8 48:1c0c3ac0:::benchmark_data_smithi119_267254_object47720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.745856+0000", 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 184.76026278000001, 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.243843488, 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:26.630 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 170930 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745856+0000", 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745856+0000", 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745854+0000", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745864+0000", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0369e-05 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978116+0000", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23225187799999999 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:26.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978133+0000", 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7152000000000001e-05 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978425+0000", 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029155099999999997 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978569+0000", 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014475600000000001 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:43:26.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978726+0000", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015679299999999999 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989529+0000", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010803241 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989591+0000", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1660000000000003e-05 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989623+0000", 2024-09-21T10:43:26.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1803e-05 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989660+0000", 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7376000000000003e-05 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989700+0000", 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9419000000000003e-05 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:26.635 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:26.636 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:43:26.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:26.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:26.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:26.767 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:26.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:26.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:26.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:26.943 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:26.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:26.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:27.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:27.074 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:27.104 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:26 smithi119 ceph-mon[105288]: pgmap v2410: 109 pgs: 109 active+clean; 5.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:43:27.104 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:26 smithi119 ceph-mon[101432]: pgmap v2410: 109 pgs: 109 active+clean; 5.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:43:27.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:27.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 62 16 89094 89078 89.7714 61.4375 0.0246546 0.0111344 2024-09-21T10:43:27.217 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:26 smithi145 ceph-mon[89359]: pgmap v2410: 109 pgs: 109 active+clean; 5.2 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:43:27.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:27.364 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:27.466 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:27.490 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:27.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:27.592 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:27.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:43:27.619 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:27.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:27.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:27.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:27.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:27.950 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:27.976 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/616471223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:28.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:28.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:28.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:28.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:28.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:28.194 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 63 16 90014 89998 89.2592 57.5 0.012684 0.0111976 2024-09-21T10:43:28.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.304 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:28.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:28.341 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/616471223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:28.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/616471223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:28.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:28.426 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:28.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:28.566 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:28.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:28.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:28.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:28.867 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:28.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:28.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:29.001 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:29.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:29.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:29.195 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 64 16 90864 90848 88.6946 53.125 0.0201292 0.0112692 2024-09-21T10:43:29.229 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:29.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:29.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:29.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:29.478 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:29.503 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:29.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.641 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:29.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:29.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:29.764 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:29 smithi145 ceph-mon[89359]: pgmap v2411: 109 pgs: 109 active+clean; 5.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:43:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:29 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:29.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:29 smithi119 ceph-mon[105288]: pgmap v2411: 109 pgs: 109 active+clean; 5.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:43:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:29 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:29 smithi119 ceph-mon[101432]: pgmap v2411: 109 pgs: 109 active+clean; 5.3 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:43:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:29 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:29.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:29.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:30.004 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:30.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:30.142 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:30.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:30.195 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 65 16 91660 91644 88.0955 49.75 0.00406816 0.0113446 2024-09-21T10:43:30.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:30.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:30.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:30.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:30.428 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:30.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:30.530 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:43:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:43:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:43:30.568 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:30.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:30.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:30.694 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:30.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:30.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:30.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:30.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:30.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:31.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:31.082 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:31.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:31.195 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 66 16 92492 92476 87.5475 52 0.0161285 0.011417 2024-09-21T10:43:31.224 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:31.325 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:31.366 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:31.467 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:31.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:31.634 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:31.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:31.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:31.782 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:31 smithi145 ceph-mon[89359]: pgmap v2412: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 987 op/s 2024-09-21T10:43:31.782 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:31.782 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:31 smithi119 ceph-mon[101432]: pgmap v2412: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 987 op/s 2024-09-21T10:43:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:31 smithi119 ceph-mon[105288]: pgmap v2412: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 62 MiB/s wr, 987 op/s 2024-09-21T10:43:31.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:31.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:31.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:31.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:32.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:32.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:32.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:32.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:32.196 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 67 16 93327 93311 87.0198 52.1875 0.0522965 0.0114831 2024-09-21T10:43:32.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:32.310 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:32.412 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:32.437 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:32.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:43:32.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:32.573 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:32.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:32.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:32.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:32.828 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:32.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:32.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:32.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:33.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:33.115 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:33.196 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 68 16 94260 94244 86.5975 58.3125 0.0201275 0.0115428 2024-09-21T10:43:33.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:33.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:33.344 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:33.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:33.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:33.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:33.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:33.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:33.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:33.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:33.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:33.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:33.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:33 smithi145 ceph-mon[89359]: pgmap v2413: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 57 MiB/s wr, 915 op/s 2024-09-21T10:43:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:33 smithi119 ceph-mon[101432]: pgmap v2413: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 57 MiB/s wr, 915 op/s 2024-09-21T10:43:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:33.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:33 smithi119 ceph-mon[105288]: pgmap v2413: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 57 MiB/s wr, 915 op/s 2024-09-21T10:43:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:33.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:33.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:33.902 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:33.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:34.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:34.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:34.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:34.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:34.196 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 69 16 95099 95083 86.1025 52.4375 0.017091 0.0116086 2024-09-21T10:43:34.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:34.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:34.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:34.397 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:34.418 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:34.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:34.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:34.633 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:43:34.633 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:43:34.633 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:43:34.633 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:34.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:34.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:34.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:34.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:34.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:43:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:43:34] "GET /metrics HTTP/1.1" 200 36859 "" "Prometheus/2.51.0" 2024-09-21T10:43:34.891 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:34.892 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:34.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:34.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:34.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:35.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:35.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:35.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:35.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:35.196 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 70 16 95940 95924 85.6234 52.5625 0.0210586 0.0116718 2024-09-21T10:43:35.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:35.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:35.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:35.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:35.476 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:35 smithi145 ceph-mon[89359]: pgmap v2414: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-21T10:43:35.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:35.613 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:35.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:43:35.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:35.757 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:35.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:35.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:35 smithi119 ceph-mon[105288]: pgmap v2414: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-21T10:43:35.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:35 smithi119 ceph-mon[101432]: pgmap v2414: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 61 MiB/s wr, 975 op/s 2024-09-21T10:43:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:35.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:35.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:35.931 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:35.981 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:36.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:36.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:36.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:36.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:36.196 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 71 16 96815 96799 85.1877 54.6875 0.00940459 0.0117335 2024-09-21T10:43:36.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:36.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:36.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:36.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:36.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:36.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:36.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:36.759 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:36.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:36.899 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:36.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:37.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.051 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:37.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:37.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:37.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:37.196 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 72 16 97680 97664 84.7554 54.0625 0.0381213 0.0117928 2024-09-21T10:43:37.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.322 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:37.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:37.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:37.471 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:37.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:37.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:37 smithi119 ceph-mon[101432]: pgmap v2415: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 54 MiB/s wr, 860 op/s 2024-09-21T10:43:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:37 smithi119 ceph-mon[105288]: pgmap v2415: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 54 MiB/s wr, 860 op/s 2024-09-21T10:43:37.662 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:37.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:37.741 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:37.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:37.779 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:37 smithi145 ceph-mon[89359]: pgmap v2415: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 54 MiB/s wr, 860 op/s 2024-09-21T10:43:37.797 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:37.797 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:37.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:37.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:37.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:37.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:38.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:38.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:38.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:38.154 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:38.179 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:38.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 73 16 98495 98479 84.2913 50.9375 0.0194048 0.011858 2024-09-21T10:43:38.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:38.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:38.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:38.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:38.445 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:38.508 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:43:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:43:38.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:38.577 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:38.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:43:38.678 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:38.701 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2438471956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:38 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:38.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:38.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:38.808 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:38.860 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:38.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2438471956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2438471956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:38.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:39.022 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:39.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:39.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:39.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:39.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 74 16 99336 99320 83.8625 52.5625 0.0161889 0.0119193 2024-09-21T10:43:39.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:39.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:39.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:39.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:39.608 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: pgmap v2416: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: osdmap e1405: 8 total, 8 up, 7 in 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: pgmap v2416: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-21T10:43:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: osdmap e1405: 8 total, 8 up, 7 in 2024-09-21T10:43:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:43:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:43:39.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:39.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:39.751 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: pgmap v2416: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: osdmap e1405: 8 total, 8 up, 7 in 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:43:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:43:39.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:39.877 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:39.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:39.941 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:39.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:40.004 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:40.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:40.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:40.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 75 16 100256 100240 83.511 57.5 0.00939754 0.0119691 2024-09-21T10:43:40.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:40.271 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:40.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:40.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:40.435 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:43:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:43:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:43:40.513 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:40.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:40.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:40.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:40.705 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:40.705 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:40.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:40.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:40.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:40.930 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:40.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:41.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:41.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:41.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:41.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:41.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 76 16 101151 101135 83.1482 55.9375 0.0177906 0.0120211 2024-09-21T10:43:41.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:41.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:41.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:41.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:41.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:41.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:41.519 INFO:teuthology.orchestra.run.smithi119.stdout:moved 0 / 327 (0%) 2024-09-21T10:43:41.519 INFO:teuthology.orchestra.run.smithi119.stdout:avg 46.7143 2024-09-21T10:43:41.519 INFO:teuthology.orchestra.run.smithi119.stdout:stddev 4.36592 -> 4.36592 (expected baseline 6.32778) 2024-09-21T10:43:41.519 INFO:teuthology.orchestra.run.smithi119.stdout:min osd.2 with 40 -> 40 pgs (0.856269 -> 0.856269 * mean) 2024-09-21T10:43:41.519 INFO:teuthology.orchestra.run.smithi119.stdout:max osd.3 with 54 -> 54 pgs (1.15596 -> 1.15596 * mean) 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stdout:oload 1000 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stdout:max_change 1 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stdout:max_change_osds 4 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stdout:average_utilization 0.0380 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stdout:overload_utilization 0.3801 2024-09-21T10:43:41.520 INFO:teuthology.orchestra.run.smithi119.stderr:no change 2024-09-21T10:43:41.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.591 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:41.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:41.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:41.717 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:41.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:41.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:41.847 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:41 smithi119 ceph-mon[105288]: pgmap v2418: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 53 MiB/s wr, 846 op/s 2024-09-21T10:43:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:41 smithi119 ceph-mon[101432]: pgmap v2418: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 53 MiB/s wr, 846 op/s 2024-09-21T10:43:41.880 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:41 smithi145 ceph-mon[89359]: pgmap v2418: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 53 MiB/s wr, 846 op/s 2024-09-21T10:43:41.916 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:43:41.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:43:41.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:41.987 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:41.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:41.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:42.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:42.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:42.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 77 16 101961 101945 82.7258 50.625 0.013611 0.0120821 2024-09-21T10:43:42.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:42.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:42.284 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:42.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:42.399 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:42.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:42.501 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:42.544 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:42.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:42 smithi119 ceph-mon[105288]: from='client.37050 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:42 smithi119 ceph-mon[101432]: from='client.37050 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:42.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:42.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:42.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:42.742 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:42 smithi145 ceph-mon[89359]: from='client.37050 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T10:43:42.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:42.811 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:42.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:42.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:42.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:42.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:43.055 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:43.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:43.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:43.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 78 16 102933 102917 82.4441 60.75 0.018896 0.0121245 2024-09-21T10:43:43.208 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:43.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:43.356 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.367 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:43.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:43.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:43.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.495 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:43.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:43.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:43.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.652 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:43.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:43.733 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:43 smithi119 ceph-mon[101432]: pgmap v2419: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s 2024-09-21T10:43:43.734 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:43 smithi119 ceph-mon[105288]: pgmap v2419: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s 2024-09-21T10:43:43.753 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:43.779 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:43.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:43.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:43 smithi145 ceph-mon[89359]: pgmap v2419: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 53 MiB/s wr, 853 op/s 2024-09-21T10:43:43.881 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:43.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:43.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:43.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:44.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:44.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:44.060 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:43:44.060 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:44.102 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:43:44.102 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:44.145 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:43:44.145 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:44.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:44.162 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:44.162 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:44.184 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:44.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:44.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 79 16 103787 103771 82.0761 53.375 0.0202015 0.0121779 2024-09-21T10:43:44.202 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:43:44.202 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:44.236 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:43:44.237 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:44.267 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:43:44.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:44.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:44.297 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:43:44.297 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:44.326 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:44.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:44.330 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:43:44.331 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:44.428 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:44.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:44.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:44.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:44.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:44.742 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:44.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/930834561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:44.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:44.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/930834561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:44.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/930834561' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:44.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:43:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:43:44] "GET /metrics HTTP/1.1" 200 36859 "" "Prometheus/2.51.0" 2024-09-21T10:43:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:44.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:44.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:44.899 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:43:44.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:45.025 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:45.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:45.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:45.197 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:43:45.196860+0000 min lat: 0.00121152 max lat: 0.319481 avg lat: 0.0122249 2024-09-21T10:43:45.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:43:45.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 80 16 104704 104688 81.7666 57.3125 0.00773171 0.0122249 2024-09-21T10:43:45.235 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:45.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:45.332 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:45.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:45.470 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:45.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:45.603 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:45.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:45.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:45.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:45 smithi145 ceph-mon[89359]: pgmap v2420: 109 pgs: 109 active+clean; 6.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 54 MiB/s wr, 863 op/s 2024-09-21T10:43:45.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:45.861 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:45 smithi119 ceph-mon[105288]: pgmap v2420: 109 pgs: 109 active+clean; 6.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 54 MiB/s wr, 863 op/s 2024-09-21T10:43:45.861 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:45 smithi119 ceph-mon[101432]: pgmap v2420: 109 pgs: 109 active+clean; 6.2 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 54 MiB/s wr, 863 op/s 2024-09-21T10:43:45.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:45.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:45.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:46.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:46.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:46.107 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:46.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:46.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 81 16 105634 105618 81.4747 58.125 0.0324026 0.0122682 2024-09-21T10:43:46.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:46.256 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:46.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:46.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:46.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:46.500 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:46.524 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:46.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:46.667 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:46.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:46.796 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:46.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:46.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:46.935 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:46.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:47.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:47.069 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:47.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:47.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 82 16 106442 106426 81.0968 50.5 0.0382924 0.0123256 2024-09-21T10:43:47.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:47.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:47.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:43:47.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.356 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:47.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:47.416 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:43:47.416 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:43:47.416 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:43:47.416 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.000000 2024-09-21T10:43:47.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 0 2024-09-21T10:43:47.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:47.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:47.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:43:47.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:47.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:47.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:47.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.826 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:47.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:47 smithi119 ceph-mon[101432]: pgmap v2421: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-09-21T10:43:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4029374252' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:47.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:47 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:47.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:47 smithi119 ceph-mon[105288]: pgmap v2421: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-09-21T10:43:47.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4029374252' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:47.871 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:47 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:47.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:47.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:47.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:47.983 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:47 smithi145 ceph-mon[89359]: pgmap v2421: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 55 MiB/s wr, 874 op/s 2024-09-21T10:43:47.983 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4029374252' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:47.983 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:47 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:43:48.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:48.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:48.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:48.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 83 16 107271 107255 80.744 51.8125 0.0301085 0.012379 2024-09-21T10:43:48.232 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:48.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:48.382 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:48.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:48.418 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:48.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:48.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227506 47.d 47:bbf06349:::benchmark_data_smithi119_238957_object102677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.433833+0000", 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.15016780299999, 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.010921754, 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227506 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433833+0000", 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433833+0000", 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433833+0000", 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433836+0000", 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5710000000000002e-06 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433845+0000", 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5420000000000005e-06 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433849+0000", 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0990000000000001e-06 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433901+0000", 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2679000000000002e-05 2024-09-21T10:43:48.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433910+0000", 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8359999999999998e-06 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.423003+0000", 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98909294699999994 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.423088+0000", 2024-09-21T10:43:48.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4950000000000005e-05 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.423166+0000", 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7825999999999997e-05 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.427677+0000", 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0045105650000000002 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444650+0000", 2024-09-21T10:43:48.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.016973647000000001 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444694+0000", 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4314000000000003e-05 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444723+0000", 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8085e-05 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444755+0000", 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2296999999999999e-05 2024-09-21T10:43:48.595 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227507 47.d 47:bca0181c:::benchmark_data_smithi119_238957_object102678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.433889+0000", 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.15011184999997, 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.011003544, 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227507 2024-09-21T10:43:48.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433889+0000", 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433889+0000", 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433889+0000", 2024-09-21T10:43:48.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433892+0000", 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.123e-06 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433898+0000", 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.392e-06 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433902+0000", 2024-09-21T10:43:48.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7270000000000001e-06 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433938+0000", 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6193999999999999e-05 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.433946+0000", 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6350000000000006e-06 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.425195+0000", 2024-09-21T10:43:48.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.99124919700000003 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.425260+0000", 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5273999999999999e-05 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.425328+0000", 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8283999999999999e-05 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.431588+0000", 2024-09-21T10:43:48.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062599750000000001 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444797+0000", 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013208667 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444841+0000", 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4082999999999998e-05 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444864+0000", 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3028e-05 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444893+0000", 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8445999999999998e-05 2024-09-21T10:43:48.602 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227522 47.d 47:bce40a9f:::benchmark_data_smithi119_238957_object102693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.447921+0000", 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.13607956200002, 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.997119019, 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227522 2024-09-21T10:43:48.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447921+0000", 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447921+0000", 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447920+0000", 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447925+0000", 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7890000000000002e-06 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447935+0000", 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0249e-05 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447940+0000", 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2849999999999996e-06 2024-09-21T10:43:48.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447975+0000", 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5958999999999999e-05 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.447983+0000", 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0679999999999999e-06 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.426539+0000", 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.97855623199999997 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.426597+0000", 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8621e-05 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.426660+0000", 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2580000000000001e-05 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.432778+0000", 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061185120000000004 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444922+0000", 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012143657 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444990+0000", 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.826e-05 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445011+0000", 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0837000000000001e-05 2024-09-21T10:43:48.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445040+0000", 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9135000000000001e-05 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227537 47.d 47:b5524a98:::benchmark_data_smithi119_238957_object102708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.460523+0000", 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.12347813500003, 2024-09-21T10:43:48.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.98472527799999998, 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227537 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460523+0000", 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460523+0000", 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460522+0000", 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460527+0000", 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5410000000000002e-06 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460537+0000", 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0122000000000001e-05 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460542+0000", 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3449999999999996e-06 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460591+0000", 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8949999999999997e-05 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.460599+0000", 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6329999999999996e-06 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.427705+0000", 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96710641200000003 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.427773+0000", 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7803000000000001e-05 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.427856+0000", 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2993999999999999e-05 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.432785+0000", 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0049288250000000004 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445084+0000", 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012299225 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445201+0000", 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011683099999999999 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445229+0000", 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8498999999999999e-05 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445248+0000", 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.891e-05 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.615 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227549 47.d 47:b38a624e:::benchmark_data_smithi119_238957_object102720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.478892+0000", 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.10510937499998, 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96643026600000004, 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227549 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478892+0000", 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478892+0000", 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478890+0000", 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478896+0000", 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5249999999999996e-06 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478906+0000", 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0570999999999999e-05 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478912+0000", 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8830000000000003e-06 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478962+0000", 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9694000000000002e-05 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.478970+0000", 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6000000000000007e-06 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.429007+0000", 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.95003690100000004 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.429070+0000", 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2255000000000006e-05 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.429130+0000", 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0826999999999999e-05 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.432790+0000", 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003659828 2024-09-21T10:43:48.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445119+0000", 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012328967 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445272+0000", 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000152907 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445300+0000", 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8326999999999998e-05 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.621 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445322+0000", 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1450999999999998e-05 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227566 47.d 47:ba4fcf60:::benchmark_data_smithi119_238957_object102737:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.496750+0000", 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.08725049700001, 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.94869806400000001, 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.622 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227566 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496750+0000", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496750+0000", 2024-09-21T10:43:48.623 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496749+0000", 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496755+0000", 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8619999999999999e-06 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.624 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496767+0000", 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2306999999999999e-05 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496773+0000", 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.592e-06 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496826+0000", 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2852000000000002e-05 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.625 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.496833+0000", 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8900000000000001e-06 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.430264+0000", 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93343077900000004 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.430351+0000", 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7066999999999998e-05 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.626 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.430435+0000", 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3913999999999997e-05 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.435455+0000", 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0050200089999999998 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445142+0000", 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0096878439999999993 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.627 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445397+0000", 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025462599999999998 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445427+0000", 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9988000000000002e-05 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445448+0000", 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1478e-05 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.628 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227570 47.d 47:b4c0570a:::benchmark_data_smithi119_238957_object102741:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.497987+0000", 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.08601377700001, 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.94756863000000002, 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227570 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.629 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.497987+0000", 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.497987+0000", 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.630 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.497986+0000", 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.497992+0000", 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3499999999999996e-06 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.498003+0000", 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1083999999999999e-05 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.631 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.498009+0000", 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0950000000000001e-06 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.498083+0000", 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4368000000000004e-05 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.498091+0000", 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2619999999999998e-06 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.632 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.431636+0000", 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93354406000000001 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.431712+0000", 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6237999999999998e-05 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.431784+0000", 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2038000000000004e-05 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.633 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.438069+0000", 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062849710000000003 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445176+0000", 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0071069599999999998 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445486+0000", 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000310599 2024-09-21T10:43:48.634 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445525+0000", 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8485999999999997e-05 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445556+0000", 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0947e-05 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.635 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227576 47.d 47:b90f8811:::benchmark_data_smithi119_238957_object102747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.504504+0000", 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.079496468, 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.94485650799999998, 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227576 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.636 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504504+0000", 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504504+0000", 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504504+0000", 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.637 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504509+0000", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5629999999999999e-06 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504526+0000", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6999000000000001e-05 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504533+0000", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7549999999999997e-06 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504625+0000", 2024-09-21T10:43:48.638 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1787999999999996e-05 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.504633+0000", 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.2029999999999992e-06 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.432836+0000", 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92820317500000005 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.432904+0000", 2024-09-21T10:43:48.639 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7652999999999998e-05 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.432972+0000", 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7811000000000005e-05 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.438076+0000", 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005104216 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.640 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445351+0000", 2024-09-21T10:43:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072753940000000001 2024-09-21T10:43:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.641 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449303+0000", 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039523149999999996 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449336+0000", 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2280999999999998e-05 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449361+0000", 2024-09-21T10:43:48.642 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5245e-05 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227600 47.d 47:b7eecf30:::benchmark_data_smithi119_238957_object102771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.531833+0000", 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.05216754000003, 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.917655789, 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.643 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227600 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531833+0000", 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531833+0000", 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.644 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531832+0000", 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531839+0000", 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1230000000000002e-06 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531854+0000", 2024-09-21T10:43:48.645 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.487e-05 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531860+0000", 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1380000000000004e-06 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531910+0000", 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0549e-05 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.646 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.531919+0000", 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0559999999999994e-06 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.434097+0000", 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.90217799799999998 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.434185+0000", 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7880000000000006e-05 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.647 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.434258+0000", 2024-09-21T10:43:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2280999999999995e-05 2024-09-21T10:43:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.648 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.441833+0000", 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0075752629999999996 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445375+0000", 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003541804 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449403+0000", 2024-09-21T10:43:48.650 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0040286499999999999 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449452+0000", 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8279000000000003e-05 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449489+0000", 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7648000000000003e-05 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.651 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227605 47.d 47:b4725a80:::benchmark_data_smithi119_238957_object102776:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.538160+0000", 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.04584123199999, 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.91139814699999999, 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227605 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.652 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538160+0000", 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538160+0000", 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538159+0000", 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.653 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538164+0000", 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4639999999999997e-06 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538175+0000", 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0358e-05 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538192+0000", 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7388e-05 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.654 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538251+0000", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8918000000000001e-05 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.538260+0000", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7609999999999996e-06 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.435491+0000", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.89723160000000002 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.655 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.435554+0000", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2419e-05 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.435617+0000", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3720000000000007e-05 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.441840+0000", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0062227369999999999 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445594+0000", 2024-09-21T10:43:48.656 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037536650000000002 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449516+0000", 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039226590000000002 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449539+0000", 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2869000000000001e-05 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.657 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449558+0000", 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8549000000000001e-05 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227608 47.d 47:b5f020ce:::benchmark_data_smithi119_238957_object102779:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.541734+0000", 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.04226689000001, 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.91727218499999996, 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.658 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227608 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541734+0000", 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541734+0000", 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.659 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541733+0000", 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541737+0000", 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8759999999999998e-06 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541746+0000", 2024-09-21T10:43:48.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.9400000000000008e-06 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541751+0000", 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8339999999999998e-06 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541806+0000", 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5334000000000002e-05 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.541822+0000", 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5353e-05 2024-09-21T10:43:48.661 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.436749+0000", 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.89492769999999999 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.436841+0000", 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1763999999999998e-05 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.436908+0000", 2024-09-21T10:43:48.662 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7230000000000005e-05 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444334+0000", 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074253310000000003 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.445627+0000", 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0012930649999999999 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.458921+0000", 2024-09-21T10:43:48.663 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013294228999999999 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.458969+0000", 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7970000000000003e-05 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.459006+0000", 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7225999999999999e-05 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.664 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227638 47.d 47:b262e035:::benchmark_data_smithi119_238957_object102809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.580842+0000", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 411.00315899899999, 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.87830156699999995, 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227638 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580842+0000", 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.665 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580842+0000", 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580840+0000", 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580847+0000", 2024-09-21T10:43:48.666 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0910000000000004e-06 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580862+0000", 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5131e-05 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580869+0000", 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.058e-06 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580917+0000", 2024-09-21T10:43:48.667 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7661999999999999e-05 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.580926+0000", 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6319999999999997e-06 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.438106+0000", 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.85718013400000004 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.438172+0000", 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5901999999999993e-05 2024-09-21T10:43:48.668 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.438238+0000", 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6162999999999997e-05 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444341+0000", 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0061027249999999998 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449188+0000", 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048470960000000004 2024-09-21T10:43:48.669 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.459057+0000", 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0098690319999999998 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.459104+0000", 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7491000000000003e-05 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.670 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.459143+0000", 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9292000000000003e-05 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227650 47.d 47:b4c5429a:::benchmark_data_smithi119_238957_object102821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1339)", 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:57.592903+0000", 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 410.99109808100002, 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.86829778899999999, 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:43:48.671 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227650 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592903+0000", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592903+0000", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592901+0000", 2024-09-21T10:43:48.672 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592907+0000", 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6269999999999997e-06 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592920+0000", 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2809999999999999e-05 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592925+0000", 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.468e-06 2024-09-21T10:43:48.673 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592968+0000", 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3010999999999999e-05 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:57.592976+0000", 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1860000000000003e-06 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.439356+0000", 2024-09-21T10:43:48.674 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.84638050799999998 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.439419+0000", 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2685e-05 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.439484+0000", 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5275000000000001e-05 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.444347+0000", 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048627949999999996 2024-09-21T10:43:48.675 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.449226+0000", 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0048792260000000004 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.461120+0000", 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011894182999999999 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.461165+0000", 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4901000000000003e-05 2024-09-21T10:43:48.676 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.461201+0000", 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5497999999999998e-05 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:73901 48.5 48:a530b83d:::benchmark_data_smithi119_267254_object73900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:39.700165+0000", 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 308.883836385, 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46753160900000001, 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.677 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 73901 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700165+0000", 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700165+0000", 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.678 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700164+0000", 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700206+0000", 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2103e-05 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700214+0000", 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6480000000000003e-06 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.679 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700218+0000", 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2390000000000004e-06 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700264+0000", 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5926e-05 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700397+0000", 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013257300000000001 2024-09-21T10:43:48.680 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700536+0000", 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013929900000000001 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.705733+0000", 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0051973979999999998 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.708126+0000", 2024-09-21T10:43:48.681 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023923410000000001 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.167633+0000", 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.45950758899999999 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.167669+0000", 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6192999999999997e-05 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.167696+0000", 2024-09-21T10:43:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6784e-05 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:73902 48.5 48:a0f83116:::benchmark_data_smithi119_267254_object73901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:39.700234+0000", 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 308.88376696099999, 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46755442800000002, 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:48.683 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 73902 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700234+0000", 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700234+0000", 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.684 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700233+0000", 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700264+0000", 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0451e-05 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700279+0000", 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4647000000000001e-05 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.685 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.700286+0000", 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6799999999999993e-06 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.702919+0000", 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002632658 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703052+0000", 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000133473 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.686 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703189+0000", 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013678099999999999 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.708093+0000", 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0049042770000000003 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.709454+0000", 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001360848 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.687 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.167726+0000", 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.45827142199999998 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.167765+0000", 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9181000000000003e-05 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.167788+0000", 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3557000000000002e-05 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.688 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:73906 48.5 48:a6e99153:::benchmark_data_smithi119_267254_object73905:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:39.703413+0000", 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 308.88058831799998, 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46519062700000002, 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 73906 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.689 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703413+0000", 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703413+0000", 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.690 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703412+0000", 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703531+0000", 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011947699999999999 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703543+0000", 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1701e-05 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.691 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.703549+0000", 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0270000000000001e-06 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.705774+0000", 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022253799999999999 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.705908+0000", 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013397499999999999 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.692 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.706044+0000", 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013601299999999999 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.708213+0000", 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021690640000000001 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:39.709628+0000", 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0014149869999999999 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.693 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.168555+0000", 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.45892634799999998 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.168582+0000", 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.783e-05 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:40.168603+0000", 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0755999999999999e-05 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.694 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.695 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:90432 48.2 48:479764fd:::benchmark_data_smithi119_267254_object90431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1365)", 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:54.391965+0000", 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 294.19203579499998, 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24263932499999999, 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 90432 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.696 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.391965+0000", 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.391965+0000", 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.391964+0000", 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.697 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.392018+0000", 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3703999999999999e-05 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.392026+0000", 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6369999999999999e-06 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.392029+0000", 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2600000000000001e-06 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.698 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.392058+0000", 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.881e-05 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.392151+0000", 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2991999999999993e-05 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.392245+0000", 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3922e-05 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.699 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.395227+0000", 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0029823499999999999 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.396983+0000", 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001756271 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.634542+0000", 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23755844200000001 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.700 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.634579+0000", 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6936e-05 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:54.634604+0000", 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5826e-05 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:192748 48.0 e1383/1354 48:0e10cdd9:::benchmark_data_smithi119_267254_object69538:head v 1383'11807, mlcod=1383'11802)", 2024-09-21T10:43:48.701 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:45.541079+0000", 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 183.04292211200001, 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.237922465, 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.541079+0000", 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.541079+0000", 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.702 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.541078+0000", 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.541084+0000", 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2439999999999998e-06 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.541095+0000", 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0676000000000001e-05 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.703 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.541100+0000", 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4729999999999999e-06 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.544318+0000", 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0032172450000000001 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.544335+0000", 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7589e-05 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.704 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.778812+0000", 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23447695499999999 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.779001+0000", 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000189013 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:192759 48.0 e1383/1354 48:0ace1b28:::benchmark_data_smithi119_267254_object69549:head v 1383'11808, mlcod=1383'11804)", 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:45.550557+0000", 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 183.03344432899999, 2024-09-21T10:43:48.705 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24044306900000001, 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550557+0000", 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550557+0000", 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.706 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550556+0000", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550561+0000", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3710000000000002e-06 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550568+0000", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4749999999999996e-06 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.707 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550572+0000", 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0990000000000001e-06 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.551915+0000", 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001343262 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.551932+0000", 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6161e-05 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.708 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.790961+0000", 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23902953900000001 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.791000+0000", 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8618000000000002e-05 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:57597 49.0 49:0fe7e42d:::benchmark_data_smithi119_301770_object57596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:43:48.709 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:57.380487+0000", 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 51.203513856000001, 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.29769361999999999, 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 57597 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380487+0000", 2024-09-21T10:43:48.710 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380487+0000", 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380486+0000", 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380534+0000", 2024-09-21T10:43:48.711 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8565e-05 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380545+0000", 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.077e-05 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380549+0000", 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9809999999999997e-06 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380586+0000", 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6424999999999998e-05 2024-09-21T10:43:48.712 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380680+0000", 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4672000000000004e-05 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380774+0000", 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3926999999999997e-05 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.382469+0000", 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016945420000000001 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.713 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.382759+0000", 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029050000000000001 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.678119+0000", 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.295359539 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.678155+0000", 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6708999999999997e-05 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:43:48.714 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.678181+0000", 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5190000000000001e-05 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:43:48.715 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:43:48.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:48.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:48.861 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:48.861 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:43:48.861 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[101432]: osdmap e1406: 8 total, 8 up, 7 in 2024-09-21T10:43:48.861 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:48.862 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[101432]: pgmap v2423: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57 MiB/s wr, 907 op/s 2024-09-21T10:43:48.862 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:48.862 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:43:48.862 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[105288]: osdmap e1406: 8 total, 8 up, 7 in 2024-09-21T10:43:48.862 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:48.862 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:48 smithi119 ceph-mon[105288]: pgmap v2423: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57 MiB/s wr, 907 op/s 2024-09-21T10:43:48.870 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:48.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:48.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:48.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:49.022 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:49.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:48 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:43:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:48 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:43:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:48 smithi145 ceph-mon[89359]: osdmap e1406: 8 total, 8 up, 7 in 2024-09-21T10:43:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:43:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:48 smithi145 ceph-mon[89359]: pgmap v2423: 109 pgs: 109 active+clean; 6.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57 MiB/s wr, 907 op/s 2024-09-21T10:43:49.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:49.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:49.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 84 16 108171 108155 80.4524 56.25 0.00878067 0.0124246 2024-09-21T10:43:49.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:49.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:49.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:49.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:49.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:49.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:49.567 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:49.669 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:49.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:49.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:49.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:49.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:49.979 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:49.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:50.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:43:50.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.122 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:50.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:50.198 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 85 16 109081 109065 80.1746 56.875 0.0148365 0.0124667 2024-09-21T10:43:50.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:50.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:50.295 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:43:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:43:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:43:50.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:43:50.376 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:50.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:50.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:50.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:50.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:50.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:50.674 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:50.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:50.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:50.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:50.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:50.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:50.984 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:50.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:51.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:50 smithi119 ceph-mon[105288]: pgmap v2424: 109 pgs: 109 active+clean; 6.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s 2024-09-21T10:43:51.120 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:50 smithi119 ceph-mon[101432]: pgmap v2424: 109 pgs: 109 active+clean; 6.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s 2024-09-21T10:43:51.127 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:51.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:51.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 86 16 109985 109969 79.8993 56.5 0.0325126 0.0125106 2024-09-21T10:43:51.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:51.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:50 smithi145 ceph-mon[89359]: pgmap v2424: 109 pgs: 109 active+clean; 6.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 887 op/s 2024-09-21T10:43:51.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:51.433 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:51.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:51.564 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:51.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:51.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.699 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:51.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:51.748 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:43:51.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:51.823 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:51.831 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:43:51.832 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:43:51.873 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:43:51.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:43:51.918 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:43:51.918 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:43:51.924 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:51.949 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:51.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:51.951 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:43:51.951 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:43:51.984 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:43:51.984 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:43:52.014 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:43:52.015 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:43:52.047 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:43:52.047 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:43:52.050 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:52.076 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:43:52.076 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:43:52.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:52.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:52.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:52.180 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:51 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1080479959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:52.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 87 16 110843 110827 79.5973 53.625 0.00906983 0.0125579 2024-09-21T10:43:52.206 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:52.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:52.311 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:52.312 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:52.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:51 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1080479959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:52.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:51 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1080479959' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:43:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:52.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:52.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:52.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:52.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:52.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:52.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:52.771 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:52.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:52.898 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:52.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:53.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:53.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:43:53.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:53.094 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:52 smithi119 ceph-mon[101432]: pgmap v2425: 109 pgs: 109 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s 2024-09-21T10:43:53.095 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:52 smithi119 ceph-mon[105288]: pgmap v2425: 109 pgs: 109 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s 2024-09-21T10:43:53.129 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:53.154 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:53.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:53.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 88 16 111751 111735 79.3376 56.75 0.0237238 0.0125983 2024-09-21T10:43:53.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:53.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:53.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:53.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:52 smithi145 ceph-mon[89359]: pgmap v2425: 109 pgs: 109 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 884 op/s 2024-09-21T10:43:53.353 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:53.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:53.417 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:53.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:53.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:53.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:53.694 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:53.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:53.822 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:53.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:53.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:53.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:54.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:54.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:54.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 89 16 112600 112584 79.0424 53.0625 0.018345 0.0126459 2024-09-21T10:43:54.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:54.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.240 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:54.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:54.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:54.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.375 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:54.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:54.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:54.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:54.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:54.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:54.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:54.758 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:54.848 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:43:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:43:54] "GET /metrics HTTP/1.1" 200 36859 "" "Prometheus/2.51.0" 2024-09-21T10:43:54.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:54.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:54.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:54.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:55.012 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:55.013 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.6 primary-affinity to 0 (802) 2024-09-21T10:43:55.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:55 smithi119 ceph-mon[101432]: pgmap v2426: 109 pgs: 109 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-21T10:43:55.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3646240465' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-21T10:43:55.107 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-21T10:43:55.107 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:55 smithi119 ceph-mon[105288]: pgmap v2426: 109 pgs: 109 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-21T10:43:55.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3646240465' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-21T10:43:55.108 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-21T10:43:55.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:55.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:55.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:43:55.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 90 6 113484 113478 78.7849 55.875 0.00507386 0.0126807 2024-09-21T10:43:55.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:55.251 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:55 smithi145 ceph-mon[89359]: pgmap v2426: 109 pgs: 109 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 55 MiB/s wr, 885 op/s 2024-09-21T10:43:55.251 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3646240465' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-21T10:43:55.251 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 0.0} : dispatch 2024-09-21T10:43:55.258 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:55.259 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:55.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:55.363 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:55.384 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:55.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:55.507 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:55.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:55.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:55.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:55.752 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:55.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:55.875 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:55.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:43:55.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:55.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:55.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:56.002 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:43:56.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:56.118 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:56.180 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:43:56.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total time run: 90.8614 2024-09-21T10:43:56.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total writes made: 113484 2024-09-21T10:43:56.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Write size: 65536 2024-09-21T10:43:56.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object size: 65536 2024-09-21T10:43:56.199 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Bandwidth (MB/sec): 78.0612 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Bandwidth: 32.4408 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max bandwidth (MB/sec): 210.312 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min bandwidth (MB/sec): 46.5625 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average IOPS: 1248 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev IOPS: 519.053 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max IOPS: 3365 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min IOPS: 745 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average Latency(s): 0.0127306 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Latency(s): 0.0137453 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max latency(s): 1.01404 2024-09-21T10:43:56.200 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min latency(s): 0.00121152 2024-09-21T10:43:56.204 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:43:56.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:56.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: osdmap e1407: 8 total, 8 up, 7 in 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2346102457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:56 smithi145 ceph-mon[89359]: osdmap e1408: 8 total, 8 up, 7 in 2024-09-21T10:43:56.347 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: osdmap e1407: 8 total, 8 up, 7 in 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2346102457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:56.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[105288]: osdmap e1408: 8 total, 8 up, 7 in 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.0}]': finished 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: osdmap e1407: 8 total, 8 up, 7 in 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2346102457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:43:56.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:56 smithi119 ceph-mon[101432]: osdmap e1408: 8 total, 8 up, 7 in 2024-09-21T10:43:56.378 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:56.480 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:56.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:56.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:56.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.648 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:56.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:56.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:56.771 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:56.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:56.897 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:56.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:56.999 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.023 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.145 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:57.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:57.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:57.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:57 smithi145 ceph-mon[89359]: pgmap v2428: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 58 MiB/s wr, 928 op/s 2024-09-21T10:43:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:57 smithi119 ceph-mon[105288]: pgmap v2428: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 58 MiB/s wr, 928 op/s 2024-09-21T10:43:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:57 smithi119 ceph-mon[101432]: pgmap v2428: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 58 MiB/s wr, 928 op/s 2024-09-21T10:43:57.385 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:43:57.553 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:43:57.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:57.686 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:57.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:57.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:57.814 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:57.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:43:57.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:57.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:43:58.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:58.075 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:58.173 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:43:58.173 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:43:58.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:58.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:58.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:58.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:58.421 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:43:58.444 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:43:58.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:43:58.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:43:58.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:58.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:58.794 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:58.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:58.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:58.941 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:58.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:59.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:59.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:59.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:59.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:59.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:43:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:59 smithi145 ceph-mon[89359]: pgmap v2430: 109 pgs: 109 active+clean; 6.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 4.2 KiB/s rd, 44 MiB/s wr, 724 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:43:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:59 smithi145 ceph-mon[89359]: 1.7 scrub starts 2024-09-21T10:43:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:43:59 smithi145 ceph-mon[89359]: 1.7 scrub ok 2024-09-21T10:43:59.313 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:43:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:59 smithi119 ceph-mon[101432]: pgmap v2430: 109 pgs: 109 active+clean; 6.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 4.2 KiB/s rd, 44 MiB/s wr, 724 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:43:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:59 smithi119 ceph-mon[101432]: 1.7 scrub starts 2024-09-21T10:43:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:43:59 smithi119 ceph-mon[101432]: 1.7 scrub ok 2024-09-21T10:43:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:59 smithi119 ceph-mon[105288]: pgmap v2430: 109 pgs: 109 active+clean; 6.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 4.2 KiB/s rd, 44 MiB/s wr, 724 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:43:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:59 smithi119 ceph-mon[105288]: 1.7 scrub starts 2024-09-21T10:43:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:43:59 smithi119 ceph-mon[105288]: 1.7 scrub ok 2024-09-21T10:43:59.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:59.425 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:43:59.450 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:59.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:59.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:59.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:43:59.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:43:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:43:59.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:43:59.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:43:59.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:43:59.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:43:59.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:43:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:43:59.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:00.099 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:00.122 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:00.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:00.249 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:00 smithi145 ceph-mon[89359]: 1.6 scrub starts 2024-09-21T10:44:00.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:00 smithi145 ceph-mon[89359]: 1.6 scrub ok 2024-09-21T10:44:00.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:44:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:44:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:44:00.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:00 smithi119 ceph-mon[101432]: 1.6 scrub starts 2024-09-21T10:44:00.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:00 smithi119 ceph-mon[101432]: 1.6 scrub ok 2024-09-21T10:44:00.336 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:00 smithi119 ceph-mon[105288]: 1.6 scrub starts 2024-09-21T10:44:00.336 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:00 smithi119 ceph-mon[105288]: 1.6 scrub ok 2024-09-21T10:44:00.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:00.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:00.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:00.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:00.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:00.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:00.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:00.669 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:00.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:00.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:00.796 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:00.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:00.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:00.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:01.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:01.089 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:01.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:01.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:01.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:01 smithi145 ceph-mon[89359]: pgmap v2431: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 4.4 KiB/s rd, 41 MiB/s wr, 840 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:44:01.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:01.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:01.341 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:01 smithi119 ceph-mon[105288]: pgmap v2431: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 4.4 KiB/s rd, 41 MiB/s wr, 840 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:44:01.341 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:01 smithi119 ceph-mon[101432]: pgmap v2431: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 4.4 KiB/s rd, 41 MiB/s wr, 840 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:44:01.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:01.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:01.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:01.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:01.702 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:01.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:01.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:01.889 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:01.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:01.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:02.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:02.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:02 smithi119 ceph-mon[101432]: 1.f scrub starts 2024-09-21T10:44:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:02 smithi119 ceph-mon[101432]: 1.f scrub ok 2024-09-21T10:44:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:02 smithi119 ceph-mon[105288]: 1.f scrub starts 2024-09-21T10:44:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:02 smithi119 ceph-mon[105288]: 1.f scrub ok 2024-09-21T10:44:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:02.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:02.162 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:02.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:02.319 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:02.421 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.444 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:02.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:02 smithi145 ceph-mon[89359]: 1.f scrub starts 2024-09-21T10:44:02.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:02 smithi145 ceph-mon[89359]: 1.f scrub ok 2024-09-21T10:44:02.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:02.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:02.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:02.695 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:02.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:02.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:02.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:44:02.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:02.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:02.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:03.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:03.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:03.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:03.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.242 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:03.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:03.292 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:03 smithi119 ceph-mon[101432]: pgmap v2432: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 4.4 KiB/s rd, 23 MiB/s wr, 682 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:44:03.292 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:03.292 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:03.292 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:03 smithi119 ceph-mon[105288]: pgmap v2432: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 4.4 KiB/s rd, 23 MiB/s wr, 682 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:44:03.293 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:03.293 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:03.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:03.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:03.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:03.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:03.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:03 smithi145 ceph-mon[89359]: pgmap v2432: 109 pgs: 109 active+clean; 6.8 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 4.4 KiB/s rd, 23 MiB/s wr, 682 op/s; 3.3 KiB/s, 0 objects/s recovering 2024-09-21T10:44:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:03.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:03.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:03.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:03.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:03.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:03.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:03.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:03.974 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:44:03.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:44:03.975 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T10:44:03.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-21T10:44:03.979 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:03.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:04.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:04.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:04.173 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:04.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:04.328 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:04.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:04.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3817535217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:04.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3817535217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:04.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:04.451 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:04.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3817535217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:04.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:04.587 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:04.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:04.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:04.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:04.795 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:44:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:44:04] "GET /metrics HTTP/1.1" 200 36881 "" "Prometheus/2.51.0" 2024-09-21T10:44:04.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:04.873 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:04.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:04.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:05.018 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:05.120 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:05.144 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:05.246 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:05.271 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:05 smithi119 ceph-mon[101432]: pgmap v2433: 109 pgs: 109 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 114 B/s rd, 13 MiB/s wr, 672 op/s 2024-09-21T10:44:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:05 smithi119 ceph-mon[105288]: pgmap v2433: 109 pgs: 109 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 114 B/s rd, 13 MiB/s wr, 672 op/s 2024-09-21T10:44:05.373 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:05.399 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:05.501 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:05 smithi145 ceph-mon[89359]: pgmap v2433: 109 pgs: 109 active+clean; 6.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 114 B/s rd, 13 MiB/s wr, 672 op/s 2024-09-21T10:44:05.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:05.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:05.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:05.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:05.801 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:05.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:05.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:05.967 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:05.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:06.067 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:06.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:06.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:06.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:06.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:06.363 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:06.399 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:06.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:06.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:06.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:06.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:06.639 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:06.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:06.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:06.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:06.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:06.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:07.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:07.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:07.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:07.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:07.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:07.225 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:44:07.225 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:44:07.225 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:122478 49.1 49:8868fa32:::benchmark_data_smithi119_301770_object8991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:07.209966+0000", 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012456814, 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012488844000000001, 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 122478 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:07.226 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.209966+0000", 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.209966+0000", 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:07.227 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.209967+0000", 2024-09-21T10:44:07.228 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.452e-06 2024-09-21T10:44:07.228 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.228 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.228 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:07.229 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.209972+0000", 2024-09-21T10:44:07.229 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8930000000000004e-06 2024-09-21T10:44:07.229 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.229 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.229 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.209983+0000", 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0733999999999999e-05 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.209989+0000", 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0360000000000003e-06 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.210665+0000", 2024-09-21T10:44:07.230 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00067566999999999996 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.210766+0000", 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100888 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.210873+0000", 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107914 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.231 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.214132+0000", 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032587359999999999 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.215045+0000", 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000912362 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:122481 49.7 e1408/1394 49:ea0ea781:::benchmark_data_smithi119_301770_object8994:head v 1408'7677, mlcod=1408'7675)", 2024-09-21T10:44:07.232 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:07.216222+0000", 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00620099, 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063119919999999998, 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216222+0000", 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216222+0000", 2024-09-21T10:44:07.233 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216223+0000", 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2999999999999999e-07 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216228+0000", 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3290000000000001e-06 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216242+0000", 2024-09-21T10:44:07.234 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3966e-05 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216248+0000", 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0580000000000003e-06 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216828+0000", 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057966599999999995 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.235 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.216845+0000", 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7601999999999999e-05 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:122485 49.3 49:c3bb83d3:::benchmark_data_smithi119_301770_object8998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:07.220308+0000", 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0021150660000000001, 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022793869999999999, 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:07.236 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 122485 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220308+0000", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220308+0000", 2024-09-21T10:44:07.237 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220310+0000", 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.883e-06 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220315+0000", 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9549999999999997e-06 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220332+0000", 2024-09-21T10:44:07.238 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6288000000000001e-05 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220339+0000", 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5519999999999999e-06 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220397+0000", 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8019000000000001e-05 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220508+0000", 2024-09-21T10:44:07.239 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110587 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.220615+0000", 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010731200000000001 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:122464 49.3 49:c0b259be:::benchmark_data_smithi119_301770_object8977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:07.186925+0000", 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.035497644000000002, 2024-09-21T10:44:07.240 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.035729714000000003, 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 122464 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.186925+0000", 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.186925+0000", 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.186926+0000", 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.268e-06 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.186932+0000", 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5620000000000003e-06 2024-09-21T10:44:07.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.186944+0000", 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2601e-05 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.186952+0000", 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5170000000000002e-06 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.189977+0000", 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030246959999999999 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.190101+0000", 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123819 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.190222+0000", 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121154 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.200048+0000", 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0098260039999999993 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.244 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.200073+0000", 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5749999999999999e-05 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:122476 49.1 49:8163c076:::benchmark_data_smithi119_301770_object8989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:07.206964+0000", 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.015458398999999999, 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015756706999999998, 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:07.245 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 122476 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.206964+0000", 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.206964+0000", 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.246 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.206966+0000", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.319e-06 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.206971+0000", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9849999999999997e-06 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.206982+0000", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.099e-05 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.206988+0000", 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8199999999999999e-06 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.207038+0000", 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9632999999999997e-05 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.207139+0000", 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000101363 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:44:07.248 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.207240+0000", 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010084700000000001 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.214125+0000", 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0068851499999999996 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:07.214934+0000", 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00080839099999999999 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:07.250 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:44:07.250 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 5 2024-09-21T10:44:07.250 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:44:07.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:07 smithi119 ceph-mon[105288]: pgmap v2434: 109 pgs: 109 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 12 MiB/s wr, 727 op/s 2024-09-21T10:44:07.250 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:07 smithi119 ceph-mon[101432]: pgmap v2434: 109 pgs: 109 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 12 MiB/s wr, 727 op/s 2024-09-21T10:44:07.297 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:07.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:07.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:07.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:07.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:07 smithi145 ceph-mon[89359]: pgmap v2434: 109 pgs: 109 active+clean; 6.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 102 B/s rd, 12 MiB/s wr, 727 op/s 2024-09-21T10:44:07.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:07.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:07.673 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:07.698 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:07.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:07.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:07.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:07.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:07.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:07.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:08.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:08.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:08.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:08.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:08.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:08.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:08.402 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:44:08.461 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:08.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:08.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:08.609 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:08.711 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:08.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:08.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:08.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:08.879 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:08.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:08.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:09.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:09.089 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:09 smithi119 ceph-mon[101432]: pgmap v2435: 109 pgs: 109 active+clean; 6.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 9.7 MiB/s wr, 745 op/s 2024-09-21T10:44:09.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:09.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:44:09.180 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:09.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:09.305 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:09.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:09 smithi119 ceph-mon[105288]: pgmap v2435: 109 pgs: 109 active+clean; 6.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 9.7 MiB/s wr, 745 op/s 2024-09-21T10:44:09.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:09.471 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:09.485 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:09.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:09 smithi145 ceph-mon[89359]: pgmap v2435: 109 pgs: 109 active+clean; 6.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 85 B/s rd, 9.7 MiB/s wr, 745 op/s 2024-09-21T10:44:09.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:09.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:09.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:09.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:09.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:09.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:09.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:09.932 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:09.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:10.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:10.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:10.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:10.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:10.321 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.363 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:10.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:10.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:10.470 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:10.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:44:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:44:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:44:10.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:10.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:10.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:10.857 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:10.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:10.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:10.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:10.932 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:10.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:11.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:11.092 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:44:11.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:11.153 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:11.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:11.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:11 smithi119 ceph-mon[101432]: pgmap v2436: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 3.6 MiB/s wr, 867 op/s 2024-09-21T10:44:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/261728808' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:11 smithi119 ceph-mon[105288]: pgmap v2436: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 3.6 MiB/s wr, 867 op/s 2024-09-21T10:44:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/261728808' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:11.395 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.421 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:11.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:11.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:11 smithi145 ceph-mon[89359]: pgmap v2436: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 3.6 MiB/s wr, 867 op/s 2024-09-21T10:44:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/261728808' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:44:11.546 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:11.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:11.647 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:11.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:11.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:11.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:11.796 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:11.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:11.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:11.937 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:11.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:12.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:12.070 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:12.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:12.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:12.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:12.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:12 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:12 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/261728808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:44:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:12 smithi119 ceph-mon[101432]: osdmap e1409: 8 total, 8 up, 7 in 2024-09-21T10:44:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:12 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:12 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/261728808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:44:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:12 smithi119 ceph-mon[105288]: osdmap e1409: 8 total, 8 up, 7 in 2024-09-21T10:44:12.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:12.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:12 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:12 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/261728808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:44:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:12 smithi145 ceph-mon[89359]: osdmap e1409: 8 total, 8 up, 7 in 2024-09-21T10:44:12.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:12.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:12.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:12.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:12.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:12.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:12.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:12.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:13.016 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:13.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:13.141 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:13 smithi119 ceph-mon[105288]: pgmap v2438: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-21T10:44:13.141 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:13 smithi119 ceph-mon[101432]: pgmap v2438: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-21T10:44:13.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:13.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:13.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:13.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:13.418 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:13.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:13 smithi145 ceph-mon[89359]: pgmap v2438: 109 pgs: 109 active+clean; 6.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 0 B/s wr, 852 op/s 2024-09-21T10:44:13.544 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:13.646 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:13.684 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:13.786 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:13.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:13.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:13.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:13.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:14.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:14.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:14.237 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:14.339 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:14.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:14.466 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:14.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:14.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:14.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:14.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:44:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:44:14] "GET /metrics HTTP/1.1" 200 36881 "" "Prometheus/2.51.0" 2024-09-21T10:44:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.631 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:14.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:14.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:14.769 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:14.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:14.871 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:14.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:14.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:14.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:14.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:44:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:44:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:44:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:44:15.133 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:15.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:15.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:15.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:15.308 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:15 smithi119 ceph-mon[101432]: pgmap v2439: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:44:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/610773605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:15 smithi119 ceph-mon[105288]: pgmap v2439: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:44:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/610773605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:15.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:15.448 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:15 smithi145 ceph-mon[89359]: pgmap v2439: 109 pgs: 109 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:44:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/610773605' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:15.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:15.588 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:15.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:15.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:15.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:15.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:15.976 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:16.001 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:16.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:16.144 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:16.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:16.269 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:16.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:16.513 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:44:16.538 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:44:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:44:16.539 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:44:16.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:16.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:16.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:16.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:16.839 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:16.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:16.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:16.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:17.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:44:17.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:17.103 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:17.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:17.221 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:44:17.221 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:44:17.256 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:17.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:17.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:17 smithi119 ceph-mon[101432]: pgmap v2440: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:44:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:17 smithi119 ceph-mon[105288]: pgmap v2440: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:44:17.408 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:17.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:17.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:17 smithi145 ceph-mon[89359]: pgmap v2440: 109 pgs: 109 active+clean; 6.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:44:17.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:17.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:17.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:17.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:17.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:17.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:17.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:44:17.864 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:44:17.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-21T10:44:17.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:17.961 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:17.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:18.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:18.099 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:18.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:18.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:18.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:18.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:18.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:18.530 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:18.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:18.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:18.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:18.790 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:18.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:18.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:44:18.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:18.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:18.974 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:18.991 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:19.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:19.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:19.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:19.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:19.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:18 smithi119 ceph-mon[101432]: pgmap v2441: 109 pgs: 109 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 908 op/s 2024-09-21T10:44:19.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:19.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:19.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:18 smithi119 ceph-mon[105288]: pgmap v2441: 109 pgs: 109 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 908 op/s 2024-09-21T10:44:19.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:19.198 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:19.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:18 smithi145 ceph-mon[89359]: pgmap v2441: 109 pgs: 109 active+clean; 5.9 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 908 op/s 2024-09-21T10:44:19.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:19.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:19.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:19.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:19.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:19.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:19.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:19.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:19.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:19.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:19.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:19.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:20.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:20.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:20.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:20.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:20.278 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:20.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:20.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:44:20.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:20.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:20.529 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:44:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:44:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:44:20.631 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:20.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:20.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:20.800 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:20.902 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:20.926 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:20.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:21.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:21.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:21.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:21.253 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:20 smithi145 ceph-mon[89359]: pgmap v2442: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-21T10:44:21.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:21.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:20 smithi119 ceph-mon[105288]: pgmap v2442: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-21T10:44:21.342 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:20 smithi119 ceph-mon[101432]: pgmap v2442: 109 pgs: 109 active+clean; 5.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 905 op/s 2024-09-21T10:44:21.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:21.470 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:21.500 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:21.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:21.630 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:21.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:21.762 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:21.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:21.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:21.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:21.992 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:22.026 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:22.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:22.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:22.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:22.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:22.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:22.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:22.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:22.565 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:22.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:22.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:22.794 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:22.815 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:22.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:22.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:22.939 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:22.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:23.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:23.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:23.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:22 smithi145 ceph-mon[89359]: pgmap v2443: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 834 op/s 2024-09-21T10:44:23.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:22 smithi119 ceph-mon[101432]: pgmap v2443: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 834 op/s 2024-09-21T10:44:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:22 smithi119 ceph-mon[105288]: pgmap v2443: 109 pgs: 109 active+clean; 5.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 834 op/s 2024-09-21T10:44:23.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:23.472 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:23.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:23.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:23.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:23.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:23.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:23.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:23.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:23.993 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:23.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:24.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:24.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:24.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:24.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:24.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:24.380 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:24.482 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:24.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:24.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:24.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:24.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:24.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:44:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:44:24] "GET /metrics HTTP/1.1" 200 36865 "" "Prometheus/2.51.0" 2024-09-21T10:44:24.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:24.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:24.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:25.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:25.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:25.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:25.227 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:25.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:25 smithi145 ceph-mon[89359]: pgmap v2444: 109 pgs: 109 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 914 op/s 2024-09-21T10:44:25.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:25.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:25 smithi119 ceph-mon[105288]: pgmap v2444: 109 pgs: 109 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 914 op/s 2024-09-21T10:44:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:25 smithi119 ceph-mon[101432]: pgmap v2444: 109 pgs: 109 active+clean; 5.6 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 914 op/s 2024-09-21T10:44:25.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:25.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:25.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:25.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:25.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:25.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:25.747 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:25.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:25.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:25.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:25.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:25.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:26.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:26.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.204 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:26.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:26.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:26.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:26.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:26.450 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:26.551 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:26.574 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:26.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:26.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:26.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.853 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:26.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:26.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:26.998 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:26.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:27.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:27.137 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:27.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:27.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:27.263 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:27.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:27 smithi145 ceph-mon[89359]: pgmap v2445: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 860 op/s 2024-09-21T10:44:27.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:27 smithi119 ceph-mon[105288]: pgmap v2445: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 860 op/s 2024-09-21T10:44:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:27 smithi119 ceph-mon[101432]: pgmap v2445: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 860 op/s 2024-09-21T10:44:27.415 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:27.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.538 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:27.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:27.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:27.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:27.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:27.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:27.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:27.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:27.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:27.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:28.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:28.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.047 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:28.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:28.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:28.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:28.290 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:28.316 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:28.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:28.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:28.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:28.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:28.566 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:28.667 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:28.693 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:28.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:28.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:28.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:28.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:28.982 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:28.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:29.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:29.112 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:29.166 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:29 smithi119 ceph-mon[105288]: pgmap v2446: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:44:29.167 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:29 smithi119 ceph-mon[101432]: pgmap v2446: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:44:29.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:29.250 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:29 smithi145 ceph-mon[89359]: pgmap v2446: 109 pgs: 109 active+clean; 5.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:44:29.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:29.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:29.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:29.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:29.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:29.698 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:29.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:29.842 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:44:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:44:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.846 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:29.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:29.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:29.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:29.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:30.023 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:44:30.080 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:30.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:30.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:30.271 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:30.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:44:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:44:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:44:30.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3063090535' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:30.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:30 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3063090535' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:30.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:30 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:30.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3063090535' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:30.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:30 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:44:30.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:30.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:30.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:30.649 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:30.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:30.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:30.807 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:30.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:30.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:30.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:30.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:31.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:31.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:31.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:44:31.207 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:44:31.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-21T10:44:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.215 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:31.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:31 smithi145 ceph-mon[89359]: pgmap v2447: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 952 op/s 2024-09-21T10:44:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:31 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:31 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:31.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:31 smithi145 ceph-mon[89359]: osdmap e1410: 8 total, 8 up, 7 in 2024-09-21T10:44:31.297 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[105288]: pgmap v2447: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 952 op/s 2024-09-21T10:44:31.297 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:31.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:31.298 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[105288]: osdmap e1410: 8 total, 8 up, 7 in 2024-09-21T10:44:31.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[101432]: pgmap v2447: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 952 op/s 2024-09-21T10:44:31.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:31.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:44:31.298 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:31 smithi119 ceph-mon[101432]: osdmap e1410: 8 total, 8 up, 7 in 2024-09-21T10:44:31.316 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:31.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:31.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:31.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:31.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:31.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:31.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:31.798 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:31.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:31.937 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:31.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:32.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:32.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:32.165 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:32.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:32.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:32.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:32.329 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:32.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:32.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:32.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:32.582 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:32.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:32.683 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:32.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:32.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:32.852 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:32.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:32.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:32.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:33.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.012 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:33.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:33.098 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:33 smithi119 ceph-mon[101432]: pgmap v2449: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 850 op/s 2024-09-21T10:44:33.098 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:33.098 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:33 smithi119 ceph-mon[105288]: pgmap v2449: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 850 op/s 2024-09-21T10:44:33.098 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:33.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.138 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:33.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:33.151 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:44:33.151 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:44:33.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:33.289 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:33 smithi145 ceph-mon[89359]: pgmap v2449: 109 pgs: 109 active+clean; 5.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 850 op/s 2024-09-21T10:44:33.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:33.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:33.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:33.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:33.573 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:33.675 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:33.700 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:33.802 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:33.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:33.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:33.953 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:44:33.953 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:33.953 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:33.991 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:44:33.991 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:33.991 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:33.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:34.026 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:44:34.026 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:34.080 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:44:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:34.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:34.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:44:34.117 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:44:34.117 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:34.134 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:34.146 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:44:34.146 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:34.179 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:44:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:34.210 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:44:34.210 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:34.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:34.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:34 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/718668144' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:34.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:34.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:34 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/718668144' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:34.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:34 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/718668144' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:34.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:34.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:34.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:34.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:44:34.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:34.673 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:34.697 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:34.798 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:34.806 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:44:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:44:34] "GET /metrics HTTP/1.1" 200 36866 "" "Prometheus/2.51.0" 2024-09-21T10:44:34.821 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:34.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:34.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:34.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:35.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.074 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:35.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:35.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:35 smithi119 ceph-mon[105288]: pgmap v2450: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 855 op/s 2024-09-21T10:44:35.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:35 smithi119 ceph-mon[101432]: pgmap v2450: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 855 op/s 2024-09-21T10:44:35.175 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:35.198 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:35.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:35 smithi145 ceph-mon[89359]: pgmap v2450: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 855 op/s 2024-09-21T10:44:35.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:35.323 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:35.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:35.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:35.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:35.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:35.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:35.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:35.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:35.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:35.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:35.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:35.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:35.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:36.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:36.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:36.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:36.273 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:36.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:36.411 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:36.513 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:36.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:36.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:36.557 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:36.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:36.645 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:36.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:36.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:36.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.808 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:36.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:36.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:36.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:36.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:37.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:37.046 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:37 smithi119 ceph-mon[101432]: pgmap v2451: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:37.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.096 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:37.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:37.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:37.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:37.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:37.327 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:37 smithi119 ceph-mon[105288]: pgmap v2451: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:37.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:37.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:37.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:37.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:37 smithi145 ceph-mon[89359]: pgmap v2451: 109 pgs: 109 active+clean; 5.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:37.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:37.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:37.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:37.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:37.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:37.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:37.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:37.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:37.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.002 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:38.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:44:38.049 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:44:38.050 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:44:38.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:44:38.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:38.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:38.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:38.285 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:38.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:38.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:38.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:38.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:38.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:38.691 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.714 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:38.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:38.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:38.852 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:38.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:38.953 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:38.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:38.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:39.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:39.143 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:39.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:39.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:39 smithi119 ceph-mon[105288]: pgmap v2452: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:44:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:39 smithi119 ceph-mon[101432]: pgmap v2452: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:44:39.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:39.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:39.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:39.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:39 smithi145 ceph-mon[89359]: pgmap v2452: 109 pgs: 109 active+clean; 4.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:44:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:39.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:39.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:39.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:44:39.711 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:44:39.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-21T10:44:39.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:39.860 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:39.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:39.986 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:39.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:40.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:40.113 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:40.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:40.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:40.369 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:40.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:40.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:44:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:44:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:44:40.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:40.678 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:40.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:40.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:40.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:40.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:40.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:40.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:41.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:41.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:41.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:41.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.346 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:41.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:41.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:41 smithi119 ceph-mon[105288]: pgmap v2453: 109 pgs: 109 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 888 op/s 2024-09-21T10:44:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:41 smithi119 ceph-mon[101432]: pgmap v2453: 109 pgs: 109 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 888 op/s 2024-09-21T10:44:41.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:41.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:41 smithi145 ceph-mon[89359]: pgmap v2453: 109 pgs: 109 active+clean; 4.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 888 op/s 2024-09-21T10:44:41.586 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:41.630 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:41.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:41.732 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:41.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:41.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:41.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:41.905 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:41.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:42.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:42.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:42.136 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:42.184 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:42.287 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:42.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:42.419 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:42.463 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:42.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:42.565 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:42.592 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:42.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:42.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:42.747 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:42.775 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:44:42.775 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:44:42.775 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.775 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:153964 49.e 49:71c7996a:::benchmark_data_smithi119_301770_object40477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1410)", 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:42.693296+0000", 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.021073812000000001, 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.021109283, 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153964 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:42.776 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.693296+0000", 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.693296+0000", 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.693298+0000", 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.781e-06 2024-09-21T10:44:42.777 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.693304+0000", 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8470000000000001e-06 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.693318+0000", 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3638999999999999e-05 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:42.778 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.693325+0000", 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5530000000000004e-06 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.698295+0000", 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004969485 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.698503+0000", 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000208717 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.698624+0000", 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000120964 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.706375+0000", 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077504560000000002 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:153969 49.e 49:75e0079a:::benchmark_data_smithi119_301770_object40482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1410)", 2024-09-21T10:44:42.780 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:42.701946+0000", 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.012424417, 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.012529903, 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153969 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:42.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.701946+0000", 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.701946+0000", 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.701947+0000", 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6449999999999999e-06 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.701953+0000", 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7629999999999997e-06 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.701970+0000", 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6813e-05 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.701978+0000", 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4970000000000004e-06 2024-09-21T10:44:42.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.702724+0000", 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00074625600000000005 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.702865+0000", 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000141416 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.702990+0000", 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012516299999999999 2024-09-21T10:44:42.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.706384+0000", 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003393891 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:153973 49.f 49:f165fe9c:::benchmark_data_smithi119_301770_object40486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1410)", 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:42.711705+0000", 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0026648869999999999, 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028321610000000001, 2024-09-21T10:44:42.785 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153973 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711705+0000", 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711705+0000", 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711708+0000", 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2450000000000001e-06 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711715+0000", 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3050000000000004e-06 2024-09-21T10:44:42.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711737+0000", 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1636000000000001e-05 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711746+0000", 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6069999999999997e-06 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.711808+0000", 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1960999999999998e-05 2024-09-21T10:44:42.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712161+0000", 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00035317299999999997 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712295+0000", 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013384599999999999 2024-09-21T10:44:42.789 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36978.0:153964 49.e e1410/1394)", 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:42.712077+0000", 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0022933279999999999, 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025139749999999999, 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:42.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712077+0000", 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712077+0000", 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712078+0000", 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.7300000000000005e-07 2024-09-21T10:44:42.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712083+0000", 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4600000000000002e-06 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712099+0000", 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6036999999999999e-05 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712115+0000", 2024-09-21T10:44:42.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6109000000000001e-05 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36978.0:153964 49.e e1410/1394)", 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:42.712287+0000", 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0020831410000000002, 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023371580000000002, 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "queued for pg", 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:42.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712287+0000", 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712287+0000", 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712288+0000", 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9400000000000005e-07 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712292+0000", 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1509999999999997e-06 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712302+0000", 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0176e-05 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.712308+0000", 2024-09-21T10:44:42.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2929999999999997e-06 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:153955 49.7 49:e9c0e547:::benchmark_data_smithi119_301770_object40468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1410)", 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:42.685936+0000", 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.028434715999999999, 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.028727490000000001, 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:44:42.796 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 153955 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.685936+0000", 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.685936+0000", 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:44:42.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.685937+0000", 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.736e-06 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.685943+0000", 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.246e-06 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.685955+0000", 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1803e-05 2024-09-21T10:44:42.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.685962+0000", 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3820000000000001e-06 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.691754+0000", 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0057925269999999996 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.694981+0000", 2024-09-21T10:44:42.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032269719999999998 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.695105+0000", 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012422400000000001 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.700355+0000", 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0052496940000000001 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:42.702357+0000", 2024-09-21T10:44:42.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020023129999999999 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 6 2024-09-21T10:44:42.801 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:44:42.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:42.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:42.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:42.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:43.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:43.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:43.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:43.311 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:43.326 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:43.335 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:43.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:43 smithi119 ceph-mon[101432]: pgmap v2454: 109 pgs: 109 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T10:44:43.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:43 smithi119 ceph-mon[105288]: pgmap v2454: 109 pgs: 109 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T10:44:43.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:43.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:43.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:43 smithi145 ceph-mon[89359]: pgmap v2454: 109 pgs: 109 active+clean; 4.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T10:44:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.577 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:43.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:43.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:43.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:43.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:43.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:43.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:44.074 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:44:44.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:44.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:44:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:44.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:44.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:44.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:44.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1420847890' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:44 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1420847890' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:44 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:44.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:44.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:44.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1420847890' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:44 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:44:44.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:44.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:44.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:44.786 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:44:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:44:44] "GET /metrics HTTP/1.1" 200 36866 "" "Prometheus/2.51.0" 2024-09-21T10:44:44.791 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:44.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:44.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:44.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:45.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:45.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:45.160 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:45.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:45.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:45.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:45.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[105288]: pgmap v2455: 109 pgs: 109 active+clean; 4.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 912 op/s 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[105288]: osdmap e1411: 8 total, 8 up, 7 in 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[101432]: pgmap v2455: 109 pgs: 109 active+clean; 4.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 912 op/s 2024-09-21T10:44:45.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:44:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[101432]: osdmap e1411: 8 total, 8 up, 7 in 2024-09-21T10:44:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:45.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:45 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:45.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:45.454 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:45 smithi145 ceph-mon[89359]: pgmap v2455: 109 pgs: 109 active+clean; 4.5 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 912 op/s 2024-09-21T10:44:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:45 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:44:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:45 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:44:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:45 smithi145 ceph-mon[89359]: osdmap e1411: 8 total, 8 up, 7 in 2024-09-21T10:44:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:45 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:44:45.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:45.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:45.727 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:45.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:45.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:45.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:45.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:45.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:45.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:46.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:46.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.161 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:46.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:46.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:46.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:46.385 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:46.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:46.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:46.535 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:46.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:46.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:46.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:46.775 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:46.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:46.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:46.928 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:46.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:47.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:47.051 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:47.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:47.175 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:47.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:47.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:44:47.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:47.329 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:47 smithi119 ceph-mon[101432]: pgmap v2457: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-21T10:44:47.329 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:47 smithi119 ceph-mon[105288]: pgmap v2457: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-21T10:44:47.422 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:47.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:47 smithi145 ceph-mon[89359]: pgmap v2457: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-21T10:44:47.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:47.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:47.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:47.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:47.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:47.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:47.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:47.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:47.985 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:48.007 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:48.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:48.145 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:48.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:48.246 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:48.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:48.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:48.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:44:48.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:48.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:48.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:48.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:48.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:48.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:48.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:48.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:48.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:49.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:49.069 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:49.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:49.206 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:49.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:49.307 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:49.331 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:49 smithi119 ceph-mon[105288]: pgmap v2458: 109 pgs: 109 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:44:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:49 smithi119 ceph-mon[101432]: pgmap v2458: 109 pgs: 109 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:44:49.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:49.459 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:49 smithi145 ceph-mon[89359]: pgmap v2458: 109 pgs: 109 active+clean; 4.3 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:44:49.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.591 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:49.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:49.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:49.717 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:49.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:49.857 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:49.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:49.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:49.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:49.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:50.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:50.128 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:50.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:50.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:50.357 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:50.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:50.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:50.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:44:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:44:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:44:50.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:50.630 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:50.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:50.732 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:50.775 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:50.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:50.901 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:50.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:50.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:51.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:51.051 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:51.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:51.178 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:51.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:51.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:51 smithi145 ceph-mon[89359]: pgmap v2459: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:44:51.307 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:51 smithi119 ceph-mon[105288]: pgmap v2459: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:44:51.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:51 smithi119 ceph-mon[101432]: pgmap v2459: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:44:51.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:51.439 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:51.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:51.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:51.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:51.811 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:51.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:51.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:51.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:51.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:51.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:52.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:52.116 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:52.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:52.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:52.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:52.383 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:52.485 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:52.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:52.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:52.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:52.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:52.807 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:52.909 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:52.916 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:52.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:52.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:53.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:53.072 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:53.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:53.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:53.251 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:53 smithi119 ceph-mon[101432]: pgmap v2460: 109 pgs: 109 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 883 op/s 2024-09-21T10:44:53.251 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:53 smithi119 ceph-mon[105288]: pgmap v2460: 109 pgs: 109 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 883 op/s 2024-09-21T10:44:53.320 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:53.344 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:53.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:53.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:53.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:53 smithi145 ceph-mon[89359]: pgmap v2460: 109 pgs: 109 active+clean; 4.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 883 op/s 2024-09-21T10:44:53.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:53.633 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:53.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:53.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:53.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:53.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:53.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:53.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:54.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:54.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:54.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:54.245 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:44:54.245 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:44:54.246 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:44:54.248 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:44:54.249 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:44:54.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:54.270 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:44:54.270 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:44:54.275 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:54.318 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:44:54.318 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:44:54.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/624412984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:54.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/624412984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:54.350 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:44:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:44:54.376 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:54.378 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:44:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:44:54.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:54.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:54.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:54.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/624412984' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:44:54.578 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:44:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:44:54] "GET /metrics HTTP/1.1" 200 36867 "" "Prometheus/2.51.0" 2024-09-21T10:44:54.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.670 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:54.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:54.771 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:54.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:54.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:54.920 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:54.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:55.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:55.046 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:55.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.185 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:55.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:55.269 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:55 smithi119 ceph-mon[105288]: pgmap v2461: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:55.269 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:55 smithi119 ceph-mon[101432]: pgmap v2461: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:55.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:55.321 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:55.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:55.423 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:55.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:55 smithi145 ceph-mon[89359]: pgmap v2461: 109 pgs: 109 active+clean; 4.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 866 op/s 2024-09-21T10:44:55.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:55.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:55.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:44:55.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:44:55.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:55.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:55.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:55.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:56.041 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:56.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:56.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:56.168 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:56.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:56.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:56.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:56.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:56.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:56.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:56.638 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:44:56.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:44:56.679 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:44:56.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-21T10:44:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:56.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:56.812 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:56.836 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:56.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:56.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:56.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:57.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:44:57.087 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:44:57.189 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:57.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:57.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:57.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:57.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:57.364 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:57 smithi119 ceph-mon[101432]: pgmap v2462: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T10:44:57.364 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:57 smithi119 ceph-mon[105288]: pgmap v2462: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T10:44:57.453 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:57.476 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:57.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:57 smithi145 ceph-mon[89359]: pgmap v2462: 109 pgs: 109 active+clean; 3.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 837 op/s 2024-09-21T10:44:57.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:57.624 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:57.726 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:57.771 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:57.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:57.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:57.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:58.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:58.024 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:58.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:58.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:58.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:58.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:58.278 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:58.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:58.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:58.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:58.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:58.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:58.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:58.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:58.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:44:58.686 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:58.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:44:58.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:44:58.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:58.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:58.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:59.025 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:44:59.025 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:44:59.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:59.098 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:59.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:44:59.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:44:59.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:44:59.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:59.365 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:44:59 smithi119 ceph-mon[105288]: pgmap v2463: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T10:44:59.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:44:59 smithi119 ceph-mon[101432]: pgmap v2463: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T10:44:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:44:59.466 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:59.491 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:44:59 smithi145 ceph-mon[89359]: pgmap v2463: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 854 op/s 2024-09-21T10:44:59.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:44:59.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:44:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:44:59.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:44:59.756 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:44:59.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:44:59.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:44:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:44:59.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:44:59.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:00.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:00.109 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:00.134 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:00.235 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:00.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:00.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:45:00.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:00.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.416 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:00.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:00.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:45:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:45:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:45:00.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:00.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:00.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:00.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:00.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:00.866 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:00.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:00.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.012 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:01.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:01.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:01.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:01.226 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:01.256 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:01.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:01.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:01.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:01.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:01.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:01 smithi145 ceph-mon[89359]: pgmap v2464: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:45:01.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:01.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:01.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:01 smithi119 ceph-mon[105288]: pgmap v2464: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:45:01.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:01 smithi119 ceph-mon[101432]: pgmap v2464: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 949 op/s 2024-09-21T10:45:01.646 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:01.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:01.655 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:01.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:01.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:01.801 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:01.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:01.926 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:01.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:01.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:02.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:02.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:02.140 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:45:02.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:02.216 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:02.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:02.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1368207710' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:02.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:02 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:02.290 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1368207710' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:02.290 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:02 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:02.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1368207710' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:02.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:02 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:02.318 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:02.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:02.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:02.466 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:02.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:02.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:02.709 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:02.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:02.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:02.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:02.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:02.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:02.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:03.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:03.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:03.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:03.296 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: pgmap v2465: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: osdmap e1412: 8 total, 8 up, 7 in 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: pgmap v2465: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:45:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:45:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:45:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: osdmap e1412: 8 total, 8 up, 7 in 2024-09-21T10:45:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:03.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:03.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:03.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:03.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: pgmap v2465: 109 pgs: 109 active+clean; 3.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 814 op/s 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: osdmap e1412: 8 total, 8 up, 7 in 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:03.570 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:03.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:03.696 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:03.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:03.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:03.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:03.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:03.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:04.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:04.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:04.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:04.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:04.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:04.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:04.385 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:04.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:04.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:04.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:04.671 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:04.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:04.799 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:04.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:45:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:45:04] "GET /metrics HTTP/1.1" 200 36865 "" "Prometheus/2.51.0" 2024-09-21T10:45:04.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:04.924 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:04.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:05.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:05.049 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:05.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:05.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:05.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:05.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:05.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:05 smithi119 ceph-mon[101432]: pgmap v2467: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:45:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:05 smithi119 ceph-mon[105288]: pgmap v2467: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:45:05.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:05.443 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:05 smithi145 ceph-mon[89359]: pgmap v2467: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 881 op/s 2024-09-21T10:45:05.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:05.567 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:05.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:05.695 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:05.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:05.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:05.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:05.933 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:05.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:05.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:06.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:06.081 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:06.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:06.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:06.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:06.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:06.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:06.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:06.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:06.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:06.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:06.751 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:06.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:06.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:06.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:06.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:06.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:07.002 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:07.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:07.129 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:07.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:07.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:07.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:07.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:07.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:07 smithi145 ceph-mon[89359]: pgmap v2468: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:45:07.593 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:07 smithi119 ceph-mon[105288]: pgmap v2468: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:45:07.593 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:07 smithi119 ceph-mon[101432]: pgmap v2468: 109 pgs: 109 active+clean; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 891 op/s 2024-09-21T10:45:07.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:07.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:07.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:07.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:07.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:07.928 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:07.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:07.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:08.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:08.053 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:08.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:08.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:08.294 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:08.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:08.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:08.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:08.470 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:08.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:08.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.609 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:08.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:08.709 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:08.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:08.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:08.858 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:08.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:08.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:08.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:09.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:09.129 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:09.231 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:09.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:09.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:09.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:09.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:09.507 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:09 smithi145 ceph-mon[89359]: pgmap v2469: 109 pgs: 109 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 885 op/s 2024-09-21T10:45:09.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:09 smithi119 ceph-mon[105288]: pgmap v2469: 109 pgs: 109 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 885 op/s 2024-09-21T10:45:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:09 smithi119 ceph-mon[101432]: pgmap v2469: 109 pgs: 109 active+clean; 3.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 0 B/s wr, 885 op/s 2024-09-21T10:45:09.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:09.734 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:09.758 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:09.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:09.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:09.897 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:09.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:09.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:10.035 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:10.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:10.174 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:10.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:10.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:10.439 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:10.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:45:10.494 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:10.494 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:10.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:45:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:45:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:45:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:10.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:10.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:10.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:10.792 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:10 smithi119 ceph-mon[101432]: pgmap v2470: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:45:10.793 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:10 smithi119 ceph-mon[105288]: pgmap v2470: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:45:10.793 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:45:10.793 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:10.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:10.832 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:45:10.832 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:10.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:10.865 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:45:10.865 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:10.899 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:45:10.899 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:10.919 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:45:10.919 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:10.948 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:45:10.949 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:10.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:10.978 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:45:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:10.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:10.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:10.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:45:11.011 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:10 smithi145 ceph-mon[89359]: pgmap v2470: 109 pgs: 109 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 882 op/s 2024-09-21T10:45:11.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:11.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.112 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:11.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:11.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:11.251 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:11.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:11.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:11.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:11.516 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:11.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.652 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:11.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:11.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:11.792 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:11.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:11.914 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:11.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:12.016 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1976929835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:12.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:12.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1976929835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1976929835' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:12.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:12.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:12.272 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.306 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:12.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:12.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:12.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:12.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:12.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:12.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:12.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:12.816 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:12.840 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:12.942 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:12.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:12.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:13.012 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:12 smithi119 ceph-mon[101432]: pgmap v2471: 109 pgs: 109 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:45:13.012 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:12 smithi119 ceph-mon[105288]: pgmap v2471: 109 pgs: 109 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:45:13.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:12 smithi145 ceph-mon[89359]: pgmap v2471: 109 pgs: 109 active+clean; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 880 op/s 2024-09-21T10:45:13.071 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:13.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:13.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:13.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:13.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:13.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:13.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:13.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:13.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:13.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:13.796 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:13.818 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:13.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:13.976 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:13.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:14.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:14.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:14.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.255 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:14.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:14.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:14.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:14.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.526 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:14.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:45:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:45:14] "GET /metrics HTTP/1.1" 200 36865 "" "Prometheus/2.51.0" 2024-09-21T10:45:14.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:45:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:45:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:45:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:14.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:14.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:14.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:14.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:14.897 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:14.921 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:14.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:15.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:15.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:15.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:15.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:15.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:15.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:15.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:15 smithi145 ceph-mon[89359]: pgmap v2472: 109 pgs: 109 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:45:15.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:15.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:15 smithi119 ceph-mon[105288]: pgmap v2472: 109 pgs: 109 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:45:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:15 smithi119 ceph-mon[101432]: pgmap v2472: 109 pgs: 109 active+clean; 2.9 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 902 op/s 2024-09-21T10:45:15.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:15.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:15.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:15.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:15.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:45:15.583 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:45:15.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-21T10:45:15.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:15.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:15.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:15.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:15.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:15.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:15.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:16.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:16.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:16.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:16.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:16.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:16.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:45:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.388 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:16.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:16.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:16.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:16.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:45:16.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:16.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:16.775 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:16.798 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:16.900 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:16.977 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:16.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:17.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:17.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:17.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:17.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:17.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[105288]: pgmap v2473: 109 pgs: 109 active+clean; 2.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 840 op/s 2024-09-21T10:45:17.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:45:17.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:45:17.289 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[101432]: pgmap v2473: 109 pgs: 109 active+clean; 2.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 840 op/s 2024-09-21T10:45:17.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:45:17.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:45:17.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:17.402 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:17.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:17.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:17 smithi145 ceph-mon[89359]: pgmap v2473: 109 pgs: 109 active+clean; 2.8 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 0 B/s wr, 840 op/s 2024-09-21T10:45:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:45:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:45:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:17.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:17.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:17.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:17.810 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:17.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:17.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:17.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:18.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:18.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:18.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:18.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:18.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:18.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:18.329 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:18.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:18.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:18.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:18.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:18.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:18.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:18.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:18.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:18.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:18.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:18.864 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:18.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:18.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.018 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:19.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:19.073 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:19.087 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:19.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:19.179 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:19.264 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:19.281 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:19.304 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:19 smithi119 ceph-mon[105288]: pgmap v2474: 109 pgs: 109 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:45:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:19 smithi119 ceph-mon[101432]: pgmap v2474: 109 pgs: 109 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:45:19.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.462 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:19.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:19 smithi145 ceph-mon[89359]: pgmap v2474: 109 pgs: 109 active+clean; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 862 op/s 2024-09-21T10:45:19.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.614 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:19.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:19.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:19.738 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:19.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:19.839 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:19.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:19.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:19.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:20.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:20.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:20.147 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:20.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:20.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:20.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:20.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:20.416 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:20.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:20.517 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:20.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:45:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:45:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:45:20.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:20.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:20.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:20.927 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:20.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:20.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:21.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:21.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:21.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:21.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:21.202 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:21.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:21.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:21.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:21 smithi145 ceph-mon[89359]: pgmap v2475: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-09-21T10:45:21.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:21.609 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:21.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:21 smithi119 ceph-mon[105288]: pgmap v2475: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-09-21T10:45:21.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:21 smithi119 ceph-mon[101432]: pgmap v2475: 109 pgs: 109 active+clean; 2.6 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 950 op/s 2024-09-21T10:45:21.710 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:21.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:21.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:21.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:21.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:21.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:22.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:22.118 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:22.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:22.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:22.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:22.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:22.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:22.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:22.535 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:22.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:22.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:22.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:22.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:22.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:22.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:22.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:22.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:23.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:23.069 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:23.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:23.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:23.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:23.253 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:45:23.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:23.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:23.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:23 smithi119 ceph-mon[101432]: pgmap v2476: 109 pgs: 109 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 810 op/s 2024-09-21T10:45:23.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/310867871' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:23 smithi119 ceph-mon[105288]: pgmap v2476: 109 pgs: 109 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 810 op/s 2024-09-21T10:45:23.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/310867871' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:23.453 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:23.476 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:23 smithi145 ceph-mon[89359]: pgmap v2476: 109 pgs: 109 active+clean; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 810 op/s 2024-09-21T10:45:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/310867871' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:45:23.577 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:23.601 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:23.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:23.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:23.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:23.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:23.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:23.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:23.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:23.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:24.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:24.128 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:24.230 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:24.272 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:24.374 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:24.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:24.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:24.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:24.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:24.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/310867871' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:24.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-mon[101432]: osdmap e1413: 8 total, 8 up, 7 in 2024-09-21T10:45:24.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:24.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/310867871' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:24.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-mon[105288]: osdmap e1413: 8 total, 8 up, 7 in 2024-09-21T10:45:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:24 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/310867871' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:45:24.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:24 smithi145 ceph-mon[89359]: osdmap e1413: 8 total, 8 up, 7 in 2024-09-21T10:45:24.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:24.660 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:24.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:24.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:24.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:24.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:45:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:45:24] "GET /metrics HTTP/1.1" 200 36861 "" "Prometheus/2.51.0" 2024-09-21T10:45:24.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:24.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:24.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:24.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:25.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:25.054 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:45:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:45:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:45:25.089 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:45:25.090 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:45:25.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:25.179 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:25.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:25.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:25.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:25.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:25.425 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:25.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:25.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:25.449 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:25.450 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:25.481 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:45:25.482 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:25.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:25 smithi119 ceph-mon[101432]: pgmap v2478: 109 pgs: 109 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:45:25.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:25 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2891901604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:25.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:25 smithi119 ceph-mon[105288]: pgmap v2478: 109 pgs: 109 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:45:25.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:25 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2891901604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:25.518 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:45:25.518 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:25 smithi145 ceph-mon[89359]: pgmap v2478: 109 pgs: 109 active+clean; 2.4 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 874 op/s 2024-09-21T10:45:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:25 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2891901604' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:25.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:25.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:25.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:25.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:25.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:25.842 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:25.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:45:25.962 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:45:25.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-21T10:45:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:25.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:25.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:26.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:26.134 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:26.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:26.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:26.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:26.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:45:26.402 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:26.504 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:26.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:26.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:26.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:26.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:26.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:27.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:27.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:27.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:27.304 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:27.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:27 smithi119 ceph-mon[105288]: pgmap v2479: 109 pgs: 109 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:45:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:27 smithi119 ceph-mon[101432]: pgmap v2479: 109 pgs: 109 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:45:27.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:27.462 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:27 smithi145 ceph-mon[89359]: pgmap v2479: 109 pgs: 109 active+clean; 2.3 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:45:27.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:27.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:27.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:27.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.748 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:27.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:27.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:27.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:27.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:27.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:27.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:28.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:28.127 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:28.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:28.253 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:28.278 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:28.379 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:28.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:28.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:28.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:28.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:28.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:28.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:28.751 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:28.772 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:28.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:28.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:28.912 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:28.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:28.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:29.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:29.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:29.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:29.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:29.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:29.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.433 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:29.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:29.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:29 smithi145 ceph-mon[89359]: pgmap v2480: 109 pgs: 109 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:45:29.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:29 smithi119 ceph-mon[105288]: pgmap v2480: 109 pgs: 109 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:45:29.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:29 smithi119 ceph-mon[101432]: pgmap v2480: 109 pgs: 109 active+clean; 2.2 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:45:29.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:29.695 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:29.796 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:29.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:29.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:29.964 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:45:29.964 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:45:29.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:29.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:30.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:30.112 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:30.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:30.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:30.246 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:30.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:30.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:30.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:30.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:30.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:45:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:45:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:45:30.598 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:30.621 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:30.723 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:30.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:30.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:30.882 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:30.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:30.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:31.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:31.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:31.149 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:31.213 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:45:31.237 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:31.237 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:31.251 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:31.272 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:31.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:31.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:31.432 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:31 smithi119 ceph-mon[101432]: pgmap v2481: 109 pgs: 109 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-21T10:45:31.433 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:31 smithi119 ceph-mon[105288]: pgmap v2481: 109 pgs: 109 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-21T10:45:31.518 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:31.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:31 smithi145 ceph-mon[89359]: pgmap v2481: 109 pgs: 109 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 892 op/s 2024-09-21T10:45:31.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:31.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:31.805 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:31.834 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:31.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:31.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:31.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:31.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:32.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:32.130 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:32.232 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:32.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:32.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:32.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:32.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:32.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:45:32.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:32.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:32.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:32.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:32.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:32.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:32.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:33.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:33.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:33.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.211 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:33.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:33.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:33.336 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:33.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:33.474 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:33 smithi145 ceph-mon[89359]: pgmap v2482: 109 pgs: 109 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:45:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:33.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:33.600 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:33 smithi119 ceph-mon[101432]: pgmap v2482: 109 pgs: 109 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:45:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:33 smithi119 ceph-mon[105288]: pgmap v2482: 109 pgs: 109 active+clean; 2.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:45:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:45:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:33.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:33.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:33.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:33.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:33.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:33.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:34.032 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:34.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:34.177 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:34.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:34.303 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:34.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:34.448 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:34.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:34.573 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:34.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:34.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:45:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:45:34] "GET /metrics HTTP/1.1" 200 36864 "" "Prometheus/2.51.0" 2024-09-21T10:45:34.675 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:34.711 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:34.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:34.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:34.851 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:no valid command found; 10 closest matches: 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:0 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:1 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:2 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:abort 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:assert 2024-09-21T10:45:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:bench [] [] [] [] 2024-09-21T10:45:34.866 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs debug_inject_read_zeros 2024-09-21T10:45:34.866 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs files list 2024-09-21T10:45:34.866 INFO:teuthology.orchestra.run.smithi119.stderr:bluefs stats 2024-09-21T10:45:34.866 INFO:teuthology.orchestra.run.smithi119.stderr:bluestore allocator dump block 2024-09-21T10:45:34.866 INFO:teuthology.orchestra.run.smithi119.stderr:Error EINVAL: invalid command 2024-09-21T10:45:34.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:34.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:34.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:35.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:35.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:35.219 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:35.296 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:45:35.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:35.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:35.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:35.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:35.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:35.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:35 smithi119 ceph-mon[105288]: pgmap v2483: 109 pgs: 109 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 960 op/s 2024-09-21T10:45:35.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1092127098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:35.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:35 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:35.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:35 smithi119 ceph-mon[101432]: pgmap v2483: 109 pgs: 109 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 960 op/s 2024-09-21T10:45:35.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1092127098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:35.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:35 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T10:45:35.585 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T10:45:35.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:35.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:35.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:35 smithi145 ceph-mon[89359]: pgmap v2483: 109 pgs: 109 active+clean; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 960 op/s 2024-09-21T10:45:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1092127098' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:35.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:35 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:45:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.825 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:35.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:35.926 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:35.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:35.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:36.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:36.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:36.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:36.315 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:36.350 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:36.414 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:36.453 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:36.474 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:36.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:36.613 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:36 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:36.613 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:36 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:36.613 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:36 smithi119 ceph-mon[105288]: osdmap e1414: 8 total, 8 up, 7 in 2024-09-21T10:45:36.614 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:36 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:36.614 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:36 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:36.614 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:36 smithi119 ceph-mon[101432]: osdmap e1414: 8 total, 8 up, 7 in 2024-09-21T10:45:36.619 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:36.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:36.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:36.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:36 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:36 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:45:36.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:36 smithi145 ceph-mon[89359]: osdmap e1414: 8 total, 8 up, 7 in 2024-09-21T10:45:36.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:36.887 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:36.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:36.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:37.013 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:37.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:37.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:37.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:37.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:37.352 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:37.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.417 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:37.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:37.518 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:37 smithi119 ceph-mon[105288]: pgmap v2485: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 867 op/s 2024-09-21T10:45:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:37 smithi119 ceph-mon[101432]: pgmap v2485: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 867 op/s 2024-09-21T10:45:37.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:37.700 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:37.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:37 smithi145 ceph-mon[89359]: pgmap v2485: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 867 op/s 2024-09-21T10:45:37.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:37.825 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:37.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:37.959 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:37.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:38.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:38.086 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:38.188 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:38.213 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:38.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:38.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:38.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:45:38.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.497 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:38.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:38.598 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:38.621 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:38.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:38.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:38.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:38.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:38.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:38.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:39.002 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:39.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:39.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:39.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:39.229 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:39.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:39.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:39.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:39.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:39.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:39 smithi119 ceph-mon[105288]: pgmap v2486: 109 pgs: 109 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 868 op/s 2024-09-21T10:45:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:39 smithi119 ceph-mon[101432]: pgmap v2486: 109 pgs: 109 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 868 op/s 2024-09-21T10:45:39.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:39.670 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:39.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:39.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:39 smithi145 ceph-mon[89359]: pgmap v2486: 109 pgs: 109 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 868 op/s 2024-09-21T10:45:39.794 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:39.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:39.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:39.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:40.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:40.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:40.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.202 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:40.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:40.246 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:45:40.246 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:40.289 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:45:40.289 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:40.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:40.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:40.334 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:45:40.334 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:40.374 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:45:40.374 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:40.404 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:45:40.405 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:40.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:40.435 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:45:40.436 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:40.452 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:40.468 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:45:40.468 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:40.498 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:45:40.498 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/271469069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:45:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:45:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:45:40.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:40.585 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:45:40.586 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:45:40.586 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T10:45:40.586 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T10:45:40.586 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T10:45:40.586 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T10:45:40.594 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/271469069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:40.594 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/271469069' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:40.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:40.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:40.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:40.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:40.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:40.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:40.911 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:45:40.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:41.006 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:45:41.006 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:45:41.006 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:45:41.006 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:45:41.006 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:45:41.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:41.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:41.139 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:41.240 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:41.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:41.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:41.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.392 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:41.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:41.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:41.535 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:41 smithi119 ceph-mon[105288]: pgmap v2487: 109 pgs: 109 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 829 op/s 2024-09-21T10:45:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:41 smithi119 ceph-mon[101432]: pgmap v2487: 109 pgs: 109 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 829 op/s 2024-09-21T10:45:41.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:41.684 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:41.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:41.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:41 smithi145 ceph-mon[89359]: pgmap v2487: 109 pgs: 109 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 829 op/s 2024-09-21T10:45:41.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:41.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:41.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:41.934 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:41.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:41.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:42.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:42.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:42.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:42.217 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:42.318 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:42.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:42.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:42.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:45:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:42.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:42.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:42.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:42.744 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:42.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:42.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:42.899 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:42.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:43.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.043 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:43.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:43.144 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:43.177 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:43.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:43.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:43.416 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.442 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:43.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:43.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:43 smithi145 ceph-mon[89359]: pgmap v2488: 109 pgs: 109 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 876 op/s 2024-09-21T10:45:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:43.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:43 smithi119 ceph-mon[105288]: pgmap v2488: 109 pgs: 109 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 876 op/s 2024-09-21T10:45:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:43 smithi119 ceph-mon[101432]: pgmap v2488: 109 pgs: 109 active+clean; 1.4 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 876 op/s 2024-09-21T10:45:43.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:45:43.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:43.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:43.892 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:43.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:43.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:44.002 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:44.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:44.124 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:45:44.124 INFO:teuthology.orchestra.run.smithi145.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T10:45:44.125 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:45:44.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:44.204 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:44.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:44.330 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:44.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:44.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:44.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:44.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:44.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:44.716 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:45:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:45:44] "GET /metrics HTTP/1.1" 200 36864 "" "Prometheus/2.51.0" 2024-09-21T10:45:44.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:44.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:44.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:44.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:44.986 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:45.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:45.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:45.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:45.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:45.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:45.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:45.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:45.457 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:45.559 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:45.584 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:45 smithi119 ceph-mon[101432]: pgmap v2489: 109 pgs: 109 active+clean; 1.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-09-21T10:45:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:45 smithi119 ceph-mon[105288]: pgmap v2489: 109 pgs: 109 active+clean; 1.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-09-21T10:45:45.687 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:45.711 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:45.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:45 smithi145 ceph-mon[89359]: pgmap v2489: 109 pgs: 109 active+clean; 1.3 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 889 op/s 2024-09-21T10:45:45.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:45.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:45.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:45.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:45.961 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:45.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:45.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:46.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:46.065 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:46.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:46.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:46.217 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:46.319 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:46.362 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:46.373 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:45:46.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:46.490 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:46.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:46.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2438970662' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:46.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:46 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2438970662' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:46.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:46 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:46.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:46.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:46.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.778 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:46.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2438970662' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:46 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:45:46.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:46.928 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:46.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:47.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:47.055 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:47.156 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:47.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:47.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:47.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:47.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:47.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:47.590 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:47.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[101432]: pgmap v2490: 109 pgs: 109 active+clean; 1.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 840 op/s 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[101432]: osdmap e1415: 8 total, 8 up, 7 in 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[105288]: pgmap v2490: 109 pgs: 109 active+clean; 1.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 840 op/s 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:45:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:47 smithi119 ceph-mon[105288]: osdmap e1415: 8 total, 8 up, 7 in 2024-09-21T10:45:47.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:47.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:47 smithi145 ceph-mon[89359]: pgmap v2490: 109 pgs: 109 active+clean; 1.2 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 840 op/s 2024-09-21T10:45:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:47 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:45:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:47 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:45:47.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:47 smithi145 ceph-mon[89359]: osdmap e1415: 8 total, 8 up, 7 in 2024-09-21T10:45:47.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:47.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:47.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:47.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.003 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:48.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:48.104 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:48.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.131 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:48.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:48.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:48.270 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:48.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:48.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:48.506 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:48.530 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:48.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:48.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:48.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:48.654 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:48.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:48.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:45:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:48.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:48.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:48.923 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:48.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:49.025 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:49.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:49.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:49.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:49.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:49.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:49.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:49.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:49.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:49.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:49.546 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:49 smithi119 ceph-mon[101432]: pgmap v2492: 109 pgs: 109 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:45:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:49 smithi119 ceph-mon[105288]: pgmap v2492: 109 pgs: 109 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:45:49.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:49.685 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:49.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:49.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:49 smithi145 ceph-mon[89359]: pgmap v2492: 109 pgs: 109 active+clean; 1.1 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 893 op/s 2024-09-21T10:45:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.823 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:49.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:49.858 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 7, 5] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:45:49.858 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:45:49.858 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 6, 7, 5] 2024-09-21T10:45:49.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd out 7 2024-09-21T10:45:49.924 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:49.947 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:49.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:50.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:50.071 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:50.173 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:50.234 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:50.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:50.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:50.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:50.498 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:45:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:45:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:45:50.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:50.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:50.722 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:50.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:50.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:50.901 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:50.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:51.002 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:51.029 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:51.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:51.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:51.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:51.259 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:51.286 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:51.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:51.422 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:51.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:51 smithi145 ceph-mon[89359]: pgmap v2493: 109 pgs: 109 active+clean; 996 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:45:51.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:51.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:51.689 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:51.791 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:51.818 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:51.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:51 smithi119 ceph-mon[101432]: pgmap v2493: 109 pgs: 109 active+clean; 996 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:45:51.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:51 smithi119 ceph-mon[105288]: pgmap v2493: 109 pgs: 109 active+clean; 996 MiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:45:51.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:51.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:51.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:51.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:52.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:52.082 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:52.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:52.224 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:52.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:52.358 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:52.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:52.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:52.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:52.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:52.621 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:52.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:52.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.747 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:52.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:52.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:52.896 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:52.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73888 48.4 48:2e75d43f:::benchmark_data_smithi119_267254_object73887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.693234+0000", 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 428.09011397799998, 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46585072999999999, 2024-09-21T10:45:52.928 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73888 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693234+0000", 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.929 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693234+0000", 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693236+0000", 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6199999999999999e-06 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693259+0000", 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3017e-05 2024-09-21T10:45:52.930 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693273+0000", 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3974999999999999e-05 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693280+0000", 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3730000000000004e-06 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.931 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693339+0000", 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9147999999999997e-05 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693482+0000", 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014265400000000001 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.693623+0000", 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014130600000000001 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.932 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158578+0000", 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46495520600000001 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158619+0000", 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0404999999999997e-05 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158998+0000", 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037948099999999998 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.933 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159047+0000", 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8637999999999997e-05 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159085+0000", 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7907000000000001e-05 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.934 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73892 48.4 48:26a06aae:::benchmark_data_smithi119_267254_object73891:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.694971+0000", 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 428.08837662100001, 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46426756200000002, 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73892 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.935 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.694971+0000", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.694971+0000", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.694973+0000", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.875e-06 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.936 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695005+0000", 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2094000000000001e-05 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695018+0000", 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2794e-05 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695026+0000", 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6070000000000002e-06 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.937 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.696896+0000", 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.001869894 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.697028+0000", 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013275900000000001 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,3", 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.697158+0000", 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000129932 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.938 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158588+0000", 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46142990699999997 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158722+0000", 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013418999999999999 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159132+0000", 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00040954899999999999 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.939 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159180+0000", 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8374000000000001e-05 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159239+0000", 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8587e-05 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73895 48.a 48:52a5ab4b:::benchmark_data_smithi119_267254_object73894:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.695044+0000", 2024-09-21T10:45:52.940 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 428.088303435, 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46376412299999997, 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73895 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695044+0000", 2024-09-21T10:45:52.941 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695044+0000", 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695045+0000", 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0299999999999999e-07 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695066+0000", 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1246999999999998e-05 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695073+0000", 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2069999999999998e-06 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695081+0000", 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8520000000000004e-06 2024-09-21T10:45:52.943 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695138+0000", 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7051e-05 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695285+0000", 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014705400000000001 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.695412+0000", 2024-09-21T10:45:52.944 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000126391 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698128+0000", 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027160639999999998 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698382+0000", 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025434299999999997 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158704+0000", 2024-09-21T10:45:52.945 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46032213100000002 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158758+0000", 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3431999999999999e-05 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158809+0000", 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0847999999999998e-05 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.946 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:73897 48.a 48:5db11cab:::benchmark_data_smithi119_267254_object73896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:39.698883+0000", 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 428.08446537499998, 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.46637884099999999, 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 73897 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.947 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698883+0000", 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698883+0000", 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698884+0000", 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.353e-06 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.948 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698905+0000", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1542000000000001e-05 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:39.698915+0000", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3179999999999996e-06 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158511+0000", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.45959662899999998 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.949 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.158894+0000", 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038238299999999997 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159034+0000", 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014023599999999999 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.159212+0000", 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017811199999999999 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.950 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165101+0000", 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005888908 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165144+0000", 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2777000000000001e-05 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165156+0000", 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1970999999999999e-05 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.951 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165211+0000", 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5526999999999999e-05 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:40.165261+0000", 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0084999999999999e-05 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.952 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:192748 48.0 48:0e10cdd9:::benchmark_data_smithi119_267254_object69538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:45.540518+0000", 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 302.242829675, 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23870176800000001, 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192748 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.953 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540518+0000", 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540518+0000", 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540520+0000", 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3799999999999999e-06 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.954 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540526+0000", 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3609999999999997e-06 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540574+0000", 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8304999999999999e-05 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540581+0000", 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0859999999999996e-06 2024-09-21T10:45:52.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540724+0000", 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000142853 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540854+0000", 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013023599999999999 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.540978+0000", 2024-09-21T10:45:52.956 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012330800000000001 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.547151+0000", 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0061727240000000001 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549395+0000", 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022446279999999998 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.957 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.779149+0000", 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.22975384600000001 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.779187+0000", 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8121000000000003e-05 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.779220+0000", 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2920000000000003e-05 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.958 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:192759 48.0 48:0ace1b28:::benchmark_data_smithi119_267254_object69549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:45.549759+0000", 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 302.23358853299999, 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24153683000000001, 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192759 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.959 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549759+0000", 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549759+0000", 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549760+0000", 2024-09-21T10:45:52.960 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1500000000000003e-07 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549765+0000", 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.515e-06 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549793+0000", 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8475000000000001e-05 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.961 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549801+0000", 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8299999999999996e-06 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549929+0000", 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000128262 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.550030+0000", 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000100591 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.962 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.550139+0000", 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108595 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556519+0000", 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0063808379999999998 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.560422+0000", 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003902918 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.963 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.791209+0000", 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.230786885 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.791256+0000", 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7173999999999999e-05 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.791296+0000", 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9832000000000001e-05 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.964 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:192761 48.a 48:5ec06fe0:::benchmark_data_smithi119_267254_object69551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:45.549968+0000", 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 302.23337945100002, 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23485099400000001, 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192761 2024-09-21T10:45:52.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549968+0000", 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549968+0000", 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.966 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549969+0000", 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8299999999999995e-07 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549973+0000", 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0389999999999998e-06 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549983+0000", 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2960000000000004e-06 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.967 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.549989+0000", 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7190000000000004e-06 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.550039+0000", 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9314000000000003e-05 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.550135+0000", 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6294999999999999e-05 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.550234+0000", 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9359999999999997e-05 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.555362+0000", 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005127754 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556025+0000", 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00066244700000000004 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.969 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.784718+0000", 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.22869368300000001 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.784777+0000", 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8430000000000001e-05 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.784819+0000", 2024-09-21T10:45:52.970 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2774000000000001e-05 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:192773 48.a 48:588b0d3d:::benchmark_data_smithi119_267254_object69563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:45.565197+0000", 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 302.21815042700001, 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21976826799999999, 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:45:52.971 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192773 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565197+0000", 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565197+0000", 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.972 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565199+0000", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7439999999999999e-06 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565206+0000", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9369999999999998e-06 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565223+0000", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7156e-05 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.973 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565232+0000", 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4149999999999999e-06 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565285+0000", 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3567999999999999e-05 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565378+0000", 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3068999999999999e-05 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.974 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.565458+0000", 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9986000000000003e-05 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.568501+0000", 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030421689999999999 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.568880+0000", 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037897199999999999 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.975 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.784870+0000", 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21599010399999999 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.784922+0000", 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2738000000000002e-05 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.784966+0000", 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3409999999999999e-05 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.976 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57594 49.8 49:127af930:::benchmark_data_smithi119_301770_object57593:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.377284+0000", 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 170.40606349399999, 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29929668399999998, 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:45:52.977 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57594 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377284+0000", 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377284+0000", 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.978 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377286+0000", 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9939999999999999e-06 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377308+0000", 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1812000000000001e-05 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377320+0000", 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2035000000000001e-05 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.979 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377327+0000", 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5459999999999997e-06 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377384+0000", 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7608999999999997e-05 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377514+0000", 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012998600000000001 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.980 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377640+0000", 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012542299999999999 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.379727+0000", 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002087499 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.379825+0000", 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7481999999999999e-05 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.676482+0000", 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29665668499999998 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.676535+0000", 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3383e-05 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.676581+0000", 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6230000000000003e-05 2024-09-21T10:45:52.982 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:119204 49.5 e1408/1394 49:af12f416:::benchmark_data_smithi119_301770_object5717:head v 1408'7463, mlcod=1408'7459)", 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.389718+0000", 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.393629739, 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21225497099999999, 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.983 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.389718+0000", 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.389718+0000", 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.389719+0000", 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2600000000000002e-07 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.984 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.389724+0000", 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0810000000000001e-06 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.389732+0000", 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.892e-06 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.389738+0000", 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0079999999999999e-06 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.985 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397331+0000", 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007592807 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397350+0000", 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9562999999999999e-05 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601921+0000", 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20457050800000001 2024-09-21T10:45:52.986 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601973+0000", 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2386000000000003e-05 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:119207 49.5 e1408/1394 49:a6a0aa42:::benchmark_data_smithi119_301770_object5720:head v 1408'7464, mlcod=1408'7460)", 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.397239+0000", 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.386108673, 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21242439299999999, 2024-09-21T10:45:52.987 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397239+0000", 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397239+0000", 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.988 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397240+0000", 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4799999999999998e-07 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397248+0000", 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.1310000000000006e-06 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397259+0000", 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1008e-05 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.989 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397267+0000", 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4649999999999997e-06 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602043+0000", 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20477610800000001 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602061+0000", 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8201000000000001e-05 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.990 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.609626+0000", 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0075652209999999996 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.609664+0000", 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7611000000000003e-05 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.991 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:119210 49.2 49:44a24368:::benchmark_data_smithi119_301770_object5723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.397822+0000", 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.385526015, 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20443257400000001, 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 119210 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.992 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397822+0000", 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397822+0000", 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397823+0000", 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.31e-06 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.993 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397828+0000", 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6090000000000003e-06 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397839+0000", 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1079e-05 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397851+0000", 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2404e-05 2024-09-21T10:45:52.994 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.397903+0000", 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1362000000000001e-05 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.398008+0000", 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010511800000000001 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.398101+0000", 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3387999999999994e-05 2024-09-21T10:45:52.995 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601995+0000", 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20389353399999999 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602008+0000", 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3351e-05 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602162+0000", 2024-09-21T10:45:52.996 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015358099999999999 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602209+0000", 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7854e-05 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602254+0000", 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4984000000000002e-05 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:52.997 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:119213 49.8 49:1c003b76:::benchmark_data_smithi119_301770_object5726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.399824+0000", 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.383523982, 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20627941999999999, 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 119213 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:52.998 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.399824+0000", 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.399824+0000", 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.399825+0000", 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2020000000000001e-06 2024-09-21T10:45:52.999 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.399829+0000", 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0099999999999997e-06 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.399841+0000", 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1815e-05 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.399854+0000", 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2726e-05 2024-09-21T10:45:53.000 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400006+0000", 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015229499999999999 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400116+0000", 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000109515 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,7", 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400221+0000", 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000105426 2024-09-21T10:45:53.001 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.605894+0000", 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20567339800000001 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.605934+0000", 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9746000000000002e-05 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:53.002 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.605944+0000", 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0178e-05 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.606054+0000", 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010953199999999999 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.606103+0000", 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9577000000000003e-05 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.003 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:119211 49.3 e1408/1394 49:c13490aa:::benchmark_data_smithi119_301770_object5724:head v 1408'7495, mlcod=1408'7493)", 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.400858+0000", 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.38248974699999, 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21654564200000001, 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400858+0000", 2024-09-21T10:45:53.004 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400858+0000", 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400859+0000", 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9600000000000008e-07 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.005 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400864+0000", 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1059999999999999e-06 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601900+0000", 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20103548199999999 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601915+0000", 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.535e-05 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.006 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602112+0000", 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019724200000000001 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602132+0000", 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9539999999999999e-05 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.617365+0000", 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.015232823 2024-09-21T10:45:53.007 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.617404+0000", 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9103000000000002e-05 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36978.0:119210 49.2 e1408/1394 ondisk, result = 0)", 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.401338+0000", 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.38201028899999, 2024-09-21T10:45:53.008 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20067122100000001, 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.401338+0000", 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.401338+0000", 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.009 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.401339+0000", 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.093e-06 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.401345+0000", 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9730000000000002e-06 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601895+0000", 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20055074000000001 2024-09-21T10:45:53.010 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601919+0000", 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3277000000000001e-05 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601974+0000", 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5674999999999998e-05 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601989+0000", 2024-09-21T10:45:53.011 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4205999999999999e-05 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602009+0000", 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0256999999999999e-05 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:119209 49.e e1408/1394 49:79e08607:::benchmark_data_smithi119_301770_object5722:head v 1408'7543, mlcod=1408'7541)", 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.404395+0000", 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 104.378953407, 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20506551000000001, 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.012 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.404395+0000", 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.404395+0000", 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.013 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.404396+0000", 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0300000000000001e-06 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.404434+0000", 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8578999999999998e-05 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601895+0000", 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.197461255 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.014 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.601921+0000", 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5304e-05 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602024+0000", 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000103198 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.602046+0000", 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.215e-05 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.015 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.609410+0000", 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007363624 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.609460+0000", 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0370000000000001e-05 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.016 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160767 49.2 49:4973d847:::benchmark_data_smithi119_301770_object47280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.469258+0000", 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 57.314089449000001, 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25982260400000001, 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160767 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.017 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469258+0000", 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469258+0000", 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469260+0000", 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.401e-06 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.018 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469265+0000", 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9389999999999996e-06 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469277+0000", 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.219e-05 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469284+0000", 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1770000000000001e-06 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469339+0000", 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4814000000000001e-05 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469453+0000", 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114369 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469577+0000", 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012403100000000001 2024-09-21T10:45:53.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.477258+0000", 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0076802609999999999 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.480090+0000", 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028321850000000001 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.728961+0000", 2024-09-21T10:45:53.021 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24887094500000001 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.729032+0000", 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1576999999999996e-05 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.729081+0000", 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8714999999999997e-05 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160774 49.2 49:4266860a:::benchmark_data_smithi119_301770_object47287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.473065+0000", 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 57.310283353999999, 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25613724300000001, 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160774 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473065+0000", 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473065+0000", 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473066+0000", 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.359e-06 2024-09-21T10:45:53.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473071+0000", 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4349999999999996e-06 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473084+0000", 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2775999999999999e-05 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473091+0000", 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1269999999999997e-06 2024-09-21T10:45:53.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473403+0000", 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00031151399999999999 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473507+0000", 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000104481 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.473616+0000", 2024-09-21T10:45:53.026 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010842500000000001 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.480101+0000", 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064856480000000001 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.480120+0000", 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9046999999999999e-05 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.027 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.729124+0000", 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.249003274 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.729167+0000", 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2848e-05 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.729202+0000", 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5308999999999997e-05 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160776 49.2 49:4547040d:::benchmark_data_smithi119_301770_object47289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.476672+0000", 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 57.306675923999997, 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25649928999999999, 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160776 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.029 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.476672+0000", 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.476672+0000", 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.476673+0000", 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2500000000000001e-06 2024-09-21T10:45:53.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.476679+0000", 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3009999999999997e-06 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.476691+0000", 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2819999999999999e-05 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.476699+0000", 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.9759999999999995e-06 2024-09-21T10:45:53.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.477281+0000", 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00058180000000000005 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.477385+0000", 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010358599999999999 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.477470+0000", 2024-09-21T10:45:53.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5117999999999994e-05 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.480201+0000", 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027312899999999999 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.480510+0000", 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000308592 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.733066+0000", 2024-09-21T10:45:53.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25255662099999998 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.733123+0000", 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6548000000000001e-05 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.733171+0000", 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8387999999999998e-05 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:179439 49.e e1412/1394 49:73d1760a:::benchmark_data_smithi119_301770_object65952:head v 1412'11463, mlcod=1412'11461)", 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:45:11.859238+0000", 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 35.924109858999998, 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.163833162, 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859238+0000", 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859238+0000", 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859239+0000", 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1100000000000004e-07 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859246+0000", 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.888e-06 2024-09-21T10:45:53.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859259+0000", 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3081999999999999e-05 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859266+0000", 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0369999999999997e-06 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859408+0000", 2024-09-21T10:45:53.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014243 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:11.859428+0000", 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9826e-05 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:12.023018+0000", 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.163589701 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:45:53.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:45:12.023071+0000", 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3287e-05 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:45:53.039 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:45:53.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:53.066 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:53.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:53.198 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:53.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:53.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:53.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:53.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:53.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:53.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:53.709 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:53.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:53.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:53 smithi145 ceph-mon[89359]: pgmap v2494: 109 pgs: 109 active+clean; 892 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:45:53.825 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:53 smithi119 ceph-mon[101432]: pgmap v2494: 109 pgs: 109 active+clean; 892 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:45:53.825 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:53 smithi119 ceph-mon[105288]: pgmap v2494: 109 pgs: 109 active+clean; 892 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:45:53.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:53.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:53.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:53.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:54.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:54.107 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:54.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:54.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.281 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:54.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:54.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:54.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:54.510 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:54.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:54.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:54.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:45:54.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:54.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:54.801 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:45:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:45:54] "GET /metrics HTTP/1.1" 200 36862 "" "Prometheus/2.51.0" 2024-09-21T10:45:54.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:54.841 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:54.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:54.942 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:55.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:55.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:55.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:55.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:55.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:55.312 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:55.328 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:55.346 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:45:55.346 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:45:55.385 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:45:55.386 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:45:55.416 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:55.421 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:45:55.422 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:45:55.439 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:55.454 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:45:55.455 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:45:55.484 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:45:55.484 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:45:55.513 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:45:55.513 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:45:55.541 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:45:55.541 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:45:55.541 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:55.564 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:55.578 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:45:55.579 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:45:55.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:55.691 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:55.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:55 smithi119 ceph-mon[105288]: pgmap v2495: 109 pgs: 109 active+clean; 773 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 887 op/s 2024-09-21T10:45:55.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1235652696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:55.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:55 smithi119 ceph-mon[101432]: pgmap v2495: 109 pgs: 109 active+clean; 773 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 887 op/s 2024-09-21T10:45:55.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1235652696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:55.793 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:55 smithi145 ceph-mon[89359]: pgmap v2495: 109 pgs: 109 active+clean; 773 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 887 op/s 2024-09-21T10:45:55.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1235652696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:45:55.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:55.921 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:55.944 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:55.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:55.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:45:56.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:56.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:56.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:56.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:56.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:56.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:56.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:56.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:56.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.650 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:56.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:56.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:56.788 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:56.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:56.929 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:56.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:57.031 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:57.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:57.061 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:57.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:57.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:57.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:57.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:57.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:57.479 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:57.581 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:57.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:57.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:57 smithi119 ceph-mon[101432]: pgmap v2496: 109 pgs: 109 active+clean; 679 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-21T10:45:57.647 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:57 smithi119 ceph-mon[105288]: pgmap v2496: 109 pgs: 109 active+clean; 679 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-21T10:45:57.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:57.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:57.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:57 smithi145 ceph-mon[89359]: pgmap v2496: 109 pgs: 109 active+clean; 679 MiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 896 op/s 2024-09-21T10:45:57.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:57.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:57.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:58.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.046 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:58.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:58.147 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:58.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:58.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:58.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:58.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:58.399 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:58.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:58.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:58.546 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:58.641 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:45:58.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:45:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.696 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:58.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:45:58.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:45:58.821 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:45:58.922 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:58.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:58.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:59.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:45:59.059 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:59.062 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:58 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:45:58.846+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1415 definitely_dead 0 2024-09-21T10:45:59.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:59.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:59.227 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:45:59.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:45:59.352 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:45:59.402 INFO:teuthology.orchestra.run.smithi119.stderr:marked out osd.7. 2024-09-21T10:45:59.454 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:45:59.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:59.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:45:59.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:45:59.598 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:45:59.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:45:59.716 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[105288]: pgmap v2497: 109 pgs: 109 active+clean; 614 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 765 op/s 2024-09-21T10:45:59.716 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/342081742' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:45:59.716 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:45:59.716 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[105288]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:45:59.717 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[101432]: pgmap v2497: 109 pgs: 109 active+clean; 614 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 765 op/s 2024-09-21T10:45:59.717 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/342081742' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:45:59.717 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:45:59.717 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:45:59 smithi119 ceph-mon[101432]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:45:59.723 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:45:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:59 smithi145 ceph-mon[89359]: pgmap v2497: 109 pgs: 109 active+clean; 614 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 765 op/s 2024-09-21T10:45:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/342081742' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:45:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:59 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["7"]} : dispatch 2024-09-21T10:45:59.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:45:59 smithi145 ceph-mon[89359]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-21T10:45:59.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:45:59.846 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:45:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:45:59.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:45:59.969 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:45:59.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:45:59.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:00.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:00.092 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:00.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:00.194 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:00.215 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:00.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:00.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:00.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:00.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:00.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:00 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:46:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:00 smithi145 ceph-mon[89359]: osdmap e1416: 8 total, 8 up, 6 in 2024-09-21T10:46:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:00 smithi145 ceph-mon[89359]: osdmap e1417: 8 total, 8 up, 6 in 2024-09-21T10:46:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:46:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:46:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:46:00.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:00.585 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:00 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:46:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:00 smithi119 ceph-mon[105288]: osdmap e1416: 8 total, 8 up, 6 in 2024-09-21T10:46:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:00 smithi119 ceph-mon[105288]: osdmap e1417: 8 total, 8 up, 6 in 2024-09-21T10:46:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:00 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-21T10:46:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:00 smithi119 ceph-mon[101432]: osdmap e1416: 8 total, 8 up, 6 in 2024-09-21T10:46:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:00 smithi119 ceph-mon[101432]: osdmap e1417: 8 total, 8 up, 6 in 2024-09-21T10:46:00.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:00.714 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:00.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:00.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:00.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:00.972 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:00.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:01.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:01.113 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:01.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:01.251 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:01.353 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:01.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:01.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:01.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:01.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:01.635 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:01.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:01.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: pgmap v2500: 109 pgs: 109 active+clean; 452 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:46:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: 1.d scrub starts 2024-09-21T10:46:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: 1.d scrub ok 2024-09-21T10:46:01.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: 49.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1415'13606 2024-09-21T10:46:01.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: 49.5 starting backfill to osd.6 from (0'0,0'0] MAX to 1415'13829 2024-09-21T10:46:01.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: osdmap e1418: 8 total, 8 up, 6 in 2024-09-21T10:46:01.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: 49.a starting backfill to osd.3 from (0'0,0'0] MAX to 1415'13651 2024-09-21T10:46:01.795 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:01 smithi145 ceph-mon[89359]: 49.9 starting backfill to osd.0 from (0'0,0'0] MAX to 1415'13607 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: pgmap v2500: 109 pgs: 109 active+clean; 452 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: 1.d scrub starts 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: 1.d scrub ok 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: 49.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1415'13606 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: 49.5 starting backfill to osd.6 from (0'0,0'0] MAX to 1415'13829 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: osdmap e1418: 8 total, 8 up, 6 in 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: 49.a starting backfill to osd.3 from (0'0,0'0] MAX to 1415'13651 2024-09-21T10:46:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[105288]: 49.9 starting backfill to osd.0 from (0'0,0'0] MAX to 1415'13607 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: pgmap v2500: 109 pgs: 109 active+clean; 452 MiB data, 6.4 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 879 op/s 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: 1.d scrub starts 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: 1.d scrub ok 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: 49.8 starting backfill to osd.6 from (0'0,0'0] MAX to 1415'13606 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: 49.5 starting backfill to osd.6 from (0'0,0'0] MAX to 1415'13829 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: osdmap e1418: 8 total, 8 up, 6 in 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: 49.a starting backfill to osd.3 from (0'0,0'0] MAX to 1415'13651 2024-09-21T10:46:01.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:01 smithi119 ceph-mon[101432]: 49.9 starting backfill to osd.0 from (0'0,0'0] MAX to 1415'13607 2024-09-21T10:46:01.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:01.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:01.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:02.005 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:02.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:02.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:02.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:02.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:02.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.310 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:02.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:02.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:02.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:02.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:02.600 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:02.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:02.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:02.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:02.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:02.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:02.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:02.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:03.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:03.115 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:03.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:03.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:03.258 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:03.258 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:03.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:03.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:03.406 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:03.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:03.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:03 smithi119 ceph-mon[101432]: pgmap v2502: 109 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 106 active+clean; 373 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 306/17943 objects misplaced (1.705%); 235 KiB/s, 3 objects/s recovering 2024-09-21T10:46:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:03 smithi119 ceph-mon[105288]: pgmap v2502: 109 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 106 active+clean; 373 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 306/17943 objects misplaced (1.705%); 235 KiB/s, 3 objects/s recovering 2024-09-21T10:46:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:03.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:03.685 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:03.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:03.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:03 smithi145 ceph-mon[89359]: pgmap v2502: 109 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 106 active+clean; 373 MiB data, 6.2 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 816 op/s; 306/17943 objects misplaced (1.705%); 235 KiB/s, 3 objects/s recovering 2024-09-21T10:46:03.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:03.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:03.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:03.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:03.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:04.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:04.140 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:04.242 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:04.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:04.367 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:04.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:04.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:04.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:04.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:04.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:46:04.617 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:04.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:04.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:04.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:04.785 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:46:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:46:04] "GET /metrics HTTP/1.1" 200 36874 "" "Prometheus/2.51.0" 2024-09-21T10:46:04.793 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:04.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:04.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:04.928 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:04.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:05.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:05.051 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:05.051 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:05.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:05.174 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:05.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:05.297 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:05.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:05.419 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:05.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:05.542 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:05.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[105288]: pgmap v2503: 109 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 105 active+clean; 321 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 407/15447 objects misplaced (2.635%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[105288]: osdmap e1419: 8 total, 8 up, 6 in 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[105288]: 1.9 scrub starts 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[105288]: 1.9 scrub ok 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[105288]: osdmap e1420: 8 total, 8 up, 6 in 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2979188057' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[101432]: pgmap v2503: 109 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 105 active+clean; 321 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 407/15447 objects misplaced (2.635%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[101432]: osdmap e1419: 8 total, 8 up, 6 in 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[101432]: 1.9 scrub starts 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[101432]: 1.9 scrub ok 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[101432]: osdmap e1420: 8 total, 8 up, 6 in 2024-09-21T10:46:05.588 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:05 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2979188057' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:05.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:05.666 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:05.668 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:46:05.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:05 smithi145 ceph-mon[89359]: pgmap v2503: 109 pgs: 2 active+remapped+backfilling, 2 remapped+peering, 105 active+clean; 321 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 0 B/s wr, 783 op/s; 407/15447 objects misplaced (2.635%); 2.4 MiB/s, 38 objects/s recovering 2024-09-21T10:46:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:05 smithi145 ceph-mon[89359]: osdmap e1419: 8 total, 8 up, 6 in 2024-09-21T10:46:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:05 smithi145 ceph-mon[89359]: 1.9 scrub starts 2024-09-21T10:46:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:05 smithi145 ceph-mon[89359]: 1.9 scrub ok 2024-09-21T10:46:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:05 smithi145 ceph-mon[89359]: osdmap e1420: 8 total, 8 up, 6 in 2024-09-21T10:46:05.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:05 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2979188057' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:05.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:05.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:05.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:05.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:05.998 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:06.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:06.089 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:06.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:06.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.230 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:06.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:06.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:06.363 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:06.465 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:06.505 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:06.607 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:06.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:06.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:06.734 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[105288]: 1.4 scrub starts 2024-09-21T10:46:06.734 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[105288]: 1.4 scrub ok 2024-09-21T10:46:06.734 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2979188057' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[105288]: osdmap e1421: 8 total, 8 up, 6 in 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[101432]: 1.4 scrub starts 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[101432]: 1.4 scrub ok 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2979188057' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:06.735 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:06 smithi119 ceph-mon[101432]: osdmap e1421: 8 total, 8 up, 6 in 2024-09-21T10:46:06.777 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:06 smithi145 ceph-mon[89359]: 1.4 scrub starts 2024-09-21T10:46:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:06 smithi145 ceph-mon[89359]: 1.4 scrub ok 2024-09-21T10:46:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:06 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:06 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:46:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2979188057' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:06.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:06 smithi145 ceph-mon[89359]: osdmap e1421: 8 total, 8 up, 6 in 2024-09-21T10:46:06.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:06.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:06.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:07.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:07.042 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:07.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:07.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:07.267 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:07.290 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:07.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:07.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:07.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:07.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:07.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:07.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:07.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:07.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:07 smithi145 ceph-mon[89359]: pgmap v2507: 109 pgs: 1 peering, 2 active+remapped+backfilling, 106 active+clean; 249 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 19 KiB/s rd, 42 KiB/s wr, 536 op/s; 407/12003 objects misplaced (3.391%); 8.3 MiB/s, 133 objects/s recovering 2024-09-21T10:46:07.798 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:07 smithi119 ceph-mon[105288]: pgmap v2507: 109 pgs: 1 peering, 2 active+remapped+backfilling, 106 active+clean; 249 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 19 KiB/s rd, 42 KiB/s wr, 536 op/s; 407/12003 objects misplaced (3.391%); 8.3 MiB/s, 133 objects/s recovering 2024-09-21T10:46:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:07 smithi119 ceph-mon[101432]: pgmap v2507: 109 pgs: 1 peering, 2 active+remapped+backfilling, 106 active+clean; 249 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 19 KiB/s rd, 42 KiB/s wr, 536 op/s; 407/12003 objects misplaced (3.391%); 8.3 MiB/s, 133 objects/s recovering 2024-09-21T10:46:07.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:07.923 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:07.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:07.977 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 5] out_osds: [4, 7] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:46:07.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:46:08.024 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:08.069 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:08.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:08.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:08.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:08.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:08.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:08.450 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Removed 113484 objects 2024-09-21T10:46:08.450 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Clean up completed and total clean up time :132.246 2024-09-21T10:46:08.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:08.512 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:08.613 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:08.635 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:08.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:08.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:08.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:08.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:08.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:08.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:09.004 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:09.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:09.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:09.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:09.250 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:09.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:09.352 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:09.374 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:09.389 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:46:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:09.407 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:46:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:09.424 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:46:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:09.440 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:46:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:09.457 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:46:09.457 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:09.472 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:46:09.473 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:09.475 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:09.488 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:46:09.488 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:09.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:09.504 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:46:09.505 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:09.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:09.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[105288]: pgmap v2508: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 182 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 519 op/s; 101/8775 objects misplaced (1.151%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[105288]: 1.e scrub starts 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[105288]: 1.e scrub ok 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/630817441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[101432]: pgmap v2508: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 182 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 519 op/s; 101/8775 objects misplaced (1.151%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[101432]: 1.e scrub starts 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[101432]: 1.e scrub ok 2024-09-21T10:46:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/630817441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:09.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:09.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:09.742 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:09 smithi145 ceph-mon[89359]: pgmap v2508: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 182 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 17 KiB/s rd, 37 KiB/s wr, 519 op/s; 101/8775 objects misplaced (1.151%); 9.8 MiB/s, 157 objects/s recovering 2024-09-21T10:46:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:09 smithi145 ceph-mon[89359]: 1.e scrub starts 2024-09-21T10:46:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:09 smithi145 ceph-mon[89359]: 1.e scrub ok 2024-09-21T10:46:09.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/630817441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:09.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:09.867 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:09.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:09.969 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:09.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:09.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:10.068 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:10.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:10.116 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:10.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:10.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:10.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:10.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:10.484 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:46:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:46:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:46:10.585 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:10.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:10.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:46:10.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:10.768 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:10.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:10.880 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-21T10:46:10.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-21T10:46:10.896 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:10.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:10.997 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:11.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:11.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:11.145 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:11.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:11.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:11.369 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:11.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:11.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:11.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:11.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:11 smithi119 ceph-mon[105288]: pgmap v2509: 109 pgs: 1 peering, 108 active+clean; 75 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 654 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-09-21T10:46:11.573 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:11 smithi119 ceph-mon[101432]: pgmap v2509: 109 pgs: 1 peering, 108 active+clean; 75 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 654 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-09-21T10:46:11.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:11.636 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:11.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:11.759 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:11.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:11 smithi145 ceph-mon[89359]: pgmap v2509: 109 pgs: 1 peering, 108 active+clean; 75 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 654 op/s; 8.5 MiB/s, 136 objects/s recovering 2024-09-21T10:46:11.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:11.881 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:11.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:11.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:12.009 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:12.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:12.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:12.234 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:12.259 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:12.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:12.386 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:12.488 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:12.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:12.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:46:12.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:12.640 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:12.742 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:12.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:12.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:12.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:12.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:13.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:13.035 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:13.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:13.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:13.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:13.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:13.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:13.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:13.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:13.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:13.520 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:13 smithi119 ceph-mon[105288]: pgmap v2510: 109 pgs: 109 active+clean; 4.9 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 595 op/s; 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:46:13.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:13 smithi119 ceph-mon[101432]: pgmap v2510: 109 pgs: 109 active+clean; 4.9 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 595 op/s; 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:46:13.532 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:13.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:13.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:13.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:13.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:13.785 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:13.792 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:46:13.792 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:46:13.792 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:46:13.792 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:46:13.792 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:46:13.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:13 smithi145 ceph-mon[89359]: pgmap v2510: 109 pgs: 109 active+clean; 4.9 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 595 op/s; 6.8 MiB/s, 109 objects/s recovering 2024-09-21T10:46:13.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:13.913 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:13.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:13.959 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:14.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:14.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:14.085 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:14.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:14.262 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:14.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:14.406 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:14.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:14.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:14.541 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:46:14.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:46:14] "GET /metrics HTTP/1.1" 200 36874 "" "Prometheus/2.51.0" 2024-09-21T10:46:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4184279998' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:14.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:14.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4184279998' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:14.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:14 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:14.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:14.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.780 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:14.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:14 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4184279998' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:14.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:14 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:46:14.881 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:14.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:14.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:14.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:14.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:46:15.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:15.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:15.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:15.156 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:15.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:15.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:15.283 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:15.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:15.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:15.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:15.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:15.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:15.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:15.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:15 smithi145 ceph-mon[89359]: pgmap v2511: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 489 op/s; 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:46:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:15 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:46:15.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:15 smithi145 ceph-mon[89359]: osdmap e1422: 8 total, 8 up, 6 in 2024-09-21T10:46:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:15 smithi119 ceph-mon[105288]: pgmap v2511: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 489 op/s; 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:46:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:15 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:46:15.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:15 smithi119 ceph-mon[105288]: osdmap e1422: 8 total, 8 up, 6 in 2024-09-21T10:46:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:15 smithi119 ceph-mon[101432]: pgmap v2511: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 489 op/s; 5.5 MiB/s, 88 objects/s recovering 2024-09-21T10:46:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:15 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:46:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:15 smithi119 ceph-mon[101432]: osdmap e1422: 8 total, 8 up, 6 in 2024-09-21T10:46:15.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:15.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:15.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:16.008 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:16.034 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:16.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:16.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:16.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:16.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:16.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:16.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:16.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:16.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:16.527 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:16.629 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:16.649 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:16.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:16.772 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:16.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:16.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:16.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:16.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:17.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:17.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:17.145 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:17.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:17.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:17.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:17.369 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:17.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:17.515 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:17.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:17.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:17 smithi119 ceph-mon[105288]: pgmap v2513: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 395 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:46:17.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:46:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:17 smithi119 ceph-mon[101432]: pgmap v2513: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 395 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:46:17.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:46:17.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:17.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:17.762 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:17.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:17 smithi145 ceph-mon[89359]: pgmap v2513: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 395 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-21T10:46:17.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:46:17.864 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:17.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:17.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:17.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:18.009 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:18.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:46:18.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:18.142 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:18.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.266 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:18.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:18.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:18.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:18.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:18.511 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:18.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:18.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:18.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:18.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:18.759 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:46:18.759 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:18.776 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:46:18.776 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:18.792 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:46:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:18.809 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:46:18.809 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:18.825 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:46:18.825 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:18.840 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:46:18.840 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:18.856 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:46:18.856 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:18.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:18.875 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:46:18.875 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:18.880 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:18.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:18.963 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[101432]: pgmap v2514: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 287 op/s; 581 KiB/s, 9 objects/s recovering 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2677400533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[105288]: pgmap v2514: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 287 op/s; 581 KiB/s, 9 objects/s recovering 2024-09-21T10:46:18.964 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2677400533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:18.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:19.002 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:19.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:19.124 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:19.226 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:19.248 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:19.257 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:19.257 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:19.257 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:18 smithi145 ceph-mon[89359]: pgmap v2514: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 287 op/s; 581 KiB/s, 9 objects/s recovering 2024-09-21T10:46:19.258 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2677400533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:19.350 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:19.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:19.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:19.497 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:19.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:19.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:19.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:19.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:19.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:19.820 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:46:19.844 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:19.865 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:19.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:19.989 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:19.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:20.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:20.112 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:19 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1286678641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:20.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:19 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:19 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1286678641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:20.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:19 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:20.119 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:19 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1286678641' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:20.119 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:19 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:46:20.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:20.235 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:20.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:20.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:20.373 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:46:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:46:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:46:20.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:20.481 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:20.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:20.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:20.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:20.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:20.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:20.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:20.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:20.859 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:20.859 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:46:20.859 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[105288]: osdmap e1423: 8 total, 8 up, 6 in 2024-09-21T10:46:20.860 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[105288]: pgmap v2516: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 8 op/s 2024-09-21T10:46:20.860 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:20.860 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:46:20.860 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[101432]: osdmap e1423: 8 total, 8 up, 6 in 2024-09-21T10:46:20.860 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:20 smithi119 ceph-mon[101432]: pgmap v2516: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 8 op/s 2024-09-21T10:46:20.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:20.974 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:20.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:21.076 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:21.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:21.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:21.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:20 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:20 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:46:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:20 smithi145 ceph-mon[89359]: osdmap e1423: 8 total, 8 up, 6 in 2024-09-21T10:46:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:20 smithi145 ceph-mon[89359]: pgmap v2516: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 8 op/s 2024-09-21T10:46:21.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:21.343 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:21.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.468 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:21.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:21.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:21.590 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:21.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:21.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:21.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.838 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:21.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:21.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:21.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:21.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:21.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:22.062 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:22.084 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:22.186 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:22.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:22.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:22.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:22.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:22.456 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:22.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:22.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:22.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:22.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:22.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:22.699 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:22.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:22.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:22.805 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:22.827 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:22.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:46:22.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:22.950 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:22.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:22.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:23.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:23.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.074 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:23.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:23.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:23.299 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:23.320 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:23.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:23.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:23.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:23 smithi119 ceph-mon[105288]: pgmap v2517: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:23.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:23.470 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:23.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:23 smithi119 ceph-mon[101432]: pgmap v2517: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:23.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:23.470 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:23 smithi145 ceph-mon[89359]: pgmap v2517: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:23.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:23.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:23.698 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:23.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:46:23.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:23.825 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:23.926 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:23.948 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:23.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:24.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:24.075 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:24.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:24.202 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:24.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:24.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:24.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:24.455 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:24.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:24.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:24.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:24.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:24.802 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:24.824 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:24.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:46:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:46:24] "GET /metrics HTTP/1.1" 200 36869 "" "Prometheus/2.51.0" 2024-09-21T10:46:24.926 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:24.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:24.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:25.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:25.070 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:25.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:25.194 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:25.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:25.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:25 smithi119 ceph-mon[105288]: pgmap v2518: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:25 smithi119 ceph-mon[101432]: pgmap v2518: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:25.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:25.441 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:25.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:25 smithi145 ceph-mon[89359]: pgmap v2518: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:25.564 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:25.666 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:25.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:25.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:25.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:25.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:25.938 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:25.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:26.023 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:26.023 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:26.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:26.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:26.185 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:26.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:26.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1031960720' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:46:26.193 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_4' removed 2024-09-21T10:46:26.286 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:26.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:26.411 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:26.433 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:26.529 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1031960720' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:46:26.529 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:26 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:46:26.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:26 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:46:26.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1031960720' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:46:26.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:26 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:46:26.558 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:26.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:26.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:26.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:26.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:26.847 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-21T10:46:26.848 INFO:tasks.radosbench:Beginning radosbench... 2024-09-21T10:46:26.848 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-21T10:46:26.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_5 16 2024-09-21T10:46:26.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:26.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:26.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:27.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:27.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:27.169 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:27.178 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:27.190 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:27.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:27.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:27.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:27.414 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:27.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:27.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:27 smithi119 ceph-mon[105288]: pgmap v2519: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:27.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:27 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:46:27.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:27 smithi119 ceph-mon[105288]: osdmap e1424: 8 total, 8 up, 6 in 2024-09-21T10:46:27.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:27 smithi119 ceph-mon[101432]: pgmap v2519: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:27.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:27 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:46:27.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:27 smithi119 ceph-mon[101432]: osdmap e1424: 8 total, 8 up, 6 in 2024-09-21T10:46:27.537 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:27 smithi145 ceph-mon[89359]: pgmap v2519: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:27 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:46:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:27 smithi145 ceph-mon[89359]: osdmap e1424: 8 total, 8 up, 6 in 2024-09-21T10:46:27.559 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:27.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:27.681 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:27.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:27.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:27.902 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:46:27.906 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:27.926 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:27.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:28.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:28.050 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:28.151 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:28.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:28.177 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:28.254 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:28.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:28.300 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:28.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:28.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:28.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:28.548 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:28.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:28.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:28.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:28.794 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:28.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:28.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:28.917 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:28.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:29.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:29.040 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:29.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:29.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:29.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:29.196 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:46:29.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:29.286 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:29.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:29 smithi145 ceph-mon[89359]: pgmap v2521: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:29.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3253390031' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:29.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:29 smithi119 ceph-mon[105288]: pgmap v2521: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:29.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3253390031' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:29.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:29 smithi119 ceph-mon[101432]: pgmap v2521: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:29.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3253390031' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:46:29.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:29.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:29.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:29.528 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:29.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:29.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:29.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:29.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:29.780 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:29.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:29.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:29.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:29.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:29.959 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:29.959 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:29.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:46:30.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:30.027 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:30.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:30.149 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:30.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:30.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:30.273 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3253390031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:46:30.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:30 smithi119 ceph-mon[101432]: osdmap e1425: 8 total, 8 up, 6 in 2024-09-21T10:46:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3253390031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:46:30.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:30 smithi119 ceph-mon[105288]: osdmap e1425: 8 total, 8 up, 6 in 2024-09-21T10:46:30.374 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:30.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:30.487 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3253390031' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:46:30.487 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:30 smithi145 ceph-mon[89359]: osdmap e1425: 8 total, 8 up, 6 in 2024-09-21T10:46:30.487 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:46:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:46:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:46:30.497 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:30.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:46:30.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.519 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:30.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:30.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:30.642 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:30.743 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:30.766 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.767 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:30.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:30.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:30.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:30.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:30.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:31.012 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:31.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:31.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:31.237 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:31.259 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:31.330 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:31 smithi119 ceph-mon[105288]: pgmap v2523: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:31.330 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:31 smithi119 ceph-mon[105288]: osdmap e1426: 8 total, 8 up, 6 in 2024-09-21T10:46:31.330 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:31 smithi119 ceph-mon[101432]: pgmap v2523: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:31.330 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:31 smithi119 ceph-mon[101432]: osdmap e1426: 8 total, 8 up, 6 in 2024-09-21T10:46:31.360 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:31.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:31.450 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:31 smithi145 ceph-mon[89359]: pgmap v2523: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:31.450 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:31 smithi145 ceph-mon[89359]: osdmap e1426: 8 total, 8 up, 6 in 2024-09-21T10:46:31.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:31.504 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:31.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:31.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:31.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:31.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:31.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:31.874 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:31.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:31.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:31.999 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:31.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:32.082 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:32.100 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:32.122 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:32.216 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:46:32.224 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:32.247 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:32.348 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:32.370 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:32.464 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:32.465 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:32.465 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/153713954' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:32.465 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:32 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:32.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:32.494 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:32.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/153713954' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:32.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:32 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:32.536 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/153713954' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:32.537 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:32 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:32.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:32.617 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:32.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:32.741 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:32.842 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:32.864 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:32.966 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:32.988 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:32.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:33.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:33.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:33.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:46:33.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:33.256 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:33.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:33.378 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:33.389 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:33.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[105288]: pgmap v2525: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:33.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:33.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:33.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[105288]: osdmap e1427: 8 total, 8 up, 6 in 2024-09-21T10:46:33.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[101432]: pgmap v2525: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[101432]: osdmap e1427: 8 total, 8 up, 6 in 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:33.458 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:33.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:33.502 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:33.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:33 smithi145 ceph-mon[89359]: pgmap v2525: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:33 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:33 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:33 smithi145 ceph-mon[89359]: osdmap e1427: 8 total, 8 up, 6 in 2024-09-21T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:33.510 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:33.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:33.626 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:33.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:33.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:33.748 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:33.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:33.871 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:33.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:33.993 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:33.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:34.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:34.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:34.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:34.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:34.343 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:34.364 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:34.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:34.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:34.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:34.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:34.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:34.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:34.818 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:46:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:46:34] "GET /metrics HTTP/1.1" 200 34440 "" "Prometheus/2.51.0" 2024-09-21T10:46:34.835 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:46:34.835 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:34.835 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:34.852 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:46:34.852 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:34.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:34.868 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:46:34.868 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:34.884 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:46:34.884 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:34.899 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:46:34.899 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:34.914 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:46:34.915 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:34.930 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:46:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:34.946 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:46:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:34.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:34.980 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:34.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:34.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:35.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:35.104 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:35.113 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:46:35.113 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:46:35.113 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:46:35.113 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227710 47.11 47:8e5c4614:::benchmark_data_smithi119_238957_object102881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.485865+0000", 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.61873878899996, 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.96921146700000005, 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227710 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.114 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485865+0000", 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485865+0000", 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485863+0000", 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.115 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485870+0000", 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.81e-06 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485883+0000", 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2697999999999999e-05 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485888+0000", 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7239999999999997e-06 2024-09-21T10:46:35.116 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485930+0000", 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1780000000000003e-05 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.485938+0000", 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0949999999999996e-06 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.417081+0000", 2024-09-21T10:46:35.117 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.93114243699999999 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.417189+0000", 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108144 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.417700+0000", 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051170399999999998 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.118 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.429700+0000", 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011999434999999999 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454935+0000", 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.025234910999999999 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454980+0000", 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4999000000000001e-05 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.119 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455026+0000", 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6554999999999997e-05 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455076+0000", 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9663999999999997e-05 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227718 47.11 47:8d67ec24:::benchmark_data_smithi119_238957_object102889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.120 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.495418+0000", 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.60918500699995, 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.95983292899999995, 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227718 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495418+0000", 2024-09-21T10:46:35.121 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495418+0000", 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495417+0000", 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.122 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495424+0000", 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7700000000000001e-06 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495439+0000", 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4962e-05 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495446+0000", 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8700000000000003e-06 2024-09-21T10:46:35.123 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495497+0000", 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1347000000000002e-05 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.495509+0000", 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1649e-05 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.420588+0000", 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.92507857999999998 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.124 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.420678+0000", 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0593000000000005e-05 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.420994+0000", 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00031541799999999998 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434783+0000", 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013789321 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.125 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455131+0000", 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020347547000000001 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455175+0000", 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4443999999999997e-05 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455216+0000", 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1389e-05 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.126 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455251+0000", 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4854999999999997e-05 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227823 47.11 47:88c12679:::benchmark_data_smithi119_238957_object102994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.643965+0000", 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.46063794300005, 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.811510175, 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.127 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227823 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643965+0000", 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.128 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643965+0000", 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643964+0000", 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643971+0000", 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7649999999999996e-06 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.129 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643984+0000", 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3444e-05 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.643989+0000", 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6720000000000003e-06 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644035+0000", 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5924999999999998e-05 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.130 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644042+0000", 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2840000000000001e-06 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.423268+0000", 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.77922614199999995 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.423378+0000", 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000109663 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.131 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.423717+0000", 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00033897799999999998 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434788+0000", 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011071009999999999 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455365+0000", 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.020577357000000001 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.132 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455418+0000", 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2346999999999999e-05 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455453+0000", 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5759000000000001e-05 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455476+0000", 2024-09-21T10:46:35.133 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2135999999999999e-05 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227826 47.11 47:8d7f136b:::benchmark_data_smithi119_238957_object102997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.644137+0000", 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.46046647399999, 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.81139437000000003, 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.134 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227826 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644137+0000", 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644137+0000", 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.135 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644136+0000", 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644140+0000", 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9110000000000003e-06 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644149+0000", 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2650000000000002e-06 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.136 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644154+0000", 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8570000000000003e-06 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644186+0000", 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1538000000000002e-05 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.644190+0000", 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4839999999999998e-06 2024-09-21T10:46:35.137 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.426298+0000", 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.78210749899999998 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.426402+0000", 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000103837 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.426777+0000", 2024-09-21T10:46:35.138 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037539600000000001 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437304+0000", 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010526598999999999 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455402+0000", 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018098521999999999 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455495+0000", 2024-09-21T10:46:35.139 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3066000000000006e-05 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455514+0000", 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9185999999999999e-05 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455531+0000", 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6874000000000001e-05 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.140 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:227980 47.11 47:8fdf9918:::benchmark_data_smithi119_238957_object103151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.865973+0000", 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.23863077500005, 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.589663194, 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 227980 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.141 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865973+0000", 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865973+0000", 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865971+0000", 2024-09-21T10:46:35.142 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.865980+0000", 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.901e-06 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866001+0000", 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1821999999999999e-05 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.143 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866009+0000", 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4359999999999996e-06 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866059+0000", 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0068000000000002e-05 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.866067+0000", 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.9589999999999995e-06 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.144 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.429784+0000", 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.56371681500000004 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.429914+0000", 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012982599999999999 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.430295+0000", 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00038109299999999999 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.145 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437313+0000", 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0070180449999999997 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455552+0000", 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.018239317000000001 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455568+0000", 2024-09-21T10:46:35.146 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5642999999999999e-05 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455603+0000", 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5519999999999999e-05 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455636+0000", 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2622e-05 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.147 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228046 47.11 47:8c47e8df:::benchmark_data_smithi119_238957_object103217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.945725+0000", 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.15887792599995, 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.510127424, 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228046 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.148 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945725+0000", 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945725+0000", 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945724+0000", 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.149 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945732+0000", 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5360000000000004e-06 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945745+0000", 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3373000000000001e-05 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945751+0000", 2024-09-21T10:46:35.150 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1330000000000004e-06 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945800+0000", 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8693999999999998e-05 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.945807+0000", 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.221e-06 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.432313+0000", 2024-09-21T10:46:35.151 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.48650629699999998 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.432392+0000", 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8780000000000001e-05 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.432698+0000", 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00030580400000000002 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444325+0000", 2024-09-21T10:46:35.152 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011627513000000001 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455683+0000", 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011357544000000001 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455767+0000", 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.4588999999999999e-05 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.153 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455819+0000", 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1533999999999998e-05 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455853+0000", 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3862000000000002e-05 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.155 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228055 47.11 47:8bc14e8b:::benchmark_data_smithi119_238957_object103226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.955945+0000", 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.14865833800002, 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.50006493799999996, 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228055 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.156 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955945+0000", 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955945+0000", 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955944+0000", 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.157 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955949+0000", 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7140000000000001e-06 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955961+0000", 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1867e-05 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.158 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955965+0000", 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1520000000000002e-06 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.955997+0000", 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2515999999999998e-05 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.956002+0000", 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8590000000000004e-06 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.159 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434830+0000", 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.47882809500000001 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.434944+0000", 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011323999999999999 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.435227+0000", 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028319100000000002 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.160 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.448939+0000", 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.013712330999999999 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455729+0000", 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0067898419999999999 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455924+0000", 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019455000000000001 2024-09-21T10:46:35.161 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455976+0000", 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2771e-05 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456010+0000", 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3640999999999999e-05 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.162 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228076 47.11 47:8c7d5a4c:::benchmark_data_smithi119_238957_object103247:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.978403+0000", 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.12620067600005, 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.477718429, 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228076 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.163 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978403+0000", 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978403+0000", 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978401+0000", 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978409+0000", 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.678e-06 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978427+0000", 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8317000000000001e-05 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978433+0000", 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9540000000000001e-06 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.165 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978484+0000", 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0674000000000003e-05 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.978493+0000", 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.645e-06 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437343+0000", 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.45884937799999997 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.166 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437420+0000", 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6736000000000005e-05 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.437662+0000", 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000242773 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.448944+0000", 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011282034999999999 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.167 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.455894+0000", 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0069497150000000004 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456051+0000", 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015661000000000001 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456092+0000", 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1384000000000002e-05 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.168 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456121+0000", 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9037999999999999e-05 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228090 47.11 47:88c3266c:::benchmark_data_smithi119_238957_object103261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.993068+0000", 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.11153486600006, 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46330631799999999, 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.169 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228090 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993068+0000", 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.170 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993068+0000", 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993066+0000", 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993076+0000", 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.004e-05 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.171 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993095+0000", 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9156000000000001e-05 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993103+0000", 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8639999999999996e-06 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993163+0000", 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0134999999999999e-05 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.172 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.993172+0000", 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5050000000000007e-06 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.439707+0000", 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.44653536300000002 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.439820+0000", 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011309 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.173 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.440102+0000", 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028139300000000001 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.448949+0000", 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0088475849999999998 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456156+0000", 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0072068439999999996 2024-09-21T10:46:35.174 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456186+0000", 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9742000000000001e-05 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456227+0000", 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0849000000000001e-05 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456375+0000", 2024-09-21T10:46:35.175 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014811699999999999 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228092 47.11 47:8fbd4fb1:::benchmark_data_smithi119_238957_object103263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:58.994568+0000", 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.11003578600003, 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.46230310600000002, 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228092 2024-09-21T10:46:35.176 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994568+0000", 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994568+0000", 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.177 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994567+0000", 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994573+0000", 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2539999999999998e-06 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994591+0000", 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7785e-05 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.178 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994597+0000", 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3090000000000001e-06 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994669+0000", 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2006000000000001e-05 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:58.994677+0000", 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1610000000000003e-06 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.179 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.441995+0000", 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.44731786099999998 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.442061+0000", 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.6523e-05 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.442289+0000", 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000227982 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.180 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454345+0000", 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.012055203 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456567+0000", 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022221900000000002 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456700+0000", 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013318800000000001 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.181 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456839+0000", 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013906500000000001 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456871+0000", 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1646000000000002e-05 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:228108 47.11 47:8bc9ff00:::benchmark_data_smithi119_238957_object103279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1340)", 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:59.020543+0000", 2024-09-21T10:46:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 576.084060142, 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.43645614300000002, 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 228108 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020543+0000", 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.183 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020543+0000", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020542+0000", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020549+0000", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5179999999999998e-06 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.184 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020565+0000", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6081000000000001e-05 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020573+0000", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4379999999999998e-06 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020619+0000", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5859999999999998e-05 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for peered", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.020625+0000", 2024-09-21T10:46:35.185 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7880000000000001e-06 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444368+0000", 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.42374235599999999 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444472+0000", 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010404300000000001 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,7", 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.444786+0000", 2024-09-21T10:46:35.186 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000313849 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.454370+0000", 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0095846319999999992 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456646+0000", 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002276255 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456906+0000", 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00025980699999999997 2024-09-21T10:46:35.187 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456969+0000", 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2584999999999998e-05 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:59.456999+0000", 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0480999999999998e-05 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.188 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:83857 48.8 48:19427d55:::benchmark_data_smithi119_267254_object83856:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:48.367161+0000", 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 466.73744218600001, 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24199303899999999, 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 83857 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.189 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367161+0000", 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367161+0000", 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367160+0000", 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.190 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367213+0000", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2580000000000001e-05 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367225+0000", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1969e-05 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367229+0000", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5070000000000002e-06 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.191 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367265+0000", 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6173000000000001e-05 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367364+0000", 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8857999999999994e-05 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.367452+0000", 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8385000000000002e-05 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.192 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369785+0000", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023325249999999998 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.373459+0000", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0036735930000000002 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.609060+0000", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.235601171 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.193 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.609114+0000", 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4073000000000002e-05 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.609154+0000", 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0379000000000001e-05 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:83863 48.8 48:1204cd08:::benchmark_data_smithi119_267254_object83862:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1364)", 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:38:48.369716+0000", 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 466.73488734400001, 2024-09-21T10:46:35.194 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.246415941, 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 83863 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369716+0000", 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.195 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369716+0000", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369715+0000", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369752+0000", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7326000000000001e-05 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.196 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369767+0000", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5194000000000001e-05 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369773+0000", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2689999999999996e-06 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369811+0000", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7265999999999999e-05 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.369932+0000", 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012138000000000001 2024-09-21T10:46:35.197 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.370074+0000", 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000141395 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.372772+0000", 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0026980929999999999 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.373561+0000", 2024-09-21T10:46:35.198 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00078955099999999999 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.616041+0000", 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24247988100000001 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.616093+0000", 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2329e-05 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:38:48.616132+0000", 2024-09-21T10:46:35.199 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8593000000000001e-05 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:170883 48.9 e1382/1354 48:978671ab:::benchmark_data_smithi119_267254_object47673:head v 1382'10478, mlcod=1382'10474)", 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.699619+0000", 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 373.40498451299999, 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.27863969300000002, 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.200 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699619+0000", 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699619+0000", 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699618+0000", 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.201 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699625+0000", 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.63e-06 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699636+0000", 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.19e-05 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699642+0000", 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8209999999999998e-06 2024-09-21T10:46:35.202 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699696+0000", 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3321e-05 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.699724+0000", 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8535999999999999e-05 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978130+0000", 2024-09-21T10:46:35.203 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.27840601500000001 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978259+0000", 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012841999999999999 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:170880 48.6 e1382/1354 48:64193ea4:::benchmark_data_smithi119_267254_object47670:head v 1382'10699, mlcod=1382'10696)", 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.701982+0000", 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 373.40262092500001, 2024-09-21T10:46:35.204 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.28656793400000002, 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701982+0000", 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701982+0000", 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.205 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701981+0000", 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.701990+0000", 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5099999999999998e-06 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.206 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.702004+0000", 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4238e-05 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.702011+0000", 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7889999999999997e-06 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978180+0000", 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.27616892900000001 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.207 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978205+0000", 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5301999999999999e-05 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.988479+0000", 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010273801000000001 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.988550+0000", 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1526999999999995e-05 2024-09-21T10:46:35.208 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.46619.0:170887 48.6 e1382/1354 48:6248d39f:::benchmark_data_smithi119_267254_object47677:head v 1382'10700, mlcod=1382'10696)", 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.708468+0000", 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 373.39613490099998, 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.28266307899999998, 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.209 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708468+0000", 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708468+0000", 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708467+0000", 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.210 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.708477+0000", 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3999999999999998e-06 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978116+0000", 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.26963935999999999 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978131+0000", 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4849e-05 2024-09-21T10:46:35.211 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.981054+0000", 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002923316 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.981070+0000", 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5781e-05 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.212 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.991078+0000", 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010008217999999999 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.991132+0000", 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3386e-05 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:170930 48.8 48:1c0c3ac0:::benchmark_data_smithi119_267254_object47720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:35.214 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:21.745856+0000", 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 373.35874712700002, 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.243843488, 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 170930 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.215 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745856+0000", 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745856+0000", 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745854+0000", 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999976 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.745864+0000", 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0369e-05 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978116+0000", 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23225187799999999 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978133+0000", 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7152000000000001e-05 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978425+0000", 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029155099999999997 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978569+0000", 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014475600000000001 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.978726+0000", 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015679299999999999 2024-09-21T10:46:35.218 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989529+0000", 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.010803241 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989591+0000", 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1660000000000003e-05 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989623+0000", 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1803e-05 2024-09-21T10:46:35.219 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989660+0000", 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7376000000000003e-05 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:21.989700+0000", 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9419000000000003e-05 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:119216 49.b 49:da5068b3:::benchmark_data_smithi119_301770_object5729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:44:03.402367+0000", 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 151.70223642900001, 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.220400869, 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 119216 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402367+0000", 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402367+0000", 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402366+0000", 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.222 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402371+0000", 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4070000000000001e-06 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402385+0000", 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3759e-05 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402391+0000", 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.8119999999999996e-06 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.223 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402438+0000", 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6703999999999999e-05 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402554+0000", 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000116759 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.402677+0000", 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012275 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.224 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.407052+0000", 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004375327 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.411787+0000", 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047350179999999997 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.622673+0000", 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.210885718 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.225 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.622726+0000", 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2522000000000003e-05 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.622768+0000", 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2137e-05 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:119218 49.b 49:dcd17871:::benchmark_data_smithi119_301770_object5731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:46:35.226 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:44:03.404458+0000", 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 151.70014576400001, 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.22652684100000001, 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 119218 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.227 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.404458+0000", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.404458+0000", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.404457+0000", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.404462+0000", 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0259999999999998e-06 2024-09-21T10:46:35.228 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.404473+0000", 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1185999999999999e-05 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.404477+0000", 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2169999999999996e-06 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.405282+0000", 2024-09-21T10:46:35.229 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000805162 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.405353+0000", 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0580000000000005e-05 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.405415+0000", 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.2260000000000004e-05 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.407191+0000", 2024-09-21T10:46:35.230 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017752689999999999 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.411946+0000", 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0047554950000000002 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.630898+0000", 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.21895145499999999 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.630945+0000", 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7633999999999998e-05 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.231 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:03.630984+0000", 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9279000000000001e-05 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:179432 49.b 49:d699a3de:::benchmark_data_smithi119_301770_object65945:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1412)", 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:45:11.853414+0000", 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 83.251189074999999, 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15994454299999999, 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:35.232 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 179432 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853414+0000", 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853414+0000", 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.233 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853414+0000", 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853418+0000", 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.172e-06 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853426+0000", 2024-09-21T10:46:35.234 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0250000000000002e-06 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853430+0000", 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1150000000000004e-06 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853470+0000", 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9886999999999999e-05 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853544+0000", 2024-09-21T10:46:35.235 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.4435000000000001e-05 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.853608+0000", 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3485000000000006e-05 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.856938+0000", 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003330539 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:11.861109+0000", 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.004171035 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.236 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:12.013270+0000", 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.152160984 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:12.013318+0000", 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7346999999999999e-05 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:45:12.013359+0000", 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1232000000000001e-05 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.237 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.238 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:35.238 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:35.238 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:46:35.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:35.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:46:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:35.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:35.382 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:35.484 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:35.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:35.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:35 smithi145 ceph-mon[89359]: pgmap v2527: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:35 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2484249697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:35.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:35 smithi119 ceph-mon[105288]: pgmap v2527: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:35.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:35 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2484249697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:35.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:35 smithi119 ceph-mon[101432]: pgmap v2527: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:35.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:35 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2484249697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:35.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:35.631 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:35.732 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:35.754 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:35.855 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:35.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:35.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:35.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:35.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:36.026 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:46:36.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:36.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:36.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:36.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:36.314 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:36.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:36.376 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:36.478 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:36.499 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:36.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:36.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:36.622 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:36.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.747 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:36.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:36.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:36.869 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:36.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:36.992 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:36.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:37.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:37.115 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:37.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:37.221 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:46:37.239 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:37.342 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:37.365 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:37.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:37.485 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:37 smithi145 ceph-mon[89359]: pgmap v2528: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:37.485 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1207734642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:46:37.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:37.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:37 smithi119 ceph-mon[105288]: pgmap v2528: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:37.511 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1207734642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:46:37.511 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:37 smithi119 ceph-mon[101432]: pgmap v2528: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:37.511 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1207734642' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:46:37.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:37.611 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:37.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:37.733 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:37.810 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:37.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:46:37.835 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:37.856 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:37.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:37.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:37.981 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:37.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:37.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:38.083 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:38.088 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:46:38.088 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:46:38.088 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:46:38.088 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:46:38.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:38.208 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:38.232 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:38.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:38.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1207734642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[105288]: osdmap e1428: 8 total, 8 up, 6 in 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 2]} : dispatch 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]} : dispatch 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1207734642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[101432]: osdmap e1428: 8 total, 8 up, 6 in 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 2]} : dispatch 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-21T10:46:38.356 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:38 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]} : dispatch 2024-09-21T10:46:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:38.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:38.479 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:38.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1207734642' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:46:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:38 smithi145 ceph-mon[89359]: osdmap e1428: 8 total, 8 up, 6 in 2024-09-21T10:46:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 2]} : dispatch 2024-09-21T10:46:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]} : dispatch 2024-09-21T10:46:38.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:38 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]} : dispatch 2024-09-21T10:46:38.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:38.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:38.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:38.725 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:38.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:46:38.826 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:38.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:38.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:38.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:38.971 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:38.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:39.005 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:39.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:39.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:39.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:39.216 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:39.228 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_5' created 2024-09-21T10:46:39.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:46:39.318 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:39.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:39.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:39.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:39 smithi145 ceph-mon[89359]: pgmap v2530: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:39.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 2]}]': finished 2024-09-21T10:46:39.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-21T10:46:39.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-09-21T10:46:39.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:39 smithi145 ceph-mon[89359]: osdmap e1429: 8 total, 8 up, 6 in 2024-09-21T10:46:39.446 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/666402904' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:46:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:39.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:39.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:39.584 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[105288]: pgmap v2530: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 2]}]': finished 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[105288]: osdmap e1429: 8 total, 8 up, 6 in 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/666402904' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[101432]: pgmap v2530: 93 pgs: 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 2]}]': finished 2024-09-21T10:46:39.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [1, 2]}]': finished 2024-09-21T10:46:39.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-09-21T10:46:39.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[101432]: osdmap e1429: 8 total, 8 up, 6 in 2024-09-21T10:46:39.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/666402904' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-21T10:46:39.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:39.707 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:39.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:39.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:39.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:39.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:39.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:40.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:40.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:40.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:40.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:40.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:40.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:40.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:40.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:40.447 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:40.530 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/666402904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:46:40.531 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:40 smithi145 ceph-mon[89359]: osdmap e1430: 8 total, 8 up, 6 in 2024-09-21T10:46:40.531 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:40 smithi145 ceph-mon[89359]: osdmap e1431: 8 total, 8 up, 6 in 2024-09-21T10:46:40.531 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:46:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:46:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:46:40.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:40.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/666402904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:46:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:40 smithi119 ceph-mon[101432]: osdmap e1430: 8 total, 8 up, 6 in 2024-09-21T10:46:40.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:40 smithi119 ceph-mon[101432]: osdmap e1431: 8 total, 8 up, 6 in 2024-09-21T10:46:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/666402904' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-21T10:46:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:40 smithi119 ceph-mon[105288]: osdmap e1430: 8 total, 8 up, 6 in 2024-09-21T10:46:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:40 smithi119 ceph-mon[105288]: osdmap e1431: 8 total, 8 up, 6 in 2024-09-21T10:46:40.673 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:40.695 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:40.762 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:46:40.762 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:46:40.762 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:46:40.762 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:46:40.762 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:46:40.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:40.807 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:40.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:40.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:40.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:40.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:41.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:41.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:41.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:41.187 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:41.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-21T10:46:41.241 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:46:41.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:41.310 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:41.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:41.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:41 smithi145 ceph-mon[89359]: pgmap v2533: 109 pgs: 16 unknown, 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:41.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:41 smithi145 ceph-mon[89359]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:46:41.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3075432006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:41.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:41 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:41.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:41 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3075432006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:46:41.358 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:41 smithi145 ceph-mon[89359]: osdmap e1432: 8 total, 8 up, 6 in 2024-09-21T10:46:41.383 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:46:41.412 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:41.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:41.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:41.570 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[101432]: pgmap v2533: 109 pgs: 16 unknown, 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[101432]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3075432006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3075432006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[101432]: osdmap e1432: 8 total, 8 up, 6 in 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[105288]: pgmap v2533: 109 pgs: 16 unknown, 93 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:41.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[105288]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T10:46:41.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3075432006' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:46:41.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:46:41.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3075432006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:46:41.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:41 smithi119 ceph-mon[105288]: osdmap e1432: 8 total, 8 up, 6 in 2024-09-21T10:46:41.626 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:41.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:41.693 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:41.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:41.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:41.816 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:41.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:41.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:41.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:42.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.063 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:42.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:42.163 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:42.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:42.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:42.309 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:42.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:42.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:42.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:42.554 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:42.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:42.677 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:42.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:42.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:42.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:42.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:42.922 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:42.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:43.024 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:43.045 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:43.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:43.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:43.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:43.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:43.316 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:43.327 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:43.357 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:43 smithi145 ceph-mon[89359]: pgmap v2536: 109 pgs: 4 creating+peering, 10 unknown, 95 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:43.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:43.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:43.437 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:46:43.437 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:46:43.437 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:46:43.437 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:46:43.437 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.437 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226789 47.e 47:70d5f70c:::benchmark_data_smithi119_238957_object101960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:46:43.438 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:56.336615+0000", 2024-09-21T10:46:43.438 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 587.01044618599997, 2024-09-21T10:46:43.438 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.245457536, 2024-09-21T10:46:43.438 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.500 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.500 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226789 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336615+0000", 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336615+0000", 2024-09-21T10:46:43.501 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336613+0000", 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336625+0000", 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2061e-05 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336644+0000", 2024-09-21T10:46:43.502 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9349999999999999e-05 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336654+0000", 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7990000000000005e-06 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336704+0000", 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0090999999999999e-05 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336823+0000", 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00011842500000000001 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.503 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.336930+0000", 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010705 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.340115+0000", 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00318534 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.342634+0000", 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0025185920000000001 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.504 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.581991+0000", 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.239357232 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582029+0000", 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7951999999999998e-05 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582073+0000", 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3699000000000003e-05 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.505 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46418.0:226796 47.e 47:7b665d5d:::benchmark_data_smithi119_238957_object101967:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1338)", 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:36:56.349363+0000", 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 586.99769825999999, 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23279523299999999, 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46418", 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/755190491", 2024-09-21T10:46:43.506 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 226796 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349363+0000", 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349363+0000", 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.507 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349361+0000", 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349376+0000", 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.466e-05 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.508 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349388+0000", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.188e-05 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349393+0000", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6319999999999999e-06 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349437+0000", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4230000000000002e-05 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.509 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349563+0000", 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012641 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.349639+0000", 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5679000000000005e-05 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.353138+0000", 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0034988020000000001 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.510 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.353630+0000", 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00049189000000000004 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582107+0000", 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.22847714799999999 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582136+0000", 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9453000000000001e-05 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.511 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:36:56.582158+0000", 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1905000000000001e-05 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:176674 48.1 48:8a53089f:::benchmark_data_smithi119_267254_object53464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.111061+0000", 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 375.23600025899998, 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.20173269799999999, 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.512 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 176674 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.111061+0000", 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.111061+0000", 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.513 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.111060+0000", 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.111066+0000", 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5289999999999997e-06 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.111078+0000", 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2377e-05 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.514 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.111085+0000", 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.319e-06 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.121013+0000", 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0099282099999999998 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.121141+0000", 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012812000000000001 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.515 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.121292+0000", 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000151261 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.124375+0000", 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0030821899999999998 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.126353+0000", 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0019784400000000001 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.516 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.310937+0000", 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.18458382200000001 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.310979+0000", 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2342000000000002e-05 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.312794+0000", 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018144789999999999 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.517 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:176698 48.1 48:8273aee0:::benchmark_data_smithi119_267254_object53488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.132299+0000", 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 375.21476207000001, 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.182139731, 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 176698 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132299+0000", 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132299+0000", 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132298+0000", 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132304+0000", 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3600000000000001e-06 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132315+0000", 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0614e-05 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132320+0000", 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.4870000000000002e-06 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132370+0000", 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0126000000000001e-05 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132484+0000", 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000113891 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.132646+0000", 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016224699999999999 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.137965+0000", 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0053188970000000004 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.139549+0000", 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015833119999999999 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.312862+0000", 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.173313509 2024-09-21T10:46:43.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.312917+0000", 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5155999999999998e-05 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.314439+0000", 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015214810000000001 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:176735 48.1 48:83c0a540:::benchmark_data_smithi119_267254_object53525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.159982+0000", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 375.18707928600003, 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15556688799999999, 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 176735 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.159982+0000", 2024-09-21T10:46:43.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.159982+0000", 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.159981+0000", 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.159986+0000", 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7200000000000003e-06 2024-09-21T10:46:43.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.160001+0000", 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4688000000000001e-05 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.160009+0000", 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.0430000000000008e-06 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.160057+0000", 2024-09-21T10:46:43.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8328000000000002e-05 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.160158+0000", 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000100226 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.160304+0000", 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000146856 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.166381+0000", 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006076377 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.166393+0000", 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2666999999999999e-05 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.314485+0000", 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14809108300000001 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.314519+0000", 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4520999999999997e-05 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.315549+0000", 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0010297629999999999 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:176737 48.1 48:80fdf112:::benchmark_data_smithi119_267254_object53527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:43.529 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.163163+0000", 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 375.18389774799999, 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15324227300000001, 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 176737 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163163+0000", 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163163+0000", 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163162+0000", 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163168+0000", 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.446e-06 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163179+0000", 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.112e-05 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163185+0000", 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4999999999999996e-06 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163471+0000", 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00028596699999999998 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163564+0000", 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.2954999999999993e-05 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.163712+0000", 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00014766200000000001 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.166461+0000", 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0027490919999999999 2024-09-21T10:46:43.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.166533+0000", 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2342000000000006e-05 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.315613+0000", 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14907970600000001 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.315646+0000", 2024-09-21T10:46:43.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2586999999999998e-05 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.316406+0000", 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00076007600000000005 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:176744 48.1 48:845a721d:::benchmark_data_smithi119_267254_object53534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1382)", 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:28.172942+0000", 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 375.17411870799998, 2024-09-21T10:46:43.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15463849399999999, 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 176744 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.172942+0000", 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.172942+0000", 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.172941+0000", 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.172949+0000", 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5820000000000004e-06 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.172964+0000", 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5829e-05 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.172972+0000", 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.1579999999999999e-06 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.173012+0000", 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0822999999999999e-05 2024-09-21T10:46:43.538 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.173086+0000", 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.3559000000000005e-05 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.173182+0000", 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.6160000000000001e-05 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.175673+0000", 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0024912250000000001 2024-09-21T10:46:43.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.175838+0000", 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00016467200000000001 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.325862+0000", 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15002359200000001 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.325899+0000", 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6937999999999998e-05 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:28.327581+0000", 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001682427 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:192757 48.1 48:80eed75d:::benchmark_data_smithi119_267254_object69547:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:45.550615+0000", 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 357.79644582600002, 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23646329399999999, 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.541 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 192757 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550615+0000", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550615+0000", 2024-09-21T10:46:43.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550614+0000", 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550622+0000", 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.1659999999999994e-06 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550635+0000", 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3179999999999999e-05 2024-09-21T10:46:43.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.550642+0000", 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7939999999999997e-06 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.554584+0000", 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039418259999999998 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.554682+0000", 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7873000000000003e-05 2024-09-21T10:46:43.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.554840+0000", 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00015860700000000001 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.559307+0000", 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044664500000000003 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.566115+0000", 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0068087479999999999 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.785225+0000", 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.219110001 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.785274+0000", 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8748000000000002e-05 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.787079+0000", 2024-09-21T10:46:43.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0018045680000000001 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:192765 48.b 48:d39a2971:::benchmark_data_smithi119_267254_object69555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:45.556273+0000", 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 357.790788459, 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.230700513, 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.547 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 192765 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556273+0000", 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556273+0000", 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556271+0000", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556281+0000", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9129999999999998e-06 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556293+0000", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2085e-05 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556299+0000", 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6840000000000001e-06 2024-09-21T10:46:43.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556369+0000", 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0234999999999994e-05 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556470+0000", 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000101131 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556569+0000", 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.9246999999999994e-05 2024-09-21T10:46:43.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.563094+0000", 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065248950000000002 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.563777+0000", 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00068303600000000004 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.786891+0000", 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.22311391699999999 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.786937+0000", 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5528999999999997e-05 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.786973+0000", 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6569999999999997e-05 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.46619.0:192767 48.b 48:d18d47dd:::benchmark_data_smithi119_267254_object69557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:40:45.556315+0000", 2024-09-21T10:46:43.552 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 357.79074588600002, 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.23326511, 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.46619", 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 192767 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556315+0000", 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556315+0000", 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556315+0000", 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556317+0000", 2024-09-21T10:46:43.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6280000000000001e-06 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556323+0000", 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6250000000000004e-06 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.556327+0000", 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.8099999999999999e-06 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.560029+0000", 2024-09-21T10:46:43.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037024940000000002 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.560148+0000", 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000118657 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.560254+0000", 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00010609 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.563222+0000", 2024-09-21T10:46:43.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002968011 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.563899+0000", 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00067708299999999998 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.789498+0000", 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.225599103 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.789541+0000", 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2349999999999999e-05 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:40:45.789580+0000", 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9721000000000001e-05 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:25172 49.a 49:5d011e1e:::benchmark_data_smithi119_301770_object25171:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-09-21T10:46:43.558 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:35.729724+0000", 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 247.617337354, 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.16255114700000001, 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 25172 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729724+0000", 2024-09-21T10:46:43.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729724+0000", 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729723+0000", 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729787+0000", 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4824000000000001e-05 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729806+0000", 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8419000000000001e-05 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729813+0000", 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.7970000000000003e-06 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.729874+0000", 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0985e-05 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730003+0000", 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012986399999999999 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730135+0000", 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000131471 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.731237+0000", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011025049999999999 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.731408+0000", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000170872 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.892116+0000", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.16070807100000001 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.892225+0000", 2024-09-21T10:46:43.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000108516 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.892275+0000", 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0080999999999997e-05 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:25174 49.5 49:ab6a7a74:::benchmark_data_smithi119_301770_object25173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:35.730603+0000", 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 247.61645802999999, 2024-09-21T10:46:43.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.162712933, 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 25174 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730603+0000", 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730603+0000", 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730602+0000", 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730689+0000", 2024-09-21T10:46:43.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.6475000000000001e-05 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730699+0000", 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0076e-05 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730704+0000", 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8600000000000001e-06 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730744+0000", 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0541999999999999e-05 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730828+0000", 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3794000000000004e-05 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.730923+0000", 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5335999999999997e-05 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.568 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.731915+0000", 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00099156200000000009 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.732181+0000", 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00026602499999999997 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.893231+0000", 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.16105039500000001 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.893278+0000", 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6845000000000003e-05 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:35.893316+0000", 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7790000000000002e-05 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.570 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:43670 49.5 49:a3753815:::benchmark_data_smithi119_301770_object43669:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:47.027056+0000", 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 236.32000542399999, 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14992391699999999, 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 43670 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.571 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027056+0000", 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027056+0000", 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027054+0000", 2024-09-21T10:46:43.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027118+0000", 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3756000000000004e-05 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027132+0000", 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3373999999999999e-05 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027138+0000", 2024-09-21T10:46:43.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1199999999999999e-06 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027179+0000", 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1724000000000002e-05 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027270+0000", 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0408999999999994e-05 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.027357+0000", 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.7346e-05 2024-09-21T10:46:43.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.028696+0000", 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00133916 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.030880+0000", 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0021837940000000002 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.176908+0000", 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.146028414 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.176955+0000", 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.6385999999999999e-05 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.176980+0000", 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4820999999999998e-05 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.576 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:43671 49.5 49:ab830f66:::benchmark_data_smithi119_301770_object43670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1399)", 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:47.029433+0000", 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 236.31762851799999, 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.154013763, 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 43671 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029433+0000", 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029433+0000", 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029432+0000", 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029486+0000", 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3798000000000002e-05 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029496+0000", 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0271999999999999e-05 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029500+0000", 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.3900000000000003e-06 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029529+0000", 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.8419e-05 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029614+0000", 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.5339000000000004e-05 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 0,2", 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.029702+0000", 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8276999999999995e-05 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.031720+0000", 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0020175710000000001 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.032096+0000", 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00037571300000000001 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.183359+0000", 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15126305300000001 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.183407+0000", 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8334999999999997e-05 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:47.183446+0000", 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9456999999999998e-05 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.582 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:55768 49.9 49:96c91b51:::benchmark_data_smithi119_301770_object55767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:55.747307+0000", 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 227.599753827, 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.15274280900000001, 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55768 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747307+0000", 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747307+0000", 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747306+0000", 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747355+0000", 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8798000000000003e-05 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747366+0000", 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1181e-05 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747372+0000", 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3680000000000001e-06 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747420+0000", 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8780999999999999e-05 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747519+0000", 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.8573999999999995e-05 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747597+0000", 2024-09-21T10:46:43.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7928000000000003e-05 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.751583+0000", 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039864439999999996 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.751617+0000", 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3479999999999998e-05 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.899960+0000", 2024-09-21T10:46:43.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.14834275 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.900019+0000", 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9576e-05 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.900050+0000", 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0978000000000001e-05 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:55772 49.9 49:9b852315:::benchmark_data_smithi119_301770_object55771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:55.747879+0000", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 227.59918253699999, 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.152295605, 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 55772 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747879+0000", 2024-09-21T10:46:43.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747879+0000", 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747878+0000", 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.590 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747967+0000", 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8969999999999997e-05 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747980+0000", 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3091000000000001e-05 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.747985+0000", 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.5090000000000001e-06 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.749563+0000", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0015774020000000001 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.749693+0000", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013047800000000001 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.749828+0000", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000135473 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.751749+0000", 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001920302 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.751955+0000", 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.000206076 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.900096+0000", 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.148141561 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.900147+0000", 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.041e-05 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:55.900174+0000", 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.7427999999999999e-05 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.594 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:57586 49.9 49:96f23a56:::benchmark_data_smithi119_301770_object57585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:57.370249+0000", 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 225.97681211400001, 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.303970713, 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 57586 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370249+0000", 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370249+0000", 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370248+0000", 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370300+0000", 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2000999999999999e-05 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370312+0000", 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2058e-05 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370317+0000", 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7419999999999997e-06 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370363+0000", 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5957000000000001e-05 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370491+0000", 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0001282 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.370630+0000", 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013901000000000001 2024-09-21T10:46:43.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.372527+0000", 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00189694 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.372579+0000", 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1882999999999998e-05 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.674137+0000", 2024-09-21T10:46:43.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.301558188 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.674182+0000", 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5334000000000003e-05 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.674220+0000", 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7509999999999998e-05 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:160772 49.9 49:9831b828:::benchmark_data_smithi119_301770_object47285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:44:50.470181+0000", 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 112.87687990800001, 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.25506346699999999, 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 160772 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470181+0000", 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470181+0000", 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470179+0000", 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470187+0000", 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.6590000000000007e-06 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470203+0000", 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5798e-05 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470210+0000", 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8600000000000004e-06 2024-09-21T10:46:43.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470259+0000", 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9416999999999998e-05 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470395+0000", 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00013556399999999999 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.470516+0000", 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00012193699999999999 2024-09-21T10:46:43.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.473712+0000", 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0031957650000000002 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.473894+0000", 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00018141699999999999 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.725147+0000", 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.25125323300000002 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.725196+0000", 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9428000000000002e-05 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.725245+0000", 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8427999999999998e-05 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:160779 49.9 49:9d3816c2:::benchmark_data_smithi119_301770_object47292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:44:50.481597+0000", 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 112.865464098, 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.24567636000000001, 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 160779 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.607 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481597+0000", 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481597+0000", 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481596+0000", 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:46:43.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481600+0000", 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9709999999999998e-06 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481607+0000", 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.075e-06 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481611+0000", 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.27e-06 2024-09-21T10:46:43.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481642+0000", 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0945000000000003e-05 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481717+0000", 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.5080000000000006e-05 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.481785+0000", 2024-09-21T10:46:43.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8695000000000006e-05 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.484677+0000", 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0028919050000000002 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.484869+0000", 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00019167699999999999 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.727182+0000", 2024-09-21T10:46:43.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.242313258 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.727233+0000", 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0970000000000002e-05 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:44:50.727273+0000", 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0283e-05 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:46:43.612 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:46:43.613 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:46:43.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:43.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:43 smithi119 ceph-mon[101432]: pgmap v2536: 109 pgs: 4 creating+peering, 10 unknown, 95 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:43 smithi119 ceph-mon[105288]: pgmap v2536: 109 pgs: 4 creating+peering, 10 unknown, 95 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:43.633 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:43.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:43.757 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:43.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:43.879 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:43.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:43.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:44.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:44.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:44.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:44.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:44.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:46:44.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:44.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:46:44.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:46:44.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:44.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:44.487 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:46:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:46:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:44.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:46:44.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:44.627 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:44.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:44.750 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:44.753 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:46:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:46:44] "GET /metrics HTTP/1.1" 200 34440 "" "Prometheus/2.51.0" 2024-09-21T10:46:44.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:44.873 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:44.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:44.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:44.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:45.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:45.119 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:45.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:45.242 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:45.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:45.336 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:46:45.344 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:45.365 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:45.468 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:45.489 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:45.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:45.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:45 smithi119 ceph-mon[101432]: pgmap v2537: 109 pgs: 4 creating+peering, 105 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2452328900' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:46:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:45 smithi119 ceph-mon[105288]: pgmap v2537: 109 pgs: 4 creating+peering, 105 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2452328900' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:46:45.683 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:45 smithi145 ceph-mon[89359]: pgmap v2537: 109 pgs: 4 creating+peering, 105 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:45.683 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2452328900' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:46:45.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:45.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:45.837 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:45.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:45.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:45.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:45.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:46.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:46.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:46.202 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:46:46.202 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:46:46.202 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:46:46.202 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:46:46.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:46.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:46.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:46.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:46.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2452328900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:46:46.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:46 smithi119 ceph-mon[101432]: osdmap e1433: 8 total, 8 up, 6 in 2024-09-21T10:46:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2452328900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:46:46.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:46 smithi119 ceph-mon[105288]: osdmap e1433: 8 total, 8 up, 6 in 2024-09-21T10:46:46.454 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:46.476 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:46.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:46:46.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:46.599 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:46.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:46.722 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:46.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2452328900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:46:46.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:46 smithi145 ceph-mon[89359]: osdmap e1433: 8 total, 8 up, 6 in 2024-09-21T10:46:46.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:46.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:46.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:46.856 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:46:46.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:46.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:46.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:47.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:47.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:47.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:47.098 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:47.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:47.214 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:47.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:47.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:47.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.338 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:47.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:47.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:47.460 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:47.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:47.529 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:47 smithi145 ceph-mon[89359]: pgmap v2539: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:47 smithi145 ceph-mon[89359]: osdmap e1434: 8 total, 8 up, 6 in 2024-09-21T10:46:47.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:47.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:47 smithi119 ceph-mon[105288]: pgmap v2539: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:47 smithi119 ceph-mon[105288]: osdmap e1434: 8 total, 8 up, 6 in 2024-09-21T10:46:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:47 smithi119 ceph-mon[101432]: pgmap v2539: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:47 smithi119 ceph-mon[101432]: osdmap e1434: 8 total, 8 up, 6 in 2024-09-21T10:46:47.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:47.707 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:47.809 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:47.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:47.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:47.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:47.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:47.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:48.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:48.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:48.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:48.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:48.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:48.324 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:48.425 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:48.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:48.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:48.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:48.569 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:48.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:48.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:48.691 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:48.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:48.807 INFO:teuthology.orchestra.run.smithi119.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-21T10:46:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.819 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:48.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:48.917 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:46:48.917 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:48.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:48.933 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:46:48.933 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:48.940 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:48.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:48.949 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:46:48.949 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:48.965 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:46:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:48.981 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:46:48.981 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:48.996 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:46:48.996 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:49.001 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:46:49.001 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:46:49.001 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:46:49.001 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:46:49.001 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:46:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:46:49.012 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:49.027 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:46:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:49.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:48 smithi145 ceph-mon[89359]: pgmap v2541: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4059075168' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:46:49.063 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[105288]: pgmap v2541: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4059075168' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:46:49.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[101432]: pgmap v2541: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:49.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4059075168' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-21T10:46:49.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:49.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:49.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.311 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:49.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:49.412 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:49.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:49.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:49.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:49.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:49.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:49.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:46:49.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:49.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:49.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:49.808 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4059075168' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:46:49.808 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:49 smithi119 ceph-mon[101432]: osdmap e1435: 8 total, 8 up, 6 in 2024-09-21T10:46:49.808 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1963230136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:49.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:49.911 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:49.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:50.032 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4059075168' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:46:50.032 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:49 smithi145 ceph-mon[89359]: osdmap e1435: 8 total, 8 up, 6 in 2024-09-21T10:46:50.032 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1963230136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:50.069 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:50.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4059075168' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T10:46:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:49 smithi119 ceph-mon[105288]: osdmap e1435: 8 total, 8 up, 6 in 2024-09-21T10:46:50.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1963230136' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:50.192 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:50.216 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:50.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:46:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:46:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:46:50.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.339 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:50.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:50.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:50.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:50.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:50.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:50.685 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:50.708 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:50.771 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:50.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:50.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:50.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:50.954 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:50.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:51.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:51.076 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:51.096 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:50 smithi145 ceph-mon[89359]: pgmap v2543: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:51.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:50 smithi119 ceph-mon[105288]: pgmap v2543: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:51.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:50 smithi119 ceph-mon[101432]: pgmap v2543: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:51.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:51.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:51.301 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:51.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:51.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:51.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:51.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:51.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:51.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:51.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:51.666 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:51.667 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:51.671 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:51.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:51.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:51.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:51.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:46:51.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:51.972 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:51.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:51.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:52.074 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:52.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:52.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:51 smithi119 ceph-mon[105288]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:46:52.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:51 smithi119 ceph-mon[101432]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:46:52.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:52.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:52.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:52.232 DEBUG:teuthology.orchestra.run.smithi119:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-21T10:46:52.235 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-21T10:46:52.235 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-21T10:46:52.281 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:hints = 1 2024-09-21T10:46:52.282 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-21T10:46:52.282 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object prefix: benchmark_data_smithi119_335063 2024-09-21T10:46:52.282 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:46:52.282 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 0 0 0 0 0 0 - 0 2024-09-21T10:46:52.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:51 smithi145 ceph-mon[89359]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T10:46:52.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:52.350 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:52.357 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:46:52.452 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:52.490 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:52.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:52.631 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:52.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:52.729 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:52.732 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:52.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:52.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:52.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:52.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:53.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.044 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:53.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:53.071 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:46:53.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:52 smithi119 ceph-mon[101432]: pgmap v2544: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:52 smithi119 ceph-mon[105288]: pgmap v2544: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:53.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:53.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:53.192 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:52 smithi145 ceph-mon[89359]: pgmap v2544: 109 pgs: 109 active+clean; 369 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail 2024-09-21T10:46:53.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:53.282 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 1 16 3093 3077 192.302 192.312 0.00659139 0.00518731 2024-09-21T10:46:53.301 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:53.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:53.442 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:53.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:53.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:53.673 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:53.699 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:53.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:53.841 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:53.845 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:46:53.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:53.970 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:53.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:54.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:54.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/333735225' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:53 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:53 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/333735225' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:53 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:53 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:54.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:54.251 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:54.282 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 2 16 5889 5873 183.518 174.75 0.00338308 0.00543764 2024-09-21T10:46:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/333735225' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:53 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:46:54.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:53 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:46:54.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:54.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:54.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:54.521 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:54.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:46:54] "GET /metrics HTTP/1.1" 200 36769 "" "Prometheus/2.51.0" 2024-09-21T10:46:54.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:54.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:54.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:54.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:54.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:54.876 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:46:54.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:54.914 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:54.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:55.016 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:55.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:55.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:55.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: osdmap e1436: 8 total, 8 up, 6 in 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: pgmap v2546: 109 pgs: 109 active+clean; 133 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s wr, 277 op/s 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2856344780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: osdmap e1436: 8 total, 8 up, 6 in 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: pgmap v2546: 109 pgs: 109 active+clean; 133 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s wr, 277 op/s 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2856344780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:46:55.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:54 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:46:55.128 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:55.128 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:55.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:46:55.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:46:55.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:46:55.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: osdmap e1436: 8 total, 8 up, 6 in 2024-09-21T10:46:55.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: pgmap v2546: 109 pgs: 109 active+clean; 133 MiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 17 MiB/s wr, 277 op/s 2024-09-21T10:46:55.128 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2856344780' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:46:55.129 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:54 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:46:55.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:55.182 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:55.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:55.283 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 3 16 8806 8790 183.04 182.312 0.00484937 0.00545466 2024-09-21T10:46:55.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:55.308 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:55.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:55.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:55.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:55.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:55.581 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:55.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:55.710 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:55.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:55.848 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:55.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:46:55.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:55.985 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:55.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:55.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:56.087 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:56.114 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:46:56.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:55 smithi119 ceph-mon[105288]: osdmap e1437: 8 total, 8 up, 6 in 2024-09-21T10:46:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:46:56.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:55 smithi119 ceph-mon[101432]: osdmap e1437: 8 total, 8 up, 6 in 2024-09-21T10:46:56.192 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:56.216 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:56.242 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:56.283 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 4 16 11376 11360 177.434 160.625 0.00599991 0.00562907 2024-09-21T10:46:56.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:46:56.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:55 smithi145 ceph-mon[89359]: osdmap e1437: 8 total, 8 up, 6 in 2024-09-21T10:46:56.344 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:56.379 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:56.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:56.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:56.619 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:56.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:56.763 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:56.789 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:56.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:46:56.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:56.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:46:57.016 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:57.038 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:56 smithi145 ceph-mon[89359]: pgmap v2548: 109 pgs: 109 active+clean; 254 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 35 MiB/s wr, 566 op/s 2024-09-21T10:46:57.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:57.162 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:57.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:57.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 5 16 13343 13327 166.535 122.938 0.00666591 0.00599411 2024-09-21T10:46:57.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:56 smithi119 ceph-mon[101432]: pgmap v2548: 109 pgs: 109 active+clean; 254 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 35 MiB/s wr, 566 op/s 2024-09-21T10:46:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:56 smithi119 ceph-mon[105288]: pgmap v2548: 109 pgs: 109 active+clean; 254 MiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 35 MiB/s wr, 566 op/s 2024-09-21T10:46:57.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:57.440 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:57.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:57.563 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:57.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:57.690 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:57.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.815 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:57.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:57.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:57.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:57.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:58.019 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:46:58.019 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:46:58.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:58.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:58.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:58.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:46:58.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 6 16 15312 15296 159.289 123.062 0.00510262 0.00627003 2024-09-21T10:46:58.320 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:58.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:58.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:58.443 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:58.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:58.573 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:58.612 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:58.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:58.714 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:58.736 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:58.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:46:58.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:46:58.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:58.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:46:58.978 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:46:59.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:46:59.078 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:46:59.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:59.142 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:59.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:46:59.265 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:46:59.283 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 7 16 16650 16634 148.48 83.625 0.00165858 0.00660551 2024-09-21T10:46:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:59 smithi145 ceph-mon[89359]: pgmap v2549: 109 pgs: 109 active+clean; 351 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 44 MiB/s wr, 701 op/s 2024-09-21T10:46:59.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:46:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3424422450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:59.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:46:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:59 smithi119 ceph-mon[101432]: pgmap v2549: 109 pgs: 109 active+clean; 351 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 44 MiB/s wr, 701 op/s 2024-09-21T10:46:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:46:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3424422450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:59 smithi119 ceph-mon[105288]: pgmap v2549: 109 pgs: 109 active+clean; 351 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 44 MiB/s wr, 701 op/s 2024-09-21T10:46:59.368 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:46:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3424422450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:46:59.401 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:46:59.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:46:59.538 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:46:59.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:59.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:59.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:46:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.814 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:46:59.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:46:59.915 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:46:59.940 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:46:59.941 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:46:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:46:59.942 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:46:59.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:00.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:00.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:00.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:00.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:00.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 8 16 18262 18246 142.514 100.75 0.00851528 0.00701095 2024-09-21T10:47:00.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:00.340 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:00.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:00.462 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:00.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:47:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:47:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:47:00.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:00.602 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:00.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:00.704 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:00.729 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:00.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:00.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:00.867 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:00.969 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:00.989 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:00.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:01.091 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:01.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:01.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:01.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:01.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 9 16 19977 19961 138.588 107.188 0.0037165 0.00720994 2024-09-21T10:47:01.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:01.356 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:01 smithi145 ceph-mon[89359]: pgmap v2550: 109 pgs: 109 active+clean; 821 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:47:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:01.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:01 smithi119 ceph-mon[101432]: pgmap v2550: 109 pgs: 109 active+clean; 821 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:47:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:01 smithi119 ceph-mon[105288]: pgmap v2550: 109 pgs: 109 active+clean; 821 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 102 MiB/s wr, 1.64k op/s 2024-09-21T10:47:01.468 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:01.510 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:01.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:01.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:01.647 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:01.749 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:01.790 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:01.892 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:01.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:01.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:02.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:02.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.272 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 10 16 21851 21835 136.441 117.125 0.00243794 0.00732142 2024-09-21T10:47:02.298 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:02.425 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:02.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.555 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:02.660 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:47:02.660 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:47:02.660 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:47:02.660 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [] 2024-09-21T10:47:02.660 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:47:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:02.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:02.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:02.820 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:02.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:02.948 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:02.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:02.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:03.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:03.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:03.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:03.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 11 16 23739 23723 134.764 118 0.00482695 0.00739086 2024-09-21T10:47:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:03 smithi145 ceph-mon[89359]: pgmap v2551: 109 pgs: 109 active+clean; 1.0 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-21T10:47:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:03.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:03.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:03.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:03.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:03.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T10:47:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:03 smithi119 ceph-mon[101432]: pgmap v2551: 109 pgs: 109 active+clean; 1.0 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-21T10:47:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:03 smithi119 ceph-mon[105288]: pgmap v2551: 109 pgs: 109 active+clean; 1.0 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 111 MiB/s wr, 1.77k op/s 2024-09-21T10:47:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:03.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:03.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:03.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:03.599 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:03.701 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:03.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:03.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:03.849 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:03.950 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:03.975 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:03.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:04.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:04.098 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:04.199 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:04.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:04.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 12 16 24639 24623 128.221 56.25 0.00165429 0.00770684 2024-09-21T10:47:04.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:04.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:04.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:04.469 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:04.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:04.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:04.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:04.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:04.738 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:04.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:47:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:47:04] "GET /metrics HTTP/1.1" 200 36829 "" "Prometheus/2.51.0" 2024-09-21T10:47:04.881 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:04.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:04.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:05.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:05.127 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:05.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:05.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:05.250 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:05.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:05.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 13 16 25725 25709 123.579 67.875 0.00158322 0.00801804 2024-09-21T10:47:05.353 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:05 smithi119 ceph-mon[105288]: pgmap v2552: 109 pgs: 109 active+clean; 1.3 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:47:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:05 smithi119 ceph-mon[101432]: pgmap v2552: 109 pgs: 109 active+clean; 1.3 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:47:05.378 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:05.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:05.479 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:05.522 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:05 smithi145 ceph-mon[89359]: pgmap v2552: 109 pgs: 109 active+clean; 1.3 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 123 MiB/s wr, 1.97k op/s 2024-09-21T10:47:05.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:05.663 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:05.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:05.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:05.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:05.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:05.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:06.053 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:06.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:47:06.092 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:06.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:06.221 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:06.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 14 16 27601 27585 123.126 117.25 0.00766356 0.00811477 2024-09-21T10:47:06.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:06.347 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:06.448 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:06.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:06.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:06.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:06.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:06.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:06.728 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:06.776 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:06.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:06.851 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:06.953 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:06.977 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:06.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:07.078 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:07.102 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:07.148 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:07 smithi119 ceph-mon[105288]: pgmap v2553: 109 pgs: 109 active+clean; 1.5 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:47:07.149 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:07 smithi119 ceph-mon[101432]: pgmap v2553: 109 pgs: 109 active+clean; 1.5 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:47:07.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:07.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:07.284 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 15 16 29419 29403 122.492 113.625 0.011787 0.00815798 2024-09-21T10:47:07.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:07.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:07.530 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:07.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:07.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:07 smithi145 ceph-mon[89359]: pgmap v2553: 109 pgs: 109 active+clean; 1.5 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 122 MiB/s wr, 1.95k op/s 2024-09-21T10:47:07.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:47:07.632 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:07.668 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:07.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:07.793 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:07.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:07.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:07.919 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:07.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:08.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:08.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:08.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:08.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:08.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:08.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:08.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 16 16 31183 31167 121.726 110.25 0.00590868 0.00820954 2024-09-21T10:47:08.299 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:08.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:08.426 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:08.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:08.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:08.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:47:08.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:08.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:08.780 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:08.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:08.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:08.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:09.013 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:09.114 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:09.139 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:09.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:09.264 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:09.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 17 16 32834 32818 120.635 103.188 0.00837715 0.00828354 2024-09-21T10:47:09.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:09 smithi119 ceph-mon[105288]: pgmap v2554: 109 pgs: 109 active+clean; 1.5 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T10:47:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:09 smithi119 ceph-mon[101432]: pgmap v2554: 109 pgs: 109 active+clean; 1.5 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T10:47:09.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:09.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:09.533 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:09.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:09 smithi145 ceph-mon[89359]: pgmap v2554: 109 pgs: 109 active+clean; 1.5 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-21T10:47:09.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:09.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:09.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:09.786 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:09.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:09.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:09.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.029 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:10.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.168 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:10.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:10.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:10.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 18 16 34549 34533 119.887 107.188 0.0167585 0.00833516 2024-09-21T10:47:10.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:10.344 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:10.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:10.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:47:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:47:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:47:10.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:10.620 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.722 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:10.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:10.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:10.885 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:10.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:10.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:10.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:11.023 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:11.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:11.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:11.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:11.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:11.277 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:11.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 19 16 36133 36117 118.788 99 0.0115964 0.00841323 2024-09-21T10:47:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:11 smithi119 ceph-mon[105288]: pgmap v2555: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-21T10:47:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:11 smithi119 ceph-mon[101432]: pgmap v2555: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-21T10:47:11.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.402 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:11.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:11.503 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:11.529 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:11 smithi145 ceph-mon[89359]: pgmap v2555: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 125 MiB/s wr, 1.99k op/s 2024-09-21T10:47:11.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:11.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:11.774 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:11.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:11.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:11.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:11.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:11.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:12.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:12.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:12.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:12.200 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:12.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:47:12.284176+0000 min lat: 0.00116109 max lat: 0.230568 avg lat: 0.00851309 2024-09-21T10:47:12.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:47:12.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 20 16 37583 37567 117.379 90.625 0.0175529 0.00851309 2024-09-21T10:47:12.301 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:12.326 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:12.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:12.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:12.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:12.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:12.596 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:12.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:12.739 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:12.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:12.879 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:12.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:12.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:12.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:13.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:13 smithi119 ceph-mon[105288]: pgmap v2556: 109 pgs: 109 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:47:13.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:13 smithi119 ceph-mon[101432]: pgmap v2556: 109 pgs: 109 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:47:13.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:13.144 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:13.245 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:13.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:13.285 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 21 16 39077 39061 116.236 93.375 0.0049365 0.0085972 2024-09-21T10:47:13.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:13.409 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:13.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:13.532 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:13.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:13.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:13 smithi145 ceph-mon[89359]: pgmap v2556: 109 pgs: 109 active+clean; 2.0 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 102 MiB/s wr, 1.62k op/s 2024-09-21T10:47:13.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:13.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:13.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:13.806 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:13.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:13.908 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:13.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:13.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:14.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:14.059 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:14.161 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:14.173 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:14.178 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:14.184 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:14.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:14.286 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 22 16 40531 40515 115.076 90.875 0.0105408 0.00868403 2024-09-21T10:47:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.333 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:14.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:14.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:14.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:14.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:14.594 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:14.617 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:14.714 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:47:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:47:14] "GET /metrics HTTP/1.1" 200 36829 "" "Prometheus/2.51.0" 2024-09-21T10:47:14.719 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:14.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:14.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:14.844 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:14.869 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:14.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:14.995 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:14.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:15.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:15.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:15.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:15.134 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:47:15.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:15.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:15.286 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 23 16 41834 41818 113.613 81.4375 0.00645959 0.00879613 2024-09-21T10:47:15.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:15 smithi119 ceph-mon[105288]: pgmap v2557: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-21T10:47:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2775344458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:15 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:15 smithi119 ceph-mon[101432]: pgmap v2557: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-21T10:47:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2775344458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:15 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:15.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:15.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:15 smithi145 ceph-mon[89359]: pgmap v2557: 109 pgs: 109 active+clean; 2.3 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 109 MiB/s wr, 1.75k op/s 2024-09-21T10:47:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2775344458' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:15 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:47:15.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:15.680 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:15.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:15.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:15.805 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:15.906 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:15.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:15.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:15.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:16.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:16.055 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:16.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:16.157 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:16.183 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:16.284 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:16.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 24 16 42933 42917 111.741 68.6875 0.00817818 0.00894397 2024-09-21T10:47:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:16.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:16.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:16.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:16 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:47:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:16 smithi145 ceph-mon[89359]: osdmap e1438: 8 total, 8 up, 6 in 2024-09-21T10:47:16.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:16.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:16 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:47:16.582 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:16 smithi119 ceph-mon[101432]: osdmap e1438: 8 total, 8 up, 6 in 2024-09-21T10:47:16.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:16 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:47:16.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:16 smithi119 ceph-mon[105288]: osdmap e1438: 8 total, 8 up, 6 in 2024-09-21T10:47:16.588 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:16.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:16.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:16.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:16.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:16.854 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:16.956 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:16.980 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:16.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:17.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:17.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:17.221 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:17.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:17.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:17.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 25 16 44480 44464 111.139 96.6875 0.00624664 0.00899056 2024-09-21T10:47:17.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:17.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:17 smithi119 ceph-mon[101432]: pgmap v2559: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T10:47:17.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:17 smithi119 ceph-mon[105288]: pgmap v2559: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T10:47:17.371 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:17.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.511 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:17.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:17.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:17 smithi145 ceph-mon[89359]: pgmap v2559: 109 pgs: 109 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 98 MiB/s wr, 1.56k op/s 2024-09-21T10:47:17.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:17.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.796 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:17.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:17.853 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:47:17.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:47:17.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:17.941 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:17.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:18.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:18.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:18.171 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:18.195 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:18.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 26 16 45913 45897 110.309 89.5625 0.00961703 0.00905955 2024-09-21T10:47:18.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:18.320 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:18.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:18.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.475 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:18.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:18.560 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:18.560 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:18.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:18.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:18.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:18.723 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:46491 50.7 50:ece7ceb5:::benchmark_data_smithi119_335063_object46490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:18.713962+0000", 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.011213923000000001, 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011245204999999999, 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:18.778 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 46491 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.713962+0000", 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.779 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.713962+0000", 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.713964+0000", 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2100000000000001e-06 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.714004+0000", 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0187e-05 2024-09-21T10:47:18.780 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.714023+0000", 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9103e-05 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.714030+0000", 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3549999999999999e-06 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:18.781 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718900+0000", 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048692650000000002 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.719023+0000", 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000123227 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.719147+0000", 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000124657 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.782 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.723339+0000", 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0041911830000000002 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.723350+0000", 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1148000000000001e-05 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.783 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:46498 50.7 50:e6051801:::benchmark_data_smithi119_335063_object46497:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:18.718574+0000", 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.006601979, 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0067077159999999999, 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 46498 2024-09-21T10:47:18.784 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718574+0000", 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718574+0000", 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:18.785 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718576+0000", 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.491e-06 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718596+0000", 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0112e-05 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718606+0000", 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6269999999999995e-06 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.786 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718611+0000", 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7379999999999999e-06 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.721131+0000", 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025196620000000002 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.721281+0000", 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015001599999999999 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.787 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.721429+0000", 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000148219 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.723393+0000", 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0019636689999999999 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.723504+0000", 2024-09-21T10:47:18.788 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000111586 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:46502 50.a 50:5c05fec7:::benchmark_data_smithi119_335063_object46501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:18.724127+0000", 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0010493080000000001, 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0012205289999999999, 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:18.789 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 46502 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724127+0000", 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724127+0000", 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.790 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724129+0000", 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4950000000000001e-06 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724151+0000", 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2509000000000001e-05 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724161+0000", 2024-09-21T10:47:18.791 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6870000000000006e-06 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724167+0000", 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.178e-06 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724219+0000", 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2540999999999997e-05 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724375+0000", 2024-09-21T10:47:18.792 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000155186 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.724517+0000", 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014248700000000001 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:46486 50.7 50:e9354b41:::benchmark_data_smithi119_335063_object46485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:18.710335+0000", 2024-09-21T10:47:18.793 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.014841744, 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01508936, 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 46486 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.710335+0000", 2024-09-21T10:47:18.794 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.710335+0000", 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.710336+0000", 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.235e-06 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.795 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.710358+0000", 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1814000000000002e-05 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.710368+0000", 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.077e-05 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.710375+0000", 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9639999999999997e-06 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.796 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.713613+0000", 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032374840000000001 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.713772+0000", 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015903299999999999 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.713921+0000", 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014893799999999999 2024-09-21T10:47:18.797 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718636+0000", 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0047148470000000003 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718827+0000", 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000191671 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.798 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:46487 50.7 50:e2c571f0:::benchmark_data_smithi119_335063_object46486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:18.711137+0000", 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.01403955, 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014354442, 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "waiting for sub ops", 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 46487 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.799 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.711137+0000", 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.711137+0000", 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.711137+0000", 2024-09-21T10:47:18.800 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1200000000000002e-07 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.711156+0000", 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8255000000000001e-05 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.711164+0000", 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2320000000000001e-06 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.711169+0000", 2024-09-21T10:47:18.801 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4720000000000003e-06 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.715831+0000", 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0046615099999999998 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.715973+0000", 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014186 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.802 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.716446+0000", 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00047318600000000001 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.718872+0000", 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024262469999999999 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:18.723298+0000", 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044261379999999996 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.803 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:18.804 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.804 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:18.804 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:47:18.804 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 5 2024-09-21T10:47:18.804 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:47:18.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:18.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:18.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:18.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:18.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:19.005 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:19.107 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:19.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:19.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.271 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:19.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:19.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 27 16 47304 47288 109.443 86.9375 0.00578907 0.00913075 2024-09-21T10:47:19.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:19.396 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:19.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:19.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:19 smithi145 ceph-mon[89359]: pgmap v2560: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T10:47:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:19 smithi119 ceph-mon[101432]: pgmap v2560: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T10:47:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:19 smithi119 ceph-mon[105288]: pgmap v2560: 109 pgs: 109 active+clean; 2.5 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-21T10:47:19.639 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:19.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:19.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:19.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:19.910 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:19.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:20.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:20.047 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:20.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:20.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:20.276 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:20.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 28 16 48716 48700 108.686 88.25 0.0107499 0.00919572 2024-09-21T10:47:20.299 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:20.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:20.424 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:20.526 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:47:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:47:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:47:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.569 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:20.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:20.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:20.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:20.836 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:20.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:20.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:20.963 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:20.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:21.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:21.087 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:21.127 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:21.189 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:47:21.198 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:21.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:21.234 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:47:21.234 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:21.250 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:47:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:21.267 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:47:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:21.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 29 16 49899 49883 107.488 73.9375 0.0576198 0.00929621 2024-09-21T10:47:21.293 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:47:21.293 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:21.326 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:47:21.326 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:21.330 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:21.356 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:47:21.356 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:21.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:21 smithi119 ceph-mon[101432]: pgmap v2561: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:47:21.357 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:21 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2167251654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:21.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:21 smithi119 ceph-mon[105288]: pgmap v2561: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:47:21.357 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:21 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2167251654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:21.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:21.388 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:47:21.389 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:21.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:21.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.498 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:21.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:21 smithi145 ceph-mon[89359]: pgmap v2561: 109 pgs: 109 active+clean; 2.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-21T10:47:21.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:21 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2167251654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:21.598 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:21.624 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:21.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:21.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:21.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:21.893 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:21.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:21.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:21.994 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:22.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:22.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:22.156 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:22.167 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:47:22.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:22.284 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:22.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 30 16 51152 51136 106.515 78.3125 0.00900983 0.00938225 2024-09-21T10:47:22.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.417 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:22.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:22.518 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2318170593' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:22.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:22 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:22.563 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2318170593' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:22.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:22 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2318170593' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:22.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:22 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:47:22.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:22.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:22.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:22.930 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:22.956 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:22.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:23.058 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:23.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:23.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:23.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:23.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:23.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 31 16 52416 52400 105.627 79 0.0297962 0.00945876 2024-09-21T10:47:23.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:23.378 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:23.480 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:23.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:23 smithi145 ceph-mon[89359]: pgmap v2562: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:47:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:23 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:47:23.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:23 smithi145 ceph-mon[89359]: osdmap e1439: 8 total, 8 up, 6 in 2024-09-21T10:47:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[105288]: pgmap v2562: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:47:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:47:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[105288]: osdmap e1439: 8 total, 8 up, 6 in 2024-09-21T10:47:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[101432]: pgmap v2562: 109 pgs: 109 active+clean; 2.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-21T10:47:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:47:23.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:23 smithi119 ceph-mon[101432]: osdmap e1439: 8 total, 8 up, 6 in 2024-09-21T10:47:23.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:23.643 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:23.745 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:23.779 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:23.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:23.881 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:23.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:23.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:24.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:24.042 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:24.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:24.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:24.266 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:24.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 32 16 53373 53357 104.195 59.8125 0.00957563 0.00959137 2024-09-21T10:47:24.290 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:24.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:24.425 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:24.527 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:24.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:24.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T10:47:24.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:24.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:24.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:47:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:47:24] "GET /metrics HTTP/1.1" 200 36836 "" "Prometheus/2.51.0" 2024-09-21T10:47:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:24.899 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:24.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:25.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:25.024 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:25.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:25.150 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:25.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:25.287 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:25.287 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 33 16 54651 54635 103.458 79.875 0.0100861 0.00966008 2024-09-21T10:47:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:25.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:25 smithi119 ceph-mon[105288]: pgmap v2564: 109 pgs: 109 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:47:25.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:25 smithi119 ceph-mon[101432]: pgmap v2564: 109 pgs: 109 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:47:25.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:25.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:25.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:25.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:25 smithi145 ceph-mon[89359]: pgmap v2564: 109 pgs: 109 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 83 MiB/s wr, 1.32k op/s 2024-09-21T10:47:25.552 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:25.654 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:25.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:25.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:25.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:25.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:25.928 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:25.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:26.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.056 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:26.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:26.157 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:26.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:26.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:26.289 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 34 15 55866 55851 102.645 76 0.0175219 0.00973659 2024-09-21T10:47:26.307 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:26.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:26.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.457 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:26.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:26.503 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:26.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.611 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:26.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:26.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:26.736 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:26.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:26.883 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:26.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:26.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:27.030 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:27.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:27.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:27.265 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:27.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:27.289 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 35 16 56960 56944 101.664 68.3125 0.00988346 0.00983048 2024-09-21T10:47:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:27 smithi119 ceph-mon[105288]: pgmap v2565: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:47:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:27 smithi119 ceph-mon[101432]: pgmap v2565: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:47:27.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:27.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:27.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:47:27.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:27.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:27 smithi145 ceph-mon[89359]: pgmap v2565: 109 pgs: 109 active+clean; 3.2 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-21T10:47:27.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:27.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:27.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:27.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:27.796 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:27.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:27.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:27.936 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:27.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:28.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:28.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:28.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:28.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:28.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:28.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:47:28.290 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 36 16 57931 57915 100.526 60.6875 0.0101787 0.00994131 2024-09-21T10:47:28.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:28.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:28.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.482 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:28.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:28.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:28.622 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:28.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:28.746 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:28.848 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:28.872 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:28.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:28.994 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:28.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:29.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:29.129 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:29.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:29.267 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:29.290 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 37 16 58954 58938 99.5367 63.9375 0.0115128 0.0100405 2024-09-21T10:47:29.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:29.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:29.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:29.518 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:29.577 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:29 smithi145 ceph-mon[89359]: pgmap v2566: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-21T10:47:29.616 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:29 smithi119 ceph-mon[105288]: pgmap v2566: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-21T10:47:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:29 smithi119 ceph-mon[101432]: pgmap v2566: 109 pgs: 109 active+clean; 3.3 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-21T10:47:29.677 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:29.701 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:29.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:29.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:29.826 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:29.928 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:29.952 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:29.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:29.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:30.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:30.077 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:30.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:30.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:30.224 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:30.291 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 38 16 60133 60117 98.8543 73.6875 0.0144143 0.0101095 2024-09-21T10:47:30.300 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:30.326 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:30.349 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:30.450 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:30.490 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:30.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:30.546 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:47:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:47:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:47:30.591 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:30.615 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:30.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:30.754 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:30.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:30.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:30.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:30.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:31.023 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:31.125 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:31.148 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:31.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:31.271 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:31.280 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:31.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 39 16 61281 61265 98.1566 71.75 0.013835 0.010181 2024-09-21T10:47:31.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:31.432 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:31.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:31 smithi119 ceph-mon[101432]: pgmap v2567: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:47:31.512 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:31 smithi119 ceph-mon[105288]: pgmap v2567: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:47:31.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:31 smithi145 ceph-mon[89359]: pgmap v2567: 109 pgs: 109 active+clean; 3.5 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-21T10:47:31.557 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:31.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:31.702 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:31.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:31.826 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:31.906 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:47:31.906 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:47:31.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:31.970 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:31.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:32.072 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:32.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:32.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:32.238 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:32.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:47:32.290979+0000 min lat: 0.00116109 max lat: 0.303819 avg lat: 0.0102588 2024-09-21T10:47:32.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:47:32.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 40 16 62382 62366 97.4229 68.8125 0.0136497 0.0102588 2024-09-21T10:47:32.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:32.377 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:32.478 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:32.618 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:32.641 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:32.743 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:32.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:32.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:32.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:32.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:32.913 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:32.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:33.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.058 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:33.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:33.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:33.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:33.210 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:33 smithi119 ceph-mon[101432]: pgmap v2568: 109 pgs: 109 active+clean; 3.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:47:33.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 41 16 63552 63536 96.8302 73.125 0.0143078 0.0103211 2024-09-21T10:47:33.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:33.354 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:33.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:33.491 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:33 smithi145 ceph-mon[89359]: pgmap v2568: 109 pgs: 109 active+clean; 3.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:47:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:33.592 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:33.616 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:33.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:33 smithi119 ceph-mon[105288]: pgmap v2568: 109 pgs: 109 active+clean; 3.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-21T10:47:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:33.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:33.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:33.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:33.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:33.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:33.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:34.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:34.109 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:34.133 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:34.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:34.276 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:34.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:34.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 42 16 64753 64737 96.3119 75.0625 0.0143988 0.0103774 2024-09-21T10:47:34.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:34.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:34.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:34.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:34.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:34.804 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:34.807 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:47:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:47:34] "GET /metrics HTTP/1.1" 200 36829 "" "Prometheus/2.51.0" 2024-09-21T10:47:34.843 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:34.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:34.968 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:34.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:34.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:35.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:35.093 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:35.195 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:35.217 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:47:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:35.222 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:47:35.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T10:47:35.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 43 16 65825 65809 95.6301 67 0.00652403 0.0104518 2024-09-21T10:47:35.319 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:35.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:35.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:35.464 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:35 smithi119 ceph-mon[101432]: pgmap v2569: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-21T10:47:35.465 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:35 smithi119 ceph-mon[105288]: pgmap v2569: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-21T10:47:35.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:35 smithi145 ceph-mon[89359]: pgmap v2569: 109 pgs: 109 active+clean; 3.8 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-21T10:47:35.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:35.592 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:35.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:35.717 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:35.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.860 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:35.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:35.961 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:35.983 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:35.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:35.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:36.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:36.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:36.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:36.253 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:36.293 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 44 16 66759 66743 94.7834 58.375 0.0144177 0.0105312 2024-09-21T10:47:36.306 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:47:36.355 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:36.377 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:36.479 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:36.502 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:36 smithi145 ceph-mon[89359]: from='client.37293 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:47:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4129457644' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:36.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:36 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:36.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:36 smithi119 ceph-mon[105288]: from='client.37293 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:47:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4129457644' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:36.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:36 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:36 smithi119 ceph-mon[101432]: from='client.37293 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:47:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4129457644' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:36.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:36 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:47:36.640 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:36.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:36.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:36.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:36.908 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:36.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:37.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:37.046 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:37.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:37.149 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:37.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:37.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:37.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 45 16 67809 67793 94.1353 65.625 0.0154551 0.0106175 2024-09-21T10:47:37.296 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[105288]: pgmap v2570: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[105288]: osdmap e1440: 8 total, 8 up, 6 in 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[101432]: pgmap v2570: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:37.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:37 smithi119 ceph-mon[101432]: osdmap e1440: 8 total, 8 up, 6 in 2024-09-21T10:47:37.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:37.422 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:37.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:37 smithi145 ceph-mon[89359]: pgmap v2570: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:37 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:47:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:37 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:47:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:37 smithi145 ceph-mon[89359]: osdmap e1440: 8 total, 8 up, 6 in 2024-09-21T10:47:37.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:37.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:37.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:37.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.795 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:37.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:37.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:37.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:38.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:38.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:38.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:38.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:38.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 46 16 68616 68600 93.1854 50.4375 0.00281908 0.0107177 2024-09-21T10:47:38.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.343 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:38.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:38.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:38.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:38.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:38.592 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:38.694 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:38.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:38.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:38.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:38.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:38.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:39.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:39.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:39.205 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:39.229 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:39.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 47 16 69716 69700 92.6654 68.75 0.0187346 0.0107864 2024-09-21T10:47:39.330 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:39.354 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:39.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:39.494 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:39 smithi145 ceph-mon[89359]: pgmap v2572: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:47:39.595 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:39 smithi119 ceph-mon[105288]: pgmap v2572: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:47:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:39 smithi119 ceph-mon[101432]: pgmap v2572: 109 pgs: 109 active+clean; 3.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-21T10:47:39.619 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:39.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:39.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:39.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:39.836 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2569,"stamp":"2024-09-21T10:47:33.999680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4095197816,"num_objects":62508,"num_object_clones":0,"num_object_copies":187524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62545,"num_write_kb":3998879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":924281,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64640,"ondisk_log_size":64640,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":17327984,"kb_used_data":11780608,"kb_used_omap":505,"kb_used_meta":5546694,"kb_avail":545142928,"statfs":{"total":575970213888,"available":558226358272,"internally_reserved":0,"allocated":12063342592,"data_stored":12038780081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":517945,"internal_metadata":5679814855},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1006960640,"num_objects":15365,"num_object_clones":0,"num_object_copies":46095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15365,"num_write_kb":983360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.829903"},"pg_stats":[{"pgid":"50.f","version":"1439'3861","reported_seq":3889,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:31.364019+0000","last_change":"2024-09-21T10:46:40.275318+0000","last_active":"2024-09-21T10:47:31.364019+0000","last_peered":"2024-09-21T10:47:31.364019+0000","last_clean":"2024-09-21T10:47:31.364019+0000","last_became_active":"2024-09-21T10:46:40.275064+0000","last_became_peered":"2024-09-21T10:46:40.275064+0000","last_unstale":"2024-09-21T10:47:31.364019+0000","last_undegraded":"2024-09-21T10:47:31.364019+0000","last_fullsized":"2024-09-21T10:47:31.364019+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:25.995350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253034496,"num_objects":3861,"num_object_clones":0,"num_object_copies":11583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3861,"num_write_kb":247104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.e","version":"1439'3722","reported_seq":3751,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:30.024640+0000","last_change":"2024-09-21T10:46:40.276274+0000","last_active":"2024-09-21T10:47:30.024640+0000","last_peered":"2024-09-21T10:47:30.024640+0000","last_clean":"2024-09-21T10:47:30.024640+0000","last_became_active":"2024-09-21T10:46:40.276005+0000","last_became_peered":"2024-09-21T10:46:40.276005+0000","last_unstale":"2024-09-21T10:47:30.024640+0000","last_undegraded":"2024-09-21T10:47:30.024640+0000","last_fullsized":"2024-09-21T10:47:30.024640+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:42.051377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243924992,"num_objects":3722,"num_object_clones":0,"num_object_copies":11166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":238208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.d","version":"1439'3972","reported_seq":3989,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.789947+0000","last_change":"2024-09-21T10:46:40.277839+0000","last_active":"2024-09-21T10:47:33.789947+0000","last_peered":"2024-09-21T10:47:33.789947+0000","last_clean":"2024-09-21T10:47:33.789947+0000","last_became_active":"2024-09-21T10:46:40.277565+0000","last_became_peered":"2024-09-21T10:46:40.277565+0000","last_unstale":"2024-09-21T10:47:33.789947+0000","last_undegraded":"2024-09-21T10:47:33.789947+0000","last_fullsized":"2024-09-21T10:47:33.789947+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3972,"log_dups_size":0,"ondisk_log_size":3972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:03.176402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260308992,"num_objects":3972,"num_object_clones":0,"num_object_copies":11916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3972,"num_write_kb":254208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.c","version":"1439'3923","reported_seq":3952,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.014469+0000","last_change":"2024-09-21T10:46:40.274270+0000","last_active":"2024-09-21T10:47:33.014469+0000","last_peered":"2024-09-21T10:47:33.014469+0000","last_clean":"2024-09-21T10:47:33.014469+0000","last_became_active":"2024-09-21T10:46:40.273989+0000","last_became_peered":"2024-09-21T10:46:40.273989+0000","last_unstale":"2024-09-21T10:47:33.014469+0000","last_undegraded":"2024-09-21T10:47:33.014469+0000","last_fullsized":"2024-09-21T10:47:33.014469+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3923,"log_dups_size":0,"ondisk_log_size":3923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.938584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257097728,"num_objects":3923,"num_object_clones":0,"num_object_copies":11769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3923,"num_write_kb":251072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.b","version":"1439'4054","reported_seq":4071,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.741527+0000","last_change":"2024-09-21T10:46:40.275045+0000","last_active":"2024-09-21T10:47:33.741527+0000","last_peered":"2024-09-21T10:47:33.741527+0000","last_clean":"2024-09-21T10:47:33.741527+0000","last_became_active":"2024-09-21T10:46:40.274835+0000","last_became_peered":"2024-09-21T10:46:40.274835+0000","last_unstale":"2024-09-21T10:47:33.741527+0000","last_undegraded":"2024-09-21T10:47:33.741527+0000","last_fullsized":"2024-09-21T10:47:33.741527+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4054,"log_dups_size":0,"ondisk_log_size":4054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:59:51.355069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265682944,"num_objects":4054,"num_object_clones":0,"num_object_copies":12162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":259456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.a","version":"1439'3901","reported_seq":3930,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.060901+0000","last_change":"2024-09-21T10:46:40.276509+0000","last_active":"2024-09-21T10:47:33.060901+0000","last_peered":"2024-09-21T10:47:33.060901+0000","last_clean":"2024-09-21T10:47:33.060901+0000","last_became_active":"2024-09-21T10:46:40.276211+0000","last_became_peered":"2024-09-21T10:46:40.276211+0000","last_unstale":"2024-09-21T10:47:33.060901+0000","last_undegraded":"2024-09-21T10:47:33.060901+0000","last_fullsized":"2024-09-21T10:47:33.060901+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:56.745364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255655936,"num_objects":3901,"num_object_clones":0,"num_object_copies":11703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":249664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.9","version":"1439'3994","reported_seq":4023,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.069061+0000","last_change":"2024-09-21T10:46:40.276505+0000","last_active":"2024-09-21T10:47:33.069061+0000","last_peered":"2024-09-21T10:47:33.069061+0000","last_clean":"2024-09-21T10:47:33.069061+0000","last_became_active":"2024-09-21T10:46:40.275398+0000","last_became_peered":"2024-09-21T10:46:40.275398+0000","last_unstale":"2024-09-21T10:47:33.069061+0000","last_undegraded":"2024-09-21T10:47:33.069061+0000","last_fullsized":"2024-09-21T10:47:33.069061+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3994,"log_dups_size":0,"ondisk_log_size":3994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:47.276665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261750784,"num_objects":3994,"num_object_clones":0,"num_object_copies":11982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":255616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.8","version":"1439'3702","reported_seq":3731,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:29.997216+0000","last_change":"2024-09-21T10:46:40.274571+0000","last_active":"2024-09-21T10:47:29.997216+0000","last_peered":"2024-09-21T10:47:29.997216+0000","last_clean":"2024-09-21T10:47:29.997216+0000","last_became_active":"2024-09-21T10:46:40.274285+0000","last_became_peered":"2024-09-21T10:46:40.274285+0000","last_unstale":"2024-09-21T10:47:29.997216+0000","last_undegraded":"2024-09-21T10:47:29.997216+0000","last_fullsized":"2024-09-21T10:47:29.997216+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:51:55.260163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242614272,"num_objects":3702,"num_object_clones":0,"num_object_copies":11106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":236928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.7","version":"1439'3958","reported_seq":3987,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.061573+0000","last_change":"2024-09-21T10:46:40.243879+0000","last_active":"2024-09-21T10:47:33.061573+0000","last_peered":"2024-09-21T10:47:33.061573+0000","last_clean":"2024-09-21T10:47:33.061573+0000","last_became_active":"2024-09-21T10:46:40.243579+0000","last_became_peered":"2024-09-21T10:46:40.243579+0000","last_unstale":"2024-09-21T10:47:33.061573+0000","last_undegraded":"2024-09-21T10:47:33.061573+0000","last_fullsized":"2024-09-21T10:47:33.061573+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3958,"log_dups_size":0,"ondisk_log_size":3958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:33.317643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259391488,"num_objects":3958,"num_object_clones":0,"num_object_copies":11874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3958,"num_write_kb":253312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.6","version":"1439'3680","reported_seq":3708,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:30.024769+0000","last_change":"2024-09-21T10:46:40.275529+0000","last_active":"2024-09-21T10:47:30.024769+0000","last_peered":"2024-09-21T10:47:30.024769+0000","last_clean":"2024-09-21T10:47:30.024769+0000","last_became_active":"2024-09-21T10:46:40.275255+0000","last_became_peered":"2024-09-21T10:46:40.275255+0000","last_unstale":"2024-09-21T10:47:30.024769+0000","last_undegraded":"2024-09-21T10:47:30.024769+0000","last_fullsized":"2024-09-21T10:47:30.024769+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3680,"log_dups_size":0,"ondisk_log_size":3680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:08.150110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241172480,"num_objects":3680,"num_object_clones":0,"num_object_copies":11040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3680,"num_write_kb":235520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.5","version":"1439'4014","reported_seq":4031,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.776038+0000","last_change":"2024-09-21T10:46:40.243663+0000","last_active":"2024-09-21T10:47:33.776038+0000","last_peered":"2024-09-21T10:47:33.776038+0000","last_clean":"2024-09-21T10:47:33.776038+0000","last_became_active":"2024-09-21T10:46:40.243417+0000","last_became_peered":"2024-09-21T10:46:40.243417+0000","last_unstale":"2024-09-21T10:47:33.776038+0000","last_undegraded":"2024-09-21T10:47:33.776038+0000","last_fullsized":"2024-09-21T10:47:33.776038+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4014,"log_dups_size":0,"ondisk_log_size":4014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:42.500241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263061504,"num_objects":4014,"num_object_clones":0,"num_object_copies":12042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4014,"num_write_kb":256896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.4","version":"1439'4043","reported_seq":4060,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.775870+0000","last_change":"2024-09-21T10:46:40.277620+0000","last_active":"2024-09-21T10:47:33.775870+0000","last_peered":"2024-09-21T10:47:33.775870+0000","last_clean":"2024-09-21T10:47:33.775870+0000","last_became_active":"2024-09-21T10:46:40.277426+0000","last_became_peered":"2024-09-21T10:46:40.277426+0000","last_unstale":"2024-09-21T10:47:33.775870+0000","last_undegraded":"2024-09-21T10:47:33.775870+0000","last_fullsized":"2024-09-21T10:47:33.775870+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.926675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264962048,"num_objects":4043,"num_object_clones":0,"num_object_copies":12129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":258752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.3","version":"1439'3865","reported_seq":3892,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:30.024043+0000","last_change":"2024-09-21T10:46:40.275398+0000","last_active":"2024-09-21T10:47:30.024043+0000","last_peered":"2024-09-21T10:47:30.024043+0000","last_clean":"2024-09-21T10:47:30.024043+0000","last_became_active":"2024-09-21T10:46:40.275096+0000","last_became_peered":"2024-09-21T10:46:40.275096+0000","last_unstale":"2024-09-21T10:47:30.024043+0000","last_undegraded":"2024-09-21T10:47:30.024043+0000","last_fullsized":"2024-09-21T10:47:30.024043+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:58.484291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":253296640,"num_objects":3865,"num_object_clones":0,"num_object_copies":11595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3865,"num_write_kb":247360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.2","version":"1439'3811","reported_seq":3839,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:31.359680+0000","last_change":"2024-09-21T10:46:40.274967+0000","last_active":"2024-09-21T10:47:31.359680+0000","last_peered":"2024-09-21T10:47:31.359680+0000","last_clean":"2024-09-21T10:47:31.359680+0000","last_became_active":"2024-09-21T10:46:40.274362+0000","last_became_peered":"2024-09-21T10:46:40.274362+0000","last_unstale":"2024-09-21T10:47:31.359680+0000","last_undegraded":"2024-09-21T10:47:31.359680+0000","last_fullsized":"2024-09-21T10:47:31.359680+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3811,"log_dups_size":0,"ondisk_log_size":3811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.876058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249757696,"num_objects":3811,"num_object_clones":0,"num_object_copies":11433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3811,"num_write_kb":243904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.1","version":"1439'3963","reported_seq":3992,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.073864+0000","last_change":"2024-09-21T10:46:40.276794+0000","last_active":"2024-09-21T10:47:33.073864+0000","last_peered":"2024-09-21T10:47:33.073864+0000","last_clean":"2024-09-21T10:47:33.073864+0000","last_became_active":"2024-09-21T10:46:40.276137+0000","last_became_peered":"2024-09-21T10:46:40.276137+0000","last_unstale":"2024-09-21T10:47:33.073864+0000","last_undegraded":"2024-09-21T10:47:33.073864+0000","last_fullsized":"2024-09-21T10:47:33.073864+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":3963,"log_dups_size":0,"ondisk_log_size":3963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:01.998423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259719168,"num_objects":3963,"num_object_clones":0,"num_object_copies":11889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3963,"num_write_kb":253632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.0","version":"1439'4019","reported_seq":4036,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:33.780327+0000","last_change":"2024-09-21T10:46:40.241307+0000","last_active":"2024-09-21T10:47:33.780327+0000","last_peered":"2024-09-21T10:47:33.780327+0000","last_clean":"2024-09-21T10:47:33.780327+0000","last_became_active":"2024-09-21T10:46:40.241061+0000","last_became_peered":"2024-09-21T10:46:40.241061+0000","last_unstale":"2024-09-21T10:47:33.780327+0000","last_undegraded":"2024-09-21T10:47:33.780327+0000","last_fullsized":"2024-09-21T10:47:33.780327+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:41.775058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263389184,"num_objects":4019,"num_object_clones":0,"num_object_copies":12057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4019,"num_write_kb":257216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":166,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187408+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:47:22.187408+0000","last_peered":"2024-09-21T10:47:22.187408+0000","last_clean":"2024-09-21T10:47:22.187408+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:47:22.187408+0000","last_undegraded":"2024-09-21T10:47:22.187408+0000","last_fullsized":"2024-09-21T10:47:22.187408+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1290,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181164+0000","last_change":"2024-09-21T10:43:56.011543+0000","last_active":"2024-09-21T10:47:22.181164+0000","last_peered":"2024-09-21T10:47:22.181164+0000","last_clean":"2024-09-21T10:47:22.181164+0000","last_became_active":"2024-09-21T10:43:56.010878+0000","last_became_peered":"2024-09-21T10:43:56.010878+0000","last_unstale":"2024-09-21T10:47:22.181164+0000","last_undegraded":"2024-09-21T10:47:22.181164+0000","last_fullsized":"2024-09-21T10:47:22.181164+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:46.231617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2203,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.192336+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:47:22.192336+0000","last_peered":"2024-09-21T10:47:22.192336+0000","last_clean":"2024-09-21T10:47:22.192336+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:47:22.192336+0000","last_undegraded":"2024-09-21T10:47:22.192336+0000","last_fullsized":"2024-09-21T10:47:22.192336+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2220,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.193275+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:47:22.193275+0000","last_peered":"2024-09-21T10:47:22.193275+0000","last_clean":"2024-09-21T10:47:22.193275+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:47:22.193275+0000","last_undegraded":"2024-09-21T10:47:22.193275+0000","last_fullsized":"2024-09-21T10:47:22.193275+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":207,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128158+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:47:15.128158+0000","last_peered":"2024-09-21T10:47:15.128158+0000","last_clean":"2024-09-21T10:47:15.128158+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:47:15.128158+0000","last_undegraded":"2024-09-21T10:47:15.128158+0000","last_fullsized":"2024-09-21T10:47:15.128158+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1404,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.196617+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:47:22.196617+0000","last_peered":"2024-09-21T10:47:22.196617+0000","last_clean":"2024-09-21T10:47:22.196617+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:47:22.196617+0000","last_undegraded":"2024-09-21T10:47:22.196617+0000","last_fullsized":"2024-09-21T10:47:22.196617+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":207,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.127118+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:47:15.127118+0000","last_peered":"2024-09-21T10:47:15.127118+0000","last_clean":"2024-09-21T10:47:15.127118+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:47:15.127118+0000","last_undegraded":"2024-09-21T10:47:15.127118+0000","last_fullsized":"2024-09-21T10:47:15.127118+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1491,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.196738+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:47:22.196738+0000","last_peered":"2024-09-21T10:47:22.196738+0000","last_clean":"2024-09-21T10:47:22.196738+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:47:22.196738+0000","last_undegraded":"2024-09-21T10:47:22.196738+0000","last_fullsized":"2024-09-21T10:47:22.196738+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":30,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.180086+0000","last_change":"2024-09-21T10:46:39.241906+0000","last_active":"2024-09-21T10:47:22.180086+0000","last_peered":"2024-09-21T10:47:22.180086+0000","last_clean":"2024-09-21T10:47:22.180086+0000","last_became_active":"2024-09-21T10:46:39.241499+0000","last_became_peered":"2024-09-21T10:46:39.241499+0000","last_unstale":"2024-09-21T10:47:22.180086+0000","last_undegraded":"2024-09-21T10:47:22.180086+0000","last_fullsized":"2024-09-21T10:47:22.180086+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:53.671658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":187,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.179460+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:47:22.179460+0000","last_peered":"2024-09-21T10:47:22.179460+0000","last_clean":"2024-09-21T10:47:22.179460+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:47:22.179460+0000","last_undegraded":"2024-09-21T10:47:22.179460+0000","last_fullsized":"2024-09-21T10:47:22.179460+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":263,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186618+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:47:22.186618+0000","last_peered":"2024-09-21T10:47:22.186618+0000","last_clean":"2024-09-21T10:47:22.186618+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:47:22.186618+0000","last_undegraded":"2024-09-21T10:47:22.186618+0000","last_fullsized":"2024-09-21T10:47:22.186618+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2210,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191819+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:47:22.191819+0000","last_peered":"2024-09-21T10:47:22.191819+0000","last_clean":"2024-09-21T10:47:22.191819+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:47:22.191819+0000","last_undegraded":"2024-09-21T10:47:22.191819+0000","last_fullsized":"2024-09-21T10:47:22.191819+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.175005+0000","last_change":"2024-09-21T10:46:39.241669+0000","last_active":"2024-09-21T10:47:22.175005+0000","last_peered":"2024-09-21T10:47:22.175005+0000","last_clean":"2024-09-21T10:47:22.175005+0000","last_became_active":"2024-09-21T10:46:39.241358+0000","last_became_peered":"2024-09-21T10:46:39.241358+0000","last_unstale":"2024-09-21T10:47:22.175005+0000","last_undegraded":"2024-09-21T10:47:22.175005+0000","last_fullsized":"2024-09-21T10:47:22.175005+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:25.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2325,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187360+0000","last_change":"2024-09-21T10:45:59.689522+0000","last_active":"2024-09-21T10:47:22.187360+0000","last_peered":"2024-09-21T10:47:22.187360+0000","last_clean":"2024-09-21T10:47:22.187360+0000","last_became_active":"2024-09-21T10:45:59.689148+0000","last_became_peered":"2024-09-21T10:45:59.689148+0000","last_unstale":"2024-09-21T10:47:22.187360+0000","last_undegraded":"2024-09-21T10:47:22.187360+0000","last_fullsized":"2024-09-21T10:47:22.187360+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:28.142643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2235,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.189647+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:47:22.189647+0000","last_peered":"2024-09-21T10:47:22.189647+0000","last_clean":"2024-09-21T10:47:22.189647+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:47:22.189647+0000","last_undegraded":"2024-09-21T10:47:22.189647+0000","last_fullsized":"2024-09-21T10:47:22.189647+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1253,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.127424+0000","last_change":"2024-09-21T10:43:56.010613+0000","last_active":"2024-09-21T10:47:15.127424+0000","last_peered":"2024-09-21T10:47:15.127424+0000","last_clean":"2024-09-21T10:47:15.127424+0000","last_became_active":"2024-09-21T10:43:56.010229+0000","last_became_peered":"2024-09-21T10:43:56.010229+0000","last_unstale":"2024-09-21T10:47:15.127424+0000","last_undegraded":"2024-09-21T10:47:15.127424+0000","last_fullsized":"2024-09-21T10:47:15.127424+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:28.116950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2215,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.189915+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:47:22.189915+0000","last_peered":"2024-09-21T10:47:22.189915+0000","last_clean":"2024-09-21T10:47:22.189915+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:47:22.189915+0000","last_undegraded":"2024-09-21T10:47:22.189915+0000","last_fullsized":"2024-09-21T10:47:22.189915+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2328,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187522+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:47:22.187522+0000","last_peered":"2024-09-21T10:47:22.187522+0000","last_clean":"2024-09-21T10:47:22.187522+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:47:22.187522+0000","last_undegraded":"2024-09-21T10:47:22.187522+0000","last_fullsized":"2024-09-21T10:47:22.187522+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":352,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.179701+0000","last_change":"2024-09-21T10:45:59.680994+0000","last_active":"2024-09-21T10:47:22.179701+0000","last_peered":"2024-09-21T10:47:22.179701+0000","last_clean":"2024-09-21T10:47:22.179701+0000","last_became_active":"2024-09-21T10:45:59.680636+0000","last_became_peered":"2024-09-21T10:45:59.680636+0000","last_unstale":"2024-09-21T10:47:22.179701+0000","last_undegraded":"2024-09-21T10:47:22.179701+0000","last_fullsized":"2024-09-21T10:47:22.179701+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:42.488110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2205,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.193531+0000","last_change":"2024-09-21T10:45:59.681544+0000","last_active":"2024-09-21T10:47:22.193531+0000","last_peered":"2024-09-21T10:47:22.193531+0000","last_clean":"2024-09-21T10:47:22.193531+0000","last_became_active":"2024-09-21T10:45:59.681218+0000","last_became_peered":"2024-09-21T10:45:59.681218+0000","last_unstale":"2024-09-21T10:47:22.193531+0000","last_undegraded":"2024-09-21T10:47:22.193531+0000","last_fullsized":"2024-09-21T10:47:22.193531+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:13.323548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":199,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.126922+0000","last_change":"2024-09-21T10:45:59.688959+0000","last_active":"2024-09-21T10:47:15.126922+0000","last_peered":"2024-09-21T10:47:15.126922+0000","last_clean":"2024-09-21T10:47:15.126922+0000","last_became_active":"2024-09-21T10:45:59.688647+0000","last_became_peered":"2024-09-21T10:45:59.688647+0000","last_unstale":"2024-09-21T10:47:15.126922+0000","last_undegraded":"2024-09-21T10:47:15.126922+0000","last_fullsized":"2024-09-21T10:47:15.126922+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:28.180582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":97,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191675+0000","last_change":"2024-09-21T10:45:59.710479+0000","last_active":"2024-09-21T10:47:22.191675+0000","last_peered":"2024-09-21T10:47:22.191675+0000","last_clean":"2024-09-21T10:47:22.191675+0000","last_became_active":"2024-09-21T10:45:59.694474+0000","last_became_peered":"2024-09-21T10:45:59.694474+0000","last_unstale":"2024-09-21T10:47:22.191675+0000","last_undegraded":"2024-09-21T10:47:22.191675+0000","last_fullsized":"2024-09-21T10:47:22.191675+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:32.554219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":37,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187607+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:47:22.187607+0000","last_peered":"2024-09-21T10:47:22.187607+0000","last_clean":"2024-09-21T10:47:22.187607+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:47:22.187607+0000","last_undegraded":"2024-09-21T10:47:22.187607+0000","last_fullsized":"2024-09-21T10:47:22.187607+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":392,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.179554+0000","last_change":"2024-09-21T10:45:59.705879+0000","last_active":"2024-09-21T10:47:22.179554+0000","last_peered":"2024-09-21T10:47:22.179554+0000","last_clean":"2024-09-21T10:47:22.179554+0000","last_became_active":"2024-09-21T10:45:59.689786+0000","last_became_peered":"2024-09-21T10:45:59.689786+0000","last_unstale":"2024-09-21T10:47:22.179554+0000","last_undegraded":"2024-09-21T10:47:22.179554+0000","last_fullsized":"2024-09-21T10:47:22.179554+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:50.473399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1254,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187064+0000","last_change":"2024-09-21T10:43:56.007984+0000","last_active":"2024-09-21T10:47:22.187064+0000","last_peered":"2024-09-21T10:47:22.187064+0000","last_clean":"2024-09-21T10:47:22.187064+0000","last_became_active":"2024-09-21T10:43:56.007667+0000","last_became_peered":"2024-09-21T10:43:56.007667+0000","last_unstale":"2024-09-21T10:47:22.187064+0000","last_undegraded":"2024-09-21T10:47:22.187064+0000","last_fullsized":"2024-09-21T10:47:22.187064+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:32.246512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":473,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.192005+0000","last_change":"2024-09-21T10:46:05.453578+0000","last_active":"2024-09-21T10:47:22.192005+0000","last_peered":"2024-09-21T10:47:22.192005+0000","last_clean":"2024-09-21T10:47:22.192005+0000","last_became_active":"2024-09-21T10:45:59.689841+0000","last_became_peered":"2024-09-21T10:45:59.689841+0000","last_unstale":"2024-09-21T10:47:22.192005+0000","last_undegraded":"2024-09-21T10:47:22.192005+0000","last_fullsized":"2024-09-21T10:47:22.192005+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:24.488516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1332,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.126725+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:47:15.126725+0000","last_peered":"2024-09-21T10:47:15.126725+0000","last_clean":"2024-09-21T10:47:15.126725+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:47:15.126725+0000","last_undegraded":"2024-09-21T10:47:15.126725+0000","last_fullsized":"2024-09-21T10:47:15.126725+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1220,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181130+0000","last_change":"2024-09-21T10:45:59.697070+0000","last_active":"2024-09-21T10:47:22.181130+0000","last_peered":"2024-09-21T10:47:22.181130+0000","last_clean":"2024-09-21T10:47:22.181130+0000","last_became_active":"2024-09-21T10:45:59.696548+0000","last_became_peered":"2024-09-21T10:45:59.696548+0000","last_unstale":"2024-09-21T10:47:22.181130+0000","last_undegraded":"2024-09-21T10:47:22.181130+0000","last_fullsized":"2024-09-21T10:47:22.181130+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:02.746989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":203,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187103+0000","last_change":"2024-09-21T10:45:59.705691+0000","last_active":"2024-09-21T10:47:22.187103+0000","last_peered":"2024-09-21T10:47:22.187103+0000","last_clean":"2024-09-21T10:47:22.187103+0000","last_became_active":"2024-09-21T10:45:59.699486+0000","last_became_peered":"2024-09-21T10:45:59.699486+0000","last_unstale":"2024-09-21T10:47:22.187103+0000","last_undegraded":"2024-09-21T10:47:22.187103+0000","last_fullsized":"2024-09-21T10:47:22.187103+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:41.452622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":247,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.176441+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:47:22.176441+0000","last_peered":"2024-09-21T10:47:22.176441+0000","last_clean":"2024-09-21T10:47:22.176441+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:47:22.176441+0000","last_undegraded":"2024-09-21T10:47:22.176441+0000","last_fullsized":"2024-09-21T10:47:22.176441+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":105,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128659+0000","last_change":"2024-09-21T10:45:59.693387+0000","last_active":"2024-09-21T10:47:15.128659+0000","last_peered":"2024-09-21T10:47:15.128659+0000","last_clean":"2024-09-21T10:47:15.128659+0000","last_became_active":"2024-09-21T10:45:59.692983+0000","last_became_peered":"2024-09-21T10:45:59.692983+0000","last_unstale":"2024-09-21T10:47:15.128659+0000","last_undegraded":"2024-09-21T10:47:15.128659+0000","last_fullsized":"2024-09-21T10:47:15.128659+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:05.623869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2317,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186730+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:47:22.186730+0000","last_peered":"2024-09-21T10:47:22.186730+0000","last_clean":"2024-09-21T10:47:22.186730+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:47:22.186730+0000","last_undegraded":"2024-09-21T10:47:22.186730+0000","last_fullsized":"2024-09-21T10:47:22.186730+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":34,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.180852+0000","last_change":"2024-09-21T10:46:39.246034+0000","last_active":"2024-09-21T10:47:22.180852+0000","last_peered":"2024-09-21T10:47:22.180852+0000","last_clean":"2024-09-21T10:47:22.180852+0000","last_became_active":"2024-09-21T10:46:39.243647+0000","last_became_peered":"2024-09-21T10:46:39.243647+0000","last_unstale":"2024-09-21T10:47:22.180852+0000","last_undegraded":"2024-09-21T10:47:22.180852+0000","last_fullsized":"2024-09-21T10:47:22.180852+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:04.275079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":401,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187196+0000","last_change":"2024-09-21T10:43:58.555202+0000","last_active":"2024-09-21T10:47:22.187196+0000","last_peered":"2024-09-21T10:47:22.187196+0000","last_clean":"2024-09-21T10:47:22.187196+0000","last_became_active":"2024-09-21T10:43:56.009648+0000","last_became_peered":"2024-09-21T10:43:56.009648+0000","last_unstale":"2024-09-21T10:47:22.187196+0000","last_undegraded":"2024-09-21T10:47:22.187196+0000","last_fullsized":"2024-09-21T10:47:22.187196+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.555120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.555120+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:45:37.803817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2327,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186635+0000","last_change":"2024-09-21T10:45:59.693562+0000","last_active":"2024-09-21T10:47:22.186635+0000","last_peered":"2024-09-21T10:47:22.186635+0000","last_clean":"2024-09-21T10:47:22.186635+0000","last_became_active":"2024-09-21T10:45:59.693223+0000","last_became_peered":"2024-09-21T10:45:59.693223+0000","last_unstale":"2024-09-21T10:47:22.186635+0000","last_undegraded":"2024-09-21T10:47:22.186635+0000","last_fullsized":"2024-09-21T10:47:22.186635+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:59.666534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1418,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186925+0000","last_change":"2024-09-21T10:45:59.694166+0000","last_active":"2024-09-21T10:47:22.186925+0000","last_peered":"2024-09-21T10:47:22.186925+0000","last_clean":"2024-09-21T10:47:22.186925+0000","last_became_active":"2024-09-21T10:45:59.693276+0000","last_became_peered":"2024-09-21T10:45:59.693276+0000","last_unstale":"2024-09-21T10:47:22.186925+0000","last_undegraded":"2024-09-21T10:47:22.186925+0000","last_fullsized":"2024-09-21T10:47:22.186925+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:12.784743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2305,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187179+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:47:22.187179+0000","last_peered":"2024-09-21T10:47:22.187179+0000","last_clean":"2024-09-21T10:47:22.187179+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:47:22.187179+0000","last_undegraded":"2024-09-21T10:47:22.187179+0000","last_fullsized":"2024-09-21T10:47:22.187179+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":412,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191956+0000","last_change":"2024-09-21T10:45:59.694509+0000","last_active":"2024-09-21T10:47:22.191956+0000","last_peered":"2024-09-21T10:47:22.191956+0000","last_clean":"2024-09-21T10:47:22.191956+0000","last_became_active":"2024-09-21T10:45:59.693089+0000","last_became_peered":"2024-09-21T10:45:59.693089+0000","last_unstale":"2024-09-21T10:47:22.191956+0000","last_undegraded":"2024-09-21T10:47:22.191956+0000","last_fullsized":"2024-09-21T10:47:22.191956+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.698401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.698401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:29.691153+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2310,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186870+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:47:22.186870+0000","last_peered":"2024-09-21T10:47:22.186870+0000","last_clean":"2024-09-21T10:47:22.186870+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:47:22.186870+0000","last_undegraded":"2024-09-21T10:47:22.186870+0000","last_fullsized":"2024-09-21T10:47:22.186870+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2312,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187577+0000","last_change":"2024-09-21T10:45:59.691645+0000","last_active":"2024-09-21T10:47:22.187577+0000","last_peered":"2024-09-21T10:47:22.187577+0000","last_clean":"2024-09-21T10:47:22.187577+0000","last_became_active":"2024-09-21T10:45:59.691334+0000","last_became_peered":"2024-09-21T10:45:59.691334+0000","last_unstale":"2024-09-21T10:47:22.187577+0000","last_undegraded":"2024-09-21T10:47:22.187577+0000","last_fullsized":"2024-09-21T10:47:22.187577+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:33.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":189,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128085+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:47:15.128085+0000","last_peered":"2024-09-21T10:47:15.128085+0000","last_clean":"2024-09-21T10:47:15.128085+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:47:15.128085+0000","last_undegraded":"2024-09-21T10:47:15.128085+0000","last_fullsized":"2024-09-21T10:47:15.128085+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":199,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.126843+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:47:15.126843+0000","last_peered":"2024-09-21T10:47:15.126843+0000","last_clean":"2024-09-21T10:47:15.126843+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:47:15.126843+0000","last_undegraded":"2024-09-21T10:47:15.126843+0000","last_fullsized":"2024-09-21T10:47:15.126843+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2302,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187259+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:47:22.187259+0000","last_peered":"2024-09-21T10:47:22.187259+0000","last_clean":"2024-09-21T10:47:22.187259+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:47:22.187259+0000","last_undegraded":"2024-09-21T10:47:22.187259+0000","last_fullsized":"2024-09-21T10:47:22.187259+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1423,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186550+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:47:22.186550+0000","last_peered":"2024-09-21T10:47:22.186550+0000","last_clean":"2024-09-21T10:47:22.186550+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:47:22.186550+0000","last_undegraded":"2024-09-21T10:47:22.186550+0000","last_fullsized":"2024-09-21T10:47:22.186550+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":193,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.126921+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:47:15.126921+0000","last_peered":"2024-09-21T10:47:15.126921+0000","last_clean":"2024-09-21T10:47:15.126921+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:47:15.126921+0000","last_undegraded":"2024-09-21T10:47:15.126921+0000","last_fullsized":"2024-09-21T10:47:15.126921+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":633,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186291+0000","last_change":"2024-09-21T10:45:59.693657+0000","last_active":"2024-09-21T10:47:22.186291+0000","last_peered":"2024-09-21T10:47:22.186291+0000","last_clean":"2024-09-21T10:47:22.186291+0000","last_became_active":"2024-09-21T10:45:59.693283+0000","last_became_peered":"2024-09-21T10:45:59.693283+0000","last_unstale":"2024-09-21T10:47:22.186291+0000","last_undegraded":"2024-09-21T10:47:22.186291+0000","last_fullsized":"2024-09-21T10:47:22.186291+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:26.099521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":37,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191309+0000","last_change":"2024-09-21T10:46:30.264517+0000","last_active":"2024-09-21T10:47:22.191309+0000","last_peered":"2024-09-21T10:47:22.191309+0000","last_clean":"2024-09-21T10:47:22.191309+0000","last_became_active":"2024-09-21T10:46:30.264122+0000","last_became_peered":"2024-09-21T10:46:30.264122+0000","last_unstale":"2024-09-21T10:47:22.191309+0000","last_undegraded":"2024-09-21T10:47:22.191309+0000","last_fullsized":"2024-09-21T10:47:22.191309+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:23.130946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2303,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186379+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:47:22.186379+0000","last_peered":"2024-09-21T10:47:22.186379+0000","last_clean":"2024-09-21T10:47:22.186379+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:47:22.186379+0000","last_undegraded":"2024-09-21T10:47:22.186379+0000","last_fullsized":"2024-09-21T10:47:22.186379+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2234,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.175263+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:47:22.175263+0000","last_peered":"2024-09-21T10:47:22.175263+0000","last_clean":"2024-09-21T10:47:22.175263+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:47:22.175263+0000","last_undegraded":"2024-09-21T10:47:22.175263+0000","last_fullsized":"2024-09-21T10:47:22.175263+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":488,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186916+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:47:22.186916+0000","last_peered":"2024-09-21T10:47:22.186916+0000","last_clean":"2024-09-21T10:47:22.186916+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:47:22.186916+0000","last_undegraded":"2024-09-21T10:47:22.186916+0000","last_fullsized":"2024-09-21T10:47:22.186916+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":159,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186950+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:47:22.186950+0000","last_peered":"2024-09-21T10:47:22.186950+0000","last_clean":"2024-09-21T10:47:22.186950+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:47:22.186950+0000","last_undegraded":"2024-09-21T10:47:22.186950+0000","last_fullsized":"2024-09-21T10:47:22.186950+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2335,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186561+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:47:22.186561+0000","last_peered":"2024-09-21T10:47:22.186561+0000","last_clean":"2024-09-21T10:47:22.186561+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:47:22.186561+0000","last_undegraded":"2024-09-21T10:47:22.186561+0000","last_fullsized":"2024-09-21T10:47:22.186561+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":580,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186378+0000","last_change":"2024-09-21T10:45:59.691940+0000","last_active":"2024-09-21T10:47:22.186378+0000","last_peered":"2024-09-21T10:47:22.186378+0000","last_clean":"2024-09-21T10:47:22.186378+0000","last_became_active":"2024-09-21T10:45:59.691610+0000","last_became_peered":"2024-09-21T10:45:59.691610+0000","last_unstale":"2024-09-21T10:47:22.186378+0000","last_undegraded":"2024-09-21T10:47:22.186378+0000","last_fullsized":"2024-09-21T10:47:22.186378+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:57.766019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1348,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187015+0000","last_change":"2024-09-21T10:46:46.607744+0000","last_active":"2024-09-21T10:47:22.187015+0000","last_peered":"2024-09-21T10:47:22.187015+0000","last_clean":"2024-09-21T10:47:22.187015+0000","last_became_active":"2024-09-21T10:46:46.607376+0000","last_became_peered":"2024-09-21T10:46:46.607376+0000","last_unstale":"2024-09-21T10:47:22.187015+0000","last_undegraded":"2024-09-21T10:47:22.187015+0000","last_fullsized":"2024-09-21T10:47:22.187015+0000","mapping_epoch":1433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:17.279053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2220,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.196733+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:47:22.196733+0000","last_peered":"2024-09-21T10:47:22.196733+0000","last_clean":"2024-09-21T10:47:22.196733+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:47:22.196733+0000","last_undegraded":"2024-09-21T10:47:22.196733+0000","last_fullsized":"2024-09-21T10:47:22.196733+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":210,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.127147+0000","last_change":"2024-09-21T10:45:59.698368+0000","last_active":"2024-09-21T10:47:15.127147+0000","last_peered":"2024-09-21T10:47:15.127147+0000","last_clean":"2024-09-21T10:47:15.127147+0000","last_became_active":"2024-09-21T10:45:59.684909+0000","last_became_peered":"2024-09-21T10:45:59.684909+0000","last_unstale":"2024-09-21T10:47:15.127147+0000","last_undegraded":"2024-09-21T10:47:15.127147+0000","last_fullsized":"2024-09-21T10:47:15.127147+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:09.098971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1571,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.185240+0000","last_change":"2024-09-21T10:45:59.692592+0000","last_active":"2024-09-21T10:47:22.185240+0000","last_peered":"2024-09-21T10:47:22.185240+0000","last_clean":"2024-09-21T10:47:22.185240+0000","last_became_active":"2024-09-21T10:45:59.692232+0000","last_became_peered":"2024-09-21T10:45:59.692232+0000","last_unstale":"2024-09-21T10:47:22.185240+0000","last_undegraded":"2024-09-21T10:47:22.185240+0000","last_fullsized":"2024-09-21T10:47:22.185240+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:26.920879+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":189,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.127011+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:47:15.127011+0000","last_peered":"2024-09-21T10:47:15.127011+0000","last_clean":"2024-09-21T10:47:15.127011+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:47:15.127011+0000","last_undegraded":"2024-09-21T10:47:15.127011+0000","last_fullsized":"2024-09-21T10:47:15.127011+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2315,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191522+0000","last_change":"2024-09-21T10:45:59.697853+0000","last_active":"2024-09-21T10:47:22.191522+0000","last_peered":"2024-09-21T10:47:22.191522+0000","last_clean":"2024-09-21T10:47:22.191522+0000","last_became_active":"2024-09-21T10:45:59.697402+0000","last_became_peered":"2024-09-21T10:45:59.697402+0000","last_unstale":"2024-09-21T10:47:22.191522+0000","last_undegraded":"2024-09-21T10:47:22.191522+0000","last_fullsized":"2024-09-21T10:47:22.191522+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:20.734347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2213,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.189762+0000","last_change":"2024-09-21T10:45:59.689063+0000","last_active":"2024-09-21T10:47:22.189762+0000","last_peered":"2024-09-21T10:47:22.189762+0000","last_clean":"2024-09-21T10:47:22.189762+0000","last_became_active":"2024-09-21T10:45:59.688749+0000","last_became_peered":"2024-09-21T10:45:59.688749+0000","last_unstale":"2024-09-21T10:47:22.189762+0000","last_undegraded":"2024-09-21T10:47:22.189762+0000","last_fullsized":"2024-09-21T10:47:22.189762+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:59.204604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":471,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186820+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:47:22.186820+0000","last_peered":"2024-09-21T10:47:22.186820+0000","last_clean":"2024-09-21T10:47:22.186820+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:47:22.186820+0000","last_undegraded":"2024-09-21T10:47:22.186820+0000","last_fullsized":"2024-09-21T10:47:22.186820+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":37,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186296+0000","last_change":"2024-09-21T10:46:30.264653+0000","last_active":"2024-09-21T10:47:22.186296+0000","last_peered":"2024-09-21T10:47:22.186296+0000","last_clean":"2024-09-21T10:47:22.186296+0000","last_became_active":"2024-09-21T10:46:30.264068+0000","last_became_peered":"2024-09-21T10:46:30.264068+0000","last_unstale":"2024-09-21T10:47:22.186296+0000","last_undegraded":"2024-09-21T10:47:22.186296+0000","last_fullsized":"2024-09-21T10:47:22.186296+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:53.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1369,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.175338+0000","last_change":"2024-09-21T10:45:59.680814+0000","last_active":"2024-09-21T10:47:22.175338+0000","last_peered":"2024-09-21T10:47:22.175338+0000","last_clean":"2024-09-21T10:47:22.175338+0000","last_became_active":"2024-09-21T10:45:59.680499+0000","last_became_peered":"2024-09-21T10:45:59.680499+0000","last_unstale":"2024-09-21T10:47:22.175338+0000","last_undegraded":"2024-09-21T10:47:22.175338+0000","last_fullsized":"2024-09-21T10:47:22.175338+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:01.487484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":193,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128787+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:47:15.128787+0000","last_peered":"2024-09-21T10:47:15.128787+0000","last_clean":"2024-09-21T10:47:15.128787+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:47:15.128787+0000","last_undegraded":"2024-09-21T10:47:15.128787+0000","last_fullsized":"2024-09-21T10:47:15.128787+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1369,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.176589+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:47:22.176589+0000","last_peered":"2024-09-21T10:47:22.176589+0000","last_clean":"2024-09-21T10:47:22.176589+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:47:22.176589+0000","last_undegraded":"2024-09-21T10:47:22.176589+0000","last_fullsized":"2024-09-21T10:47:22.176589+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1361,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186772+0000","last_change":"2024-09-21T10:43:56.012844+0000","last_active":"2024-09-21T10:47:22.186772+0000","last_peered":"2024-09-21T10:47:22.186772+0000","last_clean":"2024-09-21T10:47:22.186772+0000","last_became_active":"2024-09-21T10:43:56.012504+0000","last_became_peered":"2024-09-21T10:43:56.012504+0000","last_unstale":"2024-09-21T10:47:22.186772+0000","last_undegraded":"2024-09-21T10:47:22.186772+0000","last_fullsized":"2024-09-21T10:47:22.186772+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:02.556342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1402,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.193127+0000","last_change":"2024-09-21T10:46:04.469943+0000","last_active":"2024-09-21T10:47:22.193127+0000","last_peered":"2024-09-21T10:47:22.193127+0000","last_clean":"2024-09-21T10:47:22.193127+0000","last_became_active":"2024-09-21T10:45:59.680550+0000","last_became_peered":"2024-09-21T10:45:59.680550+0000","last_unstale":"2024-09-21T10:47:22.193127+0000","last_undegraded":"2024-09-21T10:47:22.193127+0000","last_fullsized":"2024-09-21T10:47:22.193127+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:04.469883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:04.469883+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:08.642477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1421,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.192415+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:47:22.192415+0000","last_peered":"2024-09-21T10:47:22.192415+0000","last_clean":"2024-09-21T10:47:22.192415+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:47:22.192415+0000","last_undegraded":"2024-09-21T10:47:22.192415+0000","last_fullsized":"2024-09-21T10:47:22.192415+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":434,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.196625+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:47:22.196625+0000","last_peered":"2024-09-21T10:47:22.196625+0000","last_clean":"2024-09-21T10:47:22.196625+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:47:22.196625+0000","last_undegraded":"2024-09-21T10:47:22.196625+0000","last_fullsized":"2024-09-21T10:47:22.196625+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":386,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.192189+0000","last_change":"2024-09-21T10:46:08.467544+0000","last_active":"2024-09-21T10:47:22.192189+0000","last_peered":"2024-09-21T10:47:22.192189+0000","last_clean":"2024-09-21T10:47:22.192189+0000","last_became_active":"2024-09-21T10:45:59.695828+0000","last_became_peered":"2024-09-21T10:45:59.695828+0000","last_unstale":"2024-09-21T10:47:22.192189+0000","last_undegraded":"2024-09-21T10:47:22.192189+0000","last_fullsized":"2024-09-21T10:47:22.192189+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:08.467485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:08.467485+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:36.742481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2334,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186457+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:47:22.186457+0000","last_peered":"2024-09-21T10:47:22.186457+0000","last_clean":"2024-09-21T10:47:22.186457+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:47:22.186457+0000","last_undegraded":"2024-09-21T10:47:22.186457+0000","last_fullsized":"2024-09-21T10:47:22.186457+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":189,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.127049+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:47:15.127049+0000","last_peered":"2024-09-21T10:47:15.127049+0000","last_clean":"2024-09-21T10:47:15.127049+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:47:15.127049+0000","last_undegraded":"2024-09-21T10:47:15.127049+0000","last_fullsized":"2024-09-21T10:47:15.127049+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":452,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.127306+0000","last_change":"2024-09-21T10:44:00.317312+0000","last_active":"2024-09-21T10:47:15.127306+0000","last_peered":"2024-09-21T10:47:15.127306+0000","last_clean":"2024-09-21T10:47:15.127306+0000","last_became_active":"2024-09-21T10:43:56.006731+0000","last_became_peered":"2024-09-21T10:43:56.006731+0000","last_unstale":"2024-09-21T10:47:15.127306+0000","last_undegraded":"2024-09-21T10:47:15.127306+0000","last_fullsized":"2024-09-21T10:47:15.127306+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:47.520862+0000","scrub_duration":0.012369583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1203,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186954+0000","last_change":"2024-09-21T10:45:59.688833+0000","last_active":"2024-09-21T10:47:22.186954+0000","last_peered":"2024-09-21T10:47:22.186954+0000","last_clean":"2024-09-21T10:47:22.186954+0000","last_became_active":"2024-09-21T10:45:59.686751+0000","last_became_peered":"2024-09-21T10:45:59.686751+0000","last_unstale":"2024-09-21T10:47:22.186954+0000","last_undegraded":"2024-09-21T10:47:22.186954+0000","last_fullsized":"2024-09-21T10:47:22.186954+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:04.676103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1398,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128068+0000","last_change":"2024-09-21T10:45:59.694156+0000","last_active":"2024-09-21T10:47:15.128068+0000","last_peered":"2024-09-21T10:47:15.128068+0000","last_clean":"2024-09-21T10:47:15.128068+0000","last_became_active":"2024-09-21T10:45:59.686496+0000","last_became_peered":"2024-09-21T10:45:59.686496+0000","last_unstale":"2024-09-21T10:47:15.128068+0000","last_undegraded":"2024-09-21T10:47:15.128068+0000","last_fullsized":"2024-09-21T10:47:15.128068+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:55.832764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":472,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191714+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:47:22.191714+0000","last_peered":"2024-09-21T10:47:22.191714+0000","last_clean":"2024-09-21T10:47:22.191714+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:47:22.191714+0000","last_undegraded":"2024-09-21T10:47:22.191714+0000","last_fullsized":"2024-09-21T10:47:22.191714+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1404,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.191096+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:47:22.191096+0000","last_peered":"2024-09-21T10:47:22.191096+0000","last_clean":"2024-09-21T10:47:22.191096+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:47:22.191096+0000","last_undegraded":"2024-09-21T10:47:22.191096+0000","last_fullsized":"2024-09-21T10:47:22.191096+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":71,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186577+0000","last_change":"2024-09-21T10:45:59.707648+0000","last_active":"2024-09-21T10:47:22.186577+0000","last_peered":"2024-09-21T10:47:22.186577+0000","last_clean":"2024-09-21T10:47:22.186577+0000","last_became_active":"2024-09-21T10:45:59.691996+0000","last_became_peered":"2024-09-21T10:45:59.691996+0000","last_unstale":"2024-09-21T10:47:22.186577+0000","last_undegraded":"2024-09-21T10:47:22.186577+0000","last_fullsized":"2024-09-21T10:47:22.186577+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:24.439451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":65,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187823+0000","last_change":"2024-09-21T10:46:00.158554+0000","last_active":"2024-09-21T10:47:22.187823+0000","last_peered":"2024-09-21T10:47:22.187823+0000","last_clean":"2024-09-21T10:47:22.187823+0000","last_became_active":"2024-09-21T10:45:59.680546+0000","last_became_peered":"2024-09-21T10:45:59.680546+0000","last_unstale":"2024-09-21T10:47:22.187823+0000","last_undegraded":"2024-09-21T10:47:22.187823+0000","last_fullsized":"2024-09-21T10:47:22.187823+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:00.158500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:00.158500+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:39.581987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":199,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128781+0000","last_change":"2024-09-21T10:45:59.679381+0000","last_active":"2024-09-21T10:47:15.128781+0000","last_peered":"2024-09-21T10:47:15.128781+0000","last_clean":"2024-09-21T10:47:15.128781+0000","last_became_active":"2024-09-21T10:45:59.679131+0000","last_became_peered":"2024-09-21T10:45:59.679131+0000","last_unstale":"2024-09-21T10:47:15.128781+0000","last_undegraded":"2024-09-21T10:47:15.128781+0000","last_fullsized":"2024-09-21T10:47:15.128781+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:50.700756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1428,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187463+0000","last_change":"2024-09-21T10:45:59.703618+0000","last_active":"2024-09-21T10:47:22.187463+0000","last_peered":"2024-09-21T10:47:22.187463+0000","last_clean":"2024-09-21T10:47:22.187463+0000","last_became_active":"2024-09-21T10:45:59.699339+0000","last_became_peered":"2024-09-21T10:45:59.699339+0000","last_unstale":"2024-09-21T10:47:22.187463+0000","last_undegraded":"2024-09-21T10:47:22.187463+0000","last_fullsized":"2024-09-21T10:47:22.187463+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:18:19.242317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":176,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.126674+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:47:15.126674+0000","last_peered":"2024-09-21T10:47:15.126674+0000","last_clean":"2024-09-21T10:47:15.126674+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:47:15.126674+0000","last_undegraded":"2024-09-21T10:47:15.126674+0000","last_fullsized":"2024-09-21T10:47:15.126674+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1459,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.184927+0000","last_change":"2024-09-21T10:43:56.007452+0000","last_active":"2024-09-21T10:47:22.184927+0000","last_peered":"2024-09-21T10:47:22.184927+0000","last_clean":"2024-09-21T10:47:22.184927+0000","last_became_active":"2024-09-21T10:43:56.007103+0000","last_became_peered":"2024-09-21T10:43:56.007103+0000","last_unstale":"2024-09-21T10:47:22.184927+0000","last_undegraded":"2024-09-21T10:47:22.184927+0000","last_fullsized":"2024-09-21T10:47:22.184927+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:41.710693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2202,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.192058+0000","last_change":"2024-09-21T10:45:59.696879+0000","last_active":"2024-09-21T10:47:22.192058+0000","last_peered":"2024-09-21T10:47:22.192058+0000","last_clean":"2024-09-21T10:47:22.192058+0000","last_became_active":"2024-09-21T10:45:59.696055+0000","last_became_peered":"2024-09-21T10:45:59.696055+0000","last_unstale":"2024-09-21T10:47:22.192058+0000","last_undegraded":"2024-09-21T10:47:22.192058+0000","last_fullsized":"2024-09-21T10:47:22.192058+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:10.189668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":478,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186721+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:47:22.186721+0000","last_peered":"2024-09-21T10:47:22.186721+0000","last_clean":"2024-09-21T10:47:22.186721+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:47:22.186721+0000","last_undegraded":"2024-09-21T10:47:22.186721+0000","last_fullsized":"2024-09-21T10:47:22.186721+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1257,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186239+0000","last_change":"2024-09-21T10:45:59.693730+0000","last_active":"2024-09-21T10:47:22.186239+0000","last_peered":"2024-09-21T10:47:22.186239+0000","last_clean":"2024-09-21T10:47:22.186239+0000","last_became_active":"2024-09-21T10:45:59.692582+0000","last_became_peered":"2024-09-21T10:45:59.692582+0000","last_unstale":"2024-09-21T10:47:22.186239+0000","last_undegraded":"2024-09-21T10:47:22.186239+0000","last_fullsized":"2024-09-21T10:47:22.186239+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:21.994503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2322,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.187282+0000","last_change":"2024-09-21T10:45:59.697081+0000","last_active":"2024-09-21T10:47:22.187282+0000","last_peered":"2024-09-21T10:47:22.187282+0000","last_clean":"2024-09-21T10:47:22.187282+0000","last_became_active":"2024-09-21T10:45:59.696414+0000","last_became_peered":"2024-09-21T10:45:59.696414+0000","last_unstale":"2024-09-21T10:47:22.187282+0000","last_undegraded":"2024-09-21T10:47:22.187282+0000","last_fullsized":"2024-09-21T10:47:22.187282+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:47.528914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":186,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128712+0000","last_change":"2024-09-21T10:45:59.693591+0000","last_active":"2024-09-21T10:47:15.128712+0000","last_peered":"2024-09-21T10:47:15.128712+0000","last_clean":"2024-09-21T10:47:15.128712+0000","last_became_active":"2024-09-21T10:45:59.693061+0000","last_became_peered":"2024-09-21T10:45:59.693061+0000","last_unstale":"2024-09-21T10:47:15.128712+0000","last_undegraded":"2024-09-21T10:47:15.128712+0000","last_fullsized":"2024-09-21T10:47:15.128712+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:56.259939+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2321,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186391+0000","last_change":"2024-09-21T10:45:59.700672+0000","last_active":"2024-09-21T10:47:22.186391+0000","last_peered":"2024-09-21T10:47:22.186391+0000","last_clean":"2024-09-21T10:47:22.186391+0000","last_became_active":"2024-09-21T10:45:59.699332+0000","last_became_peered":"2024-09-21T10:45:59.699332+0000","last_unstale":"2024-09-21T10:47:22.186391+0000","last_undegraded":"2024-09-21T10:47:22.186391+0000","last_fullsized":"2024-09-21T10:47:22.186391+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:48.567053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":209,"reported_epoch":1438,"state":"active+clean","last_fresh":"2024-09-21T10:47:15.128119+0000","last_change":"2024-09-21T10:45:59.680367+0000","last_active":"2024-09-21T10:47:15.128119+0000","last_peered":"2024-09-21T10:47:15.128119+0000","last_clean":"2024-09-21T10:47:15.128119+0000","last_became_active":"2024-09-21T10:45:59.679973+0000","last_became_peered":"2024-09-21T10:45:59.679973+0000","last_unstale":"2024-09-21T10:47:15.128119+0000","last_undegraded":"2024-09-21T10:47:15.128119+0000","last_fullsized":"2024-09-21T10:47:15.128119+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:18.078677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":196,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.186123+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:47:22.186123+0000","last_peered":"2024-09-21T10:47:22.186123+0000","last_clean":"2024-09-21T10:47:22.186123+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:47:22.186123+0000","last_undegraded":"2024-09-21T10:47:22.186123+0000","last_fullsized":"2024-09-21T10:47:22.186123+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":74,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.180819+0000","last_change":"2024-09-21T10:45:59.714397+0000","last_active":"2024-09-21T10:47:22.180819+0000","last_peered":"2024-09-21T10:47:22.180819+0000","last_clean":"2024-09-21T10:47:22.180819+0000","last_became_active":"2024-09-21T10:45:59.689648+0000","last_became_peered":"2024-09-21T10:45:59.689648+0000","last_unstale":"2024-09-21T10:47:22.180819+0000","last_undegraded":"2024-09-21T10:47:22.180819+0000","last_fullsized":"2024-09-21T10:47:22.180819+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:20.596738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2211,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.192160+0000","last_change":"2024-09-21T10:45:59.696141+0000","last_active":"2024-09-21T10:47:22.192160+0000","last_peered":"2024-09-21T10:47:22.192160+0000","last_clean":"2024-09-21T10:47:22.192160+0000","last_became_active":"2024-09-21T10:45:59.695761+0000","last_became_peered":"2024-09-21T10:45:59.695761+0000","last_unstale":"2024-09-21T10:47:22.192160+0000","last_undegraded":"2024-09-21T10:47:22.192160+0000","last_fullsized":"2024-09-21T10:47:22.192160+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:46.714846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":50,"num_pg":16,"stat_sum":{"num_bytes":4094820352,"num_objects":62482,"num_object_clones":0,"num_object_copies":187446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":62482,"num_write_kb":3998848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12146114560,"data_stored":12146114560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62482,"ondisk_log_size":62482,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":6214,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":26316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":210453398136,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049795,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2703056,"kb_used_data":1763208,"kb_used_omap":242,"kb_used_meta":939533,"kb_avail":91042096,"statfs":{"total":95995035648,"available":93227106304,"internally_reserved":0,"allocated":1805524992,"data_stored":1801424948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":962082015},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790594,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2698120,"kb_used_data":1796928,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":91047032,"statfs":{"total":95995035648,"available":93232160768,"internally_reserved":0,"allocated":1840054272,"data_stored":1835958688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644753,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3103744,"kb_used_data":2105848,"kb_used_omap":67,"kb_used_meta":997820,"kb_avail":90641408,"statfs":{"total":95995035648,"available":92816801792,"internally_reserved":0,"allocated":2156388352,"data_stored":2152300451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1021768541},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527771,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2836620,"kb_used_data":1896772,"kb_used_omap":75,"kb_used_meta":939764,"kb_avail":90908532,"statfs":{"total":95995035648,"available":93090336768,"internally_reserved":0,"allocated":1942294528,"data_stored":1938199543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":962318490},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274924,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2714300,"kb_used_data":1724852,"kb_used_omap":51,"kb_used_meta":989324,"kb_avail":91030852,"statfs":{"total":95995035648,"available":93215592448,"internally_reserved":0,"allocated":1766248448,"data_stored":1762163980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1013068117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354163,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3272144,"kb_used_data":2493000,"kb_used_omap":66,"kb_used_meta":779069,"kb_avail":90473008,"statfs":{"total":95995035648,"available":92644360192,"internally_reserved":0,"allocated":2552832000,"data_stored":2548732471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":797766980},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2585198592,"data_stored":2585198592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1762918400,"data_stored":1762918400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1966604288,"data_stored":1966604288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2176385024,"data_stored":2176385024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1841692672,"data_stored":1841692672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1813315584,"data_stored":1813315584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:47:39.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:47:39.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:39.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:39.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:40.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.085 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:40.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:40.186 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:40.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:40.292 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 48 16 70677 70661 91.9861 60.0625 0.0145866 0.0108657 2024-09-21T10:47:40.333 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:40.356 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:40.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:40.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.514 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:40.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:47:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:47:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:47:40.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.661 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:40.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:40.762 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:40.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:40.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:40.911 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:40.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:41.012 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:41.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:41.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:41.161 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:41.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:41.293 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 49 16 71785 71769 91.5221 69.25 0.0215701 0.0109196 2024-09-21T10:47:41.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:41.400 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:41.419 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:47:41.419 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:41.451 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:41.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:47:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:41.495 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:47:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:41.495 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:41 smithi119 ceph-mon[105288]: pgmap v2573: 109 pgs: 109 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:47:41.495 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:41 smithi119 ceph-mon[101432]: pgmap v2573: 109 pgs: 109 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:47:41.533 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:47:41.533 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:41 smithi145 ceph-mon[89359]: pgmap v2573: 109 pgs: 109 active+clean; 4.1 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-21T10:47:41.548 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:41.553 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:41.569 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:47:41.569 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:41.606 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:47:41.606 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:41.636 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:47:41.636 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:41.665 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:47:41.665 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:41.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:41.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:41.913 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:41.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:42.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:42.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:42.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:42.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:42.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:42.293 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 50 16 72746 72730 90.8927 60.0625 0.0140111 0.0109962 2024-09-21T10:47:42.316 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:42.418 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:42.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:42 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3095795953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:42.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:42.580 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:42.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:42 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3095795953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:42.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:42 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3095795953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:42.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:42.718 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:42.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:42.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:42.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:42.979 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:42.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:43.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:43.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:43.281 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:43.293 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 51 16 73822 73806 90.4289 67.25 0.0102318 0.0110524 2024-09-21T10:47:43.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:43.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:43.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:43 smithi145 ceph-mon[89359]: pgmap v2574: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:43.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:43.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:47:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:43 smithi119 ceph-mon[101432]: pgmap v2574: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:43 smithi119 ceph-mon[105288]: pgmap v2574: 109 pgs: 109 active+clean; 4.2 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:43.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:43.690 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.707 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:43.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:43.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:43.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.858 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:43.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:43.959 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:43.983 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:43.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:44.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:44.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:44.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:44.254 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:44.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 52 16 74782 74766 89.8431 60 0.0138948 0.0111257 2024-09-21T10:47:44.356 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:44.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:44.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:25233 48.3 48:cb9f643a:::benchmark_data_smithi119_267254_object25232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1358)", 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:08.768702+0000", 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 575.64923284400004, 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.37026426699999998, 2024-09-21T10:47:44.484 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 25233 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768702+0000", 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.485 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768702+0000", 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768705+0000", 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0900000000000001e-06 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768733+0000", 2024-09-21T10:47:44.486 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7912999999999999e-05 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768751+0000", 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8212000000000001e-05 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768761+0000", 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7510000000000003e-06 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.487 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.768828+0000", 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7180000000000004e-05 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136028+0000", 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.36719971499999998 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136181+0000", 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015294199999999999 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.488 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138840+0000", 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002659509 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138860+0000", 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9865e-05 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138903+0000", 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2698999999999999e-05 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.489 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138936+0000", 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3741000000000001e-05 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.138966+0000", 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9649999999999999e-05 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.490 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.46619.0:25232 48.4 e1358/1354 48:22eca9e3:::benchmark_data_smithi119_267254_object25231:head v 1358'1560, mlcod=1358'1558)", 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:38:08.769205+0000", 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 575.64872969800001, 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.37107258300000001, 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769205+0000", 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.491 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769205+0000", 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769206+0000", 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.08e-06 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:08.769250+0000", 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4456999999999998e-05 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.492 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136045+0000", 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.36679430699999999 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136053+0000", 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.909e-06 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136107+0000", 2024-09-21T10:47:44.493 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3874e-05 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.136129+0000", 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2670000000000001e-05 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.140235+0000", 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004105427 2024-09-21T10:47:44.494 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:38:09.140278+0000", 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2858999999999997e-05 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.46619.0:192769 48.3 48:ce25c197:::benchmark_data_smithi119_267254_object69559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1383)", 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:40:45.556416+0000", 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 418.86151904299999, 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.23024958800000001, 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.495 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.46619", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1034533120", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 192769 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556416+0000", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556416+0000", 2024-09-21T10:47:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556417+0000", 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.68e-06 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556421+0000", 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7299999999999999e-06 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556434+0000", 2024-09-21T10:47:44.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3057000000000001e-05 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556441+0000", 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0809999999999996e-06 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556592+0000", 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000151337 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.556881+0000", 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00028865100000000001 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 6,7", 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.557117+0000", 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000235999 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.561134+0000", 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040171859999999998 2024-09-21T10:47:44.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.563527+0000", 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0023929369999999999 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.786554+0000", 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.223026643 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.786614+0000", 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9735000000000003e-05 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:40:45.786665+0000", 2024-09-21T10:47:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1551999999999997e-05 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:119211 49.3 49:c13490aa:::benchmark_data_smithi119_301770_object5724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:47:44.502 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.400214+0000", 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 221.017720852, 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21883351000000001, 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.503 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:47:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:47:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 119211 2024-09-21T10:47:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400214+0000", 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400214+0000", 2024-09-21T10:47:44.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400215+0000", 2024-09-21T10:47:44.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.144e-06 2024-09-21T10:47:44.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400220+0000", 2024-09-21T10:47:44.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5709999999999999e-06 2024-09-21T10:47:44.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400229+0000", 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4010000000000003e-06 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400235+0000", 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.579e-06 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400583+0000", 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00034798799999999998 2024-09-21T10:47:44.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400682+0000", 2024-09-21T10:47:44.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8256000000000003e-05 2024-09-21T10:47:44.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:47:44.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.400782+0000", 2024-09-21T10:47:44.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.9869000000000002e-05 2024-09-21T10:47:44.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.406415+0000", 2024-09-21T10:47:44.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005633491 2024-09-21T10:47:44.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.407891+0000", 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014761419999999999 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.513 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.618933+0000", 2024-09-21T10:47:44.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21104145299999999 2024-09-21T10:47:44.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.619002+0000", 2024-09-21T10:47:44.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9498000000000004e-05 2024-09-21T10:47:44.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.619047+0000", 2024-09-21T10:47:44.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5117999999999997e-05 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:119227 49.3 49:cab3da16:::benchmark_data_smithi119_301770_object5740:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:47:44.516 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.416510+0000", 2024-09-21T10:47:44.517 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 221.00142415100001, 2024-09-21T10:47:44.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.206829652, 2024-09-21T10:47:44.517 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.517 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.517 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.517 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.518 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:47:44.518 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:47:44.518 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 119227 2024-09-21T10:47:44.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.518 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416510+0000", 2024-09-21T10:47:44.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416510+0000", 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416512+0000", 2024-09-21T10:47:44.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.874e-06 2024-09-21T10:47:44.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416519+0000", 2024-09-21T10:47:44.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.545e-06 2024-09-21T10:47:44.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416537+0000", 2024-09-21T10:47:44.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8031000000000001e-05 2024-09-21T10:47:44.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416547+0000", 2024-09-21T10:47:44.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5829999999999996e-06 2024-09-21T10:47:44.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416613+0000", 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6243999999999998e-05 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416738+0000", 2024-09-21T10:47:44.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000125244 2024-09-21T10:47:44.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:47:44.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.416809+0000", 2024-09-21T10:47:44.525 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0984999999999999e-05 2024-09-21T10:47:44.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.419336+0000", 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0025268690000000002 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.420181+0000", 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00084515999999999997 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.623208+0000", 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20302703799999999 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.623294+0000", 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5628000000000001e-05 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.623340+0000", 2024-09-21T10:47:44.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6451e-05 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:119232 49.3 49:cd1d3705:::benchmark_data_smithi119_301770_object5745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1408)", 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:03.426481+0000", 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 220.99145326199999, 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19696485799999999, 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.528 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 119232 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426481+0000", 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426481+0000", 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426483+0000", 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5939999999999999e-06 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426489+0000", 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.9510000000000003e-06 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426504+0000", 2024-09-21T10:47:44.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5172e-05 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426512+0000", 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.4880000000000001e-06 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426575+0000", 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3582000000000002e-05 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426678+0000", 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010262799999999999 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.426769+0000", 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1275999999999998e-05 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.429226+0000", 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024573400000000001 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.429858+0000", 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00063187699999999998 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.623374+0000", 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19351569499999999 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.623415+0000", 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1247e-05 2024-09-21T10:47:44.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:03.623446+0000", 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1007999999999999e-05 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160773 49.1 49:895b3e8c:::benchmark_data_smithi119_301770_object47286:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.471065+0000", 2024-09-21T10:47:44.534 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 173.946869591, 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25287037299999998, 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160773 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471065+0000", 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471065+0000", 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471066+0000", 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.375e-06 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471071+0000", 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2819999999999998e-06 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471085+0000", 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4222999999999999e-05 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471092+0000", 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.114e-06 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471241+0000", 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014919799999999999 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471504+0000", 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026254400000000003 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:47:44.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471833+0000", 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00032951999999999998 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.475484+0000", 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036504300000000001 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.475567+0000", 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3052000000000004e-05 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.723828+0000", 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.248261553 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.723895+0000", 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6427999999999995e-05 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.723935+0000", 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.0654000000000001e-05 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:16612 50.a 50:55718400:::benchmark_data_smithi119_335063_object16611:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:46:59.143896+0000", 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.274038246000003, 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.228424086, 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.541 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 16612 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.143896+0000", 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.143896+0000", 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.143897+0000", 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6300000000000005e-07 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.143940+0000", 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3216000000000001e-05 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.143950+0000", 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4879999999999996e-06 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.143957+0000", 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.861e-06 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.145614+0000", 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016571660000000001 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.145883+0000", 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00026913300000000001 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.148769+0000", 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028863320000000001 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.150992+0000", 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022227990000000001 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.152081+0000", 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0010893249999999999 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372222+0000", 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.220140799 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372276+0000", 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4224000000000001e-05 2024-09-21T10:47:44.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372320+0000", 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4079999999999998e-05 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:16615 50.a 50:53ff175a:::benchmark_data_smithi119_335063_object16614:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:46:59.144896+0000", 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.273038685000003, 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.227895178, 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.547 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 16615 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144896+0000", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144896+0000", 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144897+0000", 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.316e-06 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144920+0000", 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.226e-05 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144932+0000", 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.256e-05 2024-09-21T10:47:44.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144939+0000", 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.357e-06 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.149928+0000", 2024-09-21T10:47:44.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0049882479999999998 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.150111+0000", 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018301400000000001 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.150262+0000", 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015153200000000001 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.151934+0000", 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016719829999999999 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.152086+0000", 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015148599999999999 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372701+0000", 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.22061483000000001 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372752+0000", 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1279999999999997e-05 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372791+0000", 2024-09-21T10:47:44.553 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9311999999999999e-05 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:16616 50.a 50:58126766:::benchmark_data_smithi119_335063_object16615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:46:59.144953+0000", 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.272981698999999, 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.22796448999999999, 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.554 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 16616 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144953+0000", 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144953+0000", 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144953+0000", 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9200000000000001e-07 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144971+0000", 2024-09-21T10:47:44.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7958e-05 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144979+0000", 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.391e-06 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.144985+0000", 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7239999999999997e-06 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.151019+0000", 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0060341279999999997 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.151133+0000", 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011390799999999999 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.151280+0000", 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014713500000000001 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.152279+0000", 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00099919700000000006 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.152969+0000", 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00069060300000000003 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372828+0000", 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21985835100000001 2024-09-21T10:47:44.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372883+0000", 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5198999999999998e-05 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372917+0000", 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4403999999999998e-05 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:16642 50.a 50:58357829:::benchmark_data_smithi119_335063_object16641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:46:59.155975+0000", 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 45.261959189000002, 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21705448299999999, 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.561 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 16642 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.155975+0000", 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.155975+0000", 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.155977+0000", 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0580000000000001e-06 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.155996+0000", 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8992999999999999e-05 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.156005+0000", 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3980000000000005e-06 2024-09-21T10:47:44.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.156011+0000", 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2770000000000002e-06 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.156060+0000", 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8681999999999999e-05 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.156271+0000", 2024-09-21T10:47:44.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000211416 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,6", 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.156443+0000", 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000172017 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.157587+0000", 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011441890000000001 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.157641+0000", 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3463e-05 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372954+0000", 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.21531265299999999 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.372998+0000", 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3986000000000002e-05 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.566 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:46:59.373030+0000", 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.2351000000000002e-05 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:24614 50.1 50:8a9318f9:::benchmark_data_smithi119_335063_object24613:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:04.139012+0000", 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 40.278922729000001, 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19011787999999999, 2024-09-21T10:47:44.567 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 24614 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139012+0000", 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139012+0000", 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139013+0000", 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2729999999999999e-06 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139036+0000", 2024-09-21T10:47:44.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2886000000000001e-05 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139047+0000", 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0930999999999999e-05 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139054+0000", 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.72e-06 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.570 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139705+0000", 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00065152899999999998 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.139952+0000", 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00024702300000000002 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:47:44.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140079+0000", 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012676000000000001 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.156344+0000", 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.01626468 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.156458+0000", 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000114714 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.329028+0000", 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.17256982100000001 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.329084+0000", 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6042999999999999e-05 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.329130+0000", 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5500000000000001e-05 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.573 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:24616 50.1 50:87ffcb61:::benchmark_data_smithi119_335063_object24615:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:04.140149+0000", 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 40.277786100999997, 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19122307499999999, 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.574 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 24616 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140149+0000", 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140149+0000", 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.575 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140150+0000", 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.562e-06 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140178+0000", 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7398000000000001e-05 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140187+0000", 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7869999999999996e-06 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.140194+0000", 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7710000000000001e-06 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141297+0000", 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011032679999999999 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141420+0000", 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000122583 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141550+0000", 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012981099999999999 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.156423+0000", 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014873242 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.156502+0000", 2024-09-21T10:47:44.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8787000000000003e-05 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.331280+0000", 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.17477811200000001 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.331330+0000", 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0130999999999998e-05 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.331372+0000", 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1622999999999998e-05 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:24622 50.1 50:8945f6dc:::benchmark_data_smithi119_335063_object24621:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:04.141382+0000", 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 40.276552821999999, 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19011905400000001, 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.580 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 24622 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141382+0000", 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.581 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141382+0000", 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141383+0000", 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0559999999999999e-06 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141404+0000", 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0656999999999999e-05 2024-09-21T10:47:44.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141412+0000", 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0649999999999999e-06 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.141417+0000", 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4269999999999999e-06 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.154419+0000", 2024-09-21T10:47:44.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013001873000000001 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.154567+0000", 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014802500000000001 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.154715+0000", 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014824700000000001 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.156470+0000", 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0017547229999999999 2024-09-21T10:47:44.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.157171+0000", 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00070092199999999998 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.331407+0000", 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.174236052 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.331456+0000", 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9101000000000003e-05 2024-09-21T10:47:44.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.331501+0000", 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4906e-05 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:25620 50.9 50:963a68cf:::benchmark_data_smithi119_335063_object25619:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:04.969224+0000", 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 39.448710929999997, 2024-09-21T10:47:44.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.177291806, 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 25620 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.969224+0000", 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.969224+0000", 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.969225+0000", 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6499999999999998e-07 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.969245+0000", 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0004e-05 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.969255+0000", 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.005e-05 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.969261+0000", 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5640000000000002e-06 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.971452+0000", 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021912260000000001 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.589 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.971539+0000", 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6149999999999993e-05 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,6", 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.971644+0000", 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010519200000000001 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.973880+0000", 2024-09-21T10:47:44.590 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022360449999999999 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.977828+0000", 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0039486649999999996 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:05.146425+0000", 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.168596525 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:05.146474+0000", 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9416000000000003e-05 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:05.146516+0000", 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1103999999999998e-05 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:25626 50.1 50:87faaaf4:::benchmark_data_smithi119_335063_object25625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1437)", 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:04.972356+0000", 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 39.445578722, 2024-09-21T10:47:44.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.17469863599999999, 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 25626 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.972356+0000", 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.972356+0000", 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.972357+0000", 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2099999999999996e-07 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.594 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.972379+0000", 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2132000000000001e-05 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.972387+0000", 2024-09-21T10:47:44.595 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8599999999999993e-06 2024-09-21T10:47:44.596 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:44 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:47:44.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.972392+0000", 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3410000000000002e-06 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.975272+0000", 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028798729999999998 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.975375+0000", 2024-09-21T10:47:44.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010316500000000001 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,3", 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.975504+0000", 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012912399999999999 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.977916+0000", 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024122750000000002 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.598 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:04.977941+0000", 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4987e-05 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:05.146975+0000", 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.16903358399999999 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:05.147019+0000", 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3983999999999998e-05 2024-09-21T10:47:44.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:05.147055+0000", 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5689999999999999e-05 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:42595 50.7 50:ee9a528f:::benchmark_data_smithi119_335063_object42594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:15.813747+0000", 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.604187712000002, 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20361573799999999, 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.600 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 42595 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813747+0000", 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813747+0000", 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813748+0000", 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.195e-06 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813797+0000", 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8443999999999999e-05 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813808+0000", 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1324e-05 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813814+0000", 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5200000000000003e-06 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.813890+0000", 2024-09-21T10:47:44.603 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5393000000000001e-05 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.814043+0000", 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015326399999999999 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.814195+0000", 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015229600000000001 2024-09-21T10:47:44.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.816209+0000", 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020136960000000001 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.824021+0000", 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0078119950000000004 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.017256+0000", 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19323491500000001 2024-09-21T10:47:44.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.017317+0000", 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1048000000000002e-05 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.017363+0000", 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5648000000000001e-05 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.606 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:42603 50.7 50:eabd8ab7:::benchmark_data_smithi119_335063_object42602:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:15.816991+0000", 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.600943551, 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.20150675600000001, 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.607 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 42603 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.816991+0000", 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.608 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.816991+0000", 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.816993+0000", 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9659999999999999e-06 2024-09-21T10:47:44.609 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.817016+0000", 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2928000000000001e-05 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.817030+0000", 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3651e-05 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.817037+0000", 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.418e-06 2024-09-21T10:47:44.610 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.817098+0000", 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1328000000000006e-05 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.817225+0000", 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012637400000000001 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:44.611 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.817352+0000", 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000127157 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.821025+0000", 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003672589 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.824026+0000", 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0030015559999999998 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.612 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.018392+0000", 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19436593499999999 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.018454+0000", 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2347000000000005e-05 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.018498+0000", 2024-09-21T10:47:44.613 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3507000000000002e-05 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:42604 50.7 50:e1e1071c:::benchmark_data_smithi119_335063_object42603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:15.818428+0000", 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.599506893000001, 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.200651985, 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.614 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 42604 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.818428+0000", 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.615 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.818428+0000", 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.818429+0000", 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.59e-06 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.818452+0000", 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2918999999999999e-05 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.616 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.818465+0000", 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2869e-05 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.818472+0000", 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9530000000000002e-06 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.819195+0000", 2024-09-21T10:47:44.617 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00072334600000000004 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.819380+0000", 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018462 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.819513+0000", 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000133013 2024-09-21T10:47:44.618 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.822832+0000", 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0033185620000000002 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.824031+0000", 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011989920000000001 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.018991+0000", 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19496070900000001 2024-09-21T10:47:44.619 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.019042+0000", 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0628999999999999e-05 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.019080+0000", 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7783000000000001e-05 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.620 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:42618 50.7 50:ee8bff75:::benchmark_data_smithi119_335063_object42617:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1438)", 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:15.826268+0000", 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 28.591666416999999, 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.194267, 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:47:44.621 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 42618 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826268+0000", 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826268+0000", 2024-09-21T10:47:44.622 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826270+0000", 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.362e-06 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826309+0000", 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9555000000000003e-05 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.623 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826320+0000", 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1260999999999999e-05 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826326+0000", 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2360000000000004e-06 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826383+0000", 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.7781999999999997e-05 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.624 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826542+0000", 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015868800000000001 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5", 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.826688+0000", 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014617499999999999 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.828247+0000", 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0015585289999999999 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.625 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:15.828266+0000", 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9043999999999999e-05 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.020405+0000", 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.19213904100000001 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.020451+0000", 2024-09-21T10:47:44.626 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6495999999999997e-05 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:16.020535+0000", 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3831000000000004e-05 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:47:44.627 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:47:44.628 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:44 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:47:44.628 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:44 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:47:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:44.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:44.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:44.752 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:44.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:44.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:44.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:47:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:47:44] "GET /metrics HTTP/1.1" 200 36829 "" "Prometheus/2.51.0" 2024-09-21T10:47:44.901 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:44.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:45.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:45.050 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:45.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:45.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:45.180 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:45.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:45.282 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:45.293 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 53 16 75709 75693 89.241 57.9375 0.00944988 0.0111996 2024-09-21T10:47:45.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:45 smithi145 ceph-mon[89359]: pgmap v2575: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-21T10:47:45.310 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:45.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:45.411 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:45.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:45.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:45.577 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:45.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:45 smithi119 ceph-mon[101432]: pgmap v2575: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-21T10:47:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:45 smithi119 ceph-mon[105288]: pgmap v2575: 109 pgs: 109 active+clean; 4.4 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-21T10:47:45.680 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:45.705 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:45.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:47:45.810 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:45.862 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:45.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:45.963 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:45.984 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:45.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:46.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:46.107 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:46.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:46.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:46.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:46.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 54 16 76443 76427 88.438 45.875 0.00316546 0.011286 2024-09-21T10:47:46.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:46.359 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:46.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:46.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:46.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:46.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:46.600 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:46.626 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:46.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:46.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:46.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:46.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:46.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:47.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:47.030 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:47.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:47.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:47.169 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:47.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:47.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 55 16 77198 77182 87.688 47.1875 0.00821381 0.0113987 2024-09-21T10:47:47.297 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:47.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:47.442 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:47.544 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:47.545 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:47 smithi145 ceph-mon[89359]: pgmap v2576: 109 pgs: 109 active+clean; 4.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:47:47.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:47.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:47.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:47 smithi119 ceph-mon[105288]: pgmap v2576: 109 pgs: 109 active+clean; 4.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:47:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:47 smithi119 ceph-mon[101432]: pgmap v2576: 109 pgs: 109 active+clean; 4.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-21T10:47:47.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:47.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:47.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:47.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:47.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:47.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:48.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:48.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:48.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:48.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:48.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 56 16 78207 78191 87.2482 63.0625 0.0170541 0.0114566 2024-09-21T10:47:48.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:48.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:48.515 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:48.538 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:48.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:47:48.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:48.681 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:48.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:48.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T10:47:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.847 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:48.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:48.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:48.993 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:48.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:49.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:49.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:49.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:49.243 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:49.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 57 16 79118 79102 86.7164 56.9375 0.00799692 0.0115258 2024-09-21T10:47:49.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:49.355 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:49 smithi119 ceph-mon[101432]: pgmap v2577: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-21T10:47:49.355 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:49 smithi119 ceph-mon[105288]: pgmap v2577: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-21T10:47:49.375 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:49.375 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:49.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:49.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:49.534 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:49 smithi145 ceph-mon[89359]: pgmap v2577: 109 pgs: 109 active+clean; 4.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 57 MiB/s wr, 910 op/s 2024-09-21T10:47:49.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:49.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:49.750 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:47:49.751 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:47:49.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:49.784 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:49.886 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:49.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:49.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:50.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:50.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:50.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:50.226 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:50.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:50.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 58 16 79953 79937 86.1211 52.1875 0.027606 0.011606 2024-09-21T10:47:50.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:50.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.420 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:50.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:50.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:47:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:47:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:47:50.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:50.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:50.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:50.690 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:50.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:50.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[105288]: from='client.56290 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[105288]: pgmap v2578: 109 pgs: 109 active+clean; 4.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[101432]: from='client.56290 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:47:50.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:50 smithi119 ceph-mon[101432]: pgmap v2578: 109 pgs: 109 active+clean; 4.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:50.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:50.942 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:50.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:50.950 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2577,"stamp":"2024-09-21T10:47:48.005899+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4890608248,"num_objects":74645,"num_object_clones":0,"num_object_copies":223935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74682,"num_write_kb":4775647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":924281,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76777,"ondisk_log_size":76777,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":19950224,"kb_used_data":14329312,"kb_used_omap":505,"kb_used_meta":5620294,"kb_avail":542520688,"statfs":{"total":575970213888,"available":555541184512,"internally_reserved":0,"allocated":14673215488,"data_stored":14648609045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":517945,"internal_metadata":5755181255},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,5,8,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":702808064,"num_objects":10724,"num_object_clones":0,"num_object_copies":32172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10724,"num_write_kb":686336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.779536"},"pg_stats":[{"pgid":"50.f","version":"1440'4758","reported_seq":4788,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:46.427090+0000","last_change":"2024-09-21T10:46:40.275318+0000","last_active":"2024-09-21T10:47:46.427090+0000","last_peered":"2024-09-21T10:47:46.427090+0000","last_clean":"2024-09-21T10:47:46.427090+0000","last_became_active":"2024-09-21T10:46:40.275064+0000","last_became_peered":"2024-09-21T10:46:40.275064+0000","last_unstale":"2024-09-21T10:47:46.427090+0000","last_undegraded":"2024-09-21T10:47:46.427090+0000","last_fullsized":"2024-09-21T10:47:46.427090+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:25.995350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311820288,"num_objects":4758,"num_object_clones":0,"num_object_copies":14274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":304512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.e","version":"1440'4700","reported_seq":4731,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:44.995555+0000","last_change":"2024-09-21T10:46:40.276274+0000","last_active":"2024-09-21T10:47:44.995555+0000","last_peered":"2024-09-21T10:47:44.995555+0000","last_clean":"2024-09-21T10:47:44.995555+0000","last_became_active":"2024-09-21T10:46:40.276005+0000","last_became_peered":"2024-09-21T10:46:40.276005+0000","last_unstale":"2024-09-21T10:47:44.995555+0000","last_undegraded":"2024-09-21T10:47:44.995555+0000","last_fullsized":"2024-09-21T10:47:44.995555+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:42.051377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308019200,"num_objects":4700,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4700,"num_write_kb":300800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.d","version":"1440'4615","reported_seq":4633,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.751284+0000","last_change":"2024-09-21T10:46:40.277839+0000","last_active":"2024-09-21T10:47:43.751284+0000","last_peered":"2024-09-21T10:47:43.751284+0000","last_clean":"2024-09-21T10:47:43.751284+0000","last_became_active":"2024-09-21T10:46:40.277565+0000","last_became_peered":"2024-09-21T10:46:40.277565+0000","last_unstale":"2024-09-21T10:47:43.751284+0000","last_undegraded":"2024-09-21T10:47:43.751284+0000","last_fullsized":"2024-09-21T10:47:43.751284+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4615,"log_dups_size":0,"ondisk_log_size":4615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:03.176402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302448640,"num_objects":4615,"num_object_clones":0,"num_object_copies":13845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":295360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.c","version":"1440'4593","reported_seq":4624,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.072604+0000","last_change":"2024-09-21T10:46:40.274270+0000","last_active":"2024-09-21T10:47:43.072604+0000","last_peered":"2024-09-21T10:47:43.072604+0000","last_clean":"2024-09-21T10:47:43.072604+0000","last_became_active":"2024-09-21T10:46:40.273989+0000","last_became_peered":"2024-09-21T10:46:40.273989+0000","last_unstale":"2024-09-21T10:47:43.072604+0000","last_undegraded":"2024-09-21T10:47:43.072604+0000","last_fullsized":"2024-09-21T10:47:43.072604+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.938584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301006848,"num_objects":4593,"num_object_clones":0,"num_object_copies":13779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":293952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.b","version":"1440'4683","reported_seq":4701,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.774665+0000","last_change":"2024-09-21T10:46:40.275045+0000","last_active":"2024-09-21T10:47:43.774665+0000","last_peered":"2024-09-21T10:47:43.774665+0000","last_clean":"2024-09-21T10:47:43.774665+0000","last_became_active":"2024-09-21T10:46:40.274835+0000","last_became_peered":"2024-09-21T10:46:40.274835+0000","last_unstale":"2024-09-21T10:47:43.774665+0000","last_undegraded":"2024-09-21T10:47:43.774665+0000","last_fullsized":"2024-09-21T10:47:43.774665+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:59:51.355069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4683,"num_write_kb":299712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.a","version":"1440'4528","reported_seq":4559,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.040823+0000","last_change":"2024-09-21T10:46:40.276509+0000","last_active":"2024-09-21T10:47:43.040823+0000","last_peered":"2024-09-21T10:47:43.040823+0000","last_clean":"2024-09-21T10:47:43.040823+0000","last_became_active":"2024-09-21T10:46:40.276211+0000","last_became_peered":"2024-09-21T10:46:40.276211+0000","last_unstale":"2024-09-21T10:47:43.040823+0000","last_undegraded":"2024-09-21T10:47:43.040823+0000","last_fullsized":"2024-09-21T10:47:43.040823+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4528,"log_dups_size":0,"ondisk_log_size":4528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:56.745364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296747008,"num_objects":4528,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4528,"num_write_kb":289792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.9","version":"1440'4648","reported_seq":4679,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.074431+0000","last_change":"2024-09-21T10:46:40.276505+0000","last_active":"2024-09-21T10:47:43.074431+0000","last_peered":"2024-09-21T10:47:43.074431+0000","last_clean":"2024-09-21T10:47:43.074431+0000","last_became_active":"2024-09-21T10:46:40.275398+0000","last_became_peered":"2024-09-21T10:46:40.275398+0000","last_unstale":"2024-09-21T10:47:43.074431+0000","last_undegraded":"2024-09-21T10:47:43.074431+0000","last_fullsized":"2024-09-21T10:47:43.074431+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4648,"log_dups_size":0,"ondisk_log_size":4648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:47.276665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304611328,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":297472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.8","version":"1440'4667","reported_seq":4697,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:45.039990+0000","last_change":"2024-09-21T10:46:40.274571+0000","last_active":"2024-09-21T10:47:45.039990+0000","last_peered":"2024-09-21T10:47:45.039990+0000","last_clean":"2024-09-21T10:47:45.039990+0000","last_became_active":"2024-09-21T10:46:40.274285+0000","last_became_peered":"2024-09-21T10:46:40.274285+0000","last_unstale":"2024-09-21T10:47:45.039990+0000","last_undegraded":"2024-09-21T10:47:45.039990+0000","last_fullsized":"2024-09-21T10:47:45.039990+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:51:55.260163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305856512,"num_objects":4667,"num_object_clones":0,"num_object_copies":14001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4667,"num_write_kb":298688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.7","version":"1440'4621","reported_seq":4652,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.060325+0000","last_change":"2024-09-21T10:46:40.243879+0000","last_active":"2024-09-21T10:47:43.060325+0000","last_peered":"2024-09-21T10:47:43.060325+0000","last_clean":"2024-09-21T10:47:43.060325+0000","last_became_active":"2024-09-21T10:46:40.243579+0000","last_became_peered":"2024-09-21T10:46:40.243579+0000","last_unstale":"2024-09-21T10:47:43.060325+0000","last_undegraded":"2024-09-21T10:47:43.060325+0000","last_fullsized":"2024-09-21T10:47:43.060325+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:33.317643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302841856,"num_objects":4621,"num_object_clones":0,"num_object_copies":13863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4621,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4621,"num_write_kb":295744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.6","version":"1440'4611","reported_seq":4642,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:44.972143+0000","last_change":"2024-09-21T10:46:40.275529+0000","last_active":"2024-09-21T10:47:44.972143+0000","last_peered":"2024-09-21T10:47:44.972143+0000","last_clean":"2024-09-21T10:47:44.972143+0000","last_became_active":"2024-09-21T10:46:40.275255+0000","last_became_peered":"2024-09-21T10:46:40.275255+0000","last_unstale":"2024-09-21T10:47:44.972143+0000","last_undegraded":"2024-09-21T10:47:44.972143+0000","last_fullsized":"2024-09-21T10:47:44.972143+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4611,"log_dups_size":0,"ondisk_log_size":4611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:08.150110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302186496,"num_objects":4611,"num_object_clones":0,"num_object_copies":13833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4611,"num_write_kb":295104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.5","version":"1440'4637","reported_seq":4655,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.659597+0000","last_change":"2024-09-21T10:46:40.243663+0000","last_active":"2024-09-21T10:47:43.659597+0000","last_peered":"2024-09-21T10:47:43.659597+0000","last_clean":"2024-09-21T10:47:43.659597+0000","last_became_active":"2024-09-21T10:46:40.243417+0000","last_became_peered":"2024-09-21T10:46:40.243417+0000","last_unstale":"2024-09-21T10:47:43.659597+0000","last_undegraded":"2024-09-21T10:47:43.659597+0000","last_fullsized":"2024-09-21T10:47:43.659597+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:42.500241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303890432,"num_objects":4637,"num_object_clones":0,"num_object_copies":13911,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4637,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":296768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.4","version":"1440'4651","reported_seq":4669,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.796405+0000","last_change":"2024-09-21T10:46:40.277620+0000","last_active":"2024-09-21T10:47:43.796405+0000","last_peered":"2024-09-21T10:47:43.796405+0000","last_clean":"2024-09-21T10:47:43.796405+0000","last_became_active":"2024-09-21T10:46:40.277426+0000","last_became_peered":"2024-09-21T10:46:40.277426+0000","last_unstale":"2024-09-21T10:47:43.796405+0000","last_undegraded":"2024-09-21T10:47:43.796405+0000","last_fullsized":"2024-09-21T10:47:43.796405+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:54:37.926675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304807936,"num_objects":4651,"num_object_clones":0,"num_object_copies":13953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4651,"num_write_kb":297664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"50.3","version":"1440'4863","reported_seq":4894,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:45.035886+0000","last_change":"2024-09-21T10:46:40.275398+0000","last_active":"2024-09-21T10:47:45.035886+0000","last_peered":"2024-09-21T10:47:45.035886+0000","last_clean":"2024-09-21T10:47:45.035886+0000","last_became_active":"2024-09-21T10:46:40.275096+0000","last_became_peered":"2024-09-21T10:46:40.275096+0000","last_unstale":"2024-09-21T10:47:45.035886+0000","last_undegraded":"2024-09-21T10:47:45.035886+0000","last_fullsized":"2024-09-21T10:47:45.035886+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4863,"log_dups_size":0,"ondisk_log_size":4863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:58.484291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318701568,"num_objects":4863,"num_object_clones":0,"num_object_copies":14589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4863,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4863,"num_write_kb":311232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.2","version":"1440'4745","reported_seq":4773,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:46.173342+0000","last_change":"2024-09-21T10:46:40.274967+0000","last_active":"2024-09-21T10:47:46.173342+0000","last_peered":"2024-09-21T10:47:46.173342+0000","last_clean":"2024-09-21T10:47:46.173342+0000","last_became_active":"2024-09-21T10:46:40.274362+0000","last_became_peered":"2024-09-21T10:46:40.274362+0000","last_unstale":"2024-09-21T10:47:46.173342+0000","last_undegraded":"2024-09-21T10:47:46.173342+0000","last_fullsized":"2024-09-21T10:47:46.173342+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4745,"log_dups_size":0,"ondisk_log_size":4745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:41:55.876058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310968320,"num_objects":4745,"num_object_clones":0,"num_object_copies":14235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4745,"num_write_kb":303680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.1","version":"1440'4636","reported_seq":4667,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.074067+0000","last_change":"2024-09-21T10:46:40.276794+0000","last_active":"2024-09-21T10:47:43.074067+0000","last_peered":"2024-09-21T10:47:43.074067+0000","last_clean":"2024-09-21T10:47:43.074067+0000","last_became_active":"2024-09-21T10:46:40.276137+0000","last_became_peered":"2024-09-21T10:46:40.276137+0000","last_unstale":"2024-09-21T10:47:43.074067+0000","last_undegraded":"2024-09-21T10:47:43.074067+0000","last_fullsized":"2024-09-21T10:47:43.074067+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:01.998423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303824896,"num_objects":4636,"num_object_clones":0,"num_object_copies":13908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4636,"num_write_kb":296704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.0","version":"1440'4663","reported_seq":4681,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:43.731795+0000","last_change":"2024-09-21T10:46:40.241307+0000","last_active":"2024-09-21T10:47:43.731795+0000","last_peered":"2024-09-21T10:47:43.731795+0000","last_clean":"2024-09-21T10:47:43.731795+0000","last_became_active":"2024-09-21T10:46:40.241061+0000","last_became_peered":"2024-09-21T10:46:40.241061+0000","last_unstale":"2024-09-21T10:47:43.731795+0000","last_undegraded":"2024-09-21T10:47:43.731795+0000","last_fullsized":"2024-09-21T10:47:43.731795+0000","mapping_epoch":1430,"log_start":"0'0","ondisk_log_start":"0'0","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4663,"log_dups_size":0,"ondisk_log_size":4663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:41.775058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305594368,"num_objects":4663,"num_object_clones":0,"num_object_copies":13989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4663,"num_write_kb":298432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":168,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.304205+0000","last_change":"2024-09-21T10:39:00.855641+0000","last_active":"2024-09-21T10:47:36.304205+0000","last_peered":"2024-09-21T10:47:36.304205+0000","last_clean":"2024-09-21T10:47:36.304205+0000","last_became_active":"2024-09-21T10:39:00.848231+0000","last_became_peered":"2024-09-21T10:39:00.848231+0000","last_unstale":"2024-09-21T10:47:36.304205+0000","last_undegraded":"2024-09-21T10:47:36.304205+0000","last_fullsized":"2024-09-21T10:47:36.304205+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:18:02.478522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1140,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1292,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.303225+0000","last_change":"2024-09-21T10:43:56.011543+0000","last_active":"2024-09-21T10:47:36.303225+0000","last_peered":"2024-09-21T10:47:36.303225+0000","last_clean":"2024-09-21T10:47:36.303225+0000","last_became_active":"2024-09-21T10:43:56.010878+0000","last_became_peered":"2024-09-21T10:43:56.010878+0000","last_unstale":"2024-09-21T10:47:36.303225+0000","last_undegraded":"2024-09-21T10:47:36.303225+0000","last_fullsized":"2024-09-21T10:47:36.303225+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:46.231617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2205,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298220+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:47:36.298220+0000","last_peered":"2024-09-21T10:47:36.298220+0000","last_clean":"2024-09-21T10:47:36.298220+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:47:36.298220+0000","last_undegraded":"2024-09-21T10:47:36.298220+0000","last_fullsized":"2024-09-21T10:47:36.298220+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2223,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298141+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:47:36.298141+0000","last_peered":"2024-09-21T10:47:36.298141+0000","last_clean":"2024-09-21T10:47:36.298141+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:47:36.298141+0000","last_undegraded":"2024-09-21T10:47:36.298141+0000","last_fullsized":"2024-09-21T10:47:36.298141+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":208,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.184404+0000","last_change":"2024-09-21T10:36:21.872262+0000","last_active":"2024-09-21T10:47:22.184404+0000","last_peered":"2024-09-21T10:47:22.184404+0000","last_clean":"2024-09-21T10:47:22.184404+0000","last_became_active":"2024-09-21T10:36:21.858629+0000","last_became_peered":"2024-09-21T10:36:21.858629+0000","last_unstale":"2024-09-21T10:47:22.184404+0000","last_undegraded":"2024-09-21T10:47:22.184404+0000","last_fullsized":"2024-09-21T10:47:22.184404+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:23.748135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1406,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297794+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:47:36.297794+0000","last_peered":"2024-09-21T10:47:36.297794+0000","last_clean":"2024-09-21T10:47:36.297794+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:47:36.297794+0000","last_undegraded":"2024-09-21T10:47:36.297794+0000","last_fullsized":"2024-09-21T10:47:36.297794+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":208,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183738+0000","last_change":"2024-09-21T10:36:21.878749+0000","last_active":"2024-09-21T10:47:22.183738+0000","last_peered":"2024-09-21T10:47:22.183738+0000","last_clean":"2024-09-21T10:47:22.183738+0000","last_became_active":"2024-09-21T10:36:21.859021+0000","last_became_peered":"2024-09-21T10:36:21.859021+0000","last_unstale":"2024-09-21T10:47:22.183738+0000","last_undegraded":"2024-09-21T10:47:22.183738+0000","last_fullsized":"2024-09-21T10:47:22.183738+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:24:25.015429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1493,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299535+0000","last_change":"2024-09-21T10:36:21.843932+0000","last_active":"2024-09-21T10:47:36.299535+0000","last_peered":"2024-09-21T10:47:36.299535+0000","last_clean":"2024-09-21T10:47:36.299535+0000","last_became_active":"2024-09-21T10:36:21.843598+0000","last_became_peered":"2024-09-21T10:36:21.843598+0000","last_unstale":"2024-09-21T10:47:36.299535+0000","last_undegraded":"2024-09-21T10:47:36.299535+0000","last_fullsized":"2024-09-21T10:47:36.299535+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:34:05.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":32,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.303586+0000","last_change":"2024-09-21T10:46:39.241906+0000","last_active":"2024-09-21T10:47:36.303586+0000","last_peered":"2024-09-21T10:47:36.303586+0000","last_clean":"2024-09-21T10:47:36.303586+0000","last_became_active":"2024-09-21T10:46:39.241499+0000","last_became_peered":"2024-09-21T10:46:39.241499+0000","last_unstale":"2024-09-21T10:47:36.303586+0000","last_undegraded":"2024-09-21T10:47:36.303586+0000","last_fullsized":"2024-09-21T10:47:36.303586+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:53.671658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":190,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.303177+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:47:36.303177+0000","last_peered":"2024-09-21T10:47:36.303177+0000","last_clean":"2024-09-21T10:47:36.303177+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:47:36.303177+0000","last_undegraded":"2024-09-21T10:47:36.303177+0000","last_fullsized":"2024-09-21T10:47:36.303177+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":265,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298619+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:47:36.298619+0000","last_peered":"2024-09-21T10:47:36.298619+0000","last_clean":"2024-09-21T10:47:36.298619+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:47:36.298619+0000","last_undegraded":"2024-09-21T10:47:36.298619+0000","last_fullsized":"2024-09-21T10:47:36.298619+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2212,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299444+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:47:36.299444+0000","last_peered":"2024-09-21T10:47:36.299444+0000","last_clean":"2024-09-21T10:47:36.299444+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:47:36.299444+0000","last_undegraded":"2024-09-21T10:47:36.299444+0000","last_fullsized":"2024-09-21T10:47:36.299444+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":32,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.310154+0000","last_change":"2024-09-21T10:46:39.241669+0000","last_active":"2024-09-21T10:47:36.310154+0000","last_peered":"2024-09-21T10:47:36.310154+0000","last_clean":"2024-09-21T10:47:36.310154+0000","last_became_active":"2024-09-21T10:46:39.241358+0000","last_became_peered":"2024-09-21T10:46:39.241358+0000","last_unstale":"2024-09-21T10:47:36.310154+0000","last_undegraded":"2024-09-21T10:47:36.310154+0000","last_fullsized":"2024-09-21T10:47:36.310154+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:25.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2327,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301871+0000","last_change":"2024-09-21T10:45:59.689522+0000","last_active":"2024-09-21T10:47:36.301871+0000","last_peered":"2024-09-21T10:47:36.301871+0000","last_clean":"2024-09-21T10:47:36.301871+0000","last_became_active":"2024-09-21T10:45:59.689148+0000","last_became_peered":"2024-09-21T10:45:59.689148+0000","last_unstale":"2024-09-21T10:47:36.301871+0000","last_undegraded":"2024-09-21T10:47:36.301871+0000","last_fullsized":"2024-09-21T10:47:36.301871+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:28.142643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2238,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297498+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:47:36.297498+0000","last_peered":"2024-09-21T10:47:36.297498+0000","last_clean":"2024-09-21T10:47:36.297498+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:47:36.297498+0000","last_undegraded":"2024-09-21T10:47:36.297498+0000","last_fullsized":"2024-09-21T10:47:36.297498+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1254,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.184007+0000","last_change":"2024-09-21T10:43:56.010613+0000","last_active":"2024-09-21T10:47:22.184007+0000","last_peered":"2024-09-21T10:47:22.184007+0000","last_clean":"2024-09-21T10:47:22.184007+0000","last_became_active":"2024-09-21T10:43:56.010229+0000","last_became_peered":"2024-09-21T10:43:56.010229+0000","last_unstale":"2024-09-21T10:47:22.184007+0000","last_undegraded":"2024-09-21T10:47:22.184007+0000","last_fullsized":"2024-09-21T10:47:22.184007+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:10:28.116950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2217,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297217+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:47:36.297217+0000","last_peered":"2024-09-21T10:47:36.297217+0000","last_clean":"2024-09-21T10:47:36.297217+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:47:36.297217+0000","last_undegraded":"2024-09-21T10:47:36.297217+0000","last_fullsized":"2024-09-21T10:47:36.297217+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2330,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.304528+0000","last_change":"2024-09-21T10:30:09.405033+0000","last_active":"2024-09-21T10:47:36.304528+0000","last_peered":"2024-09-21T10:47:36.304528+0000","last_clean":"2024-09-21T10:47:36.304528+0000","last_became_active":"2024-09-21T10:30:09.404539+0000","last_became_peered":"2024-09-21T10:30:09.404539+0000","last_unstale":"2024-09-21T10:47:36.304528+0000","last_undegraded":"2024-09-21T10:47:36.304528+0000","last_fullsized":"2024-09-21T10:47:36.304528+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:01:54.675926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":354,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.304756+0000","last_change":"2024-09-21T10:45:59.680994+0000","last_active":"2024-09-21T10:47:36.304756+0000","last_peered":"2024-09-21T10:47:36.304756+0000","last_clean":"2024-09-21T10:47:36.304756+0000","last_became_active":"2024-09-21T10:45:59.680636+0000","last_became_peered":"2024-09-21T10:45:59.680636+0000","last_unstale":"2024-09-21T10:47:36.304756+0000","last_undegraded":"2024-09-21T10:47:36.304756+0000","last_fullsized":"2024-09-21T10:47:36.304756+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:42.488110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2207,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298305+0000","last_change":"2024-09-21T10:45:59.681544+0000","last_active":"2024-09-21T10:47:36.298305+0000","last_peered":"2024-09-21T10:47:36.298305+0000","last_clean":"2024-09-21T10:47:36.298305+0000","last_became_active":"2024-09-21T10:45:59.681218+0000","last_became_peered":"2024-09-21T10:45:59.681218+0000","last_unstale":"2024-09-21T10:47:36.298305+0000","last_undegraded":"2024-09-21T10:47:36.298305+0000","last_fullsized":"2024-09-21T10:47:36.298305+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:29:13.323548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":200,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183625+0000","last_change":"2024-09-21T10:45:59.688959+0000","last_active":"2024-09-21T10:47:22.183625+0000","last_peered":"2024-09-21T10:47:22.183625+0000","last_clean":"2024-09-21T10:47:22.183625+0000","last_became_active":"2024-09-21T10:45:59.688647+0000","last_became_peered":"2024-09-21T10:45:59.688647+0000","last_unstale":"2024-09-21T10:47:22.183625+0000","last_undegraded":"2024-09-21T10:47:22.183625+0000","last_fullsized":"2024-09-21T10:47:22.183625+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:28.180582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":99,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299283+0000","last_change":"2024-09-21T10:45:59.710479+0000","last_active":"2024-09-21T10:47:36.299283+0000","last_peered":"2024-09-21T10:47:36.299283+0000","last_clean":"2024-09-21T10:47:36.299283+0000","last_became_active":"2024-09-21T10:45:59.694474+0000","last_became_peered":"2024-09-21T10:45:59.694474+0000","last_unstale":"2024-09-21T10:47:36.299283+0000","last_undegraded":"2024-09-21T10:47:36.299283+0000","last_fullsized":"2024-09-21T10:47:36.299283+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:32.554219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":39,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298413+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:47:36.298413+0000","last_peered":"2024-09-21T10:47:36.298413+0000","last_clean":"2024-09-21T10:47:36.298413+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:47:36.298413+0000","last_undegraded":"2024-09-21T10:47:36.298413+0000","last_fullsized":"2024-09-21T10:47:36.298413+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":394,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.303237+0000","last_change":"2024-09-21T10:45:59.705879+0000","last_active":"2024-09-21T10:47:36.303237+0000","last_peered":"2024-09-21T10:47:36.303237+0000","last_clean":"2024-09-21T10:47:36.303237+0000","last_became_active":"2024-09-21T10:45:59.689786+0000","last_became_peered":"2024-09-21T10:45:59.689786+0000","last_unstale":"2024-09-21T10:47:36.303237+0000","last_undegraded":"2024-09-21T10:47:36.303237+0000","last_fullsized":"2024-09-21T10:47:36.303237+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:50.473399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1256,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299071+0000","last_change":"2024-09-21T10:43:56.007984+0000","last_active":"2024-09-21T10:47:36.299071+0000","last_peered":"2024-09-21T10:47:36.299071+0000","last_clean":"2024-09-21T10:47:36.299071+0000","last_became_active":"2024-09-21T10:43:56.007667+0000","last_became_peered":"2024-09-21T10:43:56.007667+0000","last_unstale":"2024-09-21T10:47:36.299071+0000","last_undegraded":"2024-09-21T10:47:36.299071+0000","last_fullsized":"2024-09-21T10:47:36.299071+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:49:32.246512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":475,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297399+0000","last_change":"2024-09-21T10:46:05.453578+0000","last_active":"2024-09-21T10:47:36.297399+0000","last_peered":"2024-09-21T10:47:36.297399+0000","last_clean":"2024-09-21T10:47:36.297399+0000","last_became_active":"2024-09-21T10:45:59.689841+0000","last_became_peered":"2024-09-21T10:45:59.689841+0000","last_unstale":"2024-09-21T10:47:36.297399+0000","last_undegraded":"2024-09-21T10:47:36.297399+0000","last_fullsized":"2024-09-21T10:47:36.297399+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:15:24.488516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1333,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181531+0000","last_change":"2024-09-21T10:36:21.852046+0000","last_active":"2024-09-21T10:47:22.181531+0000","last_peered":"2024-09-21T10:47:22.181531+0000","last_clean":"2024-09-21T10:47:22.181531+0000","last_became_active":"2024-09-21T10:36:21.851564+0000","last_became_peered":"2024-09-21T10:36:21.851564+0000","last_unstale":"2024-09-21T10:47:22.181531+0000","last_undegraded":"2024-09-21T10:47:22.181531+0000","last_fullsized":"2024-09-21T10:47:22.181531+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:34:21.236816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1222,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.311944+0000","last_change":"2024-09-21T10:45:59.697070+0000","last_active":"2024-09-21T10:47:36.311944+0000","last_peered":"2024-09-21T10:47:36.311944+0000","last_clean":"2024-09-21T10:47:36.311944+0000","last_became_active":"2024-09-21T10:45:59.696548+0000","last_became_peered":"2024-09-21T10:45:59.696548+0000","last_unstale":"2024-09-21T10:47:36.311944+0000","last_undegraded":"2024-09-21T10:47:36.311944+0000","last_fullsized":"2024-09-21T10:47:36.311944+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:02.746989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":205,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301598+0000","last_change":"2024-09-21T10:45:59.705691+0000","last_active":"2024-09-21T10:47:36.301598+0000","last_peered":"2024-09-21T10:47:36.301598+0000","last_clean":"2024-09-21T10:47:36.301598+0000","last_became_active":"2024-09-21T10:45:59.699486+0000","last_became_peered":"2024-09-21T10:45:59.699486+0000","last_unstale":"2024-09-21T10:47:36.301598+0000","last_undegraded":"2024-09-21T10:47:36.301598+0000","last_fullsized":"2024-09-21T10:47:36.301598+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:52:41.452622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":249,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.305007+0000","last_change":"2024-09-21T10:37:37.848003+0000","last_active":"2024-09-21T10:47:36.305007+0000","last_peered":"2024-09-21T10:47:36.305007+0000","last_clean":"2024-09-21T10:47:36.305007+0000","last_became_active":"2024-09-21T10:36:21.850611+0000","last_became_peered":"2024-09-21T10:36:21.850611+0000","last_unstale":"2024-09-21T10:47:36.305007+0000","last_undegraded":"2024-09-21T10:47:36.305007+0000","last_fullsized":"2024-09-21T10:47:36.305007+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:30.945177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":106,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181871+0000","last_change":"2024-09-21T10:45:59.693387+0000","last_active":"2024-09-21T10:47:22.181871+0000","last_peered":"2024-09-21T10:47:22.181871+0000","last_clean":"2024-09-21T10:47:22.181871+0000","last_became_active":"2024-09-21T10:45:59.692983+0000","last_became_peered":"2024-09-21T10:45:59.692983+0000","last_unstale":"2024-09-21T10:47:22.181871+0000","last_undegraded":"2024-09-21T10:47:22.181871+0000","last_fullsized":"2024-09-21T10:47:22.181871+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:05:05.623869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2319,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.300112+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:47:36.300112+0000","last_peered":"2024-09-21T10:47:36.300112+0000","last_clean":"2024-09-21T10:47:36.300112+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:47:36.300112+0000","last_undegraded":"2024-09-21T10:47:36.300112+0000","last_fullsized":"2024-09-21T10:47:36.300112+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":36,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.311306+0000","last_change":"2024-09-21T10:46:39.246034+0000","last_active":"2024-09-21T10:47:36.311306+0000","last_peered":"2024-09-21T10:47:36.311306+0000","last_clean":"2024-09-21T10:47:36.311306+0000","last_became_active":"2024-09-21T10:46:39.243647+0000","last_became_peered":"2024-09-21T10:46:39.243647+0000","last_unstale":"2024-09-21T10:47:36.311306+0000","last_undegraded":"2024-09-21T10:47:36.311306+0000","last_fullsized":"2024-09-21T10:47:36.311306+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:53:04.275079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":403,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301702+0000","last_change":"2024-09-21T10:43:58.555202+0000","last_active":"2024-09-21T10:47:36.301702+0000","last_peered":"2024-09-21T10:47:36.301702+0000","last_clean":"2024-09-21T10:47:36.301702+0000","last_became_active":"2024-09-21T10:43:56.009648+0000","last_became_peered":"2024-09-21T10:43:56.009648+0000","last_unstale":"2024-09-21T10:47:36.301702+0000","last_undegraded":"2024-09-21T10:47:36.301702+0000","last_fullsized":"2024-09-21T10:47:36.301702+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.555120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.555120+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:45:37.803817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2329,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301032+0000","last_change":"2024-09-21T10:45:59.693562+0000","last_active":"2024-09-21T10:47:36.301032+0000","last_peered":"2024-09-21T10:47:36.301032+0000","last_clean":"2024-09-21T10:47:36.301032+0000","last_became_active":"2024-09-21T10:45:59.693223+0000","last_became_peered":"2024-09-21T10:45:59.693223+0000","last_unstale":"2024-09-21T10:47:36.301032+0000","last_undegraded":"2024-09-21T10:47:36.301032+0000","last_fullsized":"2024-09-21T10:47:36.301032+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:37:59.666534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1420,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.304334+0000","last_change":"2024-09-21T10:45:59.694166+0000","last_active":"2024-09-21T10:47:36.304334+0000","last_peered":"2024-09-21T10:47:36.304334+0000","last_clean":"2024-09-21T10:47:36.304334+0000","last_became_active":"2024-09-21T10:45:59.693276+0000","last_became_peered":"2024-09-21T10:45:59.693276+0000","last_unstale":"2024-09-21T10:47:36.304334+0000","last_undegraded":"2024-09-21T10:47:36.304334+0000","last_fullsized":"2024-09-21T10:47:36.304334+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:12.784743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2307,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299974+0000","last_change":"2024-09-21T10:30:09.413672+0000","last_active":"2024-09-21T10:47:36.299974+0000","last_peered":"2024-09-21T10:47:36.299974+0000","last_clean":"2024-09-21T10:47:36.299974+0000","last_became_active":"2024-09-21T10:30:09.413359+0000","last_became_peered":"2024-09-21T10:30:09.413359+0000","last_unstale":"2024-09-21T10:47:36.299974+0000","last_undegraded":"2024-09-21T10:47:36.299974+0000","last_fullsized":"2024-09-21T10:47:36.299974+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:06:13.191102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":414,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.305475+0000","last_change":"2024-09-21T10:45:59.694509+0000","last_active":"2024-09-21T10:47:36.305475+0000","last_peered":"2024-09-21T10:47:36.305475+0000","last_clean":"2024-09-21T10:47:36.305475+0000","last_became_active":"2024-09-21T10:45:59.693089+0000","last_became_peered":"2024-09-21T10:45:59.693089+0000","last_unstale":"2024-09-21T10:47:36.305475+0000","last_undegraded":"2024-09-21T10:47:36.305475+0000","last_fullsized":"2024-09-21T10:47:36.305475+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.698401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.698401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:22:29.691153+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2312,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298854+0000","last_change":"2024-09-21T10:36:21.857238+0000","last_active":"2024-09-21T10:47:36.298854+0000","last_peered":"2024-09-21T10:47:36.298854+0000","last_clean":"2024-09-21T10:47:36.298854+0000","last_became_active":"2024-09-21T10:36:21.856882+0000","last_became_peered":"2024-09-21T10:36:21.856882+0000","last_unstale":"2024-09-21T10:47:36.298854+0000","last_undegraded":"2024-09-21T10:47:36.298854+0000","last_fullsized":"2024-09-21T10:47:36.298854+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:33:31.742112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2314,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.300135+0000","last_change":"2024-09-21T10:45:59.691645+0000","last_active":"2024-09-21T10:47:36.300135+0000","last_peered":"2024-09-21T10:47:36.300135+0000","last_clean":"2024-09-21T10:47:36.300135+0000","last_became_active":"2024-09-21T10:45:59.691334+0000","last_became_peered":"2024-09-21T10:45:59.691334+0000","last_unstale":"2024-09-21T10:47:36.300135+0000","last_undegraded":"2024-09-21T10:47:36.300135+0000","last_fullsized":"2024-09-21T10:47:36.300135+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:33.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":190,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.184310+0000","last_change":"2024-09-21T10:30:09.405766+0000","last_active":"2024-09-21T10:47:22.184310+0000","last_peered":"2024-09-21T10:47:22.184310+0000","last_clean":"2024-09-21T10:47:22.184310+0000","last_became_active":"2024-09-21T10:30:09.405486+0000","last_became_peered":"2024-09-21T10:30:09.405486+0000","last_unstale":"2024-09-21T10:47:22.184310+0000","last_undegraded":"2024-09-21T10:47:22.184310+0000","last_fullsized":"2024-09-21T10:47:22.184310+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:28.108650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":200,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181538+0000","last_change":"2024-09-21T10:33:25.163885+0000","last_active":"2024-09-21T10:47:22.181538+0000","last_peered":"2024-09-21T10:47:22.181538+0000","last_clean":"2024-09-21T10:47:22.181538+0000","last_became_active":"2024-09-21T10:30:09.402033+0000","last_became_peered":"2024-09-21T10:30:09.402033+0000","last_unstale":"2024-09-21T10:47:22.181538+0000","last_undegraded":"2024-09-21T10:47:22.181538+0000","last_fullsized":"2024-09-21T10:47:22.181538+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.163810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.163810+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:46:51.925630+0000","scrub_duration":0.001591784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2304,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.300063+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:47:36.300063+0000","last_peered":"2024-09-21T10:47:36.300063+0000","last_clean":"2024-09-21T10:47:36.300063+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:47:36.300063+0000","last_undegraded":"2024-09-21T10:47:36.300063+0000","last_fullsized":"2024-09-21T10:47:36.300063+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1425,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.300959+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:47:36.300959+0000","last_peered":"2024-09-21T10:47:36.300959+0000","last_clean":"2024-09-21T10:47:36.300959+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:47:36.300959+0000","last_undegraded":"2024-09-21T10:47:36.300959+0000","last_fullsized":"2024-09-21T10:47:36.300959+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":194,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183347+0000","last_change":"2024-09-21T10:30:09.429583+0000","last_active":"2024-09-21T10:47:22.183347+0000","last_peered":"2024-09-21T10:47:22.183347+0000","last_clean":"2024-09-21T10:47:22.183347+0000","last_became_active":"2024-09-21T10:30:09.407052+0000","last_became_peered":"2024-09-21T10:30:09.407052+0000","last_unstale":"2024-09-21T10:47:22.183347+0000","last_undegraded":"2024-09-21T10:47:22.183347+0000","last_fullsized":"2024-09-21T10:47:22.183347+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:11:38.007551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":635,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299047+0000","last_change":"2024-09-21T10:45:59.693657+0000","last_active":"2024-09-21T10:47:36.299047+0000","last_peered":"2024-09-21T10:47:36.299047+0000","last_clean":"2024-09-21T10:47:36.299047+0000","last_became_active":"2024-09-21T10:45:59.693283+0000","last_became_peered":"2024-09-21T10:45:59.693283+0000","last_unstale":"2024-09-21T10:47:36.299047+0000","last_undegraded":"2024-09-21T10:47:36.299047+0000","last_fullsized":"2024-09-21T10:47:36.299047+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:26.099521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":39,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.300976+0000","last_change":"2024-09-21T10:46:30.264517+0000","last_active":"2024-09-21T10:47:36.300976+0000","last_peered":"2024-09-21T10:47:36.300976+0000","last_clean":"2024-09-21T10:47:36.300976+0000","last_became_active":"2024-09-21T10:46:30.264122+0000","last_became_peered":"2024-09-21T10:46:30.264122+0000","last_unstale":"2024-09-21T10:47:36.300976+0000","last_undegraded":"2024-09-21T10:47:36.300976+0000","last_fullsized":"2024-09-21T10:47:36.300976+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:20:23.130946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2305,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298282+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:47:36.298282+0000","last_peered":"2024-09-21T10:47:36.298282+0000","last_clean":"2024-09-21T10:47:36.298282+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:47:36.298282+0000","last_undegraded":"2024-09-21T10:47:36.298282+0000","last_fullsized":"2024-09-21T10:47:36.298282+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2237,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.310448+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:47:36.310448+0000","last_peered":"2024-09-21T10:47:36.310448+0000","last_clean":"2024-09-21T10:47:36.310448+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:47:36.310448+0000","last_undegraded":"2024-09-21T10:47:36.310448+0000","last_fullsized":"2024-09-21T10:47:36.310448+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":490,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298301+0000","last_change":"2024-09-21T10:40:10.918077+0000","last_active":"2024-09-21T10:47:36.298301+0000","last_peered":"2024-09-21T10:47:36.298301+0000","last_clean":"2024-09-21T10:47:36.298301+0000","last_became_active":"2024-09-21T10:40:10.917840+0000","last_became_peered":"2024-09-21T10:40:10.917840+0000","last_unstale":"2024-09-21T10:47:36.298301+0000","last_undegraded":"2024-09-21T10:47:36.298301+0000","last_fullsized":"2024-09-21T10:47:36.298301+0000","mapping_epoch":1378,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1379,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:30.617798+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":162,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301484+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:47:36.301484+0000","last_peered":"2024-09-21T10:47:36.301484+0000","last_clean":"2024-09-21T10:47:36.301484+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:47:36.301484+0000","last_undegraded":"2024-09-21T10:47:36.301484+0000","last_fullsized":"2024-09-21T10:47:36.301484+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2337,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.306858+0000","last_change":"2024-09-21T10:36:21.853204+0000","last_active":"2024-09-21T10:47:36.306858+0000","last_peered":"2024-09-21T10:47:36.306858+0000","last_clean":"2024-09-21T10:47:36.306858+0000","last_became_active":"2024-09-21T10:36:21.852887+0000","last_became_peered":"2024-09-21T10:36:21.852887+0000","last_unstale":"2024-09-21T10:47:36.306858+0000","last_undegraded":"2024-09-21T10:47:36.306858+0000","last_fullsized":"2024-09-21T10:47:36.306858+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:48.666154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":582,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298372+0000","last_change":"2024-09-21T10:45:59.691940+0000","last_active":"2024-09-21T10:47:36.298372+0000","last_peered":"2024-09-21T10:47:36.298372+0000","last_clean":"2024-09-21T10:47:36.298372+0000","last_became_active":"2024-09-21T10:45:59.691610+0000","last_became_peered":"2024-09-21T10:45:59.691610+0000","last_unstale":"2024-09-21T10:47:36.298372+0000","last_undegraded":"2024-09-21T10:47:36.298372+0000","last_fullsized":"2024-09-21T10:47:36.298372+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:57.766019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1350,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.304418+0000","last_change":"2024-09-21T10:46:46.607744+0000","last_active":"2024-09-21T10:47:36.304418+0000","last_peered":"2024-09-21T10:47:36.304418+0000","last_clean":"2024-09-21T10:47:36.304418+0000","last_became_active":"2024-09-21T10:46:46.607376+0000","last_became_peered":"2024-09-21T10:46:46.607376+0000","last_unstale":"2024-09-21T10:47:36.304418+0000","last_undegraded":"2024-09-21T10:47:36.304418+0000","last_fullsized":"2024-09-21T10:47:36.304418+0000","mapping_epoch":1433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:17.279053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2222,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297894+0000","last_change":"2024-09-21T10:30:09.407066+0000","last_active":"2024-09-21T10:47:36.297894+0000","last_peered":"2024-09-21T10:47:36.297894+0000","last_clean":"2024-09-21T10:47:36.297894+0000","last_became_active":"2024-09-21T10:30:09.406711+0000","last_became_peered":"2024-09-21T10:30:09.406711+0000","last_unstale":"2024-09-21T10:47:36.297894+0000","last_undegraded":"2024-09-21T10:47:36.297894+0000","last_fullsized":"2024-09-21T10:47:36.297894+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:00.346806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":211,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183663+0000","last_change":"2024-09-21T10:45:59.698368+0000","last_active":"2024-09-21T10:47:22.183663+0000","last_peered":"2024-09-21T10:47:22.183663+0000","last_clean":"2024-09-21T10:47:22.183663+0000","last_became_active":"2024-09-21T10:45:59.684909+0000","last_became_peered":"2024-09-21T10:45:59.684909+0000","last_unstale":"2024-09-21T10:47:22.183663+0000","last_undegraded":"2024-09-21T10:47:22.183663+0000","last_fullsized":"2024-09-21T10:47:22.183663+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:09.098971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1573,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.313822+0000","last_change":"2024-09-21T10:45:59.692592+0000","last_active":"2024-09-21T10:47:36.313822+0000","last_peered":"2024-09-21T10:47:36.313822+0000","last_clean":"2024-09-21T10:47:36.313822+0000","last_became_active":"2024-09-21T10:45:59.692232+0000","last_became_peered":"2024-09-21T10:45:59.692232+0000","last_unstale":"2024-09-21T10:47:36.313822+0000","last_undegraded":"2024-09-21T10:47:36.313822+0000","last_fullsized":"2024-09-21T10:47:36.313822+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:26.920879+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":190,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183444+0000","last_change":"2024-09-21T10:30:09.407389+0000","last_active":"2024-09-21T10:47:22.183444+0000","last_peered":"2024-09-21T10:47:22.183444+0000","last_clean":"2024-09-21T10:47:22.183444+0000","last_became_active":"2024-09-21T10:30:09.406576+0000","last_became_peered":"2024-09-21T10:30:09.406576+0000","last_unstale":"2024-09-21T10:47:22.183444+0000","last_undegraded":"2024-09-21T10:47:22.183444+0000","last_fullsized":"2024-09-21T10:47:22.183444+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:30:46.810755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2317,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301172+0000","last_change":"2024-09-21T10:45:59.697853+0000","last_active":"2024-09-21T10:47:36.301172+0000","last_peered":"2024-09-21T10:47:36.301172+0000","last_clean":"2024-09-21T10:47:36.301172+0000","last_became_active":"2024-09-21T10:45:59.697402+0000","last_became_peered":"2024-09-21T10:45:59.697402+0000","last_unstale":"2024-09-21T10:47:36.301172+0000","last_undegraded":"2024-09-21T10:47:36.301172+0000","last_fullsized":"2024-09-21T10:47:36.301172+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:03:20.734347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2215,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297110+0000","last_change":"2024-09-21T10:45:59.689063+0000","last_active":"2024-09-21T10:47:36.297110+0000","last_peered":"2024-09-21T10:47:36.297110+0000","last_clean":"2024-09-21T10:47:36.297110+0000","last_became_active":"2024-09-21T10:45:59.688749+0000","last_became_peered":"2024-09-21T10:45:59.688749+0000","last_unstale":"2024-09-21T10:47:36.297110+0000","last_undegraded":"2024-09-21T10:47:36.297110+0000","last_fullsized":"2024-09-21T10:47:36.297110+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:10:59.204604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":473,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298200+0000","last_change":"2024-09-21T10:33:25.629816+0000","last_active":"2024-09-21T10:47:36.298200+0000","last_peered":"2024-09-21T10:47:36.298200+0000","last_clean":"2024-09-21T10:47:36.298200+0000","last_became_active":"2024-09-21T10:33:21.762647+0000","last_became_peered":"2024-09-21T10:33:21.762647+0000","last_unstale":"2024-09-21T10:47:36.298200+0000","last_undegraded":"2024-09-21T10:47:36.298200+0000","last_fullsized":"2024-09-21T10:47:36.298200+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:25:56.627482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":39,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299667+0000","last_change":"2024-09-21T10:46:30.264653+0000","last_active":"2024-09-21T10:47:36.299667+0000","last_peered":"2024-09-21T10:47:36.299667+0000","last_clean":"2024-09-21T10:47:36.299667+0000","last_became_active":"2024-09-21T10:46:30.264068+0000","last_became_peered":"2024-09-21T10:46:30.264068+0000","last_unstale":"2024-09-21T10:47:36.299667+0000","last_undegraded":"2024-09-21T10:47:36.299667+0000","last_fullsized":"2024-09-21T10:47:36.299667+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:53.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1371,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.310492+0000","last_change":"2024-09-21T10:45:59.680814+0000","last_active":"2024-09-21T10:47:36.310492+0000","last_peered":"2024-09-21T10:47:36.310492+0000","last_clean":"2024-09-21T10:47:36.310492+0000","last_became_active":"2024-09-21T10:45:59.680499+0000","last_became_peered":"2024-09-21T10:45:59.680499+0000","last_unstale":"2024-09-21T10:47:36.310492+0000","last_undegraded":"2024-09-21T10:47:36.310492+0000","last_fullsized":"2024-09-21T10:47:36.310492+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:48:01.487484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":194,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181651+0000","last_change":"2024-09-21T10:30:09.438340+0000","last_active":"2024-09-21T10:47:22.181651+0000","last_peered":"2024-09-21T10:47:22.181651+0000","last_clean":"2024-09-21T10:47:22.181651+0000","last_became_active":"2024-09-21T10:30:09.410177+0000","last_became_peered":"2024-09-21T10:30:09.410177+0000","last_unstale":"2024-09-21T10:47:22.181651+0000","last_undegraded":"2024-09-21T10:47:22.181651+0000","last_fullsized":"2024-09-21T10:47:22.181651+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:56:05.868829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1372,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.305180+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:47:36.305180+0000","last_peered":"2024-09-21T10:47:36.305180+0000","last_clean":"2024-09-21T10:47:36.305180+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:47:36.305180+0000","last_undegraded":"2024-09-21T10:47:36.305180+0000","last_fullsized":"2024-09-21T10:47:36.305180+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1363,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298765+0000","last_change":"2024-09-21T10:43:56.012844+0000","last_active":"2024-09-21T10:47:36.298765+0000","last_peered":"2024-09-21T10:47:36.298765+0000","last_clean":"2024-09-21T10:47:36.298765+0000","last_became_active":"2024-09-21T10:43:56.012504+0000","last_became_peered":"2024-09-21T10:43:56.012504+0000","last_unstale":"2024-09-21T10:47:36.298765+0000","last_undegraded":"2024-09-21T10:47:36.298765+0000","last_fullsized":"2024-09-21T10:47:36.298765+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:02.556342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1404,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297961+0000","last_change":"2024-09-21T10:46:04.469943+0000","last_active":"2024-09-21T10:47:36.297961+0000","last_peered":"2024-09-21T10:47:36.297961+0000","last_clean":"2024-09-21T10:47:36.297961+0000","last_became_active":"2024-09-21T10:45:59.680550+0000","last_became_peered":"2024-09-21T10:45:59.680550+0000","last_unstale":"2024-09-21T10:47:36.297961+0000","last_undegraded":"2024-09-21T10:47:36.297961+0000","last_fullsized":"2024-09-21T10:47:36.297961+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:04.469883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:04.469883+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:33:08.642477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1424,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297789+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:47:36.297789+0000","last_peered":"2024-09-21T10:47:36.297789+0000","last_clean":"2024-09-21T10:47:36.297789+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:47:36.297789+0000","last_undegraded":"2024-09-21T10:47:36.297789+0000","last_fullsized":"2024-09-21T10:47:36.297789+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":436,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299429+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:47:36.299429+0000","last_peered":"2024-09-21T10:47:36.299429+0000","last_clean":"2024-09-21T10:47:36.299429+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:47:36.299429+0000","last_undegraded":"2024-09-21T10:47:36.299429+0000","last_fullsized":"2024-09-21T10:47:36.299429+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":388,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297964+0000","last_change":"2024-09-21T10:46:08.467544+0000","last_active":"2024-09-21T10:47:36.297964+0000","last_peered":"2024-09-21T10:47:36.297964+0000","last_clean":"2024-09-21T10:47:36.297964+0000","last_became_active":"2024-09-21T10:45:59.695828+0000","last_became_peered":"2024-09-21T10:45:59.695828+0000","last_unstale":"2024-09-21T10:47:36.297964+0000","last_undegraded":"2024-09-21T10:47:36.297964+0000","last_fullsized":"2024-09-21T10:47:36.297964+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:08.467485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:08.467485+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:36.742481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2336,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.300879+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:47:36.300879+0000","last_peered":"2024-09-21T10:47:36.300879+0000","last_clean":"2024-09-21T10:47:36.300879+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:47:36.300879+0000","last_undegraded":"2024-09-21T10:47:36.300879+0000","last_fullsized":"2024-09-21T10:47:36.300879+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":190,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183655+0000","last_change":"2024-09-21T10:30:09.406727+0000","last_active":"2024-09-21T10:47:22.183655+0000","last_peered":"2024-09-21T10:47:22.183655+0000","last_clean":"2024-09-21T10:47:22.183655+0000","last_became_active":"2024-09-21T10:30:09.406453+0000","last_became_peered":"2024-09-21T10:30:09.406453+0000","last_unstale":"2024-09-21T10:47:22.183655+0000","last_undegraded":"2024-09-21T10:47:22.183655+0000","last_fullsized":"2024-09-21T10:47:22.183655+0000","mapping_epoch":1259,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:51.093235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":453,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.183912+0000","last_change":"2024-09-21T10:44:00.317312+0000","last_active":"2024-09-21T10:47:22.183912+0000","last_peered":"2024-09-21T10:47:22.183912+0000","last_clean":"2024-09-21T10:47:22.183912+0000","last_became_active":"2024-09-21T10:43:56.006731+0000","last_became_peered":"2024-09-21T10:43:56.006731+0000","last_unstale":"2024-09-21T10:47:22.183912+0000","last_undegraded":"2024-09-21T10:47:22.183912+0000","last_fullsized":"2024-09-21T10:47:22.183912+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:30:47.520862+0000","scrub_duration":0.012369583,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1205,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298937+0000","last_change":"2024-09-21T10:45:59.688833+0000","last_active":"2024-09-21T10:47:36.298937+0000","last_peered":"2024-09-21T10:47:36.298937+0000","last_clean":"2024-09-21T10:47:36.298937+0000","last_became_active":"2024-09-21T10:45:59.686751+0000","last_became_peered":"2024-09-21T10:45:59.686751+0000","last_unstale":"2024-09-21T10:47:36.298937+0000","last_undegraded":"2024-09-21T10:47:36.298937+0000","last_fullsized":"2024-09-21T10:47:36.298937+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:33:04.676103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1399,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.184249+0000","last_change":"2024-09-21T10:45:59.694156+0000","last_active":"2024-09-21T10:47:22.184249+0000","last_peered":"2024-09-21T10:47:22.184249+0000","last_clean":"2024-09-21T10:47:22.184249+0000","last_became_active":"2024-09-21T10:45:59.686496+0000","last_became_peered":"2024-09-21T10:45:59.686496+0000","last_unstale":"2024-09-21T10:47:22.184249+0000","last_undegraded":"2024-09-21T10:47:22.184249+0000","last_fullsized":"2024-09-21T10:47:22.184249+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:04:55.832764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":474,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297256+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:47:36.297256+0000","last_peered":"2024-09-21T10:47:36.297256+0000","last_clean":"2024-09-21T10:47:36.297256+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:47:36.297256+0000","last_undegraded":"2024-09-21T10:47:36.297256+0000","last_fullsized":"2024-09-21T10:47:36.297256+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1406,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301718+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:47:36.301718+0000","last_peered":"2024-09-21T10:47:36.301718+0000","last_clean":"2024-09-21T10:47:36.301718+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:47:36.301718+0000","last_undegraded":"2024-09-21T10:47:36.301718+0000","last_fullsized":"2024-09-21T10:47:36.301718+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":73,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299964+0000","last_change":"2024-09-21T10:45:59.707648+0000","last_active":"2024-09-21T10:47:36.299964+0000","last_peered":"2024-09-21T10:47:36.299964+0000","last_clean":"2024-09-21T10:47:36.299964+0000","last_became_active":"2024-09-21T10:45:59.691996+0000","last_became_peered":"2024-09-21T10:45:59.691996+0000","last_unstale":"2024-09-21T10:47:36.299964+0000","last_undegraded":"2024-09-21T10:47:36.299964+0000","last_fullsized":"2024-09-21T10:47:36.299964+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:41:24.439451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":67,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298441+0000","last_change":"2024-09-21T10:46:00.158554+0000","last_active":"2024-09-21T10:47:36.298441+0000","last_peered":"2024-09-21T10:47:36.298441+0000","last_clean":"2024-09-21T10:47:36.298441+0000","last_became_active":"2024-09-21T10:45:59.680546+0000","last_became_peered":"2024-09-21T10:45:59.680546+0000","last_unstale":"2024-09-21T10:47:36.298441+0000","last_undegraded":"2024-09-21T10:47:36.298441+0000","last_fullsized":"2024-09-21T10:47:36.298441+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:00.158500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:00.158500+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:39.581987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":200,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181720+0000","last_change":"2024-09-21T10:45:59.679381+0000","last_active":"2024-09-21T10:47:22.181720+0000","last_peered":"2024-09-21T10:47:22.181720+0000","last_clean":"2024-09-21T10:47:22.181720+0000","last_became_active":"2024-09-21T10:45:59.679131+0000","last_became_peered":"2024-09-21T10:45:59.679131+0000","last_unstale":"2024-09-21T10:47:22.181720+0000","last_undegraded":"2024-09-21T10:47:22.181720+0000","last_fullsized":"2024-09-21T10:47:22.181720+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:17:50.700756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1430,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.304322+0000","last_change":"2024-09-21T10:45:59.703618+0000","last_active":"2024-09-21T10:47:36.304322+0000","last_peered":"2024-09-21T10:47:36.304322+0000","last_clean":"2024-09-21T10:47:36.304322+0000","last_became_active":"2024-09-21T10:45:59.699339+0000","last_became_peered":"2024-09-21T10:45:59.699339+0000","last_unstale":"2024-09-21T10:47:36.304322+0000","last_undegraded":"2024-09-21T10:47:36.304322+0000","last_fullsized":"2024-09-21T10:47:36.304322+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:18:19.242317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":66,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":177,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181541+0000","last_change":"2024-09-21T10:33:38.244569+0000","last_active":"2024-09-21T10:47:22.181541+0000","last_peered":"2024-09-21T10:47:22.181541+0000","last_clean":"2024-09-21T10:47:22.181541+0000","last_became_active":"2024-09-21T10:33:20.898636+0000","last_became_peered":"2024-09-21T10:33:20.898636+0000","last_unstale":"2024-09-21T10:47:22.181541+0000","last_undegraded":"2024-09-21T10:47:22.181541+0000","last_fullsized":"2024-09-21T10:47:22.181541+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:38.244467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:38.244467+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:51.496212+0000","scrub_duration":0.0097793499999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1461,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.313352+0000","last_change":"2024-09-21T10:43:56.007452+0000","last_active":"2024-09-21T10:47:36.313352+0000","last_peered":"2024-09-21T10:47:36.313352+0000","last_clean":"2024-09-21T10:47:36.313352+0000","last_became_active":"2024-09-21T10:43:56.007103+0000","last_became_peered":"2024-09-21T10:43:56.007103+0000","last_unstale":"2024-09-21T10:47:36.313352+0000","last_undegraded":"2024-09-21T10:47:36.313352+0000","last_fullsized":"2024-09-21T10:47:36.313352+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:08:41.710693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2205,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.305611+0000","last_change":"2024-09-21T10:45:59.696879+0000","last_active":"2024-09-21T10:47:36.305611+0000","last_peered":"2024-09-21T10:47:36.305611+0000","last_clean":"2024-09-21T10:47:36.305611+0000","last_became_active":"2024-09-21T10:45:59.696055+0000","last_became_peered":"2024-09-21T10:45:59.696055+0000","last_unstale":"2024-09-21T10:47:36.305611+0000","last_undegraded":"2024-09-21T10:47:36.305611+0000","last_fullsized":"2024-09-21T10:47:36.305611+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:10.189668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":480,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.298097+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:47:36.298097+0000","last_peered":"2024-09-21T10:47:36.298097+0000","last_clean":"2024-09-21T10:47:36.298097+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:47:36.298097+0000","last_undegraded":"2024-09-21T10:47:36.298097+0000","last_fullsized":"2024-09-21T10:47:36.298097+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1259,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.306648+0000","last_change":"2024-09-21T10:45:59.693730+0000","last_active":"2024-09-21T10:47:36.306648+0000","last_peered":"2024-09-21T10:47:36.306648+0000","last_clean":"2024-09-21T10:47:36.306648+0000","last_became_active":"2024-09-21T10:45:59.692582+0000","last_became_peered":"2024-09-21T10:45:59.692582+0000","last_unstale":"2024-09-21T10:47:36.306648+0000","last_undegraded":"2024-09-21T10:47:36.306648+0000","last_fullsized":"2024-09-21T10:47:36.306648+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:21.994503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2324,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.301791+0000","last_change":"2024-09-21T10:45:59.697081+0000","last_active":"2024-09-21T10:47:36.301791+0000","last_peered":"2024-09-21T10:47:36.301791+0000","last_clean":"2024-09-21T10:47:36.301791+0000","last_became_active":"2024-09-21T10:45:59.696414+0000","last_became_peered":"2024-09-21T10:45:59.696414+0000","last_unstale":"2024-09-21T10:47:36.301791+0000","last_undegraded":"2024-09-21T10:47:36.301791+0000","last_fullsized":"2024-09-21T10:47:36.301791+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:47.528914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":187,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.181353+0000","last_change":"2024-09-21T10:45:59.693591+0000","last_active":"2024-09-21T10:47:22.181353+0000","last_peered":"2024-09-21T10:47:22.181353+0000","last_clean":"2024-09-21T10:47:22.181353+0000","last_became_active":"2024-09-21T10:45:59.693061+0000","last_became_peered":"2024-09-21T10:45:59.693061+0000","last_unstale":"2024-09-21T10:47:22.181353+0000","last_undegraded":"2024-09-21T10:47:22.181353+0000","last_fullsized":"2024-09-21T10:47:22.181353+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:56.259939+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2323,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.299145+0000","last_change":"2024-09-21T10:45:59.700672+0000","last_active":"2024-09-21T10:47:36.299145+0000","last_peered":"2024-09-21T10:47:36.299145+0000","last_clean":"2024-09-21T10:47:36.299145+0000","last_became_active":"2024-09-21T10:45:59.699332+0000","last_became_peered":"2024-09-21T10:45:59.699332+0000","last_unstale":"2024-09-21T10:47:36.299145+0000","last_undegraded":"2024-09-21T10:47:36.299145+0000","last_fullsized":"2024-09-21T10:47:36.299145+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:48.567053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":210,"reported_epoch":1439,"state":"active+clean","last_fresh":"2024-09-21T10:47:22.184336+0000","last_change":"2024-09-21T10:45:59.680367+0000","last_active":"2024-09-21T10:47:22.184336+0000","last_peered":"2024-09-21T10:47:22.184336+0000","last_clean":"2024-09-21T10:47:22.184336+0000","last_became_active":"2024-09-21T10:45:59.679973+0000","last_became_peered":"2024-09-21T10:45:59.679973+0000","last_unstale":"2024-09-21T10:47:22.184336+0000","last_undegraded":"2024-09-21T10:47:22.184336+0000","last_fullsized":"2024-09-21T10:47:22.184336+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:29:18.078677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":198,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.306553+0000","last_change":"2024-09-21T10:37:49.672276+0000","last_active":"2024-09-21T10:47:36.306553+0000","last_peered":"2024-09-21T10:47:36.306553+0000","last_clean":"2024-09-21T10:47:36.306553+0000","last_became_active":"2024-09-21T10:37:41.924744+0000","last_became_peered":"2024-09-21T10:37:41.924744+0000","last_unstale":"2024-09-21T10:47:36.306553+0000","last_undegraded":"2024-09-21T10:47:36.306553+0000","last_fullsized":"2024-09-21T10:47:36.306553+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:37:28.108197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":76,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.303687+0000","last_change":"2024-09-21T10:45:59.714397+0000","last_active":"2024-09-21T10:47:36.303687+0000","last_peered":"2024-09-21T10:47:36.303687+0000","last_clean":"2024-09-21T10:47:36.303687+0000","last_became_active":"2024-09-21T10:45:59.689648+0000","last_became_peered":"2024-09-21T10:45:59.689648+0000","last_unstale":"2024-09-21T10:47:36.303687+0000","last_undegraded":"2024-09-21T10:47:36.303687+0000","last_fullsized":"2024-09-21T10:47:36.303687+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:20.596738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2213,"reported_epoch":1440,"state":"active+clean","last_fresh":"2024-09-21T10:47:36.297497+0000","last_change":"2024-09-21T10:45:59.696141+0000","last_active":"2024-09-21T10:47:36.297497+0000","last_peered":"2024-09-21T10:47:36.297497+0000","last_clean":"2024-09-21T10:47:36.297497+0000","last_became_active":"2024-09-21T10:45:59.695761+0000","last_became_peered":"2024-09-21T10:45:59.695761+0000","last_unstale":"2024-09-21T10:47:36.297497+0000","last_undegraded":"2024-09-21T10:47:36.297497+0000","last_fullsized":"2024-09-21T10:47:36.297497+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:46.714846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":50,"num_pg":16,"stat_sum":{"num_bytes":4890230784,"num_objects":74619,"num_object_clones":0,"num_object_copies":223857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74619,"num_write_kb":4775616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14680653824,"data_stored":14680653824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74619,"ondisk_log_size":74619,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":6214,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":26316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":210453398138,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049797,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2982432,"kb_used_data":2059288,"kb_used_omap":242,"kb_used_meta":922893,"kb_avail":90762720,"statfs":{"total":95995035648,"available":92941025280,"internally_reserved":0,"allocated":2108710912,"data_stored":2104603546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":945042655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790596,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2985688,"kb_used_data":2084496,"kb_used_omap":1,"kb_used_meta":901182,"kb_avail":90759464,"statfs":{"total":95995035648,"available":92937691136,"internally_reserved":0,"allocated":2134523904,"data_stored":2130420998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922810712},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644756,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3688848,"kb_used_data":2677064,"kb_used_omap":67,"kb_used_meta":1011708,"kb_avail":90056304,"statfs":{"total":95995035648,"available":92217655296,"internally_reserved":0,"allocated":2741313536,"data_stored":2737218313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1035989853},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527774,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3371676,"kb_used_data":2405460,"kb_used_omap":75,"kb_used_meta":966132,"kb_avail":90373476,"statfs":{"total":95995035648,"available":92542439424,"internally_reserved":0,"allocated":2463191040,"data_stored":2459088733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":989319322},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274927,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3188684,"kb_used_data":2153476,"kb_used_omap":51,"kb_used_meta":1035148,"kb_avail":90556468,"statfs":{"total":95995035648,"available":92729823232,"internally_reserved":0,"allocated":2205159424,"data_stored":2201067634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1059991893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354165,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3732896,"kb_used_data":2949528,"kb_used_omap":66,"kb_used_meta":783229,"kb_avail":90012256,"statfs":{"total":95995035648,"available":92172550144,"internally_reserved":0,"allocated":3020316672,"data_stored":3016209821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":802026820},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,6],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3011772416,"data_stored":3011772416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2197684224,"data_stored":2197684224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2472411136,"data_stored":2472411136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2759000064,"data_stored":2759000064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2131820544,"data_stored":2131820544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2107965440,"data_stored":2107965440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:47:50.955 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:47:50.955 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:47:50.955 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 6, 5] 2024-09-21T10:47:50.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd out 5 2024-09-21T10:47:51.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:50 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:47:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:50 smithi145 ceph-mon[89359]: from='client.56290 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:47:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:50 smithi145 ceph-mon[89359]: pgmap v2578: 109 pgs: 109 active+clean; 4.7 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-21T10:47:51.090 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:51.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:51.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:51.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.236 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:51.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:51.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 59 16 80897 80881 85.661 59 0.0161487 0.0116676 2024-09-21T10:47:51.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:51.361 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:51.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:51.488 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:51.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:51.614 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:51.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:51.765 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:51.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:51.891 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:51.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:51.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:52.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:52.135 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:52.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:52.192 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:52.260 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:52.283 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:52.295 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:47:52.293626+0000 min lat: 0.00116109 max lat: 0.303819 avg lat: 0.0117232 2024-09-21T10:47:52.295 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:47:52.296 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 60 16 81884 81868 85.2614 61.6875 0.0264901 0.0117232 2024-09-21T10:47:52.385 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:52.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:52.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:52.533 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:52.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:52.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:52.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:52.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:52.826 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:52.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:52.990 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:52.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:53.092 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:53.115 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:53.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:53.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:53 smithi119 ceph-mon[101432]: pgmap v2579: 109 pgs: 109 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s 2024-09-21T10:47:53.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:53 smithi119 ceph-mon[105288]: pgmap v2579: 109 pgs: 109 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s 2024-09-21T10:47:53.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.259 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:53.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:53.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 61 16 82849 82833 84.8523 60.3125 0.0257266 0.0117779 2024-09-21T10:47:53.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.410 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:53.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:53.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:53 smithi145 ceph-mon[89359]: pgmap v2579: 109 pgs: 109 active+clean; 4.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 58 MiB/s wr, 933 op/s 2024-09-21T10:47:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.553 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:53.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:53.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:53.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:53.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:53.834 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:53.936 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:53.958 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:53.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:54.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.100 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:54.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:54.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:54.226 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:54.294 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 62 16 83806 83790 84.4484 59.8125 0.0173202 0.011835 2024-09-21T10:47:54.328 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:54.363 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:54.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:54.488 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:54.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:54.612 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:54.686 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:47:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:47:54] "GET /metrics HTTP/1.1" 200 36833 "" "Prometheus/2.51.0" 2024-09-21T10:47:54.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:54.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:54.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:54.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:54.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:54.984 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:55.006 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:55.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:55.131 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:55.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:55.256 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:55.295 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 63 16 84653 84637 83.9482 52.9375 0.00780595 0.0119007 2024-09-21T10:47:55.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:55 smithi119 ceph-mon[105288]: pgmap v2580: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:47:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1043867017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:55 smithi119 ceph-mon[101432]: pgmap v2580: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:47:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1043867017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:55.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:55.536 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:55 smithi145 ceph-mon[89359]: pgmap v2580: 109 pgs: 109 active+clean; 5.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-21T10:47:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1043867017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:47:55.638 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:55.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:55.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:55.799 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:55.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:55.944 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:55.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:55.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:56.046 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:56.086 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:56.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:56.188 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:47:56.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:47:56.295 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 64 16 85538 85522 83.5008 55.3125 0.0148017 0.0119696 2024-09-21T10:47:56.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:56.345 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:56.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:56.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:56.483 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:47:56.491 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:47:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:47:56.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:56.629 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:56.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:56.773 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:56.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:56.905 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:56.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:57.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:57.031 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:57.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:57.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:57.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:57.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:57.296 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 65 16 86420 86404 83.0638 55.125 0.00728344 0.0120336 2024-09-21T10:47:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:57 smithi119 ceph-mon[105288]: pgmap v2581: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:47:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:57 smithi119 ceph-mon[101432]: pgmap v2581: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:47:57.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:57.440 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:57.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:57 smithi145 ceph-mon[89359]: pgmap v2581: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 54 MiB/s wr, 861 op/s 2024-09-21T10:47:57.562 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:57.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:57.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:57.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:57.828 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:57.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:47:57.952 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:57.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:57.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:47:58.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.101 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:58.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:58.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:47:58.228 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:47:58.295 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 66 16 86888 86872 82.2485 29.25 0.0728574 0.0121483 2024-09-21T10:47:58.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:58.355 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:58.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:47:58.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:58.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:58.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:58.641 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:58.743 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:58.766 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:58.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:58.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:58.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:59.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:59.044 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:59.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:59.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:47:59 smithi119 ceph-mon[101432]: pgmap v2582: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 50 MiB/s wr, 801 op/s 2024-09-21T10:47:59.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:47:59 smithi119 ceph-mon[105288]: pgmap v2582: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 50 MiB/s wr, 801 op/s 2024-09-21T10:47:59.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:47:59.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:47:59.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:47:59.298 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 67 16 87756 87740 81.8273 54.25 0.0109246 0.0122151 2024-09-21T10:47:59.330 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:59.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:47:59.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:59.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:59.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:47:59 smithi145 ceph-mon[89359]: pgmap v2582: 109 pgs: 109 active+clean; 5.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 50 MiB/s wr, 801 op/s 2024-09-21T10:47:59.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:47:59.599 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:47:59.652 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:47:59.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:47:59.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:47:59.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:47:59.879 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:47:59.902 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:47:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:47:59.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:00.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:00.044 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:00.115 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:48:00.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:00.196 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:00.298 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:00.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 68 16 88576 88560 81.3776 51.25 0.0203021 0.0122825 2024-09-21T10:48:00.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/305712895' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:00.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:00 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/305712895' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:00.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:00 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:00.423 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:00.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:00.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/305712895' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:00 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:48:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:48:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:48:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:48:00.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:00.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:00.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:00.725 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:00.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:00.865 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:00.967 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:00.991 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:00.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:00.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:01.093 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:01.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:01.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[101432]: pgmap v2583: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 61 MiB/s wr, 971 op/s 2024-09-21T10:48:01.233 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:01.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:01.234 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[101432]: osdmap e1441: 8 total, 8 up, 6 in 2024-09-21T10:48:01.234 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[105288]: pgmap v2583: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 61 MiB/s wr, 971 op/s 2024-09-21T10:48:01.234 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:01.234 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:01.235 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:01 smithi119 ceph-mon[105288]: osdmap e1441: 8 total, 8 up, 6 in 2024-09-21T10:48:01.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:01.283 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:01.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:01.298 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 69 16 89480 89464 81.017 56.5 0.0195377 0.012337 2024-09-21T10:48:01.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:01.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:01.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:01 smithi145 ceph-mon[89359]: pgmap v2583: 109 pgs: 109 active+clean; 5.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 61 MiB/s wr, 971 op/s 2024-09-21T10:48:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:01 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:01 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:48:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:01 smithi145 ceph-mon[89359]: osdmap e1441: 8 total, 8 up, 6 in 2024-09-21T10:48:01.578 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:01.594 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:01.595 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:01.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:01.720 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:01.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:01.822 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:01.845 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:01.946 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:01.970 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:01.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:02.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:02.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:02.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:02.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:48:02.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:02.298 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 70 16 90348 90332 80.6346 54.25 0.0165618 0.012396 2024-09-21T10:48:02.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:02.387 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:02.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:02.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T10:48:02.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:02.513 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:02.615 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:02.637 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:02.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:02.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:02.918 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:02.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:02.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:48:03.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:03.041 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:03.143 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:03.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:03.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:03.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 71 16 91111 91095 80.1705 47.6875 0.012044 0.0124679 2024-09-21T10:48:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:03.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:03 smithi119 ceph-mon[105288]: pgmap v2585: 109 pgs: 109 active+clean; 5.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-21T10:48:03.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:03 smithi119 ceph-mon[101432]: pgmap v2585: 109 pgs: 109 active+clean; 5.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-21T10:48:03.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:03.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:03.476 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:03 smithi145 ceph-mon[89359]: pgmap v2585: 109 pgs: 109 active+clean; 5.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 54 MiB/s wr, 869 op/s 2024-09-21T10:48:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:03.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:03.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:03.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:03.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:03.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:03.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:03.868 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:03.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:03.969 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:03.990 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:03.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:04.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.146 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:04.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:04.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:04.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 72 16 91792 91776 79.6481 42.5625 0.0148953 0.01255 2024-09-21T10:48:04.303 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:04.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:04.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:04.440 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:04.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:04.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:04.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:04.712 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:04.797 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:04.797 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:04.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:04.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.837 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:04.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:04.870 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:48:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:48:04] "GET /metrics HTTP/1.1" 200 36833 "" "Prometheus/2.51.0" 2024-09-21T10:48:04.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:04.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:04.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:05.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:05.121 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:05.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:05.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:05.246 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:05.248 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:05 smithi119 ceph-mon[101432]: pgmap v2586: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 53 MiB/s wr, 843 op/s 2024-09-21T10:48:05.249 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:05 smithi119 ceph-mon[105288]: pgmap v2586: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 53 MiB/s wr, 843 op/s 2024-09-21T10:48:05.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 73 16 92649 92633 79.2908 53.5625 0.0289958 0.0126068 2024-09-21T10:48:05.348 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.371 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:05.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:05.472 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:05.508 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:05 smithi145 ceph-mon[89359]: pgmap v2586: 109 pgs: 109 active+clean; 5.5 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 53 MiB/s wr, 843 op/s 2024-09-21T10:48:05.610 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:05.633 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:05.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:05.783 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:05.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:05.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:06.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:06.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:06.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:06.208 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:06.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:06.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 74 16 93499 93483 78.9372 53.125 0.0185344 0.0126625 2024-09-21T10:48:06.310 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:06.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:06.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:06.484 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.511 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:06.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:06.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:06.726 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:06 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:48:06.557+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1441 definitely_dead 0 2024-09-21T10:48:06.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:06.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:06.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:06.928 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:06.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:06.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:07.029 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:07.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:07.133 INFO:teuthology.orchestra.run.smithi119.stderr:marked out osd.5. 2024-09-21T10:48:07.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:07.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:07.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 75 16 94175 94159 78.448 42.25 0.00415777 0.0127178 2024-09-21T10:48:07.313 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[105288]: pgmap v2587: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 51 MiB/s wr, 818 op/s 2024-09-21T10:48:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4251397310' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:48:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:48:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[105288]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:48:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[101432]: pgmap v2587: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 51 MiB/s wr, 818 op/s 2024-09-21T10:48:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4251397310' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:48:07.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:48:07.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:07 smithi119 ceph-mon[101432]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:48:07.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:07.436 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:07.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:07.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:07 smithi145 ceph-mon[89359]: pgmap v2587: 109 pgs: 109 active+clean; 5.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 51 MiB/s wr, 818 op/s 2024-09-21T10:48:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4251397310' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:48:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["5"]} : dispatch 2024-09-21T10:48:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:07 smithi145 ceph-mon[89359]: Client client.admin marked osd.5 out, while it was still marked up 2024-09-21T10:48:07.560 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:07.662 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:07.683 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:07.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:07.806 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:07.908 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:07.930 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:07.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:08.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:08.053 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:08.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:08.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:08.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 76 16 94175 94159 77.4159 0 - 0.0127178 2024-09-21T10:48:08.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:08.324 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:08.425 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:08.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:08 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:48:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:08 smithi145 ceph-mon[89359]: osdmap e1442: 8 total, 8 up, 5 in 2024-09-21T10:48:08.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:08 smithi145 ceph-mon[89359]: osdmap e1443: 8 total, 8 up, 5 in 2024-09-21T10:48:08.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:08.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:08 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:48:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:08 smithi119 ceph-mon[105288]: osdmap e1442: 8 total, 8 up, 5 in 2024-09-21T10:48:08.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:08 smithi119 ceph-mon[105288]: osdmap e1443: 8 total, 8 up, 5 in 2024-09-21T10:48:08.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:08 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-09-21T10:48:08.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:08 smithi119 ceph-mon[101432]: osdmap e1442: 8 total, 8 up, 5 in 2024-09-21T10:48:08.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:08 smithi119 ceph-mon[101432]: osdmap e1443: 8 total, 8 up, 5 in 2024-09-21T10:48:08.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:08.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:08.796 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:08.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:08.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:08.940 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:08.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:08.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:09.042 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:09.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:09.165 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:09.192 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:09.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:09.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 77 16 94182 94166 76.4163 0.21875 2.11522 0.0128754 2024-09-21T10:48:09.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[101432]: pgmap v2589: 109 pgs: 109 active+clean; 5.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[101432]: 1.12 scrub ok 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[101432]: osdmap e1444: 8 total, 8 up, 5 in 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[105288]: pgmap v2589: 109 pgs: 109 active+clean; 5.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[105288]: 1.12 scrub ok 2024-09-21T10:48:09.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:09 smithi119 ceph-mon[105288]: osdmap e1444: 8 total, 8 up, 5 in 2024-09-21T10:48:09.417 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:09 smithi145 ceph-mon[89359]: pgmap v2589: 109 pgs: 109 active+clean; 5.6 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 43 MiB/s wr, 690 op/s 2024-09-21T10:48:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:09 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:48:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:09 smithi145 ceph-mon[89359]: 1.12 scrub ok 2024-09-21T10:48:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:09 smithi145 ceph-mon[89359]: osdmap e1444: 8 total, 8 up, 5 in 2024-09-21T10:48:09.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:09.591 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:09.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:09.733 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:09.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:09.876 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:09.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:09.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:09.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.019 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:10.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:10.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:10.144 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:10.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:10.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:10.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:10.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 78 16 94821 94805 75.9486 39.9375 0.0179396 0.0131612 2024-09-21T10:48:10.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:10.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:10.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:10.573 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:48:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:48:10.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 1.1 scrub starts 2024-09-21T10:48:10.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 1.1 scrub ok 2024-09-21T10:48:10.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5830 2024-09-21T10:48:10.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5817 2024-09-21T10:48:10.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.b starting backfill to osd.0 from (0'0,0'0] MAX to 1441'5961 2024-09-21T10:48:10.573 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.5 starting backfill to osd.2 from (0'0,0'0] MAX to 1441'5836 2024-09-21T10:48:10.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.d starting backfill to osd.2 from (0'0,0'0] MAX to 1441'5880 2024-09-21T10:48:10.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5908 2024-09-21T10:48:10.574 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:10 smithi145 ceph-mon[89359]: 50.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1441'5917 2024-09-21T10:48:10.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 1.1 scrub starts 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 1.1 scrub ok 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5830 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5817 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.b starting backfill to osd.0 from (0'0,0'0] MAX to 1441'5961 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.5 starting backfill to osd.2 from (0'0,0'0] MAX to 1441'5836 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.d starting backfill to osd.2 from (0'0,0'0] MAX to 1441'5880 2024-09-21T10:48:10.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5908 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[101432]: 50.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1441'5917 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 1.1 scrub starts 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 1.1 scrub ok 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.4 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5830 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.2 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5817 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.b starting backfill to osd.0 from (0'0,0'0] MAX to 1441'5961 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.5 starting backfill to osd.2 from (0'0,0'0] MAX to 1441'5836 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.d starting backfill to osd.2 from (0'0,0'0] MAX to 1441'5880 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.0 starting backfill to osd.6 from (0'0,0'0] MAX to 1441'5908 2024-09-21T10:48:10.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:10 smithi119 ceph-mon[105288]: 50.7 starting backfill to osd.3 from (0'0,0'0] MAX to 1441'5917 2024-09-21T10:48:10.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:10.724 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:10.826 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:10.848 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:10.950 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:10.995 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:10.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:11.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:11.119 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:11.221 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:11.245 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:11.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 79 16 95702 95686 75.6843 55.0625 0.0137632 0.0132071 2024-09-21T10:48:11.347 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:11.370 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.472 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:11.496 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:11 smithi119 ceph-mon[101432]: pgmap v2592: 109 pgs: 1 active+remapped+backfilling, 1 unknown, 13 peering, 94 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 5.7 KiB/s rd, 29 MiB/s wr, 471 op/s; 5833/279525 objects misplaced (2.087%); 47 KiB/s, 0 objects/s recovering 2024-09-21T10:48:11.496 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:11 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:48:11.496 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:11 smithi119 ceph-mon[105288]: pgmap v2592: 109 pgs: 1 active+remapped+backfilling, 1 unknown, 13 peering, 94 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 5.7 KiB/s rd, 29 MiB/s wr, 471 op/s; 5833/279525 objects misplaced (2.087%); 47 KiB/s, 0 objects/s recovering 2024-09-21T10:48:11.496 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:11 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:48:11.501 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:11 smithi145 ceph-mon[89359]: pgmap v2592: 109 pgs: 1 active+remapped+backfilling, 1 unknown, 13 peering, 94 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 5.7 KiB/s rd, 29 MiB/s wr, 471 op/s; 5833/279525 objects misplaced (2.087%); 47 KiB/s, 0 objects/s recovering 2024-09-21T10:48:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:11 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-21T10:48:11.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:11.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:11.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:11.771 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:11.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:11.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:11.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:12.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:12.044 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:12.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:12.169 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:12.270 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:12.299 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:2024-09-21T10:48:12.298861+0000 min lat: 0.00116109 max lat: 2.25578 avg lat: 0.0132521 2024-09-21T10:48:12.300 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-21T10:48:12.300 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 80 16 96581 96565 75.4247 54.9375 0.031537 0.0132521 2024-09-21T10:48:12.322 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:12.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:12.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:12.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:12.606 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:12.708 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:12.731 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:12.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:12.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:12.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:12.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.031 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:13.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:13.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:13.172 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:13.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:13.302 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 81 16 97384 97368 75.1114 50.1875 0.0145015 0.0133074 2024-09-21T10:48:13.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:13.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:13 smithi119 ceph-mon[101432]: pgmap v2593: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 1 unknown, 13 peering, 90 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 5.7 KiB/s rd, 36 MiB/s wr, 573 op/s; 29463/284337 objects misplaced (10.362%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:48:13.365 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:13 smithi119 ceph-mon[105288]: pgmap v2593: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 1 unknown, 13 peering, 90 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 5.7 KiB/s rd, 36 MiB/s wr, 573 op/s; 29463/284337 objects misplaced (10.362%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:48:13.417 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:13.439 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:13.482 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:13.482 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:13.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:13 smithi145 ceph-mon[89359]: pgmap v2593: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 1 unknown, 13 peering, 90 active+clean; 5.8 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 5.7 KiB/s rd, 36 MiB/s wr, 573 op/s; 29463/284337 objects misplaced (10.362%); 2.5 MiB/s, 40 objects/s recovering 2024-09-21T10:48:13.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:13.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:13.719 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:13.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:13.863 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:13.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:13.965 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:13.988 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:13.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:14.091 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:14.113 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:14.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:14.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:14.254 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:14.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:14.302 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 82 16 98244 98228 74.8508 53.75 0.026163 0.013355 2024-09-21T10:48:14.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:14.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:14.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:14.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:14.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:14 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-21T10:48:14.543 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:14 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-21T10:48:14.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:14 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-21T10:48:14.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:14.687 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:14.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:14.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T10:48:14.812 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:14.840 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:48:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:48:14] "GET /metrics HTTP/1.1" 200 36833 "" "Prometheus/2.51.0" 2024-09-21T10:48:14.914 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:14.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:14.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:15.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.116 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:15.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:15.155 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:48:15.155 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:15.207 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:48:15.208 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:15.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:15.247 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:48:15.247 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:15.260 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:15.302 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:48:15.302 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:15.302 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 83 16 99027 99011 74.5386 48.9375 0.0157192 0.0134106 2024-09-21T10:48:15.329 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:48:15.329 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:15.358 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:48:15.359 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:15.359 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:15 smithi119 ceph-mon[101432]: pgmap v2594: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 37 MiB/s wr, 592 op/s; 41403/287739 objects misplaced (14.389%); 4.2 MiB/s, 69 objects/s recovering 2024-09-21T10:48:15.359 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3014168655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:15.359 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:15 smithi119 ceph-mon[105288]: pgmap v2594: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 37 MiB/s wr, 592 op/s; 41403/287739 objects misplaced (14.389%); 4.2 MiB/s, 69 objects/s recovering 2024-09-21T10:48:15.359 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3014168655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:15.362 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:15.385 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:15.396 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:48:15.396 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:15.424 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:48:15.425 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:15.486 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:15.525 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:15 smithi145 ceph-mon[89359]: pgmap v2594: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 37 MiB/s wr, 592 op/s; 41403/287739 objects misplaced (14.389%); 4.2 MiB/s, 69 objects/s recovering 2024-09-21T10:48:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3014168655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:15.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:15.654 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:15.756 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:15.785 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:15.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:15.887 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:15.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:15.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:16.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:16.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:16.168 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.191 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:16.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:16.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:16.305 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 84 16 99822 99806 74.2407 49.6875 0.0224256 0.0134639 2024-09-21T10:48:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.317 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:16.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:16.418 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.444 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:16.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:16.545 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:16.572 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:16.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.733 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:16.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:16.835 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:16.859 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:16.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:16.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.006 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:17.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:17.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:17.161 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:17.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.288 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:17.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:17.305 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 85 16 100674 100658 73.9935 53.25 0.0183183 0.0135097 2024-09-21T10:48:17.389 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:17.412 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:17.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:17.536 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:17.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:17 smithi145 ceph-mon[89359]: pgmap v2595: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 44 MiB/s wr, 710 op/s; 41150/292560 objects misplaced (14.065%); 9.7 MiB/s, 156 objects/s recovering 2024-09-21T10:48:17.545 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:17 smithi119 ceph-mon[105288]: pgmap v2595: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 44 MiB/s wr, 710 op/s; 41150/292560 objects misplaced (14.065%); 9.7 MiB/s, 156 objects/s recovering 2024-09-21T10:48:17.545 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:17 smithi119 ceph-mon[101432]: pgmap v2595: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 44 MiB/s wr, 710 op/s; 41150/292560 objects misplaced (14.065%); 9.7 MiB/s, 156 objects/s recovering 2024-09-21T10:48:17.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:17.662 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:17.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:17.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.809 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:17.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:17.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:17.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:17.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:17.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:18.055 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:18.078 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:18.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:18.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:18.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:18.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:18.283 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:48:18.306 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:18.307 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 86 16 101466 101450 73.7067 49.5 0.0318142 0.0135612 2024-09-21T10:48:18.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:18.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:18.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:18.474 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1496209935' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:18 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:18.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:18.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:18.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1496209935' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:18.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:18 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:18.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:18.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1496209935' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:18.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:18 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:48:18.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:18.740 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:18.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:18.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:18.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:18.990 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:19.013 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:19.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:19.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:19.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:19.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:19.308 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 87 16 102301 102285 73.4588 52.1875 0.00905611 0.0136078 2024-09-21T10:48:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[105288]: pgmap v2596: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 35 MiB/s wr, 560 op/s; 41480/295704 objects misplaced (14.028%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:48:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:48:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[105288]: osdmap e1445: 8 total, 8 up, 5 in 2024-09-21T10:48:19.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[101432]: pgmap v2596: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 35 MiB/s wr, 560 op/s; 41480/295704 objects misplaced (14.028%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:48:19.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:19.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:48:19.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:19.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:19 smithi119 ceph-mon[101432]: osdmap e1445: 8 total, 8 up, 5 in 2024-09-21T10:48:19.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:19.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:19.506 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:19 smithi145 ceph-mon[89359]: pgmap v2596: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 35 MiB/s wr, 560 op/s; 41480/295704 objects misplaced (14.028%); 11 MiB/s, 171 objects/s recovering 2024-09-21T10:48:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:19 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:48:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:19 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:48:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:19 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:48:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:19 smithi145 ceph-mon[89359]: osdmap e1445: 8 total, 8 up, 5 in 2024-09-21T10:48:19.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:19.623 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:19.725 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.788 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:19.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:19.889 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:19.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:19.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:20.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:20.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:48:20.036 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:20.137 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:20.159 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:20.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:20.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:20.286 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:20.308 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 88 16 103023 103007 73.1369 45.125 0.0188826 0.0136667 2024-09-21T10:48:20.388 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:20.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:20.504 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:48:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:48:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:48:20.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:20.558 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:20.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:20.684 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:20.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:48:20.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:48:20.760 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.786 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:20.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:20.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:48:20.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:48:20.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:20.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:20.959 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:20.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:21.042 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:21.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:21.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:21.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:48:21.186 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:21.210 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:21.308 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 89 16 103839 103823 72.8877 51 0.0223564 0.0137137 2024-09-21T10:48:21.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:21.360 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:21.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:21.517 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:21.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:21.619 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:21.644 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:21.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:21.766 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6] out_osds: [4, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:48:21.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:48:21.771 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:21.788 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:21 smithi119 ceph-mon[105288]: pgmap v2598: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 45 MiB/s wr, 714 op/s; 41015/300999 objects misplaced (13.626%); 18 MiB/s, 281 objects/s recovering 2024-09-21T10:48:21.788 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:21 smithi119 ceph-mon[101432]: pgmap v2598: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 45 MiB/s wr, 714 op/s; 41015/300999 objects misplaced (13.626%); 18 MiB/s, 281 objects/s recovering 2024-09-21T10:48:21.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:21 smithi145 ceph-mon[89359]: pgmap v2598: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 45 MiB/s wr, 714 op/s; 41015/300999 objects misplaced (13.626%); 18 MiB/s, 281 objects/s recovering 2024-09-21T10:48:21.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:21.896 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:21.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:21.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:21.998 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:22.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:22.121 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.165 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:22.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:22.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:22.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:22.310 INFO:tasks.radosbench.radosbench.0.smithi119.stdout: 90 16 104574 104558 72.5883 45.9375 0.0125319 0.0137659 2024-09-21T10:48:22.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:22.414 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:22.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:22.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:22.672 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:22.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:22.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:22.926 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:22.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:23.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:23.055 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:23.157 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:23.178 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:23.280 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:23.301 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:23.308 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total time run: 90.0513 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Total writes made: 104574 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Write size: 65536 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Object size: 65536 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Bandwidth (MB/sec): 72.5795 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Bandwidth: 32.8684 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max bandwidth (MB/sec): 192.312 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min bandwidth (MB/sec): 0 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average IOPS: 1161 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev IOPS: 525.906 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max IOPS: 3077 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min IOPS: 0 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Average Latency(s): 0.013773 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Stddev Latency(s): 0.0301562 2024-09-21T10:48:23.309 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Max latency(s): 2.25578 2024-09-21T10:48:23.310 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Min latency(s): 0.00116109 2024-09-21T10:48:23.315 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Cleaning up (deleting benchmark objects) 2024-09-21T10:48:23.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:23.427 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:23.529 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:23.550 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:23.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:23.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:23.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:23.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:23.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:23 smithi145 ceph-mon[89359]: pgmap v2599: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 49 MiB/s wr, 778 op/s; 41564/307719 objects misplaced (13.507%); 19 MiB/s, 298 objects/s recovering 2024-09-21T10:48:23.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:23.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:23 smithi119 ceph-mon[105288]: pgmap v2599: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 49 MiB/s wr, 778 op/s; 41564/307719 objects misplaced (13.507%); 19 MiB/s, 298 objects/s recovering 2024-09-21T10:48:23.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:23 smithi119 ceph-mon[101432]: pgmap v2599: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 49 MiB/s wr, 778 op/s; 41564/307719 objects misplaced (13.507%); 19 MiB/s, 298 objects/s recovering 2024-09-21T10:48:23.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:23.936 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:23.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:24.038 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:24.060 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:24.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:24.162 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:24.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:24.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:24.328 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:24.430 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:24.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:24.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:24.592 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:24.693 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:24.715 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:24.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.859 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:24.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:48:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:48:24] "GET /metrics HTTP/1.1" 200 36857 "" "Prometheus/2.51.0" 2024-09-21T10:48:24.960 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:24.983 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:24.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:25.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:25.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:25.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.256 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:25.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:25.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:25.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:25.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:25.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:25.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:25.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:25.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:25.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:25.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:25 smithi145 ceph-mon[89359]: pgmap v2600: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 49 MiB/s wr, 788 op/s; 41277/311388 objects misplaced (13.256%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:48:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:25 smithi119 ceph-mon[105288]: pgmap v2600: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 49 MiB/s wr, 788 op/s; 41277/311388 objects misplaced (13.256%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:48:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:25 smithi119 ceph-mon[101432]: pgmap v2600: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 49 MiB/s wr, 788 op/s; 41277/311388 objects misplaced (13.256%); 21 MiB/s, 341 objects/s recovering 2024-09-21T10:48:25.890 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:25.930 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:25.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:26.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:26.055 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:26.156 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.197 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:26.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:26.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.336 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:26.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:26.437 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:26.460 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:26.562 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:26.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:26.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:26.727 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:26.829 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:26.873 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:26.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:26.975 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:26.999 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:27.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:27.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:27.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:27.260 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:27.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:27.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:27.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:27.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:27.636 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:27.658 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:27.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:27.759 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:27.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.782 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:27.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:27.793 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:27 smithi145 ceph-mon[89359]: pgmap v2601: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 43 MiB/s wr, 739 op/s; 40201/311979 objects misplaced (12.886%); 24 MiB/s, 382 objects/s recovering 2024-09-21T10:48:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:27 smithi119 ceph-mon[105288]: pgmap v2601: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 43 MiB/s wr, 739 op/s; 40201/311979 objects misplaced (12.886%); 24 MiB/s, 382 objects/s recovering 2024-09-21T10:48:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:27 smithi119 ceph-mon[101432]: pgmap v2601: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 43 MiB/s wr, 739 op/s; 40201/311979 objects misplaced (12.886%); 24 MiB/s, 382 objects/s recovering 2024-09-21T10:48:27.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:27.922 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:27.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:28.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:28.032 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:48:28.032 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:48:28.032 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57571 49.e 49:7f608a63:::benchmark_data_smithi119_301770_object57570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.359537+0000", 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.84975881499997, 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.31887065199999998, 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.033 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57571 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.359537+0000", 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.034 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.359537+0000", 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.359539+0000", 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4929999999999999e-06 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.359561+0000", 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2175999999999999e-05 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.035 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.359573+0000", 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1919000000000001e-05 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.359580+0000", 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8510000000000001e-06 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.360121+0000", 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000540874 2024-09-21T10:48:28.036 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.360282+0000", 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016188500000000001 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.360437+0000", 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015484 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.037 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.373890+0000", 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013452761000000001 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.038 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374440+0000", 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00055026200000000004 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678317+0000", 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30387702300000002 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.039 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678369+0000", 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1756999999999999e-05 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678408+0000", 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8810999999999998e-05 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57576 49.e 49:7895bd86:::benchmark_data_smithi119_301770_object57575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.040 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.365618+0000", 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.84367787999997, 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.31300328500000002, 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57576 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.041 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365618+0000", 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365618+0000", 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365620+0000", 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.437e-06 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.042 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365647+0000", 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7636000000000001e-05 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365661+0000", 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4103e-05 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365669+0000", 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6890000000000004e-06 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.043 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.365943+0000", 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00027436399999999998 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366075+0000", 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013148399999999999 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.044 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366194+0000", 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011947600000000001 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.373897+0000", 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0077026689999999997 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374472+0000", 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00057535799999999999 2024-09-21T10:48:28.045 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678476+0000", 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30400379999999999 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678561+0000", 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.5079000000000003e-05 2024-09-21T10:48:28.046 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678621+0000", 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0189999999999998e-05 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57581 49.e 49:70a22b2d:::benchmark_data_smithi119_301770_object57580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.366010+0000", 2024-09-21T10:48:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.84328604500001, 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.31305134699999998, 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57581 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.048 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366010+0000", 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366010+0000", 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366011+0000", 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.049e-06 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.049 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366032+0000", 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1038000000000001e-05 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366044+0000", 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1653e-05 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366051+0000", 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0820000000000001e-06 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.050 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.369275+0000", 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032239439999999999 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.369420+0000", 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014554300000000001 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.369555+0000", 2024-09-21T10:48:28.051 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013482699999999999 2024-09-21T10:48:28.052 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.052 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.052 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.053 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.373901+0000", 2024-09-21T10:48:28.053 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043461369999999999 2024-09-21T10:48:28.053 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678684+0000", 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30478233999999998 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678923+0000", 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00023955 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.054 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679025+0000", 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010155299999999999 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679061+0000", 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.6631000000000002e-05 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.055 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57582 49.e 49:7f7cbc07:::benchmark_data_smithi119_301770_object57581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.366064+0000", 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.84323220599998, 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.31311253, 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57582 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.056 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366064+0000", 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366064+0000", 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366064+0000", 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1699999999999998e-07 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.057 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366082+0000", 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7893999999999998e-05 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366089+0000", 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6850000000000004e-06 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.366094+0000", 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8130000000000003e-06 2024-09-21T10:48:28.058 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371660+0000", 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0055661809999999999 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371786+0000", 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012633800000000001 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371924+0000", 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000137396 2024-09-21T10:48:28.059 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678275+0000", 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30635172100000002 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678722+0000", 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00044621999999999999 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679097+0000", 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00037572400000000002 2024-09-21T10:48:28.060 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679141+0000", 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3659000000000003e-05 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679176+0000", 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5382000000000001e-05 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.061 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57584 49.7 49:e99087fc:::benchmark_data_smithi119_301770_object57583:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.371012+0000", 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83828369700001, 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30630667299999997, 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57584 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.062 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371012+0000", 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371012+0000", 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371014+0000", 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0190000000000001e-06 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.063 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371040+0000", 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.5157e-05 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371052+0000", 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2029e-05 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371059+0000", 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.5850000000000002e-06 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.064 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371991+0000", 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00093218700000000001 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.372125+0000", 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000134046 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,5", 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.372245+0000", 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000119886 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.065 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.674380+0000", 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30213450600000002 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.674412+0000", 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1866999999999998e-05 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677228+0000", 2024-09-21T10:48:28.066 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0028161190000000002 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677275+0000", 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.727e-05 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677319+0000", 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4002000000000003e-05 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.067 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57586 49.9 e1400/1394 49:96f23a56:::benchmark_data_smithi119_301770_object57585:head v 1400'3549, mlcod=1400'3547)", 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.371127+0000", 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83816896299999, 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.302302283, 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371127+0000", 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.068 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371127+0000", 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371128+0000", 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8200000000000001e-07 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371170+0000", 2024-09-21T10:48:28.069 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2228999999999998e-05 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371181+0000", 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1106e-05 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371186+0000", 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7040000000000002e-06 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.070 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371295+0000", 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00010954399999999999 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.371313+0000", 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8399000000000001e-05 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375341+0000", 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0040273920000000003 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.071 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673429+0000", 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29808852699999999 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57587 49.e 49:79512dcc:::benchmark_data_smithi119_301770_object57586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.374745+0000", 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83455083600001, 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.304595903, 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.072 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57587 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374745+0000", 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374745+0000", 2024-09-21T10:48:28.073 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374747+0000", 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6619999999999999e-06 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374775+0000", 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7821e-05 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.074 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374787+0000", 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2768e-05 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374795+0000", 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.8830000000000006e-06 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374985+0000", 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000190096 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.075 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375115+0000", 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012946999999999999 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375237+0000", 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0001225 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678636+0000", 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30339869699999999 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.076 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678764+0000", 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012752799999999999 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679273+0000", 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000509316 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679312+0000", 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9365e-05 2024-09-21T10:48:28.077 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.679341+0000", 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8796999999999999e-05 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.078 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57590 49.4 49:2e20189d:::benchmark_data_smithi119_301770_object57589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.374816+0000", 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83448021300001, 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30266200500000001, 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57590 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.079 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374816+0000", 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374816+0000", 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374816+0000", 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5700000000000002e-07 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.080 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374872+0000", 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6047000000000001e-05 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374881+0000", 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0189999999999995e-06 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374888+0000", 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.404e-06 2024-09-21T10:48:28.081 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.374923+0000", 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.4671000000000001e-05 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375070+0000", 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014718399999999999 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375214+0000", 2024-09-21T10:48:28.082 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000144423 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.675246+0000", 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30003175999999998 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.675466+0000", 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000219574 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.083 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677339+0000", 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0018734750000000001 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677435+0000", 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.6317999999999995e-05 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677478+0000", 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2573e-05 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.084 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57592 49.e 49:7d8461d1:::benchmark_data_smithi119_301770_object57591:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.375511+0000", 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83378468500001, 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.31470445600000002, 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57592 2024-09-21T10:48:28.085 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375511+0000", 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375511+0000", 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375512+0000", 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.057e-06 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375534+0000", 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1435000000000001e-05 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672640+0000", 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29710599300000001 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.087 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672651+0000", 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0910000000000001e-05 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.683630+0000", 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010979256 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.683751+0000", 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012116999999999999 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.088 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.683883+0000", 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013150400000000001 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687145+0000", 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0032618220000000002 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.689552+0000", 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024072059999999998 2024-09-21T10:48:28.089 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.690096+0000", 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00054465600000000002 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.690168+0000", 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1997999999999997e-05 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.690216+0000", 2024-09-21T10:48:28.090 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.7448999999999999e-05 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36978.0:57571 49.e e1400/1394 ondisk, result = 0)", 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.376591+0000", 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83270520899998, 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30182794400000001, 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.091 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.376591+0000", 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.376591+0000", 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.376592+0000", 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1699999999999997e-07 2024-09-21T10:48:28.092 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.376596+0000", 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4270000000000001e-06 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672340+0000", 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.295743688 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672368+0000", 2024-09-21T10:48:28.093 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7606e-05 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678304+0000", 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0059369649999999998 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678312+0000", 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.328e-06 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.094 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678419+0000", 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107013 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop_reply(client.36978.0:57590 49.4 e1400/1394 ondisk, result = 0)", 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.377736+0000", 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 328.83156015999998, 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29751745699999999, 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.095 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377736+0000", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377736+0000", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377737+0000", 2024-09-21T10:48:28.096 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9500000000000002e-07 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.377742+0000", 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4750000000000001e-06 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672766+0000", 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29502402700000002 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.097 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672778+0000", 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1916000000000001e-05 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.675233+0000", 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024554590000000001 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.675241+0000", 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1409999999999999e-06 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.098 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.675253+0000", 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2744e-05 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160751 49.4 49:23821a15:::benchmark_data_smithi119_301770_object47264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.388094+0000", 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 215.82120161, 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.33214813999999998, 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.099 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160751 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.388094+0000", 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.100 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.388094+0000", 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.388096+0000", 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7349999999999999e-06 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.388102+0000", 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1630000000000001e-06 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.101 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451830+0000", 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.063728110000000004 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451839+0000", 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8440000000000004e-06 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.456302+0000", 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0044625649999999999 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.102 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.456410+0000", 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107972 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.456516+0000", 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000106456 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.459689+0000", 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0031730629999999998 2024-09-21T10:48:28.103 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.460125+0000", 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00043555299999999998 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.720129+0000", 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26000451899999999 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.720198+0000", 2024-09-21T10:48:28.104 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8442999999999995e-05 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.720243+0000", 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4716999999999999e-05 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160754 49.f 49:fec0a46d:::benchmark_data_smithi119_301770_object47267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.451890+0000", 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 215.757406291, 2024-09-21T10:48:28.105 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.26482756600000001, 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160754 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451890+0000", 2024-09-21T10:48:28.106 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451890+0000", 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451891+0000", 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1650000000000001e-06 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.107 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451895+0000", 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7019999999999999e-06 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451903+0000", 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.6929999999999996e-06 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451909+0000", 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.5280000000000003e-06 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.108 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.451959+0000", 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0429000000000001e-05 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.452085+0000", 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012586800000000001 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.452182+0000", 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7136e-05 2024-09-21T10:48:28.109 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.466924+0000", 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.014741384 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469095+0000", 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021711109999999999 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.110 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.716538+0000", 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24744281300000001 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.716592+0000", 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4496000000000001e-05 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.716717+0000", 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000125241 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.111 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160768 49.e 49:7ca54e28:::benchmark_data_smithi119_301770_object47281:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.469664+0000", 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 215.739631621, 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27277963999999999, 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160768 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.112 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469664+0000", 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469664+0000", 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469666+0000", 2024-09-21T10:48:28.113 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.265e-06 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469671+0000", 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8810000000000004e-06 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469680+0000", 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.7359999999999996e-06 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469687+0000", 2024-09-21T10:48:28.114 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7229999999999998e-06 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469765+0000", 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7966999999999994e-05 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469894+0000", 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012934900000000001 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.115 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470006+0000", 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011154699999999999 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.722127+0000", 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25212107700000003 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.722146+0000", 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9026999999999999e-05 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.116 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.742351+0000", 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.02020458 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.742402+0000", 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1408e-05 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.742444+0000", 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2079999999999997e-05 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.117 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160769 49.e 49:7e23f3b4:::benchmark_data_smithi119_301770_object47282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.469707+0000", 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 215.73958898800001, 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.27283665000000001, 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:48:28.118 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160769 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469707+0000", 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469707+0000", 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.119 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469708+0000", 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0900000000000002e-07 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469710+0000", 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3719999999999999e-06 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469717+0000", 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6590000000000001e-06 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.120 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.469722+0000", 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2639999999999999e-06 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.716892+0000", 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24716975599999999 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.717032+0000", 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014014799999999999 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.717160+0000", 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012853300000000001 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.722361+0000", 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.005200896 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.729448+0000", 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.007086925 2024-09-21T10:48:28.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.742477+0000", 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013028936 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.742515+0000", 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.7780000000000001e-05 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.742544+0000", 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8872000000000001e-05 2024-09-21T10:48:28.123 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:104582 50.5 50:ada6523d:::benchmark_data_smithi119_335063_object5:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1445)", 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:48:23.314472+0000", 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8948242639999999, 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.415605628, 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:48:28.124 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 104582 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.314472+0000", 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.314472+0000", 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.125 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.314473+0000", 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2470000000000001e-06 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.314478+0000", 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4769999999999997e-06 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.314488+0000", 2024-09-21T10:48:28.126 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0739e-05 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.314494+0000", 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1099999999999999e-06 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.318301+0000", 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0038068780000000001 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.127 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.318417+0000", 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000116094 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.318548+0000", 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00013054200000000001 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327683+0000", 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0091351739999999994 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.128 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327697+0000", 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4076e-05 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.329902+0000", 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022049420000000001 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726970+0000", 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.397067903 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.129 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.727030+0000", 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0458000000000003e-05 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730077+0000", 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003046988 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:104583 50.f 50:fcd29959:::benchmark_data_smithi119_335063_object6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1445)", 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:48:23.315771+0000", 2024-09-21T10:48:28.130 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8935246000000001, 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.41096917399999999, 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 104583 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.131 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315771+0000", 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315771+0000", 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315772+0000", 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9400000000000004e-07 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.132 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315775+0000", 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1159999999999999e-06 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315785+0000", 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2699999999999993e-06 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315791+0000", 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8599999999999998e-06 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.133 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315954+0000", 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000163432 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.316065+0000", 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00011052600000000001 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.316163+0000", 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8467000000000004e-05 2024-09-21T10:48:28.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.318634+0000", 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024706369999999999 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.325044+0000", 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0064099769999999999 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.135 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726653+0000", 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.40160910999999999 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726701+0000", 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8071000000000001e-05 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726741+0000", 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9814000000000002e-05 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:104586 50.5 50:afe9d1df:::benchmark_data_smithi119_335063_object9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1445)", 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:48:23.315834+0000", 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8934619550000003, 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.41445005200000001, 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 104586 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.137 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315834+0000", 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315834+0000", 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315834+0000", 2024-09-21T10:48:28.138 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.5900000000000003e-07 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315836+0000", 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.891e-06 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315842+0000", 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.4620000000000004e-06 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.139 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315854+0000", 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.2659e-05 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.321229+0000", 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0053742670000000003 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.321351+0000", 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012240800000000001 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T10:48:28.140 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.321482+0000", 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000131008 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327703+0000", 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.006221046 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327897+0000", 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00019365699999999999 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.141 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.329951+0000", 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002054054 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730129+0000", 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.40017757399999998 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730185+0000", 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6919999999999997e-05 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.142 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730284+0000", 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.8746999999999995e-05 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:104591 50.5 50:aee71bb4:::benchmark_data_smithi119_335063_object14:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1445)", 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:48:23.315884+0000", 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8934118720000002, 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.41459377200000003, 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.143 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 104591 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315884+0000", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315884+0000", 2024-09-21T10:48:28.144 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315885+0000", 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3700000000000001e-07 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315886+0000", 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7969999999999999e-06 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.145 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315892+0000", 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3299999999999998e-06 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.315896+0000", 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2429999999999999e-06 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.323857+0000", 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0079613860000000009 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.146 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.323979+0000", 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012215199999999999 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,2,5", 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.324126+0000", 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000146396 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327919+0000", 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00379291 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.147 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.328134+0000", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00021556099999999999 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.330357+0000", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.002222945 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730333+0000", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.39997595899999999 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.148 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730386+0000", 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.2877000000000002e-05 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.730478+0000", 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1879000000000006e-05 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:104594 50.f 50:f54b095c:::benchmark_data_smithi119_335063_object17:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1445)", 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:48:23.326964+0000", 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 2.8823320749999999, 2024-09-21T10:48:28.149 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.39987603799999999, 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 104594 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.326964+0000", 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.150 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.326964+0000", 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.326965+0000", 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1909999999999999e-06 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.326969+0000", 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.766e-06 2024-09-21T10:48:28.151 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.326980+0000", 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0712e-05 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.326986+0000", 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7220000000000002e-06 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327051+0000", 2024-09-21T10:48:28.152 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4595000000000001e-05 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327174+0000", 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012265799999999999 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,3", 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.327294+0000", 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012066900000000001 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.329785+0000", 2024-09-21T10:48:28.153 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0024902349999999999 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.330169+0000", 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000384189 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726772+0000", 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.39660282600000002 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.154 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726810+0000", 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8868000000000001e-05 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.726840+0000", 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9607e-05 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:48:28.155 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:48:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:28.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:28.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:28.281 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:28.383 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:28.406 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:28.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:28.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:28.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:28.692 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:28.793 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:28.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:28.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:28.954 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:28.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:29.056 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:29.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:29.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:29.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:29.305 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:29.327 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:29.429 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:29.453 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:29.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:29.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:29.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:29.697 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:29.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:29.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:29 smithi145 ceph-mon[89359]: pgmap v2602: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 38 MiB/s wr, 714 op/s; 38915/310452 objects misplaced (12.535%); 27 MiB/s, 434 objects/s recovering 2024-09-21T10:48:29.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:29.843 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:29.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:29.870 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:29 smithi119 ceph-mon[101432]: pgmap v2602: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 38 MiB/s wr, 714 op/s; 38915/310452 objects misplaced (12.535%); 27 MiB/s, 434 objects/s recovering 2024-09-21T10:48:29.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:29 smithi119 ceph-mon[105288]: pgmap v2602: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 38 MiB/s wr, 714 op/s; 38915/310452 objects misplaced (12.535%); 27 MiB/s, 434 objects/s recovering 2024-09-21T10:48:29.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:29.969 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:29.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:30.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:30.095 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:30.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:30.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:30.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:30.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:30.448 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:30.471 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:48:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:48:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:48:30.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:30.597 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:30.698 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:30.720 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:30.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:30.856 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:30.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:30.982 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:30.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:31.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:31.105 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:31.207 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:31.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.250 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:31.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:31.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:31.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:31.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:31.497 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:31.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:31.599 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:31.622 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:31.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:31.751 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:31.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:31 smithi145 ceph-mon[89359]: pgmap v2603: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 87 B/s rd, 24 MiB/s wr, 603 op/s; 37194/306090 objects misplaced (12.151%); 28 MiB/s, 444 objects/s recovering 2024-09-21T10:48:31.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:31.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:31 smithi119 ceph-mon[101432]: pgmap v2603: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 87 B/s rd, 24 MiB/s wr, 603 op/s; 37194/306090 objects misplaced (12.151%); 28 MiB/s, 444 objects/s recovering 2024-09-21T10:48:31.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:31 smithi119 ceph-mon[105288]: pgmap v2603: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.2 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 87 B/s rd, 24 MiB/s wr, 603 op/s; 37194/306090 objects misplaced (12.151%); 28 MiB/s, 444 objects/s recovering 2024-09-21T10:48:31.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:31.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:31.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:32.016 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:32.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.142 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:32.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:32.243 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:32.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:32.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:32.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:32.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:32.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:32.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:32.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:32.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:32.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:32.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:32.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:32.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:32.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:33.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:33.085 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:33.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:33.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:33.209 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:33.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:33.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:33.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:33.435 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:33.457 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:33.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:33.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:33.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.601 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:33.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:33.702 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:33.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:33.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:33 smithi145 ceph-mon[89359]: pgmap v2604: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 85 B/s rd, 22 MiB/s wr, 740 op/s; 36002/299928 objects misplaced (12.004%); 25 MiB/s, 400 objects/s recovering 2024-09-21T10:48:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:33.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:33.802 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:33 smithi119 ceph-mon[101432]: pgmap v2604: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 85 B/s rd, 22 MiB/s wr, 740 op/s; 36002/299928 objects misplaced (12.004%); 25 MiB/s, 400 objects/s recovering 2024-09-21T10:48:33.802 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:33.802 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:33.802 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:33 smithi119 ceph-mon[105288]: pgmap v2604: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.1 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 85 B/s rd, 22 MiB/s wr, 740 op/s; 36002/299928 objects misplaced (12.004%); 25 MiB/s, 400 objects/s recovering 2024-09-21T10:48:33.802 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:33.802 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:33.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:33.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T10:48:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:33.879 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:33.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:33.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:34.023 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:34.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:34.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.221 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:34.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:34.356 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:34.458 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:34.500 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:34.602 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:34.636 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:34.738 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:34.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:48:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:48:34] "GET /metrics HTTP/1.1" 200 36857 "" "Prometheus/2.51.0" 2024-09-21T10:48:34.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:34.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:34.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:35.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:35.042 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:35.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:35.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:35.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:35.291 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:35.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:35.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:35.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:35.561 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:35 smithi119 ceph-mon[101432]: pgmap v2605: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 85 B/s rd, 11 MiB/s wr, 682 op/s; 35218/295290 objects misplaced (11.927%); 24 MiB/s, 391 objects/s recovering 2024-09-21T10:48:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:35 smithi119 ceph-mon[105288]: pgmap v2605: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 85 B/s rd, 11 MiB/s wr, 682 op/s; 35218/295290 objects misplaced (11.927%); 24 MiB/s, 391 objects/s recovering 2024-09-21T10:48:35.663 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:35.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:35.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:35.809 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:35.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:35.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:35 smithi145 ceph-mon[89359]: pgmap v2605: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 6.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 85 B/s rd, 11 MiB/s wr, 682 op/s; 35218/295290 objects misplaced (11.927%); 24 MiB/s, 391 objects/s recovering 2024-09-21T10:48:36.049 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:36.070 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:36.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:36.213 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:36.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:36.315 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:36.337 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:36.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:36.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:36.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:36.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:36.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:36.688 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:36.710 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:36.811 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:36.833 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:36.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:36.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:36.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:37.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:37.121 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:37.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:37.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:37.265 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:37.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:37.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:37.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:37.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:37.634 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:37.657 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:37.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:37.781 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:37.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:37 smithi119 ceph-mon[101432]: pgmap v2606: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 4.2 MiB/s wr, 702 op/s; 34183/290919 objects misplaced (11.750%); 25 MiB/s, 394 objects/s recovering 2024-09-21T10:48:37.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:37 smithi119 ceph-mon[105288]: pgmap v2606: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 4.2 MiB/s wr, 702 op/s; 34183/290919 objects misplaced (11.750%); 25 MiB/s, 394 objects/s recovering 2024-09-21T10:48:37.883 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:37.904 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:37.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:38.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:38.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:37 smithi145 ceph-mon[89359]: pgmap v2606: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 4.2 MiB/s wr, 702 op/s; 34183/290919 objects misplaced (11.750%); 25 MiB/s, 394 objects/s recovering 2024-09-21T10:48:38.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:38.167 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:38.269 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:38.292 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:38.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:38.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:38.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:38.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:38.561 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:38.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:38.663 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:38.686 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:38.787 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:38.811 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:38.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:38 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1788044071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:38.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:38 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1788044071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:38.913 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:38.937 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:38.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:38.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:39.040 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:39.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:38 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1788044071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:48:39.062 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:39.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:39.206 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:39.295 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:48:39.295 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:48:39.295 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:48:39.295 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:48:39.295 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:48:39.295 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:48:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:48:39.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:39.347 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:39.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:39.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:39.472 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:39.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:39.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:39.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:39.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.744 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:39.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:39.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:39 smithi119 ceph-mon[105288]: pgmap v2607: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 762 KiB/s wr, 694 op/s; 33132/287823 objects misplaced (11.511%); 21 MiB/s, 328 objects/s recovering 2024-09-21T10:48:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:39 smithi119 ceph-mon[101432]: pgmap v2607: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 762 KiB/s wr, 694 op/s; 33132/287823 objects misplaced (11.511%); 21 MiB/s, 328 objects/s recovering 2024-09-21T10:48:39.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:39.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:39.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:40.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:39 smithi145 ceph-mon[89359]: pgmap v2607: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 762 KiB/s wr, 694 op/s; 33132/287823 objects misplaced (11.511%); 21 MiB/s, 328 objects/s recovering 2024-09-21T10:48:40.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:40.136 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:40.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:40.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:40.362 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:40.385 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:40.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:40.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:48:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:48:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:48:40.629 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:40.652 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:40.754 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:40.775 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:40.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:40.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:40.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:40.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:40.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:41.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:41.067 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:41.168 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:41.209 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:41.311 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:41.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:41.457 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:41.648 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:41.750 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:41.772 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:41.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:41 smithi119 ceph-mon[101432]: pgmap v2608: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 85 B/s wr, 768 op/s; 31602/282783 objects misplaced (11.175%); 21 MiB/s, 336 objects/s recovering 2024-09-21T10:48:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:41 smithi119 ceph-mon[105288]: pgmap v2608: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 85 B/s wr, 768 op/s; 31602/282783 objects misplaced (11.175%); 21 MiB/s, 336 objects/s recovering 2024-09-21T10:48:41.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:41.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:42.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:42.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:41 smithi145 ceph-mon[89359]: pgmap v2608: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.8 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 85 B/s rd, 85 B/s wr, 768 op/s; 31602/282783 objects misplaced (11.175%); 21 MiB/s, 336 objects/s recovering 2024-09-21T10:48:42.047 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:42.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:42.283 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:42.306 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:42.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:42.408 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:42.431 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:42.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:42.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:42.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:42.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:42.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:42.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:42.837 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:42.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:42.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:42.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:43.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:43.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:43.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:43.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:43.403 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:43.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:43.551 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:43.653 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:43.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:43.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:43.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.856 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:43.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:43.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:43 smithi119 ceph-mon[101432]: pgmap v2609: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 841 op/s; 30277/275802 objects misplaced (10.978%); 17 MiB/s, 266 objects/s recovering 2024-09-21T10:48:43.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:43 smithi119 ceph-mon[105288]: pgmap v2609: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 841 op/s; 30277/275802 objects misplaced (10.978%); 17 MiB/s, 266 objects/s recovering 2024-09-21T10:48:43.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:43.989 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:43.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:43.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:43 smithi145 ceph-mon[89359]: pgmap v2609: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.6 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 841 op/s; 30277/275802 objects misplaced (10.978%); 17 MiB/s, 266 objects/s recovering 2024-09-21T10:48:44.091 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:44.131 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:44.180 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:44.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:44.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:44.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:44.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:44.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.423 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:44.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:44.524 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:44.562 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:44.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:44.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:48:44.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:48:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:48:44] "GET /metrics HTTP/1.1" 200 36857 "" "Prometheus/2.51.0" 2024-09-21T10:48:44.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:44.687 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:44.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:44.810 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:44.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:44.934 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:44.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:45.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:45.058 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:45.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:45.201 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:45.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:45.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:45.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:45.449 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:45.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:45.577 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:45 smithi119 ceph-mon[101432]: pgmap v2610: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 804 op/s; 29511/270969 objects misplaced (10.891%); 17 MiB/s, 265 objects/s recovering 2024-09-21T10:48:45.597 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:45.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:45.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:45.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:45.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:45.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:45 smithi119 ceph-mon[105288]: pgmap v2610: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 804 op/s; 29511/270969 objects misplaced (10.891%); 17 MiB/s, 265 objects/s recovering 2024-09-21T10:48:45.870 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:45.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:45.995 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:45.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:45.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:45 smithi145 ceph-mon[89359]: pgmap v2610: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 0 B/s wr, 804 op/s; 29511/270969 objects misplaced (10.891%); 17 MiB/s, 265 objects/s recovering 2024-09-21T10:48:46.096 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:46.117 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:46.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:46.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:46.241 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:46.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:46.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:46.385 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:46.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:46.486 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:46.509 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:46.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:46.611 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:46.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:46.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:46.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:46.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:46.927 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:46.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:47.029 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:47.051 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:47.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:47.173 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:47.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:47.302 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:47.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:47.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:47.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:47.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:47.553 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:47.599 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:48:47.655 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:47.677 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:47.778 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:47.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:47 smithi119 ceph-mon[101432]: pgmap v2611: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 812 op/s; 28404/266034 objects misplaced (10.677%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:48:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:47 smithi119 ceph-mon[101432]: osdmap e1446: 8 total, 8 up, 5 in 2024-09-21T10:48:47.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/415974876' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:47.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:47 smithi119 ceph-mon[105288]: pgmap v2611: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 812 op/s; 28404/266034 objects misplaced (10.677%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:48:47.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:47 smithi119 ceph-mon[105288]: osdmap e1446: 8 total, 8 up, 5 in 2024-09-21T10:48:47.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/415974876' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:47.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:47.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:47.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:47 smithi145 ceph-mon[89359]: pgmap v2611: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.4 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 812 op/s; 28404/266034 objects misplaced (10.677%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:48:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:47 smithi145 ceph-mon[89359]: osdmap e1446: 8 total, 8 up, 5 in 2024-09-21T10:48:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/415974876' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:48:48.044 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:48.066 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:48.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:48.189 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:48.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:48.314 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:48.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:48.439 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:48.540 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:48.563 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:48:48.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:48.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:48.787 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:48.810 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/415974876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[101432]: osdmap e1447: 8 total, 8 up, 5 in 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1290271046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/415974876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[105288]: osdmap e1447: 8 total, 8 up, 5 in 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1290271046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:48:48.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:48.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:48.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:48.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:48.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:49.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:48 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:48:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/415974876' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:48:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:48 smithi145 ceph-mon[89359]: osdmap e1447: 8 total, 8 up, 5 in 2024-09-21T10:48:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:48 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1290271046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:48:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:48:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:48:49.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:49.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:49.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:49.179 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:49.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:49.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.348 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:49.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:49.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:49.491 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:49.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:49.593 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:49.615 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:49.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:49.740 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:49.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:49.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:49 smithi119 ceph-mon[105288]: pgmap v2614: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 27323/262947 objects misplaced (10.391%); 15 MiB/s, 246 objects/s recovering 2024-09-21T10:48:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1290271046' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:48:49.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:49 smithi119 ceph-mon[105288]: osdmap e1448: 8 total, 8 up, 5 in 2024-09-21T10:48:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:49 smithi119 ceph-mon[101432]: pgmap v2614: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 27323/262947 objects misplaced (10.391%); 15 MiB/s, 246 objects/s recovering 2024-09-21T10:48:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1290271046' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:48:49.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:49 smithi119 ceph-mon[101432]: osdmap e1448: 8 total, 8 up, 5 in 2024-09-21T10:48:49.877 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:49.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:49.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:49.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:50.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:50.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:49 smithi145 ceph-mon[89359]: pgmap v2614: 109 pgs: 4 active+remapped+backfill_wait, 3 active+remapped+backfilling, 102 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 27323/262947 objects misplaced (10.391%); 15 MiB/s, 246 objects/s recovering 2024-09-21T10:48:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1290271046' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:48:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:49 smithi145 ceph-mon[89359]: osdmap e1448: 8 total, 8 up, 5 in 2024-09-21T10:48:50.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:50.149 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:50.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:50.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:50.291 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:50.393 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:50.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:50.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:50.539 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:48:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:48:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:48:50.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:50.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:50.805 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:50.906 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:50.929 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:50.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:50.954 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:48:51.030 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:51.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:51.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:51.155 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:51.177 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:51.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:51.278 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:51.302 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:51.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:51.443 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:51.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:51.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:51.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:51.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:51.718 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:51.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:51.866 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:51.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:51.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:52.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:52.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:52.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:51 smithi145 ceph-mon[89359]: pgmap v2616: 109 pgs: 4 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 706 op/s; 26066/258249 objects misplaced (10.093%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:48:52.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:52.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:51 smithi119 ceph-mon[101432]: pgmap v2616: 109 pgs: 4 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 706 op/s; 26066/258249 objects misplaced (10.093%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:48:52.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:51 smithi119 ceph-mon[105288]: pgmap v2616: 109 pgs: 4 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 5.3 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 706 op/s; 26066/258249 objects misplaced (10.093%); 22 MiB/s, 354 objects/s recovering 2024-09-21T10:48:52.141 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:52.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:52.243 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:52.267 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:52.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.395 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:52.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:52.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:52.538 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:52.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:52.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:52.681 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:48:52.681 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:48:52.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:52.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:52.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:52.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:52.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:53.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:53.060 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:53.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:53.199 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:53.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:53.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:53.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:53.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:53.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:53.460 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:53.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:53.562 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:53.586 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:53.688 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:53.708 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:53.810 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:53.845 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:53.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:53.971 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:53.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:53 smithi145 ceph-mon[89359]: pgmap v2617: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 771 op/s; 23686/252144 objects misplaced (9.394%); 31 MiB/s, 496 objects/s recovering 2024-09-21T10:48:54.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:54.093 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:54.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:53 smithi119 ceph-mon[101432]: pgmap v2617: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 771 op/s; 23686/252144 objects misplaced (9.394%); 31 MiB/s, 496 objects/s recovering 2024-09-21T10:48:54.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:53 smithi119 ceph-mon[105288]: pgmap v2617: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 5.1 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 771 op/s; 23686/252144 objects misplaced (9.394%); 31 MiB/s, 496 objects/s recovering 2024-09-21T10:48:54.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.237 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:54.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:54.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:54.362 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:54.464 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:54.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:54.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:54.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:54.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:54.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.763 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:54.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:54.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:54.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:48:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:48:54] "GET /metrics HTTP/1.1" 200 36855 "" "Prometheus/2.51.0" 2024-09-21T10:48:54.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:54.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:55.004 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:55.046 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:55.148 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:55.171 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:55.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:55.295 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:55.397 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:55.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:55.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:55.545 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:55.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:55.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:55.668 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:55.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:55.817 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:55.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:55.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:55.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:55 smithi145 ceph-mon[89359]: pgmap v2618: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 691 op/s; 23011/247503 objects misplaced (9.297%); 24 MiB/s, 379 objects/s recovering 2024-09-21T10:48:56.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:56.086 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:55 smithi119 ceph-mon[105288]: pgmap v2618: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 691 op/s; 23011/247503 objects misplaced (9.297%); 24 MiB/s, 379 objects/s recovering 2024-09-21T10:48:56.086 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:55 smithi119 ceph-mon[101432]: pgmap v2618: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 5.0 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 0 B/s wr, 691 op/s; 23011/247503 objects misplaced (9.297%); 24 MiB/s, 379 objects/s recovering 2024-09-21T10:48:56.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:56.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:48:56.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:56.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:56.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:48:56.344 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:48:56.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:56.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:56.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:56.609 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:56.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.746 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:56.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:56.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:56.884 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:56.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:56.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:56.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:57.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.032 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:57.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:57.133 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:57.155 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:57.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:57.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:57.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T10:48:57.289 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:48:57.305 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:57.407 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:57.429 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:57.531 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.596 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:57.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:57.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:57.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:57.844 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:57 smithi119 ceph-mon[101432]: pgmap v2619: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 787 op/s; 22254/242994 objects misplaced (9.158%); 25 MiB/s, 401 objects/s recovering 2024-09-21T10:48:57.848 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:57 smithi119 ceph-mon[105288]: pgmap v2619: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 787 op/s; 22254/242994 objects misplaced (9.158%); 25 MiB/s, 401 objects/s recovering 2024-09-21T10:48:57.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:57.883 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:57.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:57.986 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:58.008 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:57 smithi145 ceph-mon[89359]: pgmap v2619: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 787 op/s; 22254/242994 objects misplaced (9.158%); 25 MiB/s, 401 objects/s recovering 2024-09-21T10:48:58.110 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:58.134 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:58.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.274 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:58.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:58.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:58.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:58.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:58.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:58.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:58.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:58.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:58.774 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:58.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:58.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:48:58.934 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:58.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:48:59.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:48:59.075 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:48:59.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:48:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:59.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:48:59.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:59.347 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:59.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:48:59.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:48:59.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:48:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.618 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:59.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:48:59.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:48:59.752 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:48:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:48:59.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:48:59.876 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:48:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:48:59.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:48:59.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:00.014 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:48:59 smithi145 ceph-mon[89359]: pgmap v2620: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 766 op/s; 21284/239955 objects misplaced (8.870%); 24 MiB/s, 380 objects/s recovering 2024-09-21T10:49:00.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:00.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:48:59 smithi119 ceph-mon[101432]: pgmap v2620: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 766 op/s; 21284/239955 objects misplaced (8.870%); 24 MiB/s, 380 objects/s recovering 2024-09-21T10:49:00.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:48:59 smithi119 ceph-mon[105288]: pgmap v2620: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 766 op/s; 21284/239955 objects misplaced (8.870%); 24 MiB/s, 380 objects/s recovering 2024-09-21T10:49:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:00.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:00.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:00.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:00.426 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:00.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:49:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:49:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:49:00.551 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:00.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:00.691 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:00.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:00.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:00.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:00.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:00.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:01.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:01.122 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:01.224 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:01.248 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:01.350 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:01.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:01.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:01.495 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:01.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:01.620 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:01.722 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:01.759 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:01.860 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:01.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:01 smithi119 ceph-mon[105288]: pgmap v2621: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 20657/234771 objects misplaced (8.799%); 19 MiB/s, 298 objects/s recovering 2024-09-21T10:49:01.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:01 smithi119 ceph-mon[101432]: pgmap v2621: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 20657/234771 objects misplaced (8.799%); 19 MiB/s, 298 objects/s recovering 2024-09-21T10:49:01.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:01.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:01.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:02.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:02.032 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:01 smithi145 ceph-mon[89359]: pgmap v2621: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.8 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 705 op/s; 20657/234771 objects misplaced (8.799%); 19 MiB/s, 298 objects/s recovering 2024-09-21T10:49:02.134 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:02.157 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:02.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:02.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:02.280 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:02.382 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:02.406 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:02.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:02.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:02.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:02.819 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:02.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:02.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:02.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:02.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:03.021 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:03.077 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:03.099 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:03.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:03.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:03.253 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:03.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:03.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:03.391 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:03.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.542 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:03.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:03.643 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.689 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:03.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:03.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:03.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:03.841 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:03 smithi119 ceph-mon[101432]: pgmap v2622: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 18712/228537 objects misplaced (8.188%); 23 MiB/s, 371 objects/s recovering 2024-09-21T10:49:03.841 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:03.842 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:03 smithi119 ceph-mon[105288]: pgmap v2622: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 18712/228537 objects misplaced (8.188%); 23 MiB/s, 371 objects/s recovering 2024-09-21T10:49:03.842 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:03.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:03.955 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:03.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:03 smithi145 ceph-mon[89359]: pgmap v2622: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 825 op/s; 18712/228537 objects misplaced (8.188%); 23 MiB/s, 371 objects/s recovering 2024-09-21T10:49:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:04.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:04.077 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:49:04.077 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:49:04.077 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:49:04.077 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:49:04.077 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:04.077 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:04.078 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:49:04.078 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:49:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:04.078 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:04.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:04.081 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:49:04.081 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:04.112 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:49:04.112 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:04.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:04.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:04.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:04.349 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:04.451 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:04.473 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:04.574 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:04.598 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:04.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:04.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:04.837 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:04.860 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:04.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:04.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2072694183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:49:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:49:04] "GET /metrics HTTP/1.1" 200 36860 "" "Prometheus/2.51.0" 2024-09-21T10:49:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2072694183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:04.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2072694183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:05.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:05.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.148 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:05.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:05.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:05.284 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:05.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:05.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:05.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:05.549 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:05.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:05.675 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:05.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:05.800 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:05 smithi119 ceph-mon[101432]: pgmap v2623: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 779 op/s; 17972/224076 objects misplaced (8.020%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:49:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:05 smithi119 ceph-mon[105288]: pgmap v2623: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 779 op/s; 17972/224076 objects misplaced (8.020%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:49:05.901 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:05.924 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:05.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:06.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:05 smithi145 ceph-mon[89359]: pgmap v2623: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 779 op/s; 17972/224076 objects misplaced (8.020%); 17 MiB/s, 271 objects/s recovering 2024-09-21T10:49:06.052 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:06.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:06.153 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:06.174 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:06.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.300 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:06.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:06.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:06.425 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:06.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:06.554 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:06.656 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:06.678 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:06.779 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:06.801 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:06.903 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:06.927 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:06.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:07.028 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:06 smithi145 ceph-mon[89359]: osdmap e1449: 8 total, 8 up, 5 in 2024-09-21T10:49:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:06 smithi145 ceph-mon[89359]: pgmap v2625: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 786 op/s; 17214/219402 objects misplaced (7.846%); 19 MiB/s, 301 objects/s recovering 2024-09-21T10:49:07.051 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:06 smithi119 ceph-mon[101432]: osdmap e1449: 8 total, 8 up, 5 in 2024-09-21T10:49:07.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:06 smithi119 ceph-mon[101432]: pgmap v2625: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 786 op/s; 17214/219402 objects misplaced (7.846%); 19 MiB/s, 301 objects/s recovering 2024-09-21T10:49:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:06 smithi119 ceph-mon[105288]: osdmap e1449: 8 total, 8 up, 5 in 2024-09-21T10:49:07.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:06 smithi119 ceph-mon[105288]: pgmap v2625: 109 pgs: 3 active+remapped+backfill_wait, 3 active+remapped+backfilling, 103 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 786 op/s; 17214/219402 objects misplaced (7.846%); 19 MiB/s, 301 objects/s recovering 2024-09-21T10:49:07.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:07.292 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:07.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:07.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:07.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:07.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:07.589 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:07.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:07.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:07.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:07.815 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:07.838 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:07.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:07.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:07.962 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:07.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:07 smithi145 ceph-mon[89359]: osdmap e1450: 8 total, 8 up, 5 in 2024-09-21T10:49:08.064 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:08.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:08.116 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:07 smithi119 ceph-mon[101432]: osdmap e1450: 8 total, 8 up, 5 in 2024-09-21T10:49:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:07 smithi119 ceph-mon[105288]: osdmap e1450: 8 total, 8 up, 5 in 2024-09-21T10:49:08.190 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:08.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:08.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:08.355 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:08.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:08.494 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:08.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:08.633 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:08.735 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:08.758 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:08.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:08.895 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:08.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:08.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:09.018 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:08 smithi145 ceph-mon[89359]: pgmap v2627: 109 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 756 op/s; 16385/216612 objects misplaced (7.564%); 21 MiB/s, 330 objects/s recovering 2024-09-21T10:49:09.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:08 smithi119 ceph-mon[101432]: pgmap v2627: 109 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 756 op/s; 16385/216612 objects misplaced (7.564%); 21 MiB/s, 330 objects/s recovering 2024-09-21T10:49:09.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:08 smithi119 ceph-mon[105288]: pgmap v2627: 109 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 4.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 756 op/s; 16385/216612 objects misplaced (7.564%); 21 MiB/s, 330 objects/s recovering 2024-09-21T10:49:09.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:09.159 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:09.261 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:09.283 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:09.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.424 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:09.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:09.525 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:09.548 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:09.649 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:09.673 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:09.774 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:09.797 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:09.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:09.938 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:09.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:10.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:10.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:10.219 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:10.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:10.361 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:10.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:10.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:49:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:49:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:49:10.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:10.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:10.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:10.778 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:10.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:10.903 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:10.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:11.004 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:11.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:11.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:11.148 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:11.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:11.250 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:11.274 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:11.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:11.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:11 smithi119 ceph-mon[101432]: pgmap v2628: 109 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 706 op/s; 15391/211569 objects misplaced (7.275%); 16 MiB/s, 263 objects/s recovering 2024-09-21T10:49:11.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:11 smithi119 ceph-mon[105288]: pgmap v2628: 109 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 706 op/s; 15391/211569 objects misplaced (7.275%); 16 MiB/s, 263 objects/s recovering 2024-09-21T10:49:11.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:11.420 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:11.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:11.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:11 smithi145 ceph-mon[89359]: pgmap v2628: 109 pgs: 1 peering, 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 103 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 706 op/s; 15391/211569 objects misplaced (7.275%); 16 MiB/s, 263 objects/s recovering 2024-09-21T10:49:11.544 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:11.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:11.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:11.668 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:11.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:11.810 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:11.912 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:11.935 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:11.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:11.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:12.037 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:12.078 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:12.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:12.201 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:12.303 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:12.325 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:12.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:12.427 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:12.448 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:12.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:12.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.621 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:12.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:12.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:12.757 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:12.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:12.880 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:12.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:12.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:13.003 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:13.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:13.106 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:49:13.126 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:13.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:13.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:13 smithi119 ceph-mon[101432]: pgmap v2629: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.2 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 780 op/s; 13929/205332 objects misplaced (6.784%); 20 MiB/s, 325 objects/s recovering 2024-09-21T10:49:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:13 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3854328616' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:13 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:13 smithi119 ceph-mon[105288]: pgmap v2629: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.2 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 780 op/s; 13929/205332 objects misplaced (6.784%); 20 MiB/s, 325 objects/s recovering 2024-09-21T10:49:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:13 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3854328616' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:13 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:13.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:13.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:13.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:13 smithi145 ceph-mon[89359]: pgmap v2629: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.2 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 780 op/s; 13929/205332 objects misplaced (6.784%); 20 MiB/s, 325 objects/s recovering 2024-09-21T10:49:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:13 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3854328616' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:13 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:49:13.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:13.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:49:13.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:13.674 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:13.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:13.799 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:13.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:13.922 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:13.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:13.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:14.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:14.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:14.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:14.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:14.209 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:14.209 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:14.209 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-mon[101432]: osdmap e1451: 8 total, 8 up, 5 in 2024-09-21T10:49:14.210 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:14.210 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:14.210 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-mon[105288]: osdmap e1451: 8 total, 8 up, 5 in 2024-09-21T10:49:14.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:14.350 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:14.452 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:14.453 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:49:14.453 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [], 2024-09-21T10:49:14.453 INFO:teuthology.orchestra.run.smithi119.stdout: "complaint_time": 30, 2024-09-21T10:49:14.453 INFO:teuthology.orchestra.run.smithi119.stdout: "num_blocked_ops": 0 2024-09-21T10:49:14.453 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:49:14.475 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:14 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:14 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:49:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:14 smithi145 ceph-mon[89359]: osdmap e1451: 8 total, 8 up, 5 in 2024-09-21T10:49:14.577 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:14.626 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:14.728 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:14.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:14.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:49:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:49:14] "GET /metrics HTTP/1.1" 200 36860 "" "Prometheus/2.51.0" 2024-09-21T10:49:14.875 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:14.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:14.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:14.977 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:15.031 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:15.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:15.138 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:49:15.191 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:15.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:15.314 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:15 smithi119 ceph-mon[101432]: pgmap v2631: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 753 op/s; 13463/201303 objects misplaced (6.688%); 17 MiB/s, 279 objects/s recovering 2024-09-21T10:49:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2658428047' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:49:15.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:15 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:49:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:15 smithi119 ceph-mon[105288]: pgmap v2631: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 753 op/s; 13463/201303 objects misplaced (6.688%); 17 MiB/s, 279 objects/s recovering 2024-09-21T10:49:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2658428047' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:49:15.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:15 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:49:15.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:15.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:15 smithi145 ceph-mon[89359]: pgmap v2631: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.1 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 0 B/s wr, 753 op/s; 13463/201303 objects misplaced (6.688%); 17 MiB/s, 279 objects/s recovering 2024-09-21T10:49:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2658428047' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:49:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:15 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:49:15.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:15.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T10:49:15.571 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:15.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:15.699 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:15.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.823 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:15.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:15.924 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:15.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:15.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:16.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:16.079 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:16.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:16.204 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:16.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:16.306 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:16.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.350 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:16.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:16 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:49:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:16 smithi119 ceph-mon[105288]: osdmap e1452: 8 total, 8 up, 5 in 2024-09-21T10:49:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:16 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:49:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:16 smithi119 ceph-mon[101432]: osdmap e1452: 8 total, 8 up, 5 in 2024-09-21T10:49:16.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:16.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:16 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:49:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:16 smithi145 ceph-mon[89359]: osdmap e1452: 8 total, 8 up, 5 in 2024-09-21T10:49:16.588 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:16.616 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:16.718 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:16.739 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:16.817 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:16.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:49:16.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.862 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:16.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:16.963 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:16.985 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:16.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:17.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:17.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:17.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:17.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:17.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:17.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:17 smithi119 ceph-mon[101432]: pgmap v2633: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 800 op/s; 12875/197385 objects misplaced (6.523%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:49:17.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:17 smithi119 ceph-mon[101432]: osdmap e1453: 8 total, 8 up, 5 in 2024-09-21T10:49:17.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:17 smithi119 ceph-mon[105288]: pgmap v2633: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 800 op/s; 12875/197385 objects misplaced (6.523%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:49:17.362 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:17 smithi119 ceph-mon[105288]: osdmap e1453: 8 total, 8 up, 5 in 2024-09-21T10:49:17.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:17.501 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:17.523 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:17.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:17 smithi145 ceph-mon[89359]: pgmap v2633: 109 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfilling, 104 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 800 op/s; 12875/197385 objects misplaced (6.523%); 19 MiB/s, 306 objects/s recovering 2024-09-21T10:49:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:17 smithi145 ceph-mon[89359]: osdmap e1453: 8 total, 8 up, 5 in 2024-09-21T10:49:17.624 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:17.646 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:17.656 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:17.748 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:17.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:17.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:17.911 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:17.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:18.013 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:18.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:49:18.035 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:18.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:18.158 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:18.260 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:18.282 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:18 smithi119 ceph-mon[105288]: osdmap e1454: 8 total, 8 up, 5 in 2024-09-21T10:49:18.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:18 smithi119 ceph-mon[101432]: osdmap e1454: 8 total, 8 up, 5 in 2024-09-21T10:49:18.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:18.383 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:18.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:18.507 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:18.528 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:18 smithi145 ceph-mon[89359]: osdmap e1454: 8 total, 8 up, 5 in 2024-09-21T10:49:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:18.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:18.674 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:18.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:18.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:18.798 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:18.900 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:18.921 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:18.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:18.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:19.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:19.063 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:19.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:19.186 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:19.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:19.327 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:19.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:19.469 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:19.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:19 smithi145 ceph-mon[89359]: pgmap v2636: 109 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 104 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 685 op/s; 11995/195168 objects misplaced (6.146%); 17 MiB/s, 267 objects/s recovering 2024-09-21T10:49:19.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:19.592 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:19.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:19 smithi119 ceph-mon[101432]: pgmap v2636: 109 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 104 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 685 op/s; 11995/195168 objects misplaced (6.146%); 17 MiB/s, 267 objects/s recovering 2024-09-21T10:49:19.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:19 smithi119 ceph-mon[105288]: pgmap v2636: 109 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 104 active+clean; 4.0 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 685 op/s; 11995/195168 objects misplaced (6.146%); 17 MiB/s, 267 objects/s recovering 2024-09-21T10:49:19.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:19.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:19.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:19.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:19.841 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:19.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:19.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:19.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:19.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:20.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:20.107 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:20.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.253 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:20.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:20.354 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:20.375 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:20.476 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:20.499 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:49:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:49:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:49:20.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:20.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:20.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:20.774 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:20.876 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:20.898 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:20.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:20.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:21.000 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:21.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:20 smithi145 ceph-mon[89359]: pgmap v2637: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 3.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 554 op/s; 10829/191313 objects misplaced (5.660%); 21 MiB/s, 343 objects/s recovering 2024-09-21T10:49:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:49:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:20 smithi119 ceph-mon[105288]: pgmap v2637: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 3.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 554 op/s; 10829/191313 objects misplaced (5.660%); 21 MiB/s, 343 objects/s recovering 2024-09-21T10:49:21.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:49:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:20 smithi119 ceph-mon[101432]: pgmap v2637: 109 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 105 active+clean; 3.9 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 554 op/s; 10829/191313 objects misplaced (5.660%); 21 MiB/s, 343 objects/s recovering 2024-09-21T10:49:21.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:49:21.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:21.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:21.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:21.290 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:21.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:21.414 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:21.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:21.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:21.552 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:21.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:21.653 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:21.676 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:21.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:21.821 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:21.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:21.946 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:21.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:22.047 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:22.083 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:22.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:22.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:22.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:22.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:22.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:22.458 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:22.485 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:22.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:22.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:22.610 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:22.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:22.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:22.737 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:22.839 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:22.863 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:22.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:22.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:22.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:23.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:23.138 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:23.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:23.261 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:23.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:23 smithi145 ceph-mon[89359]: pgmap v2638: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 727 op/s; 9358/185607 objects misplaced (5.042%); 28 MiB/s, 450 objects/s recovering 2024-09-21T10:49:23.363 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:23.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:23 smithi119 ceph-mon[101432]: pgmap v2638: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 727 op/s; 9358/185607 objects misplaced (5.042%); 28 MiB/s, 450 objects/s recovering 2024-09-21T10:49:23.369 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:23 smithi119 ceph-mon[105288]: pgmap v2638: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.8 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 727 op/s; 9358/185607 objects misplaced (5.042%); 28 MiB/s, 450 objects/s recovering 2024-09-21T10:49:23.412 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:23.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:23.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:23.553 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:23.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:23.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.705 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:23.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:23.806 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:23.828 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:23.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:23.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:23.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:24.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:24.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:24.172 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:24 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3023680075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:24.173 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:24 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3023680075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:24.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:24.221 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:24.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:24 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3023680075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:24.323 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.346 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:24.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:24.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:24.468 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:24.570 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:24.596 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:24.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.735 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:24.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:24.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:24.857 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:24.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:24.871 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:49:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:49:24] "GET /metrics HTTP/1.1" 200 36858 "" "Prometheus/2.51.0" 2024-09-21T10:49:24.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.010 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:25.051 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:49:25.051 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:49:25.051 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:25.053 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:25.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:25.121 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:25 smithi119 ceph-mon[101432]: pgmap v2639: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 682 op/s; 8808/180996 objects misplaced (4.866%); 23 MiB/s, 364 objects/s recovering 2024-09-21T10:49:25.150 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:25.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:25.292 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:25.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:25 smithi145 ceph-mon[89359]: pgmap v2639: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 682 op/s; 8808/180996 objects misplaced (4.866%); 23 MiB/s, 364 objects/s recovering 2024-09-21T10:49:25.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:25.436 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:25.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:25.580 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:25.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:25 smithi119 ceph-mon[105288]: pgmap v2639: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.7 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 0 B/s wr, 682 op/s; 8808/180996 objects misplaced (4.866%); 23 MiB/s, 364 objects/s recovering 2024-09-21T10:49:25.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:25.737 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:25.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:25.885 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:25.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:25.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:26.024 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:26.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:26.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:26.147 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:26.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:26.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:26.271 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:26.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:26.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.413 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:26.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:26.514 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:26.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:26.565 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:49:26.565 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:26 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:49:26.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:49:26.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:26 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:49:26.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:26.694 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:49:26.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:26 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:49:26.796 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:26.841 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:26.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:26.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.007 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:27.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:27.108 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:27.139 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:27.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:27.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:27.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:27.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:27.386 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:27.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:27.487 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:27.508 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:27.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:27.610 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.647 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:27.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:27.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:49:27.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:27.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:27 smithi119 ceph-mon[105288]: pgmap v2640: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 714 op/s; 8808/176166 objects misplaced (5.000%); 18 MiB/s, 294 objects/s recovering 2024-09-21T10:49:27.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:27 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2088287177' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:49:27.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:27 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:49:27.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:27 smithi119 ceph-mon[101432]: pgmap v2640: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 714 op/s; 8808/176166 objects misplaced (5.000%); 18 MiB/s, 294 objects/s recovering 2024-09-21T10:49:27.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:27 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2088287177' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:49:27.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:27 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:49:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:27 smithi145 ceph-mon[89359]: pgmap v2640: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 714 op/s; 8808/176166 objects misplaced (5.000%); 18 MiB/s, 294 objects/s recovering 2024-09-21T10:49:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:27 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2088287177' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:49:27.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:27 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:49:27.804 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:27.906 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:27.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:27.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:28.053 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.103 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:28.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:28.180 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:49:28.204 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:28.225 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:28.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:28.327 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:28.348 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:28.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:28.493 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:28.595 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:28.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:28.720 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:28.742 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:28.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:28 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:49:28.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:28 smithi145 ceph-mon[89359]: osdmap e1455: 8 total, 8 up, 5 in 2024-09-21T10:49:28.844 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:28.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:28 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:49:28.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:28 smithi119 ceph-mon[101432]: osdmap e1455: 8 total, 8 up, 5 in 2024-09-21T10:49:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:28 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:49:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:28 smithi119 ceph-mon[105288]: osdmap e1455: 8 total, 8 up, 5 in 2024-09-21T10:49:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:28.868 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:28.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:28.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:29.003 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:29.104 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.157 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:29.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:29.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:29.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:29.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:29.397 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:29.418 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:29.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:29.562 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:29.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:29.664 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:29.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:29.787 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:29.794 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:29 smithi145 ceph-mon[89359]: pgmap v2642: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 742 op/s; 7956/172881 objects misplaced (4.602%); 17 MiB/s, 278 objects/s recovering 2024-09-21T10:49:29.810 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:29.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:29 smithi119 ceph-mon[101432]: pgmap v2642: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 742 op/s; 7956/172881 objects misplaced (4.602%); 17 MiB/s, 278 objects/s recovering 2024-09-21T10:49:29.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:29 smithi119 ceph-mon[105288]: pgmap v2642: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 742 op/s; 7956/172881 objects misplaced (4.602%); 17 MiB/s, 278 objects/s recovering 2024-09-21T10:49:29.911 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:29.933 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:29.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:30.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:30.057 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:30.158 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:30.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:30.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:30.322 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:30.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:30.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:30.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:49:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:49:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:49:30.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:30.570 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:30.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:30.693 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:30.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:30.816 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:30.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:30.940 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:30.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:31.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:31.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:30 smithi145 ceph-mon[89359]: pgmap v2643: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 794 op/s; 7420/167478 objects misplaced (4.430%); 14 MiB/s, 217 objects/s recovering 2024-09-21T10:49:31.082 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:31.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:30 smithi119 ceph-mon[105288]: pgmap v2643: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 794 op/s; 7420/167478 objects misplaced (4.430%); 14 MiB/s, 217 objects/s recovering 2024-09-21T10:49:31.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:30 smithi119 ceph-mon[101432]: pgmap v2643: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.4 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 794 op/s; 7420/167478 objects misplaced (4.430%); 14 MiB/s, 217 objects/s recovering 2024-09-21T10:49:31.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:31.229 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:31.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:31.353 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:31.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:31.492 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:31.593 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:31.626 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:31.728 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:31.751 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:31.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:31.873 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:31.975 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:31.997 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:31.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:32.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:32.137 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:32.150 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:49:32.151 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:49:32.239 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:32.262 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:32.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:32.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:32.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:32.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:32.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:32.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:32.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:32.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:32.813 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:32.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:32.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:32.953 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:32.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:33.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:33.087 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:33.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:33.188 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:33.218 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:33.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:33.320 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:33 smithi119 ceph-mon[101432]: pgmap v2644: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 826 op/s; 6702/160803 objects misplaced (4.168%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:49:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:33 smithi119 ceph-mon[105288]: pgmap v2644: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 826 op/s; 6702/160803 objects misplaced (4.168%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:49:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:33.369 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:33.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:33 smithi145 ceph-mon[89359]: pgmap v2644: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.3 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 0 B/s wr, 826 op/s; 6702/160803 objects misplaced (4.168%); 9.2 MiB/s, 147 objects/s recovering 2024-09-21T10:49:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:33.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:33.652 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:33.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:33.791 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:33.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:33.893 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:33.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:33.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:34.017 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:34.039 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:34.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:34.171 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:34.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:34.302 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:34.403 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:34.446 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:34.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:34.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:34.584 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:34.616 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:49:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:49:34] "GET /metrics HTTP/1.1" 200 36862 "" "Prometheus/2.51.0" 2024-09-21T10:49:34.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:34.712 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:34.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:34.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:49:34.836 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:34.938 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:34.960 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:34.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:35.062 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:35.083 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:35.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:35.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:35.207 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:35.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:35.353 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:35 smithi119 ceph-mon[105288]: pgmap v2645: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 817 op/s; 6257/156471 objects misplaced (3.999%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:35.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:35 smithi119 ceph-mon[101432]: pgmap v2645: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 817 op/s; 6257/156471 objects misplaced (3.999%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:35.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:35.507 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:35 smithi145 ceph-mon[89359]: pgmap v2645: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.2 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 817 op/s; 6257/156471 objects misplaced (3.999%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:35.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T10:49:35.597 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:35.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:35.687 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:35.716 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:35.821 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:35.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:35.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:35.944 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:35.967 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:35.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:36.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.138 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:36.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:36.238 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:36.260 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:36.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:36.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:36.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:36.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.542 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:36.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:36.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:36.666 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:36.768 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:36.790 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:36.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:36.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:36.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:37.035 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:37.057 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:37.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:37.159 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:37.181 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:37.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:37.330 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:37 smithi119 ceph-mon[101432]: pgmap v2646: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.1 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 805 op/s; 6257/151980 objects misplaced (4.117%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:37 smithi119 ceph-mon[105288]: pgmap v2646: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.1 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 805 op/s; 6257/151980 objects misplaced (4.117%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:37.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:37.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:37 smithi145 ceph-mon[89359]: pgmap v2646: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.1 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 805 op/s; 6257/151980 objects misplaced (4.117%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:37.568 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:37.591 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:37.692 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:37.730 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:37.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:37.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:37.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:37.976 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:37.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:38.078 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:38.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:38.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:38.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:38.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.241 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:38.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:38.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:38.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:38.498 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:38.543 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:38.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:38.685 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:38.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:38.786 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.822 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:38.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:38.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:38.963 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:38.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:39.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:39.109 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:39.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:39.233 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:39.334 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:39.358 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:39.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:39.503 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:39 smithi145 ceph-mon[89359]: pgmap v2647: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.0 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 749 op/s; 5436/149052 objects misplaced (3.647%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:49:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:39 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 2]} : dispatch 2024-09-21T10:49:39.605 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:39 smithi119 ceph-mon[101432]: pgmap v2647: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.0 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 749 op/s; 5436/149052 objects misplaced (3.647%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:49:39.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:39 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 2]} : dispatch 2024-09-21T10:49:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:39 smithi119 ceph-mon[105288]: pgmap v2647: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 3.0 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 749 op/s; 5436/149052 objects misplaced (3.647%); 9.1 MiB/s, 145 objects/s recovering 2024-09-21T10:49:39.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:39 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 2]} : dispatch 2024-09-21T10:49:39.627 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:39.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:39.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:39.853 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:39.874 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:39.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:39.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.026 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:40.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:40.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:40.167 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:40.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.290 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:40.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:40 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 2]}]': finished 2024-09-21T10:49:40.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:40 smithi119 ceph-mon[101432]: osdmap e1456: 8 total, 8 up, 5 in 2024-09-21T10:49:40.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:40 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 2]}]': finished 2024-09-21T10:49:40.360 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:40 smithi119 ceph-mon[105288]: osdmap e1456: 8 total, 8 up, 5 in 2024-09-21T10:49:40.392 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.414 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:40.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:40 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [3, 2]}]': finished 2024-09-21T10:49:40.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:40 smithi145 ceph-mon[89359]: osdmap e1456: 8 total, 8 up, 5 in 2024-09-21T10:49:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:49:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:49:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:49:40.555 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:40.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:40.682 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:40.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:40.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:40.827 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:40.929 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:40.950 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:40.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:41.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:41.074 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:41.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:41.198 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:41.300 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:41.322 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:41 smithi119 ceph-mon[105288]: pgmap v2649: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 793 op/s; 4905/143652 objects misplaced (3.415%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T10:49:41.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:41 smithi119 ceph-mon[105288]: osdmap e1457: 8 total, 8 up, 5 in 2024-09-21T10:49:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:41 smithi119 ceph-mon[101432]: pgmap v2649: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 793 op/s; 4905/143652 objects misplaced (3.415%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T10:49:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:41 smithi119 ceph-mon[101432]: osdmap e1457: 8 total, 8 up, 5 in 2024-09-21T10:49:41.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:41 smithi145 ceph-mon[89359]: pgmap v2649: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 0 B/s wr, 793 op/s; 4905/143652 objects misplaced (3.415%); 9.9 MiB/s, 159 objects/s recovering 2024-09-21T10:49:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:41 smithi145 ceph-mon[89359]: osdmap e1457: 8 total, 8 up, 5 in 2024-09-21T10:49:41.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:41.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:41.709 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:41.730 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:41.832 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:41.853 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:41.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:41.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:41.977 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:41.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:42.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:42.101 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:42.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:42.238 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:42.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:42.372 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:42.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:42.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:42.495 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:42.597 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:42.618 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:42.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:42.760 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:42.862 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:42.884 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:42.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:42.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:43.024 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:43.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:43.147 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:43.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:43.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:43.270 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:43 smithi119 ceph-mon[105288]: pgmap v2651: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 849 op/s; 4137/136065 objects misplaced (3.040%); 11 MiB/s, 168 objects/s recovering 2024-09-21T10:49:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1070385339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:49:43.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:43 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:49:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:43 smithi119 ceph-mon[101432]: pgmap v2651: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 849 op/s; 4137/136065 objects misplaced (3.040%); 11 MiB/s, 168 objects/s recovering 2024-09-21T10:49:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1070385339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:49:43.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:43 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:49:43.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:43.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:43.571 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:43 smithi145 ceph-mon[89359]: pgmap v2651: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 849 op/s; 4137/136065 objects misplaced (3.040%); 11 MiB/s, 168 objects/s recovering 2024-09-21T10:49:43.571 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1070385339' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:49:43.571 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:43 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:49:43.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:43.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:43.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:43.734 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:43.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:43.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.861 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:43.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:43.962 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:43.986 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:43.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:43.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:44.088 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:44.111 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:44.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:44.235 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:44.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:44.337 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:44.359 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:44.380 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:49:44.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:49:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-mon[105288]: osdmap e1458: 8 total, 8 up, 5 in 2024-09-21T10:49:44.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/23842172' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:44.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:49:44.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-mon[101432]: osdmap e1458: 8 total, 8 up, 5 in 2024-09-21T10:49:44.543 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/23842172' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:44 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:49:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:44 smithi145 ceph-mon[89359]: osdmap e1458: 8 total, 8 up, 5 in 2024-09-21T10:49:44.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/23842172' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:49:44.601 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:44.634 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:44.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:44.758 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:44.860 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:44.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:49:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:49:44] "GET /metrics HTTP/1.1" 200 36862 "" "Prometheus/2.51.0" 2024-09-21T10:49:44.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:44.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:44.984 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:45.021 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:45.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:45.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:45.264 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.287 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:45.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:45.388 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:45.410 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:45.512 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:45 smithi145 ceph-mon[89359]: pgmap v2653: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 977 op/s; 3654/131457 objects misplaced (2.780%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:49:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:45 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:45 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/23842172' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:49:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:45 smithi145 ceph-mon[89359]: osdmap e1459: 8 total, 8 up, 5 in 2024-09-21T10:49:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:45.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[105288]: pgmap v2653: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 977 op/s; 3654/131457 objects misplaced (2.780%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:49:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/23842172' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:49:45.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[105288]: osdmap e1459: 8 total, 8 up, 5 in 2024-09-21T10:49:45.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[101432]: pgmap v2653: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.7 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 977 op/s; 3654/131457 objects misplaced (2.780%); 13 MiB/s, 213 objects/s recovering 2024-09-21T10:49:45.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:49:45.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/23842172' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:49:45.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:45 smithi119 ceph-mon[101432]: osdmap e1459: 8 total, 8 up, 5 in 2024-09-21T10:49:45.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:45.689 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:45.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:45.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:45.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:45.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:45.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:45.965 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:45.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:46.066 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:46.088 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:46.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:46.212 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:46.314 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:46.335 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:46.365 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:49:46.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:46.459 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:46.561 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:46.583 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:46.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:46.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.730 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:46.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:46.831 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:46.869 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:46.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:46.971 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:46.994 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:46.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:46.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:47.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:47.137 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:47.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:47.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:47.279 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:47 smithi119 ceph-mon[101432]: pgmap v2655: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 919 op/s; 3654/127089 objects misplaced (2.875%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:47 smithi119 ceph-mon[105288]: pgmap v2655: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 919 op/s; 3654/127089 objects misplaced (2.875%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:47.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:47.412 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:47.514 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:47.535 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:47.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:47 smithi145 ceph-mon[89359]: pgmap v2655: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.6 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 919 op/s; 3654/127089 objects misplaced (2.875%); 9.0 MiB/s, 143 objects/s recovering 2024-09-21T10:49:47.637 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:47.659 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:47.760 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:47.782 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:47.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:49:47.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:47.906 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:47.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:48.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:48.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:48.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:48.130 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:48.156 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:48.257 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:48.279 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:48 smithi119 ceph-mon[101432]: osdmap e1460: 8 total, 8 up, 5 in 2024-09-21T10:49:48.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:48 smithi119 ceph-mon[105288]: osdmap e1460: 8 total, 8 up, 5 in 2024-09-21T10:49:48.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:48.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.424 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:48.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:48.447 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:49:48.447 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:49:48.473 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:49:48.473 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:49:48.490 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:49:48.490 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:49:48.506 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:49:48.507 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:49:48.524 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:48.525 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:49:48.525 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:49:48.543 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:49:48.543 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:49:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:48 smithi145 ceph-mon[89359]: osdmap e1460: 8 total, 8 up, 5 in 2024-09-21T10:49:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:49:48.546 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:48.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:48.561 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:49:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:49:48.579 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:49:48.579 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:49:48.648 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:48.669 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:48.770 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:48.792 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:48.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:48.915 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:48.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:49.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:49.042 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:49.144 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:49.165 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:49.267 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:49.289 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:49.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:49.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:49.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:49.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:49 smithi145 ceph-mon[89359]: pgmap v2657: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 658 op/s; 2977/124203 objects misplaced (2.397%); 8.8 MiB/s, 141 objects/s recovering 2024-09-21T10:49:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:49 smithi145 ceph-mon[89359]: osdmap e1461: 8 total, 8 up, 5 in 2024-09-21T10:49:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1521502741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:49.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:49 smithi119 ceph-mon[101432]: pgmap v2657: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 658 op/s; 2977/124203 objects misplaced (2.397%); 8.8 MiB/s, 141 objects/s recovering 2024-09-21T10:49:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:49 smithi119 ceph-mon[101432]: osdmap e1461: 8 total, 8 up, 5 in 2024-09-21T10:49:49.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1521502741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:49 smithi119 ceph-mon[105288]: pgmap v2657: 109 pgs: 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 106 active+clean; 2.5 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 0 B/s wr, 658 op/s; 2977/124203 objects misplaced (2.397%); 8.8 MiB/s, 141 objects/s recovering 2024-09-21T10:49:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:49 smithi119 ceph-mon[105288]: osdmap e1461: 8 total, 8 up, 5 in 2024-09-21T10:49:49.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1521502741' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:49:49.673 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:49.695 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:49.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:49.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:49.842 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:49.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:49.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:50.068 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:50.106 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:50.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:50.208 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:50.229 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:50.331 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:50.375 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:50.452 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:50.476 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:50.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:50 smithi119 ceph-mon[105288]: osdmap e1462: 8 total, 8 up, 5 in 2024-09-21T10:49:50.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:50 smithi119 ceph-mon[101432]: osdmap e1462: 8 total, 8 up, 5 in 2024-09-21T10:49:50.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.540 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:50.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:50.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:50 smithi145 ceph-mon[89359]: osdmap e1462: 8 total, 8 up, 5 in 2024-09-21T10:49:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:49:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:49:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:49:50.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:50.663 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:50.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:50.790 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:50.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:50.929 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:50.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:51.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:51.072 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:51.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:51.212 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:51.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:51.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:51.358 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:51.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:51.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:51.506 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:51.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:51 smithi119 ceph-mon[101432]: pgmap v2660: 109 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 106 active+clean; 2.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 675 op/s; 2443/119538 objects misplaced (2.044%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:49:51.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:51 smithi119 ceph-mon[105288]: pgmap v2660: 109 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 106 active+clean; 2.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 675 op/s; 2443/119538 objects misplaced (2.044%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:49:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:51 smithi145 ceph-mon[89359]: pgmap v2660: 109 pgs: 1 peering, 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 106 active+clean; 2.4 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 675 op/s; 2443/119538 objects misplaced (2.044%); 9.9 MiB/s, 158 objects/s recovering 2024-09-21T10:49:51.569 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:51.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:51.632 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:51.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.781 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:51.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:51.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:51.919 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:51.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:51.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:52.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:52.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:52.157 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:52.180 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:52.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:52.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.323 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:52.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:52.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:52.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:52.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:49:52.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:52.586 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:52.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:52.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:52.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:52.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:52.833 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:52.889 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:52.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:52.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:53.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:53.115 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.163 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:53.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:53.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:53.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:53.412 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:53.434 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:53.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:53 smithi145 ceph-mon[89359]: pgmap v2661: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 695 op/s; 791/114570 objects misplaced (0.690%); 26 MiB/s, 413 objects/s recovering 2024-09-21T10:49:53.557 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:53.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:53.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:49:53.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:53 smithi119 ceph-mon[101432]: pgmap v2661: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 695 op/s; 791/114570 objects misplaced (0.690%); 26 MiB/s, 413 objects/s recovering 2024-09-21T10:49:53.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:53 smithi119 ceph-mon[105288]: pgmap v2661: 109 pgs: 1 peering, 1 active+remapped+backfilling, 107 active+clean; 2.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 695 op/s; 791/114570 objects misplaced (0.690%); 26 MiB/s, 413 objects/s recovering 2024-09-21T10:49:53.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.707 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:53.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:53.808 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:53.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:53.930 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:53.952 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:53.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:53.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:54.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:54.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:54.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:54.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:54.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:54.439 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:54.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.462 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:54.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:54.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:54.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:54.733 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:54.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:54.834 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:54.856 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:54.865 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:49:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:49:54] "GET /metrics HTTP/1.1" 200 36858 "" "Prometheus/2.51.0" 2024-09-21T10:49:54.958 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:54.980 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:54.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:54.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:55.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:55.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:55.227 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.271 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:55.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:55.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:55 smithi119 ceph-mon[101432]: pgmap v2662: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 663 op/s; 791/110508 objects misplaced (0.716%); 19 MiB/s, 299 objects/s recovering 2024-09-21T10:49:55.370 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:55 smithi119 ceph-mon[105288]: pgmap v2662: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 663 op/s; 791/110508 objects misplaced (0.716%); 19 MiB/s, 299 objects/s recovering 2024-09-21T10:49:55.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:55.393 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:55.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:55 smithi145 ceph-mon[89359]: pgmap v2662: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 663 op/s; 791/110508 objects misplaced (0.716%); 19 MiB/s, 299 objects/s recovering 2024-09-21T10:49:55.550 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:55.652 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:55.687 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:55.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:55.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:55.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:55.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:55.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:56.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:56.094 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:56.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:56.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:56.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:56.320 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:56.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:56.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:56.465 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:56.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:56.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:56.717 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:56.718 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:49:56.727 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:56.738 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:49:56.738 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:49:56.738 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.738 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:176476 50.2 e1462/1443 50:4bf44262:::benchmark_data_smithi119_335063_object71899:head v 1462'10920, mlcod=1462'10918)", 2024-09-21T10:49:56.738 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:49:56.724818+0000", 2024-09-21T10:49:56.738 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0094997320000000003, 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095291530000000003, 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724818+0000", 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.739 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724818+0000", 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724818+0000", 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4799999999999998e-07 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724823+0000", 2024-09-21T10:49:56.740 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9169999999999997e-06 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724830+0000", 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3260000000000001e-06 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724834+0000", 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3370000000000001e-06 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.741 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725529+0000", 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00069544699999999997 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.726993+0000", 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0014636849999999999 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.742 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:176477 50.2 e1462/1443 50:43a8adf6:::benchmark_data_smithi119_335063_object71900:head v 1462'10921, mlcod=1462'10918)", 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:49:56.724860+0000", 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0094573829999999998, 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095431130000000006, 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724860+0000", 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.743 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724860+0000", 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724860+0000", 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.7399999999999999e-07 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724865+0000", 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.865e-06 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724871+0000", 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.0170000000000002e-06 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.724876+0000", 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4599999999999996e-06 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733114+0000", 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0082382670000000005 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.745 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733138+0000", 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4136999999999999e-05 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:176472 50.a e1462/1430 50:571809d3:::benchmark_data_smithi119_335063_object71895:head v 1462'10886, mlcod=1462'10884)", 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:49:56.725519+0000", 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0087989499999999998, 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0089350929999999999, 2024-09-21T10:49:56.746 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725519+0000", 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725519+0000", 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.747 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725520+0000", 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0020000000000001e-06 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725527+0000", 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6569999999999997e-06 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725540+0000", 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3148e-05 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.748 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.725548+0000", 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.2490000000000004e-06 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.730496+0000", 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.004947913 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.730515+0000", 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9009e-05 2024-09-21T10:49:56.749 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:176482 50.7 e1462/1461 50:e7efe667:::benchmark_data_smithi119_335063_object71905:head v 1462'11077, mlcod=1462'11075)", 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:49:56.731775+0000", 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.0025427739999999998, 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0027241990000000001, 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.750 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731775+0000", 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731775+0000", 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731776+0000", 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7179999999999999e-06 2024-09-21T10:49:56.751 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731786+0000", 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.5799999999999998e-06 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731807+0000", 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0452e-05 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731817+0000", 2024-09-21T10:49:56.752 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0677e-05 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731911+0000", 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.3449000000000006e-05 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.731932+0000", 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1301000000000002e-05 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.753 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:176478 50.f e1462/1430 50:faf8bcc4:::benchmark_data_smithi119_335063_object71901:head v 1462'10964, mlcod=1462'10962)", 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:49:56.733922+0000", 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 0.00039538300000000002, 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00062233999999999998, 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "started", 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733922+0000", 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733922+0000", 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733923+0000", 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.125e-06 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733930+0000", 2024-09-21T10:49:56.755 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1029999999999996e-06 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733946+0000", 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5634999999999999e-05 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.733954+0000", 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.7840000000000004e-06 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:49:56.756 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.734244+0000", 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029027600000000001 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:49:56.734263+0000", 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9074000000000001e-05 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout: "num_ops": 5 2024-09-21T10:49:56.757 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:49:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.758 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:56.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:56.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:56.882 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:56.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:56.983 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.005 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:57.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:57.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:57.128 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:57.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:57.253 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:57.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:57.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:57.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:57 smithi119 ceph-mon[101432]: pgmap v2663: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 762 op/s; 791/105888 objects misplaced (0.747%); 16 MiB/s, 257 objects/s recovering 2024-09-21T10:49:57.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:57 smithi119 ceph-mon[105288]: pgmap v2663: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 762 op/s; 791/105888 objects misplaced (0.747%); 16 MiB/s, 257 objects/s recovering 2024-09-21T10:49:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.396 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:57.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:57.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:57 smithi145 ceph-mon[89359]: pgmap v2663: 109 pgs: 1 active+remapped+backfilling, 108 active+clean; 2.2 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 762 op/s; 791/105888 objects misplaced (0.747%); 16 MiB/s, 257 objects/s recovering 2024-09-21T10:49:57.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:49:57.644 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:49:57.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:57.767 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:57.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:57.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:57.891 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:57.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:57.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:57.993 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:58.014 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:58.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:58.139 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:58.241 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:58.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:58.368 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:49:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:58.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:49:58.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:58.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:58.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:58.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:58 smithi145 ceph-mon[89359]: osdmap e1463: 8 total, 8 up, 5 in 2024-09-21T10:49:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:58 smithi119 ceph-mon[105288]: osdmap e1463: 8 total, 8 up, 5 in 2024-09-21T10:49:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:58 smithi119 ceph-mon[101432]: osdmap e1463: 8 total, 8 up, 5 in 2024-09-21T10:49:58.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:58.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:58.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:49:58.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:58.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:49:58.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:58.907 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:58.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:58.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:49:59.009 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:59.030 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:59.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:59.153 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:59.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:49:59.276 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:59.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:49:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:59 smithi119 ceph-mon[101432]: pgmap v2665: 109 pgs: 1 active+clean+remapped, 108 active+clean; 2.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 651 op/s; 16 MiB/s, 258 objects/s recovering 2024-09-21T10:49:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:49:59 smithi119 ceph-mon[101432]: osdmap e1464: 8 total, 8 up, 5 in 2024-09-21T10:49:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:59 smithi119 ceph-mon[105288]: pgmap v2665: 109 pgs: 1 active+clean+remapped, 108 active+clean; 2.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 651 op/s; 16 MiB/s, 258 objects/s recovering 2024-09-21T10:49:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:49:59 smithi119 ceph-mon[105288]: osdmap e1464: 8 total, 8 up, 5 in 2024-09-21T10:49:59.378 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:59.400 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:59.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:59.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:49:59.524 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:49:59.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:49:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:59 smithi145 ceph-mon[89359]: pgmap v2665: 109 pgs: 1 active+clean+remapped, 108 active+clean; 2.1 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 0 B/s wr, 651 op/s; 16 MiB/s, 258 objects/s recovering 2024-09-21T10:49:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:49:59 smithi145 ceph-mon[89359]: osdmap e1464: 8 total, 8 up, 5 in 2024-09-21T10:49:59.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:49:59.669 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:49:59.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:49:59.810 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:59.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:49:59.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:49:59.957 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:49:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:49:59.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:00.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:00.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:00.218 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:00.320 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:00.341 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:00.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.482 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:00.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:50:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:50:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:50:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:00 smithi145 ceph-mon[89359]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-21T10:50:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:00 smithi145 ceph-mon[89359]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T10:50:00.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:00 smithi119 ceph-mon[105288]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-21T10:50:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:00 smithi119 ceph-mon[105288]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T10:50:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:00 smithi119 ceph-mon[101432]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-09-21T10:50:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:00 smithi119 ceph-mon[101432]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-21T10:50:00.622 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:00.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:00.758 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:00.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:00.881 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:00.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:00.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.007 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:01.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:01.108 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:01.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:01.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:01.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:01.252 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:01.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:01.400 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:01.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:01.502 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:01.525 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:01.528 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:01 smithi145 ceph-mon[89359]: pgmap v2667: 109 pgs: 1 active+clean+remapped, 108 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 684 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:01.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:01 smithi119 ceph-mon[101432]: pgmap v2667: 109 pgs: 1 active+clean+remapped, 108 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 684 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:01 smithi119 ceph-mon[105288]: pgmap v2667: 109 pgs: 1 active+clean+remapped, 108 active+clean; 2.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 684 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:01.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:01.649 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:01.751 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:01.772 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:01.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:01.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:01.918 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:01.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:02.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.073 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:02.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:02.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:02.196 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:02.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:02.338 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:02.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.497 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:02.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:02.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.662 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:02.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:02.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.818 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:02.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:02.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:02.941 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:02.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:03.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:03.064 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:03.166 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:03.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:03.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.311 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:03.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:03.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:03.435 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:03.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:03 smithi145 ceph-mon[89359]: pgmap v2668: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 758 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:03.571 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[105288]: pgmap v2668: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 758 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[101432]: pgmap v2668: 109 pgs: 109 active+clean; 1.9 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 758 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:03.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:03.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:03.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:03.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.836 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:03.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:03.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:03.978 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:03.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:03.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:04.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:04.122 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:04.224 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:04.261 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:04.362 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:04.398 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:04.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:04.541 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:04.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:04.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:04.766 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:04.788 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:50:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:50:04] "GET /metrics HTTP/1.1" 200 36847 "" "Prometheus/2.51.0" 2024-09-21T10:50:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:04 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:04.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:04 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:04.890 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:04.912 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:04.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:04.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:05.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:04 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:05.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:05.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:05.210 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:05.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:05.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:05.333 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:05.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:05.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T10:50:05.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:05.515 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:05.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:05.616 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:05.638 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:05.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:05.795 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:05.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[105288]: pgmap v2669: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 758 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:50:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:50:05.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[101432]: pgmap v2669: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 758 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:05.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:50:05.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:50:05.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:05.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:05.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:06.041 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:05 smithi145 ceph-mon[89359]: pgmap v2669: 109 pgs: 109 active+clean; 1.8 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 0 B/s wr, 758 op/s; 5.9 MiB/s, 93 objects/s recovering 2024-09-21T10:50:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:50:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:50:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:06.063 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:06.069 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:50:06.070 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:50:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:06.100 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:50:06.100 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:06.152 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:50:06.152 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:06.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:06.190 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:50:06.190 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:06.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:06.216 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:50:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:06.247 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:50:06.247 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:06.274 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:50:06.274 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:06.301 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:06.323 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:06.424 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:06.445 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:06.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:06.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:06.583 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:06.684 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:06.706 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:06.808 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.833 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:06.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:06.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:06 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3017519543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:06.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:06 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3017519543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:06.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:06.974 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:06.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:06 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3017519543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:07.075 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:07.097 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:07.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:07.199 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:07.220 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:07.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:07.322 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:07.344 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:07.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:07.467 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:07.568 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:07.611 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:07.616 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:50:07.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:07.735 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:07.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:07.858 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:07.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:07.866 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:07 smithi119 ceph-mon[101432]: pgmap v2670: 109 pgs: 109 active+clean; 1.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 739 op/s 2024-09-21T10:50:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3017795119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:07.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:07 smithi119 ceph-mon[105288]: pgmap v2670: 109 pgs: 109 active+clean; 1.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 739 op/s 2024-09-21T10:50:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3017795119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:07.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:07.959 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:07.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:07.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:07 smithi145 ceph-mon[89359]: pgmap v2670: 109 pgs: 109 active+clean; 1.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 739 op/s 2024-09-21T10:50:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3017795119' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:08.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:08.120 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:08.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:08.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:08.366 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:08.388 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:08.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:08.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:08.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:08.650 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:08 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:50:08.650 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:08 smithi119 ceph-mon[101432]: osdmap e1465: 8 total, 8 up, 5 in 2024-09-21T10:50:08.654 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:08.756 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.778 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:08.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:08.879 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:08.902 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:08.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:09.003 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:09.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:08 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:50:09.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:08 smithi119 ceph-mon[105288]: osdmap e1465: 8 total, 8 up, 5 in 2024-09-21T10:50:09.025 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:09.043 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:08 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:50:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:08 smithi145 ceph-mon[89359]: osdmap e1465: 8 total, 8 up, 5 in 2024-09-21T10:50:09.126 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:09.166 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:09.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:09.289 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:09.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.434 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:09.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:09.535 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:09.556 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:09.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:09.679 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:09.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:09.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:09.904 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:09.942 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:09.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:09.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:09 smithi119 ceph-mon[101432]: pgmap v2672: 109 pgs: 109 active+clean; 1.6 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 767 op/s 2024-09-21T10:50:09.964 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:09 smithi119 ceph-mon[105288]: pgmap v2672: 109 pgs: 109 active+clean; 1.6 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 767 op/s 2024-09-21T10:50:10.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:10.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:09 smithi145 ceph-mon[89359]: pgmap v2672: 109 pgs: 109 active+clean; 1.6 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 767 op/s 2024-09-21T10:50:10.065 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:10.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.189 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:10.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:10.290 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:10.311 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:10.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:10.449 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:50:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:50:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:50:10.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:10.593 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:10.680 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:50:10.694 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:50:10.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:50:10.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:10.716 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:10.817 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.839 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:10.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:10.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:10.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2077758163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:11.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:10 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:11.085 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:11.106 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2077758163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:11.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:10 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2077758163' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:11.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:10 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:11.207 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:11.228 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:11.330 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:11.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:11.453 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:11.474 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:11.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:11.622 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:11.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:11.724 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:11.745 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:11.847 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:11.869 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:11.970 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:11.992 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:11.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:11 smithi145 ceph-mon[89359]: pgmap v2673: 109 pgs: 109 active+clean; 1.5 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 790 op/s 2024-09-21T10:50:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:11 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:50:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:11 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:50:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:11 smithi145 ceph-mon[89359]: osdmap e1466: 8 total, 8 up, 5 in 2024-09-21T10:50:12.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:12.115 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[101432]: pgmap v2673: 109 pgs: 109 active+clean; 1.5 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 790 op/s 2024-09-21T10:50:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:50:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:50:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:12.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[101432]: osdmap e1466: 8 total, 8 up, 5 in 2024-09-21T10:50:12.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[105288]: pgmap v2673: 109 pgs: 109 active+clean; 1.5 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 0 B/s wr, 790 op/s 2024-09-21T10:50:12.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:50:12.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:50:12.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:12.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:11 smithi119 ceph-mon[105288]: osdmap e1466: 8 total, 8 up, 5 in 2024-09-21T10:50:12.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:50:12.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:12.239 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:12.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:12.363 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:12.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:12.465 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:12.486 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:12.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:12.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:12.641 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:12.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.794 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:12.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:12.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:12.930 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:12.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:13.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:13.054 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:13.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:13.176 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:13.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:13.278 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.315 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:13.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:13.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:13.454 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:13.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:13.595 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:13.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:13.718 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:13.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:13.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:13.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.843 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:13.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:13.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:13.981 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:13.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:13 smithi145 ceph-mon[89359]: pgmap v2675: 109 pgs: 109 active+clean; 1.4 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:50:14.083 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:14.104 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:13 smithi119 ceph-mon[101432]: pgmap v2675: 109 pgs: 109 active+clean; 1.4 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:50:14.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:13 smithi119 ceph-mon[105288]: pgmap v2675: 109 pgs: 109 active+clean; 1.4 GiB data, 9.4 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 871 op/s 2024-09-21T10:50:14.206 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.534 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:14.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:14.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.683 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:14.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:14.769 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:50:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:50:14] "GET /metrics HTTP/1.1" 200 36847 "" "Prometheus/2.51.0" 2024-09-21T10:50:14.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:14.823 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:14.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:14.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:14.968 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:14.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:15.070 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:15.111 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:15.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:15.233 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:15.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:15.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:15.395 INFO:teuthology.orchestra.run.smithi119.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:15.395 INFO:teuthology.orchestra.run.smithi119.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:15.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:15.496 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:15.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:15.594 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:15.597 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:15.599 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:15.665 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:15.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:50:15.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:15.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:15.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:15.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:16.000 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:15 smithi145 ceph-mon[89359]: pgmap v2676: 109 pgs: 109 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 897 op/s 2024-09-21T10:50:16.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:16.116 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:15 smithi119 ceph-mon[105288]: pgmap v2676: 109 pgs: 109 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 897 op/s 2024-09-21T10:50:16.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:15 smithi119 ceph-mon[101432]: pgmap v2676: 109 pgs: 109 active+clean; 1.3 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 897 op/s 2024-09-21T10:50:16.124 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:16.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:16.196 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T10:50:16.226 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:16.294 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:16.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:16.437 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:16.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:16.539 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:16.561 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:16.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:16.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:16.814 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:16.835 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:16.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:16.980 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:16.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:17.082 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:17.127 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:17.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:17.266 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:17.368 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.390 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:17.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:17.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.532 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:17.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:17.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:17.656 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:17.757 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:17.778 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:17.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:17.915 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:17.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:18.016 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:17 smithi145 ceph-mon[89359]: pgmap v2677: 109 pgs: 109 active+clean; 1.1 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 932 op/s 2024-09-21T10:50:18.054 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:18.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:17 smithi119 ceph-mon[105288]: pgmap v2677: 109 pgs: 109 active+clean; 1.1 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 932 op/s 2024-09-21T10:50:18.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:17 smithi119 ceph-mon[101432]: pgmap v2677: 109 pgs: 109 active+clean; 1.1 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 932 op/s 2024-09-21T10:50:18.156 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:18.195 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:18.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:18.297 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:18.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:18.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:18.461 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:18.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:18.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:18.607 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:18.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:18.737 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:18.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:18.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:18.921 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:18.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:18.980 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:18.980 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:18.981 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:18 smithi119 ceph-mon[101432]: pgmap v2678: 109 pgs: 109 active+clean; 1.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:50:18.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:18.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:18.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:18 smithi119 ceph-mon[105288]: pgmap v2678: 109 pgs: 109 active+clean; 1.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:50:19.021 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:19.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:19.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:19.166 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:19.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:19.289 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:18 smithi145 ceph-mon[89359]: pgmap v2678: 109 pgs: 109 active+clean; 1.1 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 0 B/s wr, 899 op/s 2024-09-21T10:50:19.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:19.431 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:19.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:19.574 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:19.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:19.718 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:19.820 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:19.842 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:19.943 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:19.965 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:19.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:20.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:20.096 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:20 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/119619869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:20.097 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:20 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/119619869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:20.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:20.212 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:20.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.234 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:20.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:20.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:20 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/119619869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:50:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:50:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:50:20.336 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:20.357 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:20.459 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:20.481 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:20.582 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:20.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:20.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:20.770 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:20.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:20.894 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:20.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:20.996 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:21.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:21.130 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:21.152 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:21.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:21.293 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:21.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:21 smithi145 ceph-mon[89359]: pgmap v2679: 109 pgs: 109 active+clean; 977 MiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 913 op/s 2024-09-21T10:50:21.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:21 smithi119 ceph-mon[105288]: pgmap v2679: 109 pgs: 109 active+clean; 977 MiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 913 op/s 2024-09-21T10:50:21.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:21 smithi119 ceph-mon[101432]: pgmap v2679: 109 pgs: 109 active+clean; 977 MiB data, 8.0 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 913 op/s 2024-09-21T10:50:21.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:21.415 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:21.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:21.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.539 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:21.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:21.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:21.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:21.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:21.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:21.803 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:21.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:21.945 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:21.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:22.047 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.070 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:22.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:22.171 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:22.193 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:22.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:22.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:22.464 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:22.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:22.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:22.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:22.713 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:22.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:22.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:22.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:23.100 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:23.120 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:23.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:23.259 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:23.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:23.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:23.381 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:23.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:23.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:23.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.506 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:23.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:23.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:23 smithi145 ceph-mon[89359]: pgmap v2680: 109 pgs: 109 active+clean; 821 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:50:23.607 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:23.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:23 smithi119 ceph-mon[101432]: pgmap v2680: 109 pgs: 109 active+clean; 821 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:50:23.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:23 smithi119 ceph-mon[105288]: pgmap v2680: 109 pgs: 109 active+clean; 821 MiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 895 op/s 2024-09-21T10:50:23.629 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:23.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:23.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:23.767 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:23.869 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:23.890 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:23.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:23.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:24.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:24.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:24.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:24.156 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:24.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:24.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:24.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:24.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:24.399 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:24.420 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:24.522 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:24.544 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:24.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:24.646 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:24.692 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:24.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:24.831 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:24.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:24.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:50:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:50:24] "GET /metrics HTTP/1.1" 200 36832 "" "Prometheus/2.51.0" 2024-09-21T10:50:24.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:24.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:24.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:50:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:50:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:50:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:50:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:50:25.057 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:50:25.058 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:50:25.059 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:25.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:25.132 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:25.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:25.255 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:25.357 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:25.378 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:25.390 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:25 smithi119 ceph-mon[105288]: pgmap v2681: 109 pgs: 109 active+clean; 721 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:50:25.390 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:25 smithi119 ceph-mon[101432]: pgmap v2681: 109 pgs: 109 active+clean; 721 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:50:25.480 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:25.522 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:25.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:25 smithi145 ceph-mon[89359]: pgmap v2681: 109 pgs: 109 active+clean; 721 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 890 op/s 2024-09-21T10:50:25.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:25.659 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:25.761 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:25.783 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:25.884 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:25.905 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:25.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:26.007 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:26.028 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:26.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:26.172 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:26.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:26.307 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:26.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:26.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:26.566 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:26.588 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:26.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:26.731 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:26.833 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:26.854 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:26.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:26.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:26.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:27.015 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:50:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:50:27.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:27.164 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:27.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:27.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:27.303 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:27.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:27 smithi119 ceph-mon[105288]: pgmap v2682: 109 pgs: 109 active+clean; 622 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 878 op/s 2024-09-21T10:50:27.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:27 smithi119 ceph-mon[101432]: pgmap v2682: 109 pgs: 109 active+clean; 622 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 878 op/s 2024-09-21T10:50:27.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:27.426 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:27.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:27.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:27 smithi145 ceph-mon[89359]: pgmap v2682: 109 pgs: 109 active+clean; 622 MiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 878 op/s 2024-09-21T10:50:27.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:27.650 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:27.671 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:27.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:27.773 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:27.794 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:27.896 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:27.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:50:27.917 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:27.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:27.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:28.019 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:28.040 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:28.142 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:28.164 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:28.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:28.180 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:50:28.265 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:28.295 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2682,"stamp":"2024-09-21T10:50:26.076424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652526224,"num_objects":9978,"num_object_clones":0,"num_object_copies":29934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9978,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199261,"num_write_kb":6692768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30672,"num_bytes_recovered":2007047412,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78174,"ondisk_log_size":78174,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":7433272,"kb_used_data":2006416,"kb_used_omap":504,"kb_used_meta":5426055,"kb_avail":461292488,"statfs":{"total":479975178240,"available":472363507712,"internally_reserved":0,"allocated":2054569984,"data_stored":2033100441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516241,"internal_metadata":5556281199},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,1,10,12,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-691404800,"num_objects":-10550,"num_object_clones":0,"num_object_copies":-31650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10550,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005823"},"pg_stats":[{"pgid":"50.f","version":"1466'12197","reported_seq":12280,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:21.309603+0000","last_change":"2024-09-21T10:46:40.275318+0000","last_active":"2024-09-21T10:50:21.309603+0000","last_peered":"2024-09-21T10:50:21.309603+0000","last_clean":"2024-09-21T10:50:21.309603+0000","last_became_active":"2024-09-21T10:46:40.275064+0000","last_became_peered":"2024-09-21T10:46:40.275064+0000","last_unstale":"2024-09-21T10:50:21.309603+0000","last_undegraded":"2024-09-21T10:50:21.309603+0000","last_fullsized":"2024-09-21T10:50:21.309603+0000","mapping_epoch":1430,"log_start":"1447'7650","ondisk_log_start":"1447'7650","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:25.995350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46465024,"num_objects":709,"num_object_clones":0,"num_object_copies":2127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12197,"num_write_kb":412992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.e","version":"1466'12513","reported_seq":12597,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:25.459242+0000","last_change":"2024-09-21T10:46:40.276274+0000","last_active":"2024-09-21T10:50:25.459242+0000","last_peered":"2024-09-21T10:50:25.459242+0000","last_clean":"2024-09-21T10:50:25.459242+0000","last_became_active":"2024-09-21T10:46:40.276005+0000","last_became_peered":"2024-09-21T10:46:40.276005+0000","last_unstale":"2024-09-21T10:50:25.459242+0000","last_undegraded":"2024-09-21T10:50:25.459242+0000","last_fullsized":"2024-09-21T10:50:25.459242+0000","mapping_epoch":1430,"log_start":"1445'7507","ondisk_log_start":"1445'7507","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:42.051377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35586048,"num_objects":543,"num_object_clones":0,"num_object_copies":1629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.d","version":"1466'12357","reported_seq":23040,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:23.345061+0000","last_change":"2024-09-21T10:49:16.172243+0000","last_active":"2024-09-21T10:50:23.345061+0000","last_peered":"2024-09-21T10:50:23.345061+0000","last_clean":"2024-09-21T10:50:23.345061+0000","last_became_active":"2024-09-21T10:49:16.154591+0000","last_became_peered":"2024-09-21T10:49:16.154591+0000","last_unstale":"2024-09-21T10:50:23.345061+0000","last_undegraded":"2024-09-21T10:50:23.345061+0000","last_fullsized":"2024-09-21T10:50:23.345061+0000","mapping_epoch":1452,"log_start":"1445'7593","ondisk_log_start":"1445'7593","created":1430,"last_epoch_clean":1453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:01.553893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38010904,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12357,"num_write_kb":413953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347144216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.c","version":"1466'12509","reported_seq":12593,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:23.350359+0000","last_change":"2024-09-21T10:46:40.274270+0000","last_active":"2024-09-21T10:50:23.350359+0000","last_peered":"2024-09-21T10:50:23.350359+0000","last_clean":"2024-09-21T10:50:23.350359+0000","last_became_active":"2024-09-21T10:46:40.273989+0000","last_became_peered":"2024-09-21T10:46:40.273989+0000","last_unstale":"2024-09-21T10:50:23.350359+0000","last_undegraded":"2024-09-21T10:50:23.350359+0000","last_fullsized":"2024-09-21T10:50:23.350359+0000","mapping_epoch":1430,"log_start":"1445'7733","ondisk_log_start":"1445'7733","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.938584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12509,"num_write_kb":421440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.b","version":"1466'12478","reported_seq":23959,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:21.386943+0000","last_change":"2024-09-21T10:49:06.725075+0000","last_active":"2024-09-21T10:50:21.386943+0000","last_peered":"2024-09-21T10:50:21.386943+0000","last_clean":"2024-09-21T10:50:21.386943+0000","last_became_active":"2024-09-21T10:49:06.705032+0000","last_became_peered":"2024-09-21T10:49:06.705032+0000","last_unstale":"2024-09-21T10:50:21.386943+0000","last_undegraded":"2024-09-21T10:50:21.386943+0000","last_fullsized":"2024-09-21T10:50:21.386943+0000","mapping_epoch":1449,"log_start":"1448'7984","ondisk_log_start":"1448'7984","created":1430,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:59.569378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48627712,"num_objects":742,"num_object_clones":0,"num_object_copies":2226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12478,"num_write_kb":423040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5698,"num_bytes_recovered":373424128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.a","version":"1466'12292","reported_seq":12376,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:23.317837+0000","last_change":"2024-09-21T10:46:40.276509+0000","last_active":"2024-09-21T10:50:23.317837+0000","last_peered":"2024-09-21T10:50:23.317837+0000","last_clean":"2024-09-21T10:50:23.317837+0000","last_became_active":"2024-09-21T10:46:40.276211+0000","last_became_peered":"2024-09-21T10:46:40.276211+0000","last_unstale":"2024-09-21T10:50:23.317837+0000","last_undegraded":"2024-09-21T10:50:23.317837+0000","last_fullsized":"2024-09-21T10:50:23.317837+0000","mapping_epoch":1430,"log_start":"1445'7524","ondisk_log_start":"1445'7524","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:56.745364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43253760,"num_objects":660,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12292,"num_write_kb":414464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.9","version":"1466'12616","reported_seq":12699,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:23.353801+0000","last_change":"2024-09-21T10:46:40.276505+0000","last_active":"2024-09-21T10:50:23.353801+0000","last_peered":"2024-09-21T10:50:23.353801+0000","last_clean":"2024-09-21T10:50:23.353801+0000","last_became_active":"2024-09-21T10:46:40.275398+0000","last_became_peered":"2024-09-21T10:46:40.275398+0000","last_unstale":"2024-09-21T10:50:23.353801+0000","last_undegraded":"2024-09-21T10:50:23.353801+0000","last_fullsized":"2024-09-21T10:50:23.353801+0000","mapping_epoch":1430,"log_start":"1447'7876","ondisk_log_start":"1447'7876","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:47.276665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12616,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.8","version":"1466'12519","reported_seq":12603,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:25.522456+0000","last_change":"2024-09-21T10:46:40.274571+0000","last_active":"2024-09-21T10:50:25.522456+0000","last_peered":"2024-09-21T10:50:25.522456+0000","last_clean":"2024-09-21T10:50:25.522456+0000","last_became_active":"2024-09-21T10:46:40.274285+0000","last_became_peered":"2024-09-21T10:46:40.274285+0000","last_unstale":"2024-09-21T10:50:25.522456+0000","last_undegraded":"2024-09-21T10:50:25.522456+0000","last_fullsized":"2024-09-21T10:50:25.522456+0000","mapping_epoch":1430,"log_start":"1445'7503","ondisk_log_start":"1445'7503","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:51:55.260163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":553,"num_object_clones":0,"num_object_copies":1659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12519,"num_write_kb":418304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.7","version":"1466'12489","reported_seq":19130,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:23.320651+0000","last_change":"2024-09-21T10:49:49.217806+0000","last_active":"2024-09-21T10:50:23.320651+0000","last_peered":"2024-09-21T10:50:23.320651+0000","last_clean":"2024-09-21T10:50:23.320651+0000","last_became_active":"2024-09-21T10:49:49.193506+0000","last_became_peered":"2024-09-21T10:49:49.193506+0000","last_unstale":"2024-09-21T10:50:23.320651+0000","last_undegraded":"2024-09-21T10:50:23.320651+0000","last_fullsized":"2024-09-21T10:50:23.320651+0000","mapping_epoch":1461,"log_start":"1445'7730","ondisk_log_start":"1445'7730","created":1430,"last_epoch_clean":1462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:12:24.255016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42270720,"num_objects":645,"num_object_clones":0,"num_object_copies":1935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12489,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":214171648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.6","version":"1466'12285","reported_seq":12369,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:25.506361+0000","last_change":"2024-09-21T10:46:40.275529+0000","last_active":"2024-09-21T10:50:25.506361+0000","last_peered":"2024-09-21T10:50:25.506361+0000","last_clean":"2024-09-21T10:50:25.506361+0000","last_became_active":"2024-09-21T10:46:40.275255+0000","last_became_peered":"2024-09-21T10:46:40.275255+0000","last_unstale":"2024-09-21T10:50:25.506361+0000","last_undegraded":"2024-09-21T10:50:25.506361+0000","last_fullsized":"2024-09-21T10:50:25.506361+0000","mapping_epoch":1430,"log_start":"1445'7322","ondisk_log_start":"1445'7322","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:08.150110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":477,"num_object_clones":0,"num_object_copies":1431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12285,"num_write_kb":408384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.5","version":"1466'12254","reported_seq":18971,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:21.298182+0000","last_change":"2024-09-21T10:49:48.213220+0000","last_active":"2024-09-21T10:50:21.298182+0000","last_peered":"2024-09-21T10:50:21.298182+0000","last_clean":"2024-09-21T10:50:21.298182+0000","last_became_active":"2024-09-21T10:49:48.190551+0000","last_became_peered":"2024-09-21T10:49:48.190551+0000","last_unstale":"2024-09-21T10:50:21.298182+0000","last_undegraded":"2024-09-21T10:50:21.298182+0000","last_fullsized":"2024-09-21T10:50:21.298182+0000","mapping_epoch":1460,"log_start":"1448'7759","ondisk_log_start":"1448'7759","created":1430,"last_epoch_clean":1461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4495,"log_dups_size":0,"ondisk_log_size":4495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:41:12.288633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12254,"num_write_kb":415808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3314,"num_bytes_recovered":217186304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.4","version":"1466'12444","reported_seq":24696,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:24.692561+0000","last_change":"2024-09-21T10:48:47.619926+0000","last_active":"2024-09-21T10:50:24.692561+0000","last_peered":"2024-09-21T10:50:24.692561+0000","last_clean":"2024-09-21T10:50:24.692561+0000","last_became_active":"2024-09-21T10:48:47.605894+0000","last_became_peered":"2024-09-21T10:48:47.605894+0000","last_unstale":"2024-09-21T10:50:24.692561+0000","last_undegraded":"2024-09-21T10:50:24.692561+0000","last_fullsized":"2024-09-21T10:50:24.692561+0000","mapping_epoch":1446,"log_start":"1448'7983","ondisk_log_start":"1448'7983","created":1430,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4461,"log_dups_size":0,"ondisk_log_size":4461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:46.710060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38928384,"num_objects":594,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":417216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6084,"num_bytes_recovered":398721024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"50.3","version":"1466'12911","reported_seq":12995,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:25.523222+0000","last_change":"2024-09-21T10:46:40.275398+0000","last_active":"2024-09-21T10:50:25.523222+0000","last_peered":"2024-09-21T10:50:25.523222+0000","last_clean":"2024-09-21T10:50:25.523222+0000","last_became_active":"2024-09-21T10:46:40.275096+0000","last_became_peered":"2024-09-21T10:46:40.275096+0000","last_unstale":"2024-09-21T10:50:25.523222+0000","last_undegraded":"2024-09-21T10:50:25.523222+0000","last_fullsized":"2024-09-21T10:50:25.523222+0000","mapping_epoch":1430,"log_start":"1447'7936","ondisk_log_start":"1447'7936","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4975,"log_dups_size":0,"ondisk_log_size":4975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:58.484291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12911,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.2","version":"1466'12193","reported_seq":16876,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:21.282780+0000","last_change":"2024-09-21T10:49:58.269301+0000","last_active":"2024-09-21T10:50:21.282780+0000","last_peered":"2024-09-21T10:50:21.282780+0000","last_clean":"2024-09-21T10:50:21.282780+0000","last_became_active":"2024-09-21T10:49:58.247816+0000","last_became_peered":"2024-09-21T10:49:58.247816+0000","last_unstale":"2024-09-21T10:50:21.282780+0000","last_undegraded":"2024-09-21T10:50:21.282780+0000","last_fullsized":"2024-09-21T10:50:21.282780+0000","mapping_epoch":1463,"log_start":"1445'7640","ondisk_log_start":"1445'7640","created":1430,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:58.250248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47120384,"num_objects":719,"num_object_clones":0,"num_object_copies":2157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12193,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":150208512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.1","version":"1466'12529","reported_seq":12613,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:23.327810+0000","last_change":"2024-09-21T10:46:40.276794+0000","last_active":"2024-09-21T10:50:23.327810+0000","last_peered":"2024-09-21T10:50:23.327810+0000","last_clean":"2024-09-21T10:50:23.327810+0000","last_became_active":"2024-09-21T10:46:40.276137+0000","last_became_peered":"2024-09-21T10:46:40.276137+0000","last_unstale":"2024-09-21T10:50:23.327810+0000","last_undegraded":"2024-09-21T10:50:23.327810+0000","last_fullsized":"2024-09-21T10:50:23.327810+0000","mapping_epoch":1430,"log_start":"1448'7763","ondisk_log_start":"1448'7763","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:01.998423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12529,"num_write_kb":420672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.0","version":"1466'12612","reported_seq":22015,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:25.537973+0000","last_change":"2024-09-21T10:49:17.180145+0000","last_active":"2024-09-21T10:50:25.537973+0000","last_peered":"2024-09-21T10:50:25.537973+0000","last_clean":"2024-09-21T10:50:25.537973+0000","last_became_active":"2024-09-21T10:49:17.157202+0000","last_became_peered":"2024-09-21T10:49:17.157202+0000","last_unstale":"2024-09-21T10:50:25.537973+0000","last_undegraded":"2024-09-21T10:50:25.537973+0000","last_fullsized":"2024-09-21T10:50:25.537973+0000","mapping_epoch":1453,"log_start":"1445'7679","ondisk_log_start":"1445'7679","created":1430,"last_epoch_clean":1454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:54:17.833068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34078720,"num_objects":520,"num_object_clones":0,"num_object_copies":1560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12612,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4658,"num_bytes_recovered":305266688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":234,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681778+0000","last_change":"2024-09-21T10:48:08.207623+0000","last_active":"2024-09-21T10:50:10.681778+0000","last_peered":"2024-09-21T10:50:10.681778+0000","last_clean":"2024-09-21T10:50:10.681778+0000","last_became_active":"2024-09-21T10:48:08.152668+0000","last_became_peered":"2024-09-21T10:48:08.152668+0000","last_unstale":"2024-09-21T10:50:10.681778+0000","last_undegraded":"2024-09-21T10:50:10.681778+0000","last_fullsized":"2024-09-21T10:50:10.681778+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:46.388407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1345,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.676940+0000","last_change":"2024-09-21T10:43:56.011543+0000","last_active":"2024-09-21T10:50:10.676940+0000","last_peered":"2024-09-21T10:50:10.676940+0000","last_clean":"2024-09-21T10:50:10.676940+0000","last_became_active":"2024-09-21T10:43:56.010878+0000","last_became_peered":"2024-09-21T10:43:56.010878+0000","last_unstale":"2024-09-21T10:50:10.676940+0000","last_undegraded":"2024-09-21T10:50:10.676940+0000","last_fullsized":"2024-09-21T10:50:10.676940+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:46.231617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2258,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689661+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:50:10.689661+0000","last_peered":"2024-09-21T10:50:10.689661+0000","last_clean":"2024-09-21T10:50:10.689661+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:50:10.689661+0000","last_undegraded":"2024-09-21T10:50:10.689661+0000","last_fullsized":"2024-09-21T10:50:10.689661+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2276,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689889+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:50:10.689889+0000","last_peered":"2024-09-21T10:50:10.689889+0000","last_clean":"2024-09-21T10:50:10.689889+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:50:10.689889+0000","last_undegraded":"2024-09-21T10:50:10.689889+0000","last_fullsized":"2024-09-21T10:50:10.689889+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":147,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693104+0000","last_change":"2024-09-21T10:48:08.200826+0000","last_active":"2024-09-21T10:50:10.693104+0000","last_peered":"2024-09-21T10:50:10.693104+0000","last_clean":"2024-09-21T10:50:10.693104+0000","last_became_active":"2024-09-21T10:48:08.165760+0000","last_became_peered":"2024-09-21T10:48:08.165760+0000","last_unstale":"2024-09-21T10:50:10.693104+0000","last_undegraded":"2024-09-21T10:50:10.693104+0000","last_fullsized":"2024-09-21T10:50:10.693104+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:35.607267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1459,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.690475+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:50:10.690475+0000","last_peered":"2024-09-21T10:50:10.690475+0000","last_clean":"2024-09-21T10:50:10.690475+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:50:10.690475+0000","last_undegraded":"2024-09-21T10:50:10.690475+0000","last_fullsized":"2024-09-21T10:50:10.690475+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":96,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.701918+0000","last_change":"2024-09-21T10:48:08.196786+0000","last_active":"2024-09-21T10:50:10.701918+0000","last_peered":"2024-09-21T10:50:10.701918+0000","last_clean":"2024-09-21T10:50:10.701918+0000","last_became_active":"2024-09-21T10:48:08.161165+0000","last_became_peered":"2024-09-21T10:48:08.161165+0000","last_unstale":"2024-09-21T10:50:10.701918+0000","last_undegraded":"2024-09-21T10:50:10.701918+0000","last_fullsized":"2024-09-21T10:50:10.701918+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:35:51.828935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1553,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691321+0000","last_change":"2024-09-21T10:48:08.171396+0000","last_active":"2024-09-21T10:50:10.691321+0000","last_peered":"2024-09-21T10:50:10.691321+0000","last_clean":"2024-09-21T10:50:10.691321+0000","last_became_active":"2024-09-21T10:48:08.165828+0000","last_became_peered":"2024-09-21T10:48:08.165828+0000","last_unstale":"2024-09-21T10:50:10.691321+0000","last_undegraded":"2024-09-21T10:50:10.691321+0000","last_fullsized":"2024-09-21T10:50:10.691321+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:17.505484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":85,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671526+0000","last_change":"2024-09-21T10:46:39.241906+0000","last_active":"2024-09-21T10:50:10.671526+0000","last_peered":"2024-09-21T10:50:10.671526+0000","last_clean":"2024-09-21T10:50:10.671526+0000","last_became_active":"2024-09-21T10:46:39.241499+0000","last_became_peered":"2024-09-21T10:46:39.241499+0000","last_unstale":"2024-09-21T10:50:10.671526+0000","last_undegraded":"2024-09-21T10:50:10.671526+0000","last_fullsized":"2024-09-21T10:50:10.671526+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:53.671658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":243,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671254+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:50:10.671254+0000","last_peered":"2024-09-21T10:50:10.671254+0000","last_clean":"2024-09-21T10:50:10.671254+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:50:10.671254+0000","last_undegraded":"2024-09-21T10:50:10.671254+0000","last_fullsized":"2024-09-21T10:50:10.671254+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":318,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.694037+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:50:10.694037+0000","last_peered":"2024-09-21T10:50:10.694037+0000","last_clean":"2024-09-21T10:50:10.694037+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:50:10.694037+0000","last_undegraded":"2024-09-21T10:50:10.694037+0000","last_fullsized":"2024-09-21T10:50:10.694037+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2265,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.690159+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:50:10.690159+0000","last_peered":"2024-09-21T10:50:10.690159+0000","last_clean":"2024-09-21T10:50:10.690159+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:50:10.690159+0000","last_undegraded":"2024-09-21T10:50:10.690159+0000","last_fullsized":"2024-09-21T10:50:10.690159+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":85,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.675522+0000","last_change":"2024-09-21T10:46:39.241669+0000","last_active":"2024-09-21T10:50:10.675522+0000","last_peered":"2024-09-21T10:50:10.675522+0000","last_clean":"2024-09-21T10:50:10.675522+0000","last_became_active":"2024-09-21T10:46:39.241358+0000","last_became_peered":"2024-09-21T10:46:39.241358+0000","last_unstale":"2024-09-21T10:50:10.675522+0000","last_undegraded":"2024-09-21T10:50:10.675522+0000","last_fullsized":"2024-09-21T10:50:10.675522+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:25.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2381,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.682022+0000","last_change":"2024-09-21T10:45:59.689522+0000","last_active":"2024-09-21T10:50:10.682022+0000","last_peered":"2024-09-21T10:50:10.682022+0000","last_clean":"2024-09-21T10:50:10.682022+0000","last_became_active":"2024-09-21T10:45:59.689148+0000","last_became_peered":"2024-09-21T10:45:59.689148+0000","last_unstale":"2024-09-21T10:50:10.682022+0000","last_undegraded":"2024-09-21T10:50:10.682022+0000","last_fullsized":"2024-09-21T10:50:10.682022+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:28.142643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2291,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689464+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:50:10.689464+0000","last_peered":"2024-09-21T10:50:10.689464+0000","last_clean":"2024-09-21T10:50:10.689464+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:50:10.689464+0000","last_undegraded":"2024-09-21T10:50:10.689464+0000","last_fullsized":"2024-09-21T10:50:10.689464+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1273,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.704126+0000","last_change":"2024-09-21T10:48:08.161215+0000","last_active":"2024-09-21T10:50:10.704126+0000","last_peered":"2024-09-21T10:50:10.704126+0000","last_clean":"2024-09-21T10:50:10.704126+0000","last_became_active":"2024-09-21T10:48:08.160600+0000","last_became_peered":"2024-09-21T10:48:08.160600+0000","last_unstale":"2024-09-21T10:50:10.704126+0000","last_undegraded":"2024-09-21T10:50:10.704126+0000","last_fullsized":"2024-09-21T10:50:10.704126+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:35.127563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2270,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689638+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:50:10.689638+0000","last_peered":"2024-09-21T10:50:10.689638+0000","last_clean":"2024-09-21T10:50:10.689638+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:50:10.689638+0000","last_undegraded":"2024-09-21T10:50:10.689638+0000","last_fullsized":"2024-09-21T10:50:10.689638+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2397,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.682126+0000","last_change":"2024-09-21T10:49:40.171098+0000","last_active":"2024-09-21T10:50:10.682126+0000","last_peered":"2024-09-21T10:50:10.682126+0000","last_clean":"2024-09-21T10:50:10.682126+0000","last_became_active":"2024-09-21T10:49:40.169461+0000","last_became_peered":"2024-09-21T10:49:40.169461+0000","last_unstale":"2024-09-21T10:50:10.682126+0000","last_undegraded":"2024-09-21T10:50:10.682126+0000","last_fullsized":"2024-09-21T10:50:10.682126+0000","mapping_epoch":1456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:09.774760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":407,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671411+0000","last_change":"2024-09-21T10:45:59.680994+0000","last_active":"2024-09-21T10:50:10.671411+0000","last_peered":"2024-09-21T10:50:10.671411+0000","last_clean":"2024-09-21T10:50:10.671411+0000","last_became_active":"2024-09-21T10:45:59.680636+0000","last_became_peered":"2024-09-21T10:45:59.680636+0000","last_unstale":"2024-09-21T10:50:10.671411+0000","last_undegraded":"2024-09-21T10:50:10.671411+0000","last_fullsized":"2024-09-21T10:50:10.671411+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:42.488110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2267,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.690144+0000","last_change":"2024-09-21T10:48:08.152845+0000","last_active":"2024-09-21T10:50:10.690144+0000","last_peered":"2024-09-21T10:50:10.690144+0000","last_clean":"2024-09-21T10:50:10.690144+0000","last_became_active":"2024-09-21T10:48:08.152542+0000","last_became_peered":"2024-09-21T10:48:08.152542+0000","last_unstale":"2024-09-21T10:50:10.690144+0000","last_undegraded":"2024-09-21T10:50:10.690144+0000","last_fullsized":"2024-09-21T10:50:10.690144+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:27.471208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":557,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693318+0000","last_change":"2024-09-21T10:48:08.177384+0000","last_active":"2024-09-21T10:50:10.693318+0000","last_peered":"2024-09-21T10:50:10.693318+0000","last_clean":"2024-09-21T10:50:10.693318+0000","last_became_active":"2024-09-21T10:48:08.176576+0000","last_became_peered":"2024-09-21T10:48:08.176576+0000","last_unstale":"2024-09-21T10:50:10.693318+0000","last_undegraded":"2024-09-21T10:50:10.693318+0000","last_fullsized":"2024-09-21T10:50:10.693318+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:35.528635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":152,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689997+0000","last_change":"2024-09-21T10:45:59.710479+0000","last_active":"2024-09-21T10:50:10.689997+0000","last_peered":"2024-09-21T10:50:10.689997+0000","last_clean":"2024-09-21T10:50:10.689997+0000","last_became_active":"2024-09-21T10:45:59.694474+0000","last_became_peered":"2024-09-21T10:45:59.694474+0000","last_unstale":"2024-09-21T10:50:10.689997+0000","last_undegraded":"2024-09-21T10:50:10.689997+0000","last_fullsized":"2024-09-21T10:50:10.689997+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:32.554219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":92,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693798+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:50:10.693798+0000","last_peered":"2024-09-21T10:50:10.693798+0000","last_clean":"2024-09-21T10:50:10.693798+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:50:10.693798+0000","last_undegraded":"2024-09-21T10:50:10.693798+0000","last_fullsized":"2024-09-21T10:50:10.693798+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":447,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671310+0000","last_change":"2024-09-21T10:45:59.705879+0000","last_active":"2024-09-21T10:50:10.671310+0000","last_peered":"2024-09-21T10:50:10.671310+0000","last_clean":"2024-09-21T10:50:10.671310+0000","last_became_active":"2024-09-21T10:45:59.689786+0000","last_became_peered":"2024-09-21T10:45:59.689786+0000","last_unstale":"2024-09-21T10:50:10.671310+0000","last_undegraded":"2024-09-21T10:50:10.671310+0000","last_fullsized":"2024-09-21T10:50:10.671310+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:50.473399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1316,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.694024+0000","last_change":"2024-09-21T10:48:08.155603+0000","last_active":"2024-09-21T10:50:10.694024+0000","last_peered":"2024-09-21T10:50:10.694024+0000","last_clean":"2024-09-21T10:50:10.694024+0000","last_became_active":"2024-09-21T10:48:08.155285+0000","last_became_peered":"2024-09-21T10:48:08.155285+0000","last_unstale":"2024-09-21T10:50:10.694024+0000","last_undegraded":"2024-09-21T10:50:10.694024+0000","last_fullsized":"2024-09-21T10:50:10.694024+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:40:25.014519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":535,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691359+0000","last_change":"2024-09-21T10:48:08.167192+0000","last_active":"2024-09-21T10:50:10.691359+0000","last_peered":"2024-09-21T10:50:10.691359+0000","last_clean":"2024-09-21T10:50:10.691359+0000","last_became_active":"2024-09-21T10:48:08.166566+0000","last_became_peered":"2024-09-21T10:48:08.166566+0000","last_unstale":"2024-09-21T10:50:10.691359+0000","last_undegraded":"2024-09-21T10:50:10.691359+0000","last_fullsized":"2024-09-21T10:50:10.691359+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:19.985042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1273,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.683890+0000","last_change":"2024-09-21T10:48:08.152147+0000","last_active":"2024-09-21T10:50:10.683890+0000","last_peered":"2024-09-21T10:50:10.683890+0000","last_clean":"2024-09-21T10:50:10.683890+0000","last_became_active":"2024-09-21T10:48:08.151842+0000","last_became_peered":"2024-09-21T10:48:08.151842+0000","last_unstale":"2024-09-21T10:50:10.683890+0000","last_undegraded":"2024-09-21T10:50:10.683890+0000","last_fullsized":"2024-09-21T10:50:10.683890+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:39.526853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1275,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.673540+0000","last_change":"2024-09-21T10:45:59.697070+0000","last_active":"2024-09-21T10:50:10.673540+0000","last_peered":"2024-09-21T10:50:10.673540+0000","last_clean":"2024-09-21T10:50:10.673540+0000","last_became_active":"2024-09-21T10:45:59.696548+0000","last_became_peered":"2024-09-21T10:45:59.696548+0000","last_unstale":"2024-09-21T10:50:10.673540+0000","last_undegraded":"2024-09-21T10:50:10.673540+0000","last_fullsized":"2024-09-21T10:50:10.673540+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:02.746989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":269,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681629+0000","last_change":"2024-09-21T10:48:08.210449+0000","last_active":"2024-09-21T10:50:10.681629+0000","last_peered":"2024-09-21T10:50:10.681629+0000","last_clean":"2024-09-21T10:50:10.681629+0000","last_became_active":"2024-09-21T10:48:08.162433+0000","last_became_peered":"2024-09-21T10:48:08.162433+0000","last_unstale":"2024-09-21T10:50:10.681629+0000","last_undegraded":"2024-09-21T10:50:10.681629+0000","last_fullsized":"2024-09-21T10:50:10.681629+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:55.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":309,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671756+0000","last_change":"2024-09-21T10:48:08.155548+0000","last_active":"2024-09-21T10:50:10.671756+0000","last_peered":"2024-09-21T10:50:10.671756+0000","last_clean":"2024-09-21T10:50:10.671756+0000","last_became_active":"2024-09-21T10:48:08.154428+0000","last_became_peered":"2024-09-21T10:48:08.154428+0000","last_unstale":"2024-09-21T10:50:10.671756+0000","last_undegraded":"2024-09-21T10:50:10.671756+0000","last_fullsized":"2024-09-21T10:50:10.671756+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:23.699561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":59,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.688773+0000","last_change":"2024-09-21T10:48:08.155960+0000","last_active":"2024-09-21T10:50:10.688773+0000","last_peered":"2024-09-21T10:50:10.688773+0000","last_clean":"2024-09-21T10:50:10.688773+0000","last_became_active":"2024-09-21T10:48:08.154801+0000","last_became_peered":"2024-09-21T10:48:08.154801+0000","last_unstale":"2024-09-21T10:50:10.688773+0000","last_undegraded":"2024-09-21T10:50:10.688773+0000","last_fullsized":"2024-09-21T10:50:10.688773+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:46.418044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2373,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693397+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:50:10.693397+0000","last_peered":"2024-09-21T10:50:10.693397+0000","last_clean":"2024-09-21T10:50:10.693397+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:50:10.693397+0000","last_undegraded":"2024-09-21T10:50:10.693397+0000","last_fullsized":"2024-09-21T10:50:10.693397+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":100,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.673351+0000","last_change":"2024-09-21T10:48:08.194127+0000","last_active":"2024-09-21T10:50:10.673351+0000","last_peered":"2024-09-21T10:50:10.673351+0000","last_clean":"2024-09-21T10:50:10.673351+0000","last_became_active":"2024-09-21T10:48:08.161824+0000","last_became_peered":"2024-09-21T10:48:08.161824+0000","last_unstale":"2024-09-21T10:50:10.673351+0000","last_undegraded":"2024-09-21T10:50:10.673351+0000","last_fullsized":"2024-09-21T10:50:10.673351+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.139417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":456,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681724+0000","last_change":"2024-09-21T10:43:58.555202+0000","last_active":"2024-09-21T10:50:10.681724+0000","last_peered":"2024-09-21T10:50:10.681724+0000","last_clean":"2024-09-21T10:50:10.681724+0000","last_became_active":"2024-09-21T10:43:56.009648+0000","last_became_peered":"2024-09-21T10:43:56.009648+0000","last_unstale":"2024-09-21T10:50:10.681724+0000","last_undegraded":"2024-09-21T10:50:10.681724+0000","last_fullsized":"2024-09-21T10:50:10.681724+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.555120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.555120+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:45:37.803817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2390,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681473+0000","last_change":"2024-09-21T10:48:08.162201+0000","last_active":"2024-09-21T10:50:10.681473+0000","last_peered":"2024-09-21T10:50:10.681473+0000","last_clean":"2024-09-21T10:50:10.681473+0000","last_became_active":"2024-09-21T10:48:08.161884+0000","last_became_peered":"2024-09-21T10:48:08.161884+0000","last_unstale":"2024-09-21T10:50:10.681473+0000","last_undegraded":"2024-09-21T10:50:10.681473+0000","last_fullsized":"2024-09-21T10:50:10.681473+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:08.759002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1474,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.682612+0000","last_change":"2024-09-21T10:45:59.694166+0000","last_active":"2024-09-21T10:50:10.682612+0000","last_peered":"2024-09-21T10:50:10.682612+0000","last_clean":"2024-09-21T10:50:10.682612+0000","last_became_active":"2024-09-21T10:45:59.693276+0000","last_became_peered":"2024-09-21T10:45:59.693276+0000","last_unstale":"2024-09-21T10:50:10.682612+0000","last_undegraded":"2024-09-21T10:50:10.682612+0000","last_fullsized":"2024-09-21T10:50:10.682612+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:12.784743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2367,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.702013+0000","last_change":"2024-09-21T10:48:08.161699+0000","last_active":"2024-09-21T10:50:10.702013+0000","last_peered":"2024-09-21T10:50:10.702013+0000","last_clean":"2024-09-21T10:50:10.702013+0000","last_became_active":"2024-09-21T10:48:08.160423+0000","last_became_peered":"2024-09-21T10:48:08.160423+0000","last_unstale":"2024-09-21T10:50:10.702013+0000","last_undegraded":"2024-09-21T10:50:10.702013+0000","last_fullsized":"2024-09-21T10:50:10.702013+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:28.698737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":474,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.688924+0000","last_change":"2024-09-21T10:48:08.157705+0000","last_active":"2024-09-21T10:50:10.688924+0000","last_peered":"2024-09-21T10:50:10.688924+0000","last_clean":"2024-09-21T10:50:10.688924+0000","last_became_active":"2024-09-21T10:48:08.157385+0000","last_became_peered":"2024-09-21T10:48:08.157385+0000","last_unstale":"2024-09-21T10:50:10.688924+0000","last_undegraded":"2024-09-21T10:50:10.688924+0000","last_fullsized":"2024-09-21T10:50:10.688924+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.698401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.698401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:48.903892+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2372,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.694245+0000","last_change":"2024-09-21T10:48:08.161856+0000","last_active":"2024-09-21T10:50:10.694245+0000","last_peered":"2024-09-21T10:50:10.694245+0000","last_clean":"2024-09-21T10:50:10.694245+0000","last_became_active":"2024-09-21T10:48:08.160283+0000","last_became_peered":"2024-09-21T10:48:08.160283+0000","last_unstale":"2024-09-21T10:50:10.694245+0000","last_undegraded":"2024-09-21T10:50:10.694245+0000","last_fullsized":"2024-09-21T10:50:10.694245+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:09.474670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2367,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.702158+0000","last_change":"2024-09-21T10:45:59.691645+0000","last_active":"2024-09-21T10:50:10.702158+0000","last_peered":"2024-09-21T10:50:10.702158+0000","last_clean":"2024-09-21T10:50:10.702158+0000","last_became_active":"2024-09-21T10:45:59.691334+0000","last_became_peered":"2024-09-21T10:45:59.691334+0000","last_unstale":"2024-09-21T10:50:10.702158+0000","last_undegraded":"2024-09-21T10:50:10.702158+0000","last_fullsized":"2024-09-21T10:50:10.702158+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:33.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":797,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.690117+0000","last_change":"2024-09-21T10:48:08.144825+0000","last_active":"2024-09-21T10:50:10.690117+0000","last_peered":"2024-09-21T10:50:10.690117+0000","last_clean":"2024-09-21T10:50:10.690117+0000","last_became_active":"2024-09-21T10:48:08.142212+0000","last_became_peered":"2024-09-21T10:48:08.142212+0000","last_unstale":"2024-09-21T10:50:10.690117+0000","last_undegraded":"2024-09-21T10:50:10.690117+0000","last_fullsized":"2024-09-21T10:50:10.690117+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:46.899049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":452,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.684086+0000","last_change":"2024-09-21T10:48:08.960298+0000","last_active":"2024-09-21T10:50:10.684086+0000","last_peered":"2024-09-21T10:50:10.684086+0000","last_clean":"2024-09-21T10:50:10.684086+0000","last_became_active":"2024-09-21T10:48:08.151776+0000","last_became_peered":"2024-09-21T10:48:08.151776+0000","last_unstale":"2024-09-21T10:50:10.684086+0000","last_undegraded":"2024-09-21T10:50:10.684086+0000","last_fullsized":"2024-09-21T10:50:10.684086+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.960250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.960250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:25.581994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2357,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.702087+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:50:10.702087+0000","last_peered":"2024-09-21T10:50:10.702087+0000","last_clean":"2024-09-21T10:50:10.702087+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:50:10.702087+0000","last_undegraded":"2024-09-21T10:50:10.702087+0000","last_fullsized":"2024-09-21T10:50:10.702087+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1478,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681384+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:50:10.681384+0000","last_peered":"2024-09-21T10:50:10.681384+0000","last_clean":"2024-09-21T10:50:10.681384+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:50:10.681384+0000","last_undegraded":"2024-09-21T10:50:10.681384+0000","last_fullsized":"2024-09-21T10:50:10.681384+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":197,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.694200+0000","last_change":"2024-09-21T10:48:08.212451+0000","last_active":"2024-09-21T10:50:10.694200+0000","last_peered":"2024-09-21T10:50:10.694200+0000","last_clean":"2024-09-21T10:50:10.694200+0000","last_became_active":"2024-09-21T10:48:08.161435+0000","last_became_peered":"2024-09-21T10:48:08.161435+0000","last_unstale":"2024-09-21T10:50:10.694200+0000","last_undegraded":"2024-09-21T10:50:10.694200+0000","last_fullsized":"2024-09-21T10:50:10.694200+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:44.511679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":695,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681293+0000","last_change":"2024-09-21T10:48:08.172490+0000","last_active":"2024-09-21T10:50:10.681293+0000","last_peered":"2024-09-21T10:50:10.681293+0000","last_clean":"2024-09-21T10:50:10.681293+0000","last_became_active":"2024-09-21T10:48:08.172117+0000","last_became_peered":"2024-09-21T10:48:08.172117+0000","last_unstale":"2024-09-21T10:50:10.681293+0000","last_undegraded":"2024-09-21T10:50:10.681293+0000","last_fullsized":"2024-09-21T10:50:10.681293+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:35.639980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":99,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.700976+0000","last_change":"2024-09-21T10:48:08.182097+0000","last_active":"2024-09-21T10:50:10.700976+0000","last_peered":"2024-09-21T10:50:10.700976+0000","last_clean":"2024-09-21T10:50:10.700976+0000","last_became_active":"2024-09-21T10:48:08.181514+0000","last_became_peered":"2024-09-21T10:48:08.181514+0000","last_unstale":"2024-09-21T10:50:10.700976+0000","last_undegraded":"2024-09-21T10:50:10.700976+0000","last_fullsized":"2024-09-21T10:50:10.700976+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:59.851982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2358,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693534+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:50:10.693534+0000","last_peered":"2024-09-21T10:50:10.693534+0000","last_clean":"2024-09-21T10:50:10.693534+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:50:10.693534+0000","last_undegraded":"2024-09-21T10:50:10.693534+0000","last_fullsized":"2024-09-21T10:50:10.693534+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2290,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.675960+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:50:10.675960+0000","last_peered":"2024-09-21T10:50:10.675960+0000","last_clean":"2024-09-21T10:50:10.675960+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:50:10.675960+0000","last_undegraded":"2024-09-21T10:50:10.675960+0000","last_fullsized":"2024-09-21T10:50:10.675960+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":550,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693929+0000","last_change":"2024-09-21T10:48:08.182699+0000","last_active":"2024-09-21T10:50:10.693929+0000","last_peered":"2024-09-21T10:50:10.693929+0000","last_clean":"2024-09-21T10:50:10.693929+0000","last_became_active":"2024-09-21T10:48:08.182308+0000","last_became_peered":"2024-09-21T10:48:08.182308+0000","last_unstale":"2024-09-21T10:50:10.693929+0000","last_undegraded":"2024-09-21T10:50:10.693929+0000","last_fullsized":"2024-09-21T10:50:10.693929+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:32.985403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":215,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681483+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:50:10.681483+0000","last_peered":"2024-09-21T10:50:10.681483+0000","last_clean":"2024-09-21T10:50:10.681483+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:50:10.681483+0000","last_undegraded":"2024-09-21T10:50:10.681483+0000","last_fullsized":"2024-09-21T10:50:10.681483+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2398,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.684137+0000","last_change":"2024-09-21T10:48:08.166778+0000","last_active":"2024-09-21T10:50:10.684137+0000","last_peered":"2024-09-21T10:50:10.684137+0000","last_clean":"2024-09-21T10:50:10.684137+0000","last_became_active":"2024-09-21T10:48:08.166031+0000","last_became_peered":"2024-09-21T10:48:08.166031+0000","last_unstale":"2024-09-21T10:50:10.684137+0000","last_undegraded":"2024-09-21T10:50:10.684137+0000","last_fullsized":"2024-09-21T10:50:10.684137+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.713934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":636,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693617+0000","last_change":"2024-09-21T10:45:59.691940+0000","last_active":"2024-09-21T10:50:10.693617+0000","last_peered":"2024-09-21T10:50:10.693617+0000","last_clean":"2024-09-21T10:50:10.693617+0000","last_became_active":"2024-09-21T10:45:59.691610+0000","last_became_peered":"2024-09-21T10:45:59.691610+0000","last_unstale":"2024-09-21T10:50:10.693617+0000","last_undegraded":"2024-09-21T10:50:10.693617+0000","last_fullsized":"2024-09-21T10:50:10.693617+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:57.766019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1404,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.682689+0000","last_change":"2024-09-21T10:46:46.607744+0000","last_active":"2024-09-21T10:50:10.682689+0000","last_peered":"2024-09-21T10:50:10.682689+0000","last_clean":"2024-09-21T10:50:10.682689+0000","last_became_active":"2024-09-21T10:46:46.607376+0000","last_became_peered":"2024-09-21T10:46:46.607376+0000","last_unstale":"2024-09-21T10:50:10.682689+0000","last_undegraded":"2024-09-21T10:50:10.682689+0000","last_fullsized":"2024-09-21T10:50:10.682689+0000","mapping_epoch":1433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:17.279053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2282,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.690557+0000","last_change":"2024-09-21T10:48:08.158793+0000","last_active":"2024-09-21T10:50:10.690557+0000","last_peered":"2024-09-21T10:50:10.690557+0000","last_clean":"2024-09-21T10:50:10.690557+0000","last_became_active":"2024-09-21T10:48:08.158487+0000","last_became_peered":"2024-09-21T10:48:08.158487+0000","last_unstale":"2024-09-21T10:50:10.690557+0000","last_undegraded":"2024-09-21T10:50:10.690557+0000","last_fullsized":"2024-09-21T10:50:10.690557+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:53.537691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":195,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671719+0000","last_change":"2024-09-21T10:48:08.214131+0000","last_active":"2024-09-21T10:50:10.671719+0000","last_peered":"2024-09-21T10:50:10.671719+0000","last_clean":"2024-09-21T10:50:10.671719+0000","last_became_active":"2024-09-21T10:48:08.157618+0000","last_became_peered":"2024-09-21T10:48:08.157618+0000","last_unstale":"2024-09-21T10:50:10.671719+0000","last_undegraded":"2024-09-21T10:50:10.671719+0000","last_fullsized":"2024-09-21T10:50:10.671719+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:36.574106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1627,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693029+0000","last_change":"2024-09-21T10:45:59.692592+0000","last_active":"2024-09-21T10:50:10.693029+0000","last_peered":"2024-09-21T10:50:10.693029+0000","last_clean":"2024-09-21T10:50:10.693029+0000","last_became_active":"2024-09-21T10:45:59.692232+0000","last_became_peered":"2024-09-21T10:45:59.692232+0000","last_unstale":"2024-09-21T10:50:10.693029+0000","last_undegraded":"2024-09-21T10:50:10.693029+0000","last_fullsized":"2024-09-21T10:50:10.693029+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:26.920879+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":127,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671309+0000","last_change":"2024-09-21T10:48:08.148499+0000","last_active":"2024-09-21T10:50:10.671309+0000","last_peered":"2024-09-21T10:50:10.671309+0000","last_clean":"2024-09-21T10:50:10.671309+0000","last_became_active":"2024-09-21T10:48:08.148141+0000","last_became_peered":"2024-09-21T10:48:08.148141+0000","last_unstale":"2024-09-21T10:50:10.671309+0000","last_undegraded":"2024-09-21T10:50:10.671309+0000","last_fullsized":"2024-09-21T10:50:10.671309+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:35.860469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2377,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.701149+0000","last_change":"2024-09-21T10:48:08.171805+0000","last_active":"2024-09-21T10:50:10.701149+0000","last_peered":"2024-09-21T10:50:10.701149+0000","last_clean":"2024-09-21T10:50:10.701149+0000","last_became_active":"2024-09-21T10:48:08.170840+0000","last_became_peered":"2024-09-21T10:48:08.170840+0000","last_unstale":"2024-09-21T10:50:10.701149+0000","last_undegraded":"2024-09-21T10:50:10.701149+0000","last_fullsized":"2024-09-21T10:50:10.701149+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:18.802870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2276,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689561+0000","last_change":"2024-09-21T10:48:08.155588+0000","last_active":"2024-09-21T10:50:10.689561+0000","last_peered":"2024-09-21T10:50:10.689561+0000","last_clean":"2024-09-21T10:50:10.689561+0000","last_became_active":"2024-09-21T10:48:08.153652+0000","last_became_peered":"2024-09-21T10:48:08.153652+0000","last_unstale":"2024-09-21T10:50:10.689561+0000","last_undegraded":"2024-09-21T10:50:10.689561+0000","last_fullsized":"2024-09-21T10:50:10.689561+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:58.415174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":534,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693920+0000","last_change":"2024-09-21T10:48:08.181118+0000","last_active":"2024-09-21T10:50:10.693920+0000","last_peered":"2024-09-21T10:50:10.693920+0000","last_clean":"2024-09-21T10:50:10.693920+0000","last_became_active":"2024-09-21T10:48:08.180184+0000","last_became_peered":"2024-09-21T10:48:08.180184+0000","last_unstale":"2024-09-21T10:50:10.693920+0000","last_undegraded":"2024-09-21T10:50:10.693920+0000","last_fullsized":"2024-09-21T10:50:10.693920+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:03:18.370832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":92,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.692971+0000","last_change":"2024-09-21T10:46:30.264653+0000","last_active":"2024-09-21T10:50:10.692971+0000","last_peered":"2024-09-21T10:50:10.692971+0000","last_clean":"2024-09-21T10:50:10.692971+0000","last_became_active":"2024-09-21T10:46:30.264068+0000","last_became_peered":"2024-09-21T10:46:30.264068+0000","last_unstale":"2024-09-21T10:50:10.692971+0000","last_undegraded":"2024-09-21T10:50:10.692971+0000","last_fullsized":"2024-09-21T10:50:10.692971+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:53.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1431,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.675882+0000","last_change":"2024-09-21T10:48:08.171234+0000","last_active":"2024-09-21T10:50:10.675882+0000","last_peered":"2024-09-21T10:50:10.675882+0000","last_clean":"2024-09-21T10:50:10.675882+0000","last_became_active":"2024-09-21T10:48:08.170908+0000","last_became_peered":"2024-09-21T10:48:08.170908+0000","last_unstale":"2024-09-21T10:50:10.675882+0000","last_undegraded":"2024-09-21T10:50:10.675882+0000","last_fullsized":"2024-09-21T10:50:10.675882+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:28.112802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":142,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.677061+0000","last_change":"2024-09-21T10:48:08.215096+0000","last_active":"2024-09-21T10:50:10.677061+0000","last_peered":"2024-09-21T10:50:10.677061+0000","last_clean":"2024-09-21T10:50:10.677061+0000","last_became_active":"2024-09-21T10:48:08.145535+0000","last_became_peered":"2024-09-21T10:48:08.145535+0000","last_unstale":"2024-09-21T10:50:10.677061+0000","last_undegraded":"2024-09-21T10:50:10.677061+0000","last_fullsized":"2024-09-21T10:50:10.677061+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:55.688095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1425,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671904+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:50:10.671904+0000","last_peered":"2024-09-21T10:50:10.671904+0000","last_clean":"2024-09-21T10:50:10.671904+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:50:10.671904+0000","last_undegraded":"2024-09-21T10:50:10.671904+0000","last_fullsized":"2024-09-21T10:50:10.671904+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1416,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.700635+0000","last_change":"2024-09-21T10:43:56.012844+0000","last_active":"2024-09-21T10:50:10.700635+0000","last_peered":"2024-09-21T10:50:10.700635+0000","last_clean":"2024-09-21T10:50:10.700635+0000","last_became_active":"2024-09-21T10:43:56.012504+0000","last_became_peered":"2024-09-21T10:43:56.012504+0000","last_unstale":"2024-09-21T10:50:10.700635+0000","last_undegraded":"2024-09-21T10:50:10.700635+0000","last_fullsized":"2024-09-21T10:50:10.700635+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:02.556342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1465,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689070+0000","last_change":"2024-09-21T10:48:08.153599+0000","last_active":"2024-09-21T10:50:10.689070+0000","last_peered":"2024-09-21T10:50:10.689070+0000","last_clean":"2024-09-21T10:50:10.689070+0000","last_became_active":"2024-09-21T10:48:08.153255+0000","last_became_peered":"2024-09-21T10:48:08.153255+0000","last_unstale":"2024-09-21T10:50:10.689070+0000","last_undegraded":"2024-09-21T10:50:10.689070+0000","last_fullsized":"2024-09-21T10:50:10.689070+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:04.469883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:04.469883+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:12:53.690175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1477,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691618+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:50:10.691618+0000","last_peered":"2024-09-21T10:50:10.691618+0000","last_clean":"2024-09-21T10:50:10.691618+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:50:10.691618+0000","last_undegraded":"2024-09-21T10:50:10.691618+0000","last_fullsized":"2024-09-21T10:50:10.691618+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":489,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691216+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:50:10.691216+0000","last_peered":"2024-09-21T10:50:10.691216+0000","last_clean":"2024-09-21T10:50:10.691216+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:50:10.691216+0000","last_undegraded":"2024-09-21T10:50:10.691216+0000","last_fullsized":"2024-09-21T10:50:10.691216+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":441,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689482+0000","last_change":"2024-09-21T10:46:08.467544+0000","last_active":"2024-09-21T10:50:10.689482+0000","last_peered":"2024-09-21T10:50:10.689482+0000","last_clean":"2024-09-21T10:50:10.689482+0000","last_became_active":"2024-09-21T10:45:59.695828+0000","last_became_peered":"2024-09-21T10:45:59.695828+0000","last_unstale":"2024-09-21T10:50:10.689482+0000","last_undegraded":"2024-09-21T10:50:10.689482+0000","last_fullsized":"2024-09-21T10:50:10.689482+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:08.467485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:08.467485+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:36.742481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2389,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681300+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:50:10.681300+0000","last_peered":"2024-09-21T10:50:10.681300+0000","last_clean":"2024-09-21T10:50:10.681300+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:50:10.681300+0000","last_undegraded":"2024-09-21T10:50:10.681300+0000","last_fullsized":"2024-09-21T10:50:10.681300+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1332,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691489+0000","last_change":"2024-09-21T10:48:08.166901+0000","last_active":"2024-09-21T10:50:10.691489+0000","last_peered":"2024-09-21T10:50:10.691489+0000","last_clean":"2024-09-21T10:50:10.691489+0000","last_became_active":"2024-09-21T10:48:08.166122+0000","last_became_peered":"2024-09-21T10:48:08.166122+0000","last_unstale":"2024-09-21T10:50:10.691489+0000","last_undegraded":"2024-09-21T10:50:10.691489+0000","last_fullsized":"2024-09-21T10:50:10.691489+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:18.037036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":524,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.688946+0000","last_change":"2024-09-21T10:48:08.158541+0000","last_active":"2024-09-21T10:50:10.688946+0000","last_peered":"2024-09-21T10:50:10.688946+0000","last_clean":"2024-09-21T10:50:10.688946+0000","last_became_active":"2024-09-21T10:48:08.158099+0000","last_became_peered":"2024-09-21T10:48:08.158099+0000","last_unstale":"2024-09-21T10:50:10.688946+0000","last_undegraded":"2024-09-21T10:50:10.688946+0000","last_fullsized":"2024-09-21T10:50:10.688946+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:31.818227+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1265,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.694317+0000","last_change":"2024-09-21T10:48:08.154134+0000","last_active":"2024-09-21T10:50:10.694317+0000","last_peered":"2024-09-21T10:50:10.694317+0000","last_clean":"2024-09-21T10:50:10.694317+0000","last_became_active":"2024-09-21T10:48:08.153798+0000","last_became_peered":"2024-09-21T10:48:08.153798+0000","last_unstale":"2024-09-21T10:50:10.694317+0000","last_undegraded":"2024-09-21T10:50:10.694317+0000","last_fullsized":"2024-09-21T10:50:10.694317+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:59:30.113364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1427,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.689945+0000","last_change":"2024-09-21T10:48:08.170002+0000","last_active":"2024-09-21T10:50:10.689945+0000","last_peered":"2024-09-21T10:50:10.689945+0000","last_clean":"2024-09-21T10:50:10.689945+0000","last_became_active":"2024-09-21T10:48:08.152043+0000","last_became_peered":"2024-09-21T10:48:08.152043+0000","last_unstale":"2024-09-21T10:50:10.689945+0000","last_undegraded":"2024-09-21T10:50:10.689945+0000","last_fullsized":"2024-09-21T10:50:10.689945+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:55.281864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":528,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691217+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:50:10.691217+0000","last_peered":"2024-09-21T10:50:10.691217+0000","last_clean":"2024-09-21T10:50:10.691217+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:50:10.691217+0000","last_undegraded":"2024-09-21T10:50:10.691217+0000","last_fullsized":"2024-09-21T10:50:10.691217+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1459,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693470+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:50:10.693470+0000","last_peered":"2024-09-21T10:50:10.693470+0000","last_clean":"2024-09-21T10:50:10.693470+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:50:10.693470+0000","last_undegraded":"2024-09-21T10:50:10.693470+0000","last_fullsized":"2024-09-21T10:50:10.693470+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":137,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693226+0000","last_change":"2024-09-21T10:48:08.204361+0000","last_active":"2024-09-21T10:50:10.693226+0000","last_peered":"2024-09-21T10:50:10.693226+0000","last_clean":"2024-09-21T10:50:10.693226+0000","last_became_active":"2024-09-21T10:48:08.177064+0000","last_became_peered":"2024-09-21T10:48:08.177064+0000","last_unstale":"2024-09-21T10:50:10.693226+0000","last_undegraded":"2024-09-21T10:50:10.693226+0000","last_fullsized":"2024-09-21T10:50:10.693226+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:12.137967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":120,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693212+0000","last_change":"2024-09-21T10:46:00.158554+0000","last_active":"2024-09-21T10:50:10.693212+0000","last_peered":"2024-09-21T10:50:10.693212+0000","last_clean":"2024-09-21T10:50:10.693212+0000","last_became_active":"2024-09-21T10:45:59.680546+0000","last_became_peered":"2024-09-21T10:45:59.680546+0000","last_unstale":"2024-09-21T10:50:10.693212+0000","last_undegraded":"2024-09-21T10:50:10.693212+0000","last_fullsized":"2024-09-21T10:50:10.693212+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:00.158500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:00.158500+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:39.581987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":228,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.688993+0000","last_change":"2024-09-21T10:48:08.158317+0000","last_active":"2024-09-21T10:50:10.688993+0000","last_peered":"2024-09-21T10:50:10.688993+0000","last_clean":"2024-09-21T10:50:10.688993+0000","last_became_active":"2024-09-21T10:48:08.154052+0000","last_became_peered":"2024-09-21T10:48:08.154052+0000","last_unstale":"2024-09-21T10:50:10.688993+0000","last_undegraded":"2024-09-21T10:50:10.688993+0000","last_fullsized":"2024-09-21T10:50:10.688993+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:28.045098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1494,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.682262+0000","last_change":"2024-09-21T10:48:08.213283+0000","last_active":"2024-09-21T10:50:10.682262+0000","last_peered":"2024-09-21T10:50:10.682262+0000","last_clean":"2024-09-21T10:50:10.682262+0000","last_became_active":"2024-09-21T10:48:08.166573+0000","last_became_peered":"2024-09-21T10:48:08.166573+0000","last_unstale":"2024-09-21T10:50:10.682262+0000","last_undegraded":"2024-09-21T10:50:10.682262+0000","last_fullsized":"2024-09-21T10:50:10.682262+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:43:51.213203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":68,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693946+0000","last_change":"2024-09-21T10:48:08.843710+0000","last_active":"2024-09-21T10:50:10.693946+0000","last_peered":"2024-09-21T10:50:10.693946+0000","last_clean":"2024-09-21T10:50:10.693946+0000","last_became_active":"2024-09-21T10:48:08.162916+0000","last_became_peered":"2024-09-21T10:48:08.162916+0000","last_unstale":"2024-09-21T10:50:10.693946+0000","last_undegraded":"2024-09-21T10:50:10.693946+0000","last_fullsized":"2024-09-21T10:50:10.693946+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.843662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.843662+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:40:35.024010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1531,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.692833+0000","last_change":"2024-09-21T10:48:08.193479+0000","last_active":"2024-09-21T10:50:10.692833+0000","last_peered":"2024-09-21T10:50:10.692833+0000","last_clean":"2024-09-21T10:50:10.692833+0000","last_became_active":"2024-09-21T10:48:08.149775+0000","last_became_peered":"2024-09-21T10:48:08.149775+0000","last_unstale":"2024-09-21T10:50:10.692833+0000","last_undegraded":"2024-09-21T10:50:10.692833+0000","last_fullsized":"2024-09-21T10:50:10.692833+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:00.302830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2258,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.688995+0000","last_change":"2024-09-21T10:45:59.696879+0000","last_active":"2024-09-21T10:50:10.688995+0000","last_peered":"2024-09-21T10:50:10.688995+0000","last_clean":"2024-09-21T10:50:10.688995+0000","last_became_active":"2024-09-21T10:45:59.696055+0000","last_became_peered":"2024-09-21T10:45:59.696055+0000","last_unstale":"2024-09-21T10:50:10.688995+0000","last_undegraded":"2024-09-21T10:50:10.688995+0000","last_fullsized":"2024-09-21T10:50:10.688995+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:10.189668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":533,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.693823+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:50:10.693823+0000","last_peered":"2024-09-21T10:50:10.693823+0000","last_clean":"2024-09-21T10:50:10.693823+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:50:10.693823+0000","last_undegraded":"2024-09-21T10:50:10.693823+0000","last_fullsized":"2024-09-21T10:50:10.693823+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1312,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.683753+0000","last_change":"2024-09-21T10:45:59.693730+0000","last_active":"2024-09-21T10:50:10.683753+0000","last_peered":"2024-09-21T10:50:10.683753+0000","last_clean":"2024-09-21T10:50:10.683753+0000","last_became_active":"2024-09-21T10:45:59.692582+0000","last_became_peered":"2024-09-21T10:45:59.692582+0000","last_unstale":"2024-09-21T10:50:10.683753+0000","last_undegraded":"2024-09-21T10:50:10.683753+0000","last_fullsized":"2024-09-21T10:50:10.683753+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:21.994503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2377,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681818+0000","last_change":"2024-09-21T10:45:59.697081+0000","last_active":"2024-09-21T10:50:10.681818+0000","last_peered":"2024-09-21T10:50:10.681818+0000","last_clean":"2024-09-21T10:50:10.681818+0000","last_became_active":"2024-09-21T10:45:59.696414+0000","last_became_peered":"2024-09-21T10:45:59.696414+0000","last_unstale":"2024-09-21T10:50:10.681818+0000","last_undegraded":"2024-09-21T10:50:10.681818+0000","last_fullsized":"2024-09-21T10:50:10.681818+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:47.528914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":214,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.692747+0000","last_change":"2024-09-21T10:48:08.150768+0000","last_active":"2024-09-21T10:50:10.692747+0000","last_peered":"2024-09-21T10:50:10.692747+0000","last_clean":"2024-09-21T10:50:10.692747+0000","last_became_active":"2024-09-21T10:48:08.150235+0000","last_became_peered":"2024-09-21T10:48:08.150235+0000","last_unstale":"2024-09-21T10:50:10.692747+0000","last_undegraded":"2024-09-21T10:50:10.692747+0000","last_fullsized":"2024-09-21T10:50:10.692747+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:58.640157+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2376,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.681385+0000","last_change":"2024-09-21T10:45:59.700672+0000","last_active":"2024-09-21T10:50:10.681385+0000","last_peered":"2024-09-21T10:50:10.681385+0000","last_clean":"2024-09-21T10:50:10.681385+0000","last_became_active":"2024-09-21T10:45:59.699332+0000","last_became_peered":"2024-09-21T10:45:59.699332+0000","last_unstale":"2024-09-21T10:50:10.681385+0000","last_undegraded":"2024-09-21T10:50:10.681385+0000","last_fullsized":"2024-09-21T10:50:10.681385+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:48.567053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":237,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671206+0000","last_change":"2024-09-21T10:48:08.155880+0000","last_active":"2024-09-21T10:50:10.671206+0000","last_peered":"2024-09-21T10:50:10.671206+0000","last_clean":"2024-09-21T10:50:10.671206+0000","last_became_active":"2024-09-21T10:48:08.154917+0000","last_became_peered":"2024-09-21T10:48:08.154917+0000","last_unstale":"2024-09-21T10:50:10.671206+0000","last_undegraded":"2024-09-21T10:50:10.671206+0000","last_fullsized":"2024-09-21T10:50:10.671206+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:37:14.509661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":258,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.683642+0000","last_change":"2024-09-21T10:48:08.167083+0000","last_active":"2024-09-21T10:50:10.683642+0000","last_peered":"2024-09-21T10:50:10.683642+0000","last_clean":"2024-09-21T10:50:10.683642+0000","last_became_active":"2024-09-21T10:48:08.166441+0000","last_became_peered":"2024-09-21T10:48:08.166441+0000","last_unstale":"2024-09-21T10:50:10.683642+0000","last_undegraded":"2024-09-21T10:50:10.683642+0000","last_fullsized":"2024-09-21T10:50:10.683642+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:53:57.413991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":129,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.671774+0000","last_change":"2024-09-21T10:45:59.714397+0000","last_active":"2024-09-21T10:50:10.671774+0000","last_peered":"2024-09-21T10:50:10.671774+0000","last_clean":"2024-09-21T10:50:10.671774+0000","last_became_active":"2024-09-21T10:45:59.689648+0000","last_became_peered":"2024-09-21T10:45:59.689648+0000","last_unstale":"2024-09-21T10:50:10.671774+0000","last_undegraded":"2024-09-21T10:50:10.671774+0000","last_fullsized":"2024-09-21T10:50:10.671774+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:20.596738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2266,"reported_epoch":1466,"state":"active+clean","last_fresh":"2024-09-21T10:50:10.691424+0000","last_change":"2024-09-21T10:45:59.696141+0000","last_active":"2024-09-21T10:50:10.691424+0000","last_peered":"2024-09-21T10:50:10.691424+0000","last_clean":"2024-09-21T10:50:10.691424+0000","last_became_active":"2024-09-21T10:45:59.695761+0000","last_became_peered":"2024-09-21T10:45:59.695761+0000","last_unstale":"2024-09-21T10:50:10.691424+0000","last_undegraded":"2024-09-21T10:50:10.691424+0000","last_fullsized":"2024-09-21T10:50:10.691424+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:46.714846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":50,"num_pg":16,"stat_sum":{"num_bytes":652148760,"num_objects":9952,"num_object_clones":0,"num_object_copies":29856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9952,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":199198,"num_write_kb":6692737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30612,"num_bytes_recovered":2006122520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2756063232,"data_stored":2756051016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76016,"ondisk_log_size":76016,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":6806,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":26316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":210453398170,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049829,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662408,"kb_used_data":446720,"kb_used_omap":242,"kb_used_meta":1215437,"kb_avail":92082744,"statfs":{"total":95995035648,"available":94292729856,"internally_reserved":0,"allocated":457441280,"data_stored":453139929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":1244607711},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":158913790628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644788,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1484792,"kb_used_data":370928,"kb_used_omap":67,"kb_used_meta":1113660,"kb_avail":92260360,"statfs":{"total":95995035648,"available":94474608640,"internally_reserved":0,"allocated":379830272,"data_stored":375539996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1140388701},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527806,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1453696,"kb_used_data":345656,"kb_used_omap":75,"kb_used_meta":1107828,"kb_avail":92291456,"statfs":{"total":95995035648,"available":94506450944,"internally_reserved":0,"allocated":353951744,"data_stored":349658458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":1134416026},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274958,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1430572,"kb_used_data":346020,"kb_used_omap":51,"kb_used_meta":1084492,"kb_avail":92314580,"statfs":{"total":95995035648,"available":94530129920,"internally_reserved":0,"allocated":354324480,"data_stored":350045783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1110520149},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354197,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1401804,"kb_used_data":497092,"kb_used_omap":66,"kb_used_meta":904637,"kb_avail":92343348,"statfs":{"total":95995035648,"available":94559588352,"internally_reserved":0,"allocated":509022208,"data_stored":504716275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":926348612},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,7,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":466882560,"data_stored":466878488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":334430208,"data_stored":334430208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":338366464,"data_stored":338362392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":364646400,"data_stored":364642328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":816775168,"data_stored":816775168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":434962432,"data_stored":434962432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:50:28.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:50:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:28.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:28.487 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:28 smithi145 ceph-mon[89359]: from='client.37410 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:50:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:28 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3042503561' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:28.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:28 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:28.589 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:28.596 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:28 smithi119 ceph-mon[101432]: from='client.37410 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:50:28.596 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:28 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3042503561' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:28.596 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:28 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:28.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:28 smithi119 ceph-mon[105288]: from='client.37410 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:50:28.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:28 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3042503561' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:28.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:28 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:50:28.610 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:28.712 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:28.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.755 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:28.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:28.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:28.892 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:28.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:28.994 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.019 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:29.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:29.121 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:29.143 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:29.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:29.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.282 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:29.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:29.383 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:29.405 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:29.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.529 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:29.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:29 smithi145 ceph-mon[89359]: pgmap v2683: 109 pgs: 109 active+clean; 554 MiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:50:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:29 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:29 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:50:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:29 smithi145 ceph-mon[89359]: osdmap e1467: 8 total, 8 up, 5 in 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[101432]: pgmap v2683: 109 pgs: 109 active+clean; 554 MiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[101432]: osdmap e1467: 8 total, 8 up, 5 in 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[105288]: pgmap v2683: 109 pgs: 109 active+clean; 554 MiB data, 7.0 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 828 op/s 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:50:29.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:29 smithi119 ceph-mon[105288]: osdmap e1467: 8 total, 8 up, 5 in 2024-09-21T10:50:29.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:29.651 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:29.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:29.793 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:29.896 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:29.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:29.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:30.049 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.072 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:30.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:30.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:30.196 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:30.297 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:30.321 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:30.423 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:30.446 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:30.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:30.495 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-21T10:50:30.495 INFO:teuthology.orchestra.run.smithi119.stdout: "size": 20, 2024-09-21T10:50:30.495 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 600, 2024-09-21T10:50:30.495 INFO:teuthology.orchestra.run.smithi119.stdout: "ops": [ 2024-09-21T10:50:30.495 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57587 49.e e1400/1394 49:79512dcc:::benchmark_data_smithi119_301770_object57586:head v 1400'3729, mlcod=1400'3723)", 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 453.11357389900002, 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29749656200000002, 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.496 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375441+0000", 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.3000000000000002e-07 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375459+0000", 2024-09-21T10:50:30.497 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8499999999999999e-05 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375466+0000", 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2179999999999996e-06 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.375470+0000", 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5229999999999997e-06 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.498 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378768+0000", 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.003297763 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378786+0000", 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8192000000000001e-05 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380891+0000", 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0021050470000000001 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.499 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672937+0000", 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29204598900000001 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57594 49.8 e1400/1394 49:127af930:::benchmark_data_smithi119_301770_object57593:head v 1400'3519, mlcod=1400'3517)", 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.378651+0000", 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 453.11036309600001, 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29406905999999999, 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.500 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378651+0000", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378651+0000", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378652+0000", 2024-09-21T10:50:30.501 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.0400000000000005e-07 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378674+0000", 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.1175999999999999e-05 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378685+0000", 2024-09-21T10:50:30.502 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1063000000000001e-05 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378691+0000", 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.618e-06 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378813+0000", 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000121687 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.378831+0000", 2024-09-21T10:50:30.503 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8059000000000001e-05 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380832+0000", 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0020010470000000002 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672721+0000", 2024-09-21T10:50:30.504 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29188850599999999 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57596 49.6 49:6ba8cc7d:::benchmark_data_smithi119_301770_object57595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.380340+0000", 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 453.108674361, 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30705738599999999, 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.505 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57596 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380340+0000", 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380340+0000", 2024-09-21T10:50:30.506 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380342+0000", 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.747e-06 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380365+0000", 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3220000000000001e-05 2024-09-21T10:50:30.507 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380379+0000", 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3472000000000001e-05 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380386+0000", 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1160000000000001e-06 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.508 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380444+0000", 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8150999999999999e-05 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380593+0000", 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014878600000000001 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:50:30.509 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380735+0000", 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00014229100000000001 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673023+0000", 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29228787099999998 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673778+0000", 2024-09-21T10:50:30.510 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00075495100000000002 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687296+0000", 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.013517972 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687352+0000", 2024-09-21T10:50:30.511 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6647000000000002e-05 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.687398+0000", 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5161999999999999e-05 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.512 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:57598 49.c 49:3fc93c6d:::benchmark_data_smithi119_301770_object57597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.380860+0000", 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 453.10815471500001, 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.303667727, 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:50:30.513 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 57598 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380860+0000", 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.514 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380860+0000", 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380861+0000", 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1600000000000001e-07 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380879+0000", 2024-09-21T10:50:30.515 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7966999999999999e-05 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380893+0000", 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4806e-05 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380903+0000", 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.4390000000000006e-06 2024-09-21T10:50:30.516 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673605+0000", 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29270185599999998 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673994+0000", 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00038972699999999998 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 0,6", 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.674177+0000", 2024-09-21T10:50:30.517 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018300900000000001 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.683774+0000", 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0095964210000000008 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684421+0000", 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00064724400000000005 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684431+0000", 2024-09-21T10:50:30.518 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0114e-05 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684481+0000", 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.0355000000000002e-05 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.684528+0000", 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6072999999999997e-05 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.519 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.36978.0:57597 49.0 e1400/1394 49:0fe7e42d:::benchmark_data_smithi119_301770_object57596:head v 1400'3700, mlcod=1400'3698)", 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:42:57.380894+0000", 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 453.10812070399999, 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29711208099999997, 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.520 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380894+0000", 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380894+0000", 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380895+0000", 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3900000000000004e-07 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.521 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.380965+0000", 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0871000000000006e-05 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672802+0000", 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29183669200000001 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.672811+0000", 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2969999999999992e-06 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.522 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673118+0000", 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000306726 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.673142+0000", 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.3787000000000001e-05 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.677964+0000", 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0048218999999999996 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.523 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:42:57.678006+0000", 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.2069e-05 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.36978.0:160771 49.6 49:62a3a975:::benchmark_data_smithi119_301770_object47284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1411)", 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:44:50.470348+0000", 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 340.018666462, 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25296919699999998, 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.524 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.36978", 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 160771 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.525 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470348+0000", 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470348+0000", 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470349+0000", 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.879999999999999e-07 2024-09-21T10:50:30.526 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470353+0000", 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.1150000000000004e-06 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470365+0000", 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1382e-05 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470372+0000", 2024-09-21T10:50:30.527 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9700000000000002e-06 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470556+0000", 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018472200000000001 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.470812+0000", 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00025620599999999998 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 1,5", 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.471249+0000", 2024-09-21T10:50:30.528 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00043646800000000002 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.474884+0000", 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036346249999999998 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.474994+0000", 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000110469 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.723218+0000", 2024-09-21T10:50:30.529 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.24822361300000001 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.723272+0000", 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3897000000000003e-05 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:44:50.723317+0000", 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5741999999999997e-05 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.530 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:76351 50.b e1440/1430 50:df787275:::benchmark_data_smithi119_335063_object76350:head v 1440'4821, mlcod=1440'4818)", 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:45.908201+0000", 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 164.58081304500001, 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25674179400000002, 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.908201+0000", 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.531 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.908201+0000", 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.908202+0000", 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.47e-07 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.908271+0000", 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9072000000000005e-05 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.532 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.908283+0000", 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.128e-05 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.908289+0000", 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.5230000000000001e-06 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.912591+0000", 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0043020319999999999 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.533 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.912612+0000", 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.0463e-05 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.164858+0000", 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25224667299999998 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.534 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.164943+0000", 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.4903999999999999e-05 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:76356 50.b e1440/1430 50:d87d21e4:::benchmark_data_smithi119_335063_object76355:head v 1440'4822, mlcod=1440'4818)", 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:45.919028+0000", 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 164.56998647399999, 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.252696166, 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.535 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919028+0000", 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919028+0000", 2024-09-21T10:50:30.536 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919029+0000", 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1200000000000001e-07 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919075+0000", 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.5731e-05 2024-09-21T10:50:30.537 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919085+0000", 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0693000000000001e-05 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919092+0000", 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.0659999999999998e-06 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.162637+0000", 2024-09-21T10:50:30.538 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.243544699 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.162666+0000", 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.8473999999999999e-05 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.171672+0000", 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0090066320000000005 2024-09-21T10:50:30.539 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.171724+0000", 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1959000000000002e-05 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:76355 50.2 e1440/1430 50:4e5baa05:::benchmark_data_smithi119_335063_object76354:head v 1440'4739, mlcod=1440'4737)", 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:47:45.919247+0000", 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 164.56976783900001, 2024-09-21T10:50:30.540 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25274052400000002, 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919247+0000", 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.541 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919247+0000", 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919247+0000", 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.8000000000000001e-07 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.542 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919265+0000", 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8368000000000001e-05 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919273+0000", 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.1740000000000003e-06 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919277+0000", 2024-09-21T10:50:30.543 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.8670000000000002e-06 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919410+0000", 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000132946 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:45.919427+0000", 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.6351999999999999e-05 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.544 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.171943+0000", 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.25251640800000003 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:47:46.171987+0000", 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4029000000000001e-05 2024-09-21T10:50:30.545 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:104601 50.4 50:285866f4:::benchmark_data_smithi119_335063_object24:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1445)", 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:48:23.336415+0000", 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 127.152599655, 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.38860185000000003, 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.546 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 104601 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.336415+0000", 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.547 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.336415+0000", 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.336416+0000", 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.4449999999999999e-06 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.548 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.336421+0000", 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.6369999999999998e-06 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.336432+0000", 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1168e-05 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.336437+0000", 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.1730000000000003e-06 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.549 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.337117+0000", 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00067945799999999997 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.337205+0000", 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.8535999999999994e-05 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,5,6", 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.337297+0000", 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.1207999999999998e-05 2024-09-21T10:50:30.550 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.339514+0000", 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0022170390000000001 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.339567+0000", 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.3430000000000002e-05 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.340209+0000", 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00064204400000000003 2024-09-21T10:50:30.551 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.724831+0000", 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.38462215999999999 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.724905+0000", 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.3577000000000004e-05 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:48:23.725017+0000", 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000111975 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.552 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:197399 50.a e1466/1430 50:5183e37a:::benchmark_data_smithi119_335063_object92822:head v 1466'12214, mlcod=1466'12209)", 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.377176+0000", 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1118388469999996, 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.301111077, 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.553 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.377176+0000", 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.377176+0000", 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.377177+0000", 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.64e-07 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.554 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.377183+0000", 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.494e-06 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.377195+0000", 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1721e-05 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.555 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.377202+0000", 2024-09-21T10:50:30.556 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:30.556 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8410000000000002e-06 2024-09-21T10:50:30.556 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.378347+0000", 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0011455320000000001 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.378367+0000", 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9585e-05 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678244+0000", 2024-09-21T10:50:30.557 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29987688200000001 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678287+0000", 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.3058e-05 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:197408 50.7 e1466/1461 50:e16d363f:::benchmark_data_smithi119_335063_object92831:head v 1466'12399, mlcod=1466'12397)", 2024-09-21T10:50:30.558 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.382665+0000", 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1063498210000002, 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29551317100000002, 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382665+0000", 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.559 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382665+0000", 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382666+0000", 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0780000000000001e-06 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382672+0000", 2024-09-21T10:50:30.560 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.3690000000000003e-06 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382684+0000", 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1929999999999999e-05 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382691+0000", 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6359999999999997e-06 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.561 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382990+0000", 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00029910599999999999 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.383008+0000", 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8338e-05 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.386705+0000", 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0036970789999999998 2024-09-21T10:50:30.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678178+0000", 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29147263499999998 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:197407 50.a e1466/1430 50:52911827:::benchmark_data_smithi119_335063_object92830:head v 1466'12215, mlcod=1466'12209)", 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.382714+0000", 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.106300075, 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29846853800000001, 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.563 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382714+0000", 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382714+0000", 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.564 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382715+0000", 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.91e-07 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382718+0000", 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.9110000000000002e-06 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382723+0000", 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.6949999999999997e-06 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.382729+0000", 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.22e-06 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.566 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.384396+0000", 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0016672270000000001 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.384415+0000", 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9483999999999998e-05 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.681094+0000", 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29667822100000002 2024-09-21T10:50:30.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.681183+0000", 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.9488999999999996e-05 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:197412 50.a e1466/1430 50:53af34e8:::benchmark_data_smithi119_335063_object92835:head v 1466'12216, mlcod=1466'12211)", 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.385432+0000", 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1035820510000001, 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29805094100000001, 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.569 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.385432+0000", 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.385432+0000", 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.385433+0000", 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 9.2600000000000001e-07 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.385440+0000", 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1739999999999997e-06 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.385450+0000", 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.06e-05 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.571 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.385458+0000", 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6199999999999999e-06 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678439+0000", 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29298106200000001 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678458+0000", 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9021e-05 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.572 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.683452+0000", 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0049936299999999998 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.683483+0000", 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.1908e-05 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:197414 50.3 e1466/1430 50:ce09486a:::benchmark_data_smithi119_335063_object92837:head v 1466'12682, mlcod=1466'12680)", 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.388131+0000", 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1008839459999997, 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.295546894, 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.574 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388131+0000", 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388131+0000", 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.576 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.576 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388131+0000", 2024-09-21T10:50:30.576 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 7.6000000000000003e-07 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388136+0000", 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9450000000000001e-06 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.578 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.578 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678170+0000", 2024-09-21T10:50:30.578 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29003330100000002 2024-09-21T10:50:30.578 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.578 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.579 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.579 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678178+0000", 2024-09-21T10:50:30.579 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.2810000000000008e-06 2024-09-21T10:50:30.579 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.579 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.580 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.580 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678286+0000", 2024-09-21T10:50:30.580 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000108317 2024-09-21T10:50:30.580 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.580 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.581 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.581 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678305+0000", 2024-09-21T10:50:30.581 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9298000000000001e-05 2024-09-21T10:50:30.581 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.581 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.582 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.683647+0000", 2024-09-21T10:50:30.582 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0053413899999999997 2024-09-21T10:50:30.582 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.582 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.582 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.683677+0000", 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.0602000000000003e-05 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_op(client.37269.0:197416 50.4 50:24c2e034:::benchmark_data_smithi119_335063_object92839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1466)", 2024-09-21T10:50:30.583 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.388299+0000", 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.1007151759999996, 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.30988947700000002, 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "client_info": { 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "client": "client.37269", 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "client_addr": "172.21.15.119:0/2191711833", 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "tid": 197416 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.584 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388299+0000", 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388299+0000", 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388301+0000", 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.61e-06 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.585 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.388307+0000", 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 5.8830000000000003e-06 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678148+0000", 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.28984151699999999 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678164+0000", 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.5823000000000001e-05 2024-09-21T10:50:30.586 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678228+0000", 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.4109999999999994e-05 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678368+0000", 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000139352 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "waiting for subops from 2,6", 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678493+0000", 2024-09-21T10:50:30.587 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00012568200000000001 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.689569+0000", 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.011075481999999999 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.689630+0000", 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1465999999999997e-05 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "op_commit", 2024-09-21T10:50:30.588 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.698095+0000", 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0084648550000000003 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.698145+0000", 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.9573999999999997e-05 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.698189+0000", 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 4.4122999999999998e-05 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.589 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:197419 50.d e1466/1452 50:b4d06f8e:::benchmark_data_smithi119_335063_object92842:head v 1466'12269, mlcod=1466'12267)", 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:21.389080+0000", 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 9.0999348300000005, 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.29642460700000001, 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.590 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.389080+0000", 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.389080+0000", 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.389081+0000", 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.0800000000000004e-07 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.389087+0000", 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.1920000000000002e-06 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678150+0000", 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.28906322899999998 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678163+0000", 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.3329000000000001e-05 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.592 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678226+0000", 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.2949999999999999e-05 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.678251+0000", 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.4811e-05 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.685464+0000", 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0072133539999999999 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.593 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:21.685504+0000", 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 3.9934000000000001e-05 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:204212 50.a e1467/1430 50:5bc650e4:::benchmark_data_smithi119_335063_object99635:head v 1467'12639, mlcod=1467'12637)", 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:29.441395+0000", 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.047619297, 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.124069838, 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.594 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441395+0000", 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441395+0000", 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.595 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441396+0000", 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1739999999999999e-06 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441403+0000", 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.6460000000000004e-06 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441414+0000", 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.134e-05 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.596 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441421+0000", 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.8129999999999998e-06 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441594+0000", 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00017281299999999999 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.597 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441616+0000", 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 2.2121999999999999e-05 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.448263+0000", 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.0066472550000000004 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565465+0000", 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.11720167500000001 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.598 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:204204 50.3 e1467/1430 50:c0f8e431:::benchmark_data_smithi119_335063_object99627:head v 1467'13100, mlcod=1467'13096)", 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:29.441465+0000", 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.047549906, 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.12408382, 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441465+0000", 2024-09-21T10:50:30.599 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441465+0000", 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441465+0000", 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 8.3699999999999999e-07 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441472+0000", 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.7730000000000002e-06 2024-09-21T10:50:30.600 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441483+0000", 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.1202e-05 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441494+0000", 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.0356e-05 2024-09-21T10:50:30.601 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441650+0000", 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00015636799999999999 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.441668+0000", 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8122e-05 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565441+0000", 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.123772403 2024-09-21T10:50:30.602 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565548+0000", 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.000107759 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "description": "osd_repop(client.37269.0:204219 50.a e1467/1430 50:5a0ffc46:::benchmark_data_smithi119_335063_object99642:head v 1467'12641, mlcod=1467'12637)", 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "initiated_at": "2024-09-21T10:50:29.450232+0000", 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "age": 1.0387820640000001, 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.12594786599999999, 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "continuous": false, 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "type_data": { 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "events": [ 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "initiated", 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.450232+0000", 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "header_read", 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.450232+0000", 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "throttled", 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.450234+0000", 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.9199999999999998e-06 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "all_read", 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.450241+0000", 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 6.9840000000000004e-06 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "dispatched", 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565392+0000", 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.11515057099999999 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "queued_for_pg", 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565409+0000", 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.7091e-05 2024-09-21T10:50:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "reached_pg", 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565573+0000", 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00016422299999999999 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "started", 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.565592+0000", 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 1.8861000000000001e-05 2024-09-21T10:50:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "commit_sent", 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.575993+0000", 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.010400603 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: "event": "done", 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: "time": "2024-09-21T10:50:29.576180+0000", 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: "duration": 0.00018761299999999999 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-21T10:50:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-21T10:50:30.609 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-21T10:50:30.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:30.610 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:50:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:50:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:50:30.655 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:30.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:30.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.794 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:30.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:30.895 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:30.939 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:30.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:31.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.077 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:31.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:31.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:31.226 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:31.328 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:31.351 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:31.407 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:31 smithi119 ceph-mon[101432]: pgmap v2685: 109 pgs: 109 active+clean; 434 MiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:50:31.407 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:31 smithi119 ceph-mon[105288]: pgmap v2685: 109 pgs: 109 active+clean; 434 MiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:50:31.453 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:31.489 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:31 smithi145 ceph-mon[89359]: pgmap v2685: 109 pgs: 109 active+clean; 434 MiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:50:31.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:31.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:31.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:31.738 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:31.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:31.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:31.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:31.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:32.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:32.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T10:50:32.044 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:50:31 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a[76972]: logger=cleanup t=2024-09-21T10:50:31.650431434Z level=info msg="Completed cleanup jobs" duration=12.520065ms 2024-09-21T10:50:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.049 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:32.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:32.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:32.200 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:32.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:32.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:32.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:32.459 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:32.482 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:32.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:32.605 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:32.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:32.743 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:32.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T10:50:32.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:32.896 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:32.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:32.997 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:33.036 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:33.109 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:33.138 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:33.155 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:33 smithi145 ceph-mon[89359]: pgmap v2686: 109 pgs: 109 active+clean; 283 MiB data, 6.2 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 860 op/s 2024-09-21T10:50:33.155 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:33.155 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:33.160 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:33.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:33.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:33.285 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:33.387 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:33.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:33 smithi119 ceph-mon[105288]: pgmap v2686: 109 pgs: 109 active+clean; 283 MiB data, 6.2 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 860 op/s 2024-09-21T10:50:33.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:33.418 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:33.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:33 smithi119 ceph-mon[101432]: pgmap v2686: 109 pgs: 109 active+clean; 283 MiB data, 6.2 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 860 op/s 2024-09-21T10:50:33.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:50:33.419 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:33.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:33.546 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:33.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:33.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:33.671 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:33.692 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:33.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:33.832 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:33.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:33.977 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:33.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:34.079 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:34.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:34.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:34.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:34.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:34.399 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:34.501 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:34.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.526 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:34.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:34.533 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:50:34.537 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:50:34.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:34.650 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:34.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:34.751 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:50:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:50:34] "GET /metrics HTTP/1.1" 200 36826 "" "Prometheus/2.51.0" 2024-09-21T10:50:34.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.776 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:34.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:34.877 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:34.898 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:34.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:34.942 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Removed 104574 objects 2024-09-21T10:50:34.942 INFO:tasks.radosbench.radosbench.0.smithi119.stdout:Clean up completed and total clean up time :131.627 2024-09-21T10:50:34.999 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:35.020 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:35.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:35.122 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.145 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:35.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:35.216 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:50:35.216 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:50:35.217 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:50:35.217 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:50:35.217 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:35 smithi145 ceph-mon[89359]: pgmap v2687: 109 pgs: 109 active+clean; 177 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 870 op/s 2024-09-21T10:50:35.217 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:35 smithi145 ceph-mon[89359]: from='client.47327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:50:35.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:35.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:35.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:35.393 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:35.494 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:35.518 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:35 smithi119 ceph-mon[105288]: pgmap v2687: 109 pgs: 109 active+clean; 177 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 870 op/s 2024-09-21T10:50:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:35 smithi119 ceph-mon[105288]: from='client.47327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:50:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:35 smithi119 ceph-mon[101432]: pgmap v2687: 109 pgs: 109 active+clean; 177 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 870 op/s 2024-09-21T10:50:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:35 smithi119 ceph-mon[101432]: from='client.47327 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:50:35.620 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:35.642 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:35.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:35.769 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:35.836 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T10:50:35.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:35.900 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:35.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:36.001 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:36.028 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:36.082 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:36.131 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.153 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:36.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:36.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:36.280 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:36.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:36.381 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:36.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:36.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:36.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:36.531 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:36.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:36.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:36.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.654 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:36.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:36.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:36.777 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:36.879 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:36.899 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:36.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:36.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:36.926 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:37.001 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:37.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:37.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:37.151 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:37.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:37.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:37.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:37 smithi145 ceph-mon[89359]: pgmap v2688: 109 pgs: 109 active+clean; 79 MiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:50:37.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:37.407 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:37.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:37.509 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.531 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:37.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:37.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:37 smithi119 ceph-mon[105288]: pgmap v2688: 109 pgs: 109 active+clean; 79 MiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:50:37.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:37 smithi119 ceph-mon[101432]: pgmap v2688: 109 pgs: 109 active+clean; 79 MiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 869 op/s 2024-09-21T10:50:37.631 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:37.657 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:37.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:37.784 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:37.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:37.806 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:50:37.806 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:50:37.806 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:50:37.806 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:50:37.806 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:50:37.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:37.909 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:37.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:37.981 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-21T10:50:37.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-21T10:50:38.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:38.032 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:38.134 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:38.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:38.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.278 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:38.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:38.379 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:38.401 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:38.492 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T10:50:38.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:38.524 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:38.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:38.647 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:38.738 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:38.748 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:38.770 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:38.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:38.894 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:38.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:38.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:39.022 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:39.123 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:39.145 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:39.211 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:50:39.212 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:39.228 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:50:39.229 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:39.244 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:50:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:39.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:39.261 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:50:39.261 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:39.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:39.277 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:50:39.277 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:39.293 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:50:39.293 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:39.308 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:50:39.308 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:39.324 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:50:39.324 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:39.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:39 smithi119 ceph-mon[105288]: pgmap v2689: 109 pgs: 109 active+clean; 34 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 831 op/s 2024-09-21T10:50:39.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:39 smithi119 ceph-mon[101432]: pgmap v2689: 109 pgs: 109 active+clean; 34 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 831 op/s 2024-09-21T10:50:39.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:39.392 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:39.494 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:39.516 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:39 smithi145 ceph-mon[89359]: pgmap v2689: 109 pgs: 109 active+clean; 34 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 831 op/s 2024-09-21T10:50:39.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:39.639 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:39.741 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:39.763 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:39.865 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:39.886 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:39.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:39.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:39.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:40.010 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:40.112 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:40.235 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:40.257 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:40.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:40.380 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:40.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:40.457 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2493751340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:40.457 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:50:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:50:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:50:40.461 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:40.462 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:40.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:40.502 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:40.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:40.616 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2493751340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:40.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2493751340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:40.625 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:40.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:40.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:40.748 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:40.850 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:40.872 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:40.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:40.995 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:40.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:41.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:41.099 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T10:50:41.117 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:41.219 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:41.241 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:41.343 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:41.343 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:41.365 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:41.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:41 smithi119 ceph-mon[105288]: pgmap v2690: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:50:41.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:41 smithi119 ceph-mon[101432]: pgmap v2690: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:50:41.464 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:41 smithi145 ceph-mon[89359]: pgmap v2690: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 743 op/s 2024-09-21T10:50:41.466 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:41.487 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:41.588 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:41.609 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:41.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:41.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:41.774 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2687,"stamp":"2024-09-21T10:50:34.079130+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":185320080,"num_objects":2849,"num_object_clones":0,"num_object_copies":8547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2849,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206390,"num_write_kb":6692768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30672,"num_bytes_recovered":2007047412,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78103,"ondisk_log_size":78103,"up":327,"acting":327,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":327,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":6391184,"kb_used_data":687392,"kb_used_omap":504,"kb_used_meta":5702983,"kb_avail":462334576,"statfs":{"total":479975178240,"available":473430605824,"internally_reserved":0,"allocated":703889408,"data_stored":682383110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516241,"internal_metadata":5839855471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,0,4,7,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-570490880,"num_objects":-8705,"num_object_clones":0,"num_object_copies":-26115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8705,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003217"},"pg_stats":[{"pgid":"50.f","version":"1467'12720","reported_seq":12805,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:31.638452+0000","last_change":"2024-09-21T10:46:40.275318+0000","last_active":"2024-09-21T10:50:31.638452+0000","last_peered":"2024-09-21T10:50:31.638452+0000","last_clean":"2024-09-21T10:50:31.638452+0000","last_became_active":"2024-09-21T10:46:40.275064+0000","last_became_peered":"2024-09-21T10:46:40.275064+0000","last_unstale":"2024-09-21T10:50:31.638452+0000","last_undegraded":"2024-09-21T10:50:31.638452+0000","last_fullsized":"2024-09-21T10:50:31.638452+0000","mapping_epoch":1430,"log_start":"1448'8150","ondisk_log_start":"1448'8150","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:25.995350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12189696,"num_objects":186,"num_object_clones":0,"num_object_copies":558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12720,"num_write_kb":412992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.e","version":"1467'12801","reported_seq":12886,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:30.529830+0000","last_change":"2024-09-21T10:46:40.276274+0000","last_active":"2024-09-21T10:50:30.529830+0000","last_peered":"2024-09-21T10:50:30.529830+0000","last_clean":"2024-09-21T10:50:30.529830+0000","last_became_active":"2024-09-21T10:46:40.276005+0000","last_became_peered":"2024-09-21T10:46:40.276005+0000","last_unstale":"2024-09-21T10:50:30.529830+0000","last_undegraded":"2024-09-21T10:50:30.529830+0000","last_fullsized":"2024-09-21T10:50:30.529830+0000","mapping_epoch":1430,"log_start":"1448'7807","ondisk_log_start":"1448'7807","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:42.051377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12801,"num_write_kb":417792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.d","version":"1467'12862","reported_seq":23547,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:33.338007+0000","last_change":"2024-09-21T10:49:16.172243+0000","last_active":"2024-09-21T10:50:33.338007+0000","last_peered":"2024-09-21T10:50:33.338007+0000","last_clean":"2024-09-21T10:50:33.338007+0000","last_became_active":"2024-09-21T10:49:16.154591+0000","last_became_peered":"2024-09-21T10:49:16.154591+0000","last_unstale":"2024-09-21T10:50:33.338007+0000","last_undegraded":"2024-09-21T10:50:33.338007+0000","last_fullsized":"2024-09-21T10:50:33.338007+0000","mapping_epoch":1452,"log_start":"1448'8093","ondisk_log_start":"1448'8093","created":1430,"last_epoch_clean":1453,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4769,"log_dups_size":0,"ondisk_log_size":4769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:01.553893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4915224,"num_objects":76,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12862,"num_write_kb":413953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347144216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.c","version":"1467'13083","reported_seq":13169,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:33.370647+0000","last_change":"2024-09-21T10:46:40.274270+0000","last_active":"2024-09-21T10:50:33.370647+0000","last_peered":"2024-09-21T10:50:33.370647+0000","last_clean":"2024-09-21T10:50:33.370647+0000","last_became_active":"2024-09-21T10:46:40.273989+0000","last_became_peered":"2024-09-21T10:46:40.273989+0000","last_unstale":"2024-09-21T10:50:33.370647+0000","last_undegraded":"2024-09-21T10:50:33.370647+0000","last_fullsized":"2024-09-21T10:50:33.370647+0000","mapping_epoch":1430,"log_start":"1448'8333","ondisk_log_start":"1448'8333","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4750,"log_dups_size":0,"ondisk_log_size":4750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:56.938584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13083,"num_write_kb":421440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.b","version":"1467'13036","reported_seq":24520,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:31.635130+0000","last_change":"2024-09-21T10:49:06.725075+0000","last_active":"2024-09-21T10:50:31.635130+0000","last_peered":"2024-09-21T10:50:31.635130+0000","last_clean":"2024-09-21T10:50:31.635130+0000","last_became_active":"2024-09-21T10:49:06.705032+0000","last_became_peered":"2024-09-21T10:49:06.705032+0000","last_unstale":"2024-09-21T10:50:31.635130+0000","last_undegraded":"2024-09-21T10:50:31.635130+0000","last_fullsized":"2024-09-21T10:50:31.635130+0000","mapping_epoch":1449,"log_start":"1448'8484","ondisk_log_start":"1448'8484","created":1430,"last_epoch_clean":1450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4552,"log_dups_size":0,"ondisk_log_size":4552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:59.569378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058624,"num_objects":184,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13036,"num_write_kb":423040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5698,"num_bytes_recovered":373424128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.a","version":"1467'12867","reported_seq":12953,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:33.337412+0000","last_change":"2024-09-21T10:46:40.276509+0000","last_active":"2024-09-21T10:50:33.337412+0000","last_peered":"2024-09-21T10:50:33.337412+0000","last_clean":"2024-09-21T10:50:33.337412+0000","last_became_active":"2024-09-21T10:46:40.276211+0000","last_became_peered":"2024-09-21T10:46:40.276211+0000","last_unstale":"2024-09-21T10:50:33.337412+0000","last_undegraded":"2024-09-21T10:50:33.337412+0000","last_fullsized":"2024-09-21T10:50:33.337412+0000","mapping_epoch":1430,"log_start":"1448'8124","ondisk_log_start":"1448'8124","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4743,"log_dups_size":0,"ondisk_log_size":4743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:13:56.745364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5570560,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12867,"num_write_kb":414464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.9","version":"1467'13191","reported_seq":13277,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:33.351825+0000","last_change":"2024-09-21T10:46:40.276505+0000","last_active":"2024-09-21T10:50:33.351825+0000","last_peered":"2024-09-21T10:50:33.351825+0000","last_clean":"2024-09-21T10:50:33.351825+0000","last_became_active":"2024-09-21T10:46:40.275398+0000","last_became_peered":"2024-09-21T10:46:40.275398+0000","last_unstale":"2024-09-21T10:50:33.351825+0000","last_undegraded":"2024-09-21T10:50:33.351825+0000","last_fullsized":"2024-09-21T10:50:33.351825+0000","mapping_epoch":1430,"log_start":"1448'8476","ondisk_log_start":"1448'8476","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:47.276665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13191,"num_write_kb":425216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.8","version":"1467'12819","reported_seq":12905,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:30.551441+0000","last_change":"2024-09-21T10:46:40.274571+0000","last_active":"2024-09-21T10:50:30.551441+0000","last_peered":"2024-09-21T10:50:30.551441+0000","last_clean":"2024-09-21T10:50:30.551441+0000","last_became_active":"2024-09-21T10:46:40.274285+0000","last_became_peered":"2024-09-21T10:46:40.274285+0000","last_unstale":"2024-09-21T10:50:30.551441+0000","last_undegraded":"2024-09-21T10:50:30.551441+0000","last_fullsized":"2024-09-21T10:50:30.551441+0000","mapping_epoch":1430,"log_start":"1448'7803","ondisk_log_start":"1448'7803","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":5016,"log_dups_size":0,"ondisk_log_size":5016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:51:55.260163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16580608,"num_objects":253,"num_object_clones":0,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12819,"num_write_kb":418304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.7","version":"1467'13042","reported_seq":19685,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:33.369263+0000","last_change":"2024-09-21T10:49:49.217806+0000","last_active":"2024-09-21T10:50:33.369263+0000","last_peered":"2024-09-21T10:50:33.369263+0000","last_clean":"2024-09-21T10:50:33.369263+0000","last_became_active":"2024-09-21T10:49:49.193506+0000","last_became_peered":"2024-09-21T10:49:49.193506+0000","last_unstale":"2024-09-21T10:50:33.369263+0000","last_undegraded":"2024-09-21T10:50:33.369263+0000","last_fullsized":"2024-09-21T10:50:33.369263+0000","mapping_epoch":1461,"log_start":"1448'8330","ondisk_log_start":"1448'8330","created":1430,"last_epoch_clean":1462,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:12:24.255016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13042,"num_write_kb":420288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3268,"num_bytes_recovered":214171648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.6","version":"1467'12531","reported_seq":12616,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:30.512449+0000","last_change":"2024-09-21T10:46:40.275529+0000","last_active":"2024-09-21T10:50:30.512449+0000","last_peered":"2024-09-21T10:50:30.512449+0000","last_clean":"2024-09-21T10:50:30.512449+0000","last_became_active":"2024-09-21T10:46:40.275255+0000","last_became_peered":"2024-09-21T10:46:40.275255+0000","last_unstale":"2024-09-21T10:50:30.512449+0000","last_undegraded":"2024-09-21T10:50:30.512449+0000","last_fullsized":"2024-09-21T10:50:30.512449+0000","mapping_epoch":1430,"log_start":"1447'7522","ondisk_log_start":"1447'7522","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:34:08.150110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12531,"num_write_kb":408384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.5","version":"1467'12806","reported_seq":19525,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:31.614721+0000","last_change":"2024-09-21T10:49:48.213220+0000","last_active":"2024-09-21T10:50:31.614721+0000","last_peered":"2024-09-21T10:50:31.614721+0000","last_clean":"2024-09-21T10:50:31.614721+0000","last_became_active":"2024-09-21T10:49:48.190551+0000","last_became_peered":"2024-09-21T10:49:48.190551+0000","last_unstale":"2024-09-21T10:50:31.614721+0000","last_undegraded":"2024-09-21T10:50:31.614721+0000","last_fullsized":"2024-09-21T10:50:31.614721+0000","mapping_epoch":1460,"log_start":"1448'8259","ondisk_log_start":"1448'8259","created":1430,"last_epoch_clean":1461,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:41:12.288633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12806,"num_write_kb":415808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3314,"num_bytes_recovered":217186304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.4","version":"1467'12735","reported_seq":24989,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:29.671901+0000","last_change":"2024-09-21T10:48:47.619926+0000","last_active":"2024-09-21T10:50:29.671901+0000","last_peered":"2024-09-21T10:50:29.671901+0000","last_clean":"2024-09-21T10:50:29.671901+0000","last_became_active":"2024-09-21T10:48:47.605894+0000","last_became_peered":"2024-09-21T10:48:47.605894+0000","last_unstale":"2024-09-21T10:50:29.671901+0000","last_undegraded":"2024-09-21T10:50:29.671901+0000","last_fullsized":"2024-09-21T10:50:29.671901+0000","mapping_epoch":1446,"log_start":"1448'8283","ondisk_log_start":"1448'8283","created":1430,"last_epoch_clean":1447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:23:46.710060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12735,"num_write_kb":417216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6084,"num_bytes_recovered":398721024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"50.3","version":"1467'13164","reported_seq":13250,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:30.519695+0000","last_change":"2024-09-21T10:46:40.275398+0000","last_active":"2024-09-21T10:50:30.519695+0000","last_peered":"2024-09-21T10:50:30.519695+0000","last_clean":"2024-09-21T10:50:30.519695+0000","last_became_active":"2024-09-21T10:46:40.275096+0000","last_became_peered":"2024-09-21T10:46:40.275096+0000","last_unstale":"2024-09-21T10:50:30.519695+0000","last_undegraded":"2024-09-21T10:50:30.519695+0000","last_fullsized":"2024-09-21T10:50:30.519695+0000","mapping_epoch":1430,"log_start":"1448'8236","ondisk_log_start":"1448'8236","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4928,"log_dups_size":0,"ondisk_log_size":4928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:58.484291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13164,"num_write_kb":429760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"50.2","version":"1467'12747","reported_seq":17432,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:31.665129+0000","last_change":"2024-09-21T10:49:58.269301+0000","last_active":"2024-09-21T10:50:31.665129+0000","last_peered":"2024-09-21T10:50:31.665129+0000","last_clean":"2024-09-21T10:50:31.665129+0000","last_became_active":"2024-09-21T10:49:58.247816+0000","last_became_peered":"2024-09-21T10:49:58.247816+0000","last_unstale":"2024-09-21T10:50:31.665129+0000","last_undegraded":"2024-09-21T10:50:31.665129+0000","last_fullsized":"2024-09-21T10:50:31.665129+0000","mapping_epoch":1463,"log_start":"1448'8240","ondisk_log_start":"1448'8240","created":1430,"last_epoch_clean":1464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:58.250248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12747,"num_write_kb":413184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":150208512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"50.1","version":"1467'13067","reported_seq":13153,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:33.324765+0000","last_change":"2024-09-21T10:46:40.276794+0000","last_active":"2024-09-21T10:50:33.324765+0000","last_peered":"2024-09-21T10:50:33.324765+0000","last_clean":"2024-09-21T10:50:33.324765+0000","last_became_active":"2024-09-21T10:46:40.276137+0000","last_became_peered":"2024-09-21T10:46:40.276137+0000","last_unstale":"2024-09-21T10:50:33.324765+0000","last_undegraded":"2024-09-21T10:50:33.324765+0000","last_fullsized":"2024-09-21T10:50:33.324765+0000","mapping_epoch":1430,"log_start":"1448'8363","ondisk_log_start":"1448'8363","created":1430,"last_epoch_clean":1431,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:10:01.998423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13067,"num_write_kb":420672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"50.0","version":"1467'12856","reported_seq":22261,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:30.553581+0000","last_change":"2024-09-21T10:49:17.180145+0000","last_active":"2024-09-21T10:50:30.553581+0000","last_peered":"2024-09-21T10:50:30.553581+0000","last_clean":"2024-09-21T10:50:30.553581+0000","last_became_active":"2024-09-21T10:49:17.157202+0000","last_became_peered":"2024-09-21T10:49:17.157202+0000","last_unstale":"2024-09-21T10:50:30.553581+0000","last_undegraded":"2024-09-21T10:50:30.553581+0000","last_fullsized":"2024-09-21T10:50:30.553581+0000","mapping_epoch":1453,"log_start":"1448'7879","ondisk_log_start":"1448'7879","created":1430,"last_epoch_clean":1454,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T10:46:39.225242+0000","last_clean_scrub_stamp":"2024-09-21T10:46:39.225242+0000","objects_scrubbed":0,"log_size":4977,"log_dups_size":0,"ondisk_log_size":4977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:54:17.833068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18087936,"num_objects":276,"num_object_clones":0,"num_object_copies":828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":420224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4658,"num_bytes_recovered":305266688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":236,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.196680+0000","last_change":"2024-09-21T10:48:08.207623+0000","last_active":"2024-09-21T10:50:28.196680+0000","last_peered":"2024-09-21T10:50:28.196680+0000","last_clean":"2024-09-21T10:50:28.196680+0000","last_became_active":"2024-09-21T10:48:08.152668+0000","last_became_peered":"2024-09-21T10:48:08.152668+0000","last_unstale":"2024-09-21T10:50:28.196680+0000","last_undegraded":"2024-09-21T10:50:28.196680+0000","last_fullsized":"2024-09-21T10:50:28.196680+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:46.388407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1347,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.204800+0000","last_change":"2024-09-21T10:43:56.011543+0000","last_active":"2024-09-21T10:50:28.204800+0000","last_peered":"2024-09-21T10:50:28.204800+0000","last_clean":"2024-09-21T10:50:28.204800+0000","last_became_active":"2024-09-21T10:43:56.010878+0000","last_became_peered":"2024-09-21T10:43:56.010878+0000","last_unstale":"2024-09-21T10:50:28.204800+0000","last_undegraded":"2024-09-21T10:50:28.204800+0000","last_fullsized":"2024-09-21T10:50:28.204800+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:36:46.231617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2260,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.194089+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:50:28.194089+0000","last_peered":"2024-09-21T10:50:28.194089+0000","last_clean":"2024-09-21T10:50:28.194089+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:50:28.194089+0000","last_undegraded":"2024-09-21T10:50:28.194089+0000","last_fullsized":"2024-09-21T10:50:28.194089+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2278,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191297+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:50:28.191297+0000","last_peered":"2024-09-21T10:50:28.191297+0000","last_clean":"2024-09-21T10:50:28.191297+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:50:28.191297+0000","last_undegraded":"2024-09-21T10:50:28.191297+0000","last_fullsized":"2024-09-21T10:50:28.191297+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":149,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.185972+0000","last_change":"2024-09-21T10:48:08.200826+0000","last_active":"2024-09-21T10:50:28.185972+0000","last_peered":"2024-09-21T10:50:28.185972+0000","last_clean":"2024-09-21T10:50:28.185972+0000","last_became_active":"2024-09-21T10:48:08.165760+0000","last_became_peered":"2024-09-21T10:48:08.165760+0000","last_unstale":"2024-09-21T10:50:28.185972+0000","last_undegraded":"2024-09-21T10:50:28.185972+0000","last_fullsized":"2024-09-21T10:50:28.185972+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:35.607267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1461,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198504+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:50:28.198504+0000","last_peered":"2024-09-21T10:50:28.198504+0000","last_clean":"2024-09-21T10:50:28.198504+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:50:28.198504+0000","last_undegraded":"2024-09-21T10:50:28.198504+0000","last_fullsized":"2024-09-21T10:50:28.198504+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":98,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.188844+0000","last_change":"2024-09-21T10:48:08.196786+0000","last_active":"2024-09-21T10:50:28.188844+0000","last_peered":"2024-09-21T10:50:28.188844+0000","last_clean":"2024-09-21T10:50:28.188844+0000","last_became_active":"2024-09-21T10:48:08.161165+0000","last_became_peered":"2024-09-21T10:48:08.161165+0000","last_unstale":"2024-09-21T10:50:28.188844+0000","last_undegraded":"2024-09-21T10:50:28.188844+0000","last_fullsized":"2024-09-21T10:50:28.188844+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:35:51.828935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1555,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198238+0000","last_change":"2024-09-21T10:48:08.171396+0000","last_active":"2024-09-21T10:50:28.198238+0000","last_peered":"2024-09-21T10:50:28.198238+0000","last_clean":"2024-09-21T10:50:28.198238+0000","last_became_active":"2024-09-21T10:48:08.165828+0000","last_became_peered":"2024-09-21T10:48:08.165828+0000","last_unstale":"2024-09-21T10:50:28.198238+0000","last_undegraded":"2024-09-21T10:50:28.198238+0000","last_fullsized":"2024-09-21T10:50:28.198238+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:17.505484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":87,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198149+0000","last_change":"2024-09-21T10:46:39.241906+0000","last_active":"2024-09-21T10:50:28.198149+0000","last_peered":"2024-09-21T10:50:28.198149+0000","last_clean":"2024-09-21T10:50:28.198149+0000","last_became_active":"2024-09-21T10:46:39.241499+0000","last_became_peered":"2024-09-21T10:46:39.241499+0000","last_unstale":"2024-09-21T10:50:28.198149+0000","last_undegraded":"2024-09-21T10:50:28.198149+0000","last_fullsized":"2024-09-21T10:50:28.198149+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:53.671658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":245,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199494+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:50:28.199494+0000","last_peered":"2024-09-21T10:50:28.199494+0000","last_clean":"2024-09-21T10:50:28.199494+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:50:28.199494+0000","last_undegraded":"2024-09-21T10:50:28.199494+0000","last_fullsized":"2024-09-21T10:50:28.199494+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":320,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189245+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:50:28.189245+0000","last_peered":"2024-09-21T10:50:28.189245+0000","last_clean":"2024-09-21T10:50:28.189245+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:50:28.189245+0000","last_undegraded":"2024-09-21T10:50:28.189245+0000","last_fullsized":"2024-09-21T10:50:28.189245+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2267,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192307+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:50:28.192307+0000","last_peered":"2024-09-21T10:50:28.192307+0000","last_clean":"2024-09-21T10:50:28.192307+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:50:28.192307+0000","last_undegraded":"2024-09-21T10:50:28.192307+0000","last_fullsized":"2024-09-21T10:50:28.192307+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":87,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.194986+0000","last_change":"2024-09-21T10:46:39.241669+0000","last_active":"2024-09-21T10:50:28.194986+0000","last_peered":"2024-09-21T10:50:28.194986+0000","last_clean":"2024-09-21T10:50:28.194986+0000","last_became_active":"2024-09-21T10:46:39.241358+0000","last_became_peered":"2024-09-21T10:46:39.241358+0000","last_unstale":"2024-09-21T10:50:28.194986+0000","last_undegraded":"2024-09-21T10:50:28.194986+0000","last_fullsized":"2024-09-21T10:50:28.194986+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:25.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2383,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.197553+0000","last_change":"2024-09-21T10:45:59.689522+0000","last_active":"2024-09-21T10:50:28.197553+0000","last_peered":"2024-09-21T10:50:28.197553+0000","last_clean":"2024-09-21T10:50:28.197553+0000","last_became_active":"2024-09-21T10:45:59.689148+0000","last_became_peered":"2024-09-21T10:45:59.689148+0000","last_unstale":"2024-09-21T10:50:28.197553+0000","last_undegraded":"2024-09-21T10:50:28.197553+0000","last_fullsized":"2024-09-21T10:50:28.197553+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:28.142643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2293,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191343+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:50:28.191343+0000","last_peered":"2024-09-21T10:50:28.191343+0000","last_clean":"2024-09-21T10:50:28.191343+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:50:28.191343+0000","last_undegraded":"2024-09-21T10:50:28.191343+0000","last_fullsized":"2024-09-21T10:50:28.191343+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1275,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189918+0000","last_change":"2024-09-21T10:48:08.161215+0000","last_active":"2024-09-21T10:50:28.189918+0000","last_peered":"2024-09-21T10:50:28.189918+0000","last_clean":"2024-09-21T10:50:28.189918+0000","last_became_active":"2024-09-21T10:48:08.160600+0000","last_became_peered":"2024-09-21T10:48:08.160600+0000","last_unstale":"2024-09-21T10:50:28.189918+0000","last_undegraded":"2024-09-21T10:50:28.189918+0000","last_fullsized":"2024-09-21T10:50:28.189918+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:05:35.127563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2272,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191525+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:50:28.191525+0000","last_peered":"2024-09-21T10:50:28.191525+0000","last_clean":"2024-09-21T10:50:28.191525+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:50:28.191525+0000","last_undegraded":"2024-09-21T10:50:28.191525+0000","last_fullsized":"2024-09-21T10:50:28.191525+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2399,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.190640+0000","last_change":"2024-09-21T10:49:40.171098+0000","last_active":"2024-09-21T10:50:28.190640+0000","last_peered":"2024-09-21T10:50:28.190640+0000","last_clean":"2024-09-21T10:50:28.190640+0000","last_became_active":"2024-09-21T10:49:40.169461+0000","last_became_peered":"2024-09-21T10:49:40.169461+0000","last_unstale":"2024-09-21T10:50:28.190640+0000","last_undegraded":"2024-09-21T10:50:28.190640+0000","last_fullsized":"2024-09-21T10:50:28.190640+0000","mapping_epoch":1456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:09.774760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":409,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199690+0000","last_change":"2024-09-21T10:45:59.680994+0000","last_active":"2024-09-21T10:50:28.199690+0000","last_peered":"2024-09-21T10:50:28.199690+0000","last_clean":"2024-09-21T10:50:28.199690+0000","last_became_active":"2024-09-21T10:45:59.680636+0000","last_became_peered":"2024-09-21T10:45:59.680636+0000","last_unstale":"2024-09-21T10:50:28.199690+0000","last_undegraded":"2024-09-21T10:50:28.199690+0000","last_fullsized":"2024-09-21T10:50:28.199690+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:42.488110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2269,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192502+0000","last_change":"2024-09-21T10:48:08.152845+0000","last_active":"2024-09-21T10:50:28.192502+0000","last_peered":"2024-09-21T10:50:28.192502+0000","last_clean":"2024-09-21T10:50:28.192502+0000","last_became_active":"2024-09-21T10:48:08.152542+0000","last_became_peered":"2024-09-21T10:48:08.152542+0000","last_unstale":"2024-09-21T10:50:28.192502+0000","last_undegraded":"2024-09-21T10:50:28.192502+0000","last_fullsized":"2024-09-21T10:50:28.192502+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:27.471208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":559,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.184674+0000","last_change":"2024-09-21T10:48:08.177384+0000","last_active":"2024-09-21T10:50:28.184674+0000","last_peered":"2024-09-21T10:50:28.184674+0000","last_clean":"2024-09-21T10:50:28.184674+0000","last_became_active":"2024-09-21T10:48:08.176576+0000","last_became_peered":"2024-09-21T10:48:08.176576+0000","last_unstale":"2024-09-21T10:50:28.184674+0000","last_undegraded":"2024-09-21T10:50:28.184674+0000","last_fullsized":"2024-09-21T10:50:28.184674+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:35.528635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":154,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192119+0000","last_change":"2024-09-21T10:45:59.710479+0000","last_active":"2024-09-21T10:50:28.192119+0000","last_peered":"2024-09-21T10:50:28.192119+0000","last_clean":"2024-09-21T10:50:28.192119+0000","last_became_active":"2024-09-21T10:45:59.694474+0000","last_became_peered":"2024-09-21T10:45:59.694474+0000","last_unstale":"2024-09-21T10:50:28.192119+0000","last_undegraded":"2024-09-21T10:50:28.192119+0000","last_fullsized":"2024-09-21T10:50:28.192119+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:32.554219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":94,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.188980+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:50:28.188980+0000","last_peered":"2024-09-21T10:50:28.188980+0000","last_clean":"2024-09-21T10:50:28.188980+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:50:28.188980+0000","last_undegraded":"2024-09-21T10:50:28.188980+0000","last_fullsized":"2024-09-21T10:50:28.188980+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":449,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199588+0000","last_change":"2024-09-21T10:45:59.705879+0000","last_active":"2024-09-21T10:50:28.199588+0000","last_peered":"2024-09-21T10:50:28.199588+0000","last_clean":"2024-09-21T10:50:28.199588+0000","last_became_active":"2024-09-21T10:45:59.689786+0000","last_became_peered":"2024-09-21T10:45:59.689786+0000","last_unstale":"2024-09-21T10:50:28.199588+0000","last_undegraded":"2024-09-21T10:50:28.199588+0000","last_fullsized":"2024-09-21T10:50:28.199588+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:49:50.473399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4744,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1318,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.186307+0000","last_change":"2024-09-21T10:48:08.155603+0000","last_active":"2024-09-21T10:50:28.186307+0000","last_peered":"2024-09-21T10:50:28.186307+0000","last_clean":"2024-09-21T10:50:28.186307+0000","last_became_active":"2024-09-21T10:48:08.155285+0000","last_became_peered":"2024-09-21T10:48:08.155285+0000","last_unstale":"2024-09-21T10:50:28.186307+0000","last_undegraded":"2024-09-21T10:50:28.186307+0000","last_fullsized":"2024-09-21T10:50:28.186307+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:40:25.014519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":537,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192611+0000","last_change":"2024-09-21T10:48:08.167192+0000","last_active":"2024-09-21T10:50:28.192611+0000","last_peered":"2024-09-21T10:50:28.192611+0000","last_clean":"2024-09-21T10:50:28.192611+0000","last_became_active":"2024-09-21T10:48:08.166566+0000","last_became_peered":"2024-09-21T10:48:08.166566+0000","last_unstale":"2024-09-21T10:50:28.192611+0000","last_undegraded":"2024-09-21T10:50:28.192611+0000","last_fullsized":"2024-09-21T10:50:28.192611+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:19.985042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1275,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.201103+0000","last_change":"2024-09-21T10:48:08.152147+0000","last_active":"2024-09-21T10:50:28.201103+0000","last_peered":"2024-09-21T10:50:28.201103+0000","last_clean":"2024-09-21T10:50:28.201103+0000","last_became_active":"2024-09-21T10:48:08.151842+0000","last_became_peered":"2024-09-21T10:48:08.151842+0000","last_unstale":"2024-09-21T10:50:28.201103+0000","last_undegraded":"2024-09-21T10:50:28.201103+0000","last_fullsized":"2024-09-21T10:50:28.201103+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:39.526853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1278,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199651+0000","last_change":"2024-09-21T10:45:59.697070+0000","last_active":"2024-09-21T10:50:28.199651+0000","last_peered":"2024-09-21T10:50:28.199651+0000","last_clean":"2024-09-21T10:50:28.199651+0000","last_became_active":"2024-09-21T10:45:59.696548+0000","last_became_peered":"2024-09-21T10:45:59.696548+0000","last_unstale":"2024-09-21T10:50:28.199651+0000","last_undegraded":"2024-09-21T10:50:28.199651+0000","last_fullsized":"2024-09-21T10:50:28.199651+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:15:02.746989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,3],"acting":[2,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":271,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.197019+0000","last_change":"2024-09-21T10:48:08.210449+0000","last_active":"2024-09-21T10:50:28.197019+0000","last_peered":"2024-09-21T10:50:28.197019+0000","last_clean":"2024-09-21T10:50:28.197019+0000","last_became_active":"2024-09-21T10:48:08.162433+0000","last_became_peered":"2024-09-21T10:48:08.162433+0000","last_unstale":"2024-09-21T10:50:28.197019+0000","last_undegraded":"2024-09-21T10:50:28.197019+0000","last_fullsized":"2024-09-21T10:50:28.197019+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:55.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":311,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198867+0000","last_change":"2024-09-21T10:48:08.155548+0000","last_active":"2024-09-21T10:50:28.198867+0000","last_peered":"2024-09-21T10:50:28.198867+0000","last_clean":"2024-09-21T10:50:28.198867+0000","last_became_active":"2024-09-21T10:48:08.154428+0000","last_became_peered":"2024-09-21T10:48:08.154428+0000","last_unstale":"2024-09-21T10:50:28.198867+0000","last_undegraded":"2024-09-21T10:50:28.198867+0000","last_fullsized":"2024-09-21T10:50:28.198867+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:23.699561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":61,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.190987+0000","last_change":"2024-09-21T10:48:08.155960+0000","last_active":"2024-09-21T10:50:28.190987+0000","last_peered":"2024-09-21T10:50:28.190987+0000","last_clean":"2024-09-21T10:50:28.190987+0000","last_became_active":"2024-09-21T10:48:08.154801+0000","last_became_peered":"2024-09-21T10:48:08.154801+0000","last_unstale":"2024-09-21T10:50:28.190987+0000","last_undegraded":"2024-09-21T10:50:28.190987+0000","last_fullsized":"2024-09-21T10:50:28.190987+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:46.418044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2375,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.184704+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:50:28.184704+0000","last_peered":"2024-09-21T10:50:28.184704+0000","last_clean":"2024-09-21T10:50:28.184704+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:50:28.184704+0000","last_undegraded":"2024-09-21T10:50:28.184704+0000","last_fullsized":"2024-09-21T10:50:28.184704+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":102,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199416+0000","last_change":"2024-09-21T10:48:08.194127+0000","last_active":"2024-09-21T10:50:28.199416+0000","last_peered":"2024-09-21T10:50:28.199416+0000","last_clean":"2024-09-21T10:50:28.199416+0000","last_became_active":"2024-09-21T10:48:08.161824+0000","last_became_peered":"2024-09-21T10:48:08.161824+0000","last_unstale":"2024-09-21T10:50:28.199416+0000","last_undegraded":"2024-09-21T10:50:28.199416+0000","last_fullsized":"2024-09-21T10:50:28.199416+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.139417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":459,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.197151+0000","last_change":"2024-09-21T10:43:58.555202+0000","last_active":"2024-09-21T10:50:28.197151+0000","last_peered":"2024-09-21T10:50:28.197151+0000","last_clean":"2024-09-21T10:50:28.197151+0000","last_became_active":"2024-09-21T10:43:56.009648+0000","last_became_peered":"2024-09-21T10:43:56.009648+0000","last_unstale":"2024-09-21T10:50:28.197151+0000","last_undegraded":"2024-09-21T10:50:28.197151+0000","last_fullsized":"2024-09-21T10:50:28.197151+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.555120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.555120+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:45:37.803817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2392,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199333+0000","last_change":"2024-09-21T10:48:08.162201+0000","last_active":"2024-09-21T10:50:28.199333+0000","last_peered":"2024-09-21T10:50:28.199333+0000","last_clean":"2024-09-21T10:50:28.199333+0000","last_became_active":"2024-09-21T10:48:08.161884+0000","last_became_peered":"2024-09-21T10:48:08.161884+0000","last_unstale":"2024-09-21T10:50:28.199333+0000","last_undegraded":"2024-09-21T10:50:28.199333+0000","last_fullsized":"2024-09-21T10:50:28.199333+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:08.759002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1476,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.196888+0000","last_change":"2024-09-21T10:45:59.694166+0000","last_active":"2024-09-21T10:50:28.196888+0000","last_peered":"2024-09-21T10:50:28.196888+0000","last_clean":"2024-09-21T10:50:28.196888+0000","last_became_active":"2024-09-21T10:45:59.693276+0000","last_became_peered":"2024-09-21T10:45:59.693276+0000","last_unstale":"2024-09-21T10:50:28.196888+0000","last_undegraded":"2024-09-21T10:50:28.196888+0000","last_fullsized":"2024-09-21T10:50:28.196888+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:12.784743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2370,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.188997+0000","last_change":"2024-09-21T10:48:08.161699+0000","last_active":"2024-09-21T10:50:28.188997+0000","last_peered":"2024-09-21T10:50:28.188997+0000","last_clean":"2024-09-21T10:50:28.188997+0000","last_became_active":"2024-09-21T10:48:08.160423+0000","last_became_peered":"2024-09-21T10:48:08.160423+0000","last_unstale":"2024-09-21T10:50:28.188997+0000","last_undegraded":"2024-09-21T10:50:28.188997+0000","last_fullsized":"2024-09-21T10:50:28.188997+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:28.698737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":477,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.203684+0000","last_change":"2024-09-21T10:48:08.157705+0000","last_active":"2024-09-21T10:50:28.203684+0000","last_peered":"2024-09-21T10:50:28.203684+0000","last_clean":"2024-09-21T10:50:28.203684+0000","last_became_active":"2024-09-21T10:48:08.157385+0000","last_became_peered":"2024-09-21T10:48:08.157385+0000","last_unstale":"2024-09-21T10:50:28.203684+0000","last_undegraded":"2024-09-21T10:50:28.203684+0000","last_fullsized":"2024-09-21T10:50:28.203684+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.698401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.698401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:18:48.903892+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2375,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189667+0000","last_change":"2024-09-21T10:48:08.161856+0000","last_active":"2024-09-21T10:50:28.189667+0000","last_peered":"2024-09-21T10:50:28.189667+0000","last_clean":"2024-09-21T10:50:28.189667+0000","last_became_active":"2024-09-21T10:48:08.160283+0000","last_became_peered":"2024-09-21T10:48:08.160283+0000","last_unstale":"2024-09-21T10:50:28.189667+0000","last_undegraded":"2024-09-21T10:50:28.189667+0000","last_fullsized":"2024-09-21T10:50:28.189667+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:09.474670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2370,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189155+0000","last_change":"2024-09-21T10:45:59.691645+0000","last_active":"2024-09-21T10:50:28.189155+0000","last_peered":"2024-09-21T10:50:28.189155+0000","last_clean":"2024-09-21T10:50:28.189155+0000","last_became_active":"2024-09-21T10:45:59.691334+0000","last_became_peered":"2024-09-21T10:45:59.691334+0000","last_unstale":"2024-09-21T10:50:28.189155+0000","last_undegraded":"2024-09-21T10:50:28.189155+0000","last_fullsized":"2024-09-21T10:50:28.189155+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:33.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":799,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192213+0000","last_change":"2024-09-21T10:48:08.144825+0000","last_active":"2024-09-21T10:50:28.192213+0000","last_peered":"2024-09-21T10:50:28.192213+0000","last_clean":"2024-09-21T10:50:28.192213+0000","last_became_active":"2024-09-21T10:48:08.142212+0000","last_became_peered":"2024-09-21T10:48:08.142212+0000","last_unstale":"2024-09-21T10:50:28.192213+0000","last_undegraded":"2024-09-21T10:50:28.192213+0000","last_fullsized":"2024-09-21T10:50:28.192213+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:46.899049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":454,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.201269+0000","last_change":"2024-09-21T10:48:08.960298+0000","last_active":"2024-09-21T10:50:28.201269+0000","last_peered":"2024-09-21T10:50:28.201269+0000","last_clean":"2024-09-21T10:50:28.201269+0000","last_became_active":"2024-09-21T10:48:08.151776+0000","last_became_peered":"2024-09-21T10:48:08.151776+0000","last_unstale":"2024-09-21T10:50:28.201269+0000","last_undegraded":"2024-09-21T10:50:28.201269+0000","last_fullsized":"2024-09-21T10:50:28.201269+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.960250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.960250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:25.581994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2360,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189077+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:50:28.189077+0000","last_peered":"2024-09-21T10:50:28.189077+0000","last_clean":"2024-09-21T10:50:28.189077+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:50:28.189077+0000","last_undegraded":"2024-09-21T10:50:28.189077+0000","last_fullsized":"2024-09-21T10:50:28.189077+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1480,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199239+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:50:28.199239+0000","last_peered":"2024-09-21T10:50:28.199239+0000","last_clean":"2024-09-21T10:50:28.199239+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:50:28.199239+0000","last_undegraded":"2024-09-21T10:50:28.199239+0000","last_fullsized":"2024-09-21T10:50:28.199239+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":199,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189339+0000","last_change":"2024-09-21T10:48:08.212451+0000","last_active":"2024-09-21T10:50:28.189339+0000","last_peered":"2024-09-21T10:50:28.189339+0000","last_clean":"2024-09-21T10:50:28.189339+0000","last_became_active":"2024-09-21T10:48:08.161435+0000","last_became_peered":"2024-09-21T10:48:08.161435+0000","last_unstale":"2024-09-21T10:50:28.189339+0000","last_undegraded":"2024-09-21T10:50:28.189339+0000","last_fullsized":"2024-09-21T10:50:28.189339+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:44.511679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":697,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199905+0000","last_change":"2024-09-21T10:48:08.172490+0000","last_active":"2024-09-21T10:50:28.199905+0000","last_peered":"2024-09-21T10:50:28.199905+0000","last_clean":"2024-09-21T10:50:28.199905+0000","last_became_active":"2024-09-21T10:48:08.172117+0000","last_became_peered":"2024-09-21T10:48:08.172117+0000","last_unstale":"2024-09-21T10:50:28.199905+0000","last_undegraded":"2024-09-21T10:50:28.199905+0000","last_fullsized":"2024-09-21T10:50:28.199905+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:35.639980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":101,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189594+0000","last_change":"2024-09-21T10:48:08.182097+0000","last_active":"2024-09-21T10:50:28.189594+0000","last_peered":"2024-09-21T10:50:28.189594+0000","last_clean":"2024-09-21T10:50:28.189594+0000","last_became_active":"2024-09-21T10:48:08.181514+0000","last_became_peered":"2024-09-21T10:48:08.181514+0000","last_unstale":"2024-09-21T10:50:28.189594+0000","last_undegraded":"2024-09-21T10:50:28.189594+0000","last_fullsized":"2024-09-21T10:50:28.189594+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:59.851982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2360,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.184971+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:50:28.184971+0000","last_peered":"2024-09-21T10:50:28.184971+0000","last_clean":"2024-09-21T10:50:28.184971+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:50:28.184971+0000","last_undegraded":"2024-09-21T10:50:28.184971+0000","last_fullsized":"2024-09-21T10:50:28.184971+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2292,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.195251+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:50:28.195251+0000","last_peered":"2024-09-21T10:50:28.195251+0000","last_clean":"2024-09-21T10:50:28.195251+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:50:28.195251+0000","last_undegraded":"2024-09-21T10:50:28.195251+0000","last_fullsized":"2024-09-21T10:50:28.195251+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":552,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.186140+0000","last_change":"2024-09-21T10:48:08.182699+0000","last_active":"2024-09-21T10:50:28.186140+0000","last_peered":"2024-09-21T10:50:28.186140+0000","last_clean":"2024-09-21T10:50:28.186140+0000","last_became_active":"2024-09-21T10:48:08.182308+0000","last_became_peered":"2024-09-21T10:48:08.182308+0000","last_unstale":"2024-09-21T10:50:28.186140+0000","last_undegraded":"2024-09-21T10:50:28.186140+0000","last_fullsized":"2024-09-21T10:50:28.186140+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:32.985403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":217,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.196485+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:50:28.196485+0000","last_peered":"2024-09-21T10:50:28.196485+0000","last_clean":"2024-09-21T10:50:28.196485+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:50:28.196485+0000","last_undegraded":"2024-09-21T10:50:28.196485+0000","last_fullsized":"2024-09-21T10:50:28.196485+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2400,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.201355+0000","last_change":"2024-09-21T10:48:08.166778+0000","last_active":"2024-09-21T10:50:28.201355+0000","last_peered":"2024-09-21T10:50:28.201355+0000","last_clean":"2024-09-21T10:50:28.201355+0000","last_became_active":"2024-09-21T10:48:08.166031+0000","last_became_peered":"2024-09-21T10:48:08.166031+0000","last_unstale":"2024-09-21T10:50:28.201355+0000","last_undegraded":"2024-09-21T10:50:28.201355+0000","last_fullsized":"2024-09-21T10:50:28.201355+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.713934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":638,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.185152+0000","last_change":"2024-09-21T10:45:59.691940+0000","last_active":"2024-09-21T10:50:28.185152+0000","last_peered":"2024-09-21T10:50:28.185152+0000","last_clean":"2024-09-21T10:50:28.185152+0000","last_became_active":"2024-09-21T10:45:59.691610+0000","last_became_peered":"2024-09-21T10:45:59.691610+0000","last_unstale":"2024-09-21T10:50:28.185152+0000","last_undegraded":"2024-09-21T10:50:28.185152+0000","last_fullsized":"2024-09-21T10:50:28.185152+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:57.766019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1406,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.196964+0000","last_change":"2024-09-21T10:46:46.607744+0000","last_active":"2024-09-21T10:50:28.196964+0000","last_peered":"2024-09-21T10:50:28.196964+0000","last_clean":"2024-09-21T10:50:28.196964+0000","last_became_active":"2024-09-21T10:46:46.607376+0000","last_became_peered":"2024-09-21T10:46:46.607376+0000","last_unstale":"2024-09-21T10:50:28.196964+0000","last_undegraded":"2024-09-21T10:50:28.196964+0000","last_fullsized":"2024-09-21T10:50:28.196964+0000","mapping_epoch":1433,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:04:17.279053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2284,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198579+0000","last_change":"2024-09-21T10:48:08.158793+0000","last_active":"2024-09-21T10:50:28.198579+0000","last_peered":"2024-09-21T10:50:28.198579+0000","last_clean":"2024-09-21T10:50:28.198579+0000","last_became_active":"2024-09-21T10:48:08.158487+0000","last_became_peered":"2024-09-21T10:48:08.158487+0000","last_unstale":"2024-09-21T10:50:28.198579+0000","last_undegraded":"2024-09-21T10:50:28.198579+0000","last_fullsized":"2024-09-21T10:50:28.198579+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:53.537691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":197,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198332+0000","last_change":"2024-09-21T10:48:08.214131+0000","last_active":"2024-09-21T10:50:28.198332+0000","last_peered":"2024-09-21T10:50:28.198332+0000","last_clean":"2024-09-21T10:50:28.198332+0000","last_became_active":"2024-09-21T10:48:08.157618+0000","last_became_peered":"2024-09-21T10:48:08.157618+0000","last_unstale":"2024-09-21T10:50:28.198332+0000","last_undegraded":"2024-09-21T10:50:28.198332+0000","last_fullsized":"2024-09-21T10:50:28.198332+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:36.574106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1629,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199645+0000","last_change":"2024-09-21T10:45:59.692592+0000","last_active":"2024-09-21T10:50:28.199645+0000","last_peered":"2024-09-21T10:50:28.199645+0000","last_clean":"2024-09-21T10:50:28.199645+0000","last_became_active":"2024-09-21T10:45:59.692232+0000","last_became_peered":"2024-09-21T10:45:59.692232+0000","last_unstale":"2024-09-21T10:50:28.199645+0000","last_undegraded":"2024-09-21T10:50:28.199645+0000","last_fullsized":"2024-09-21T10:50:28.199645+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:26.920879+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":129,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.195597+0000","last_change":"2024-09-21T10:48:08.148499+0000","last_active":"2024-09-21T10:50:28.195597+0000","last_peered":"2024-09-21T10:50:28.195597+0000","last_clean":"2024-09-21T10:50:28.195597+0000","last_became_active":"2024-09-21T10:48:08.148141+0000","last_became_peered":"2024-09-21T10:48:08.148141+0000","last_unstale":"2024-09-21T10:50:28.195597+0000","last_undegraded":"2024-09-21T10:50:28.195597+0000","last_fullsized":"2024-09-21T10:50:28.195597+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:35.860469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2379,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189933+0000","last_change":"2024-09-21T10:48:08.171805+0000","last_active":"2024-09-21T10:50:28.189933+0000","last_peered":"2024-09-21T10:50:28.189933+0000","last_clean":"2024-09-21T10:50:28.189933+0000","last_became_active":"2024-09-21T10:48:08.170840+0000","last_became_peered":"2024-09-21T10:48:08.170840+0000","last_unstale":"2024-09-21T10:50:28.189933+0000","last_undegraded":"2024-09-21T10:50:28.189933+0000","last_fullsized":"2024-09-21T10:50:28.189933+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:18.802870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2278,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191439+0000","last_change":"2024-09-21T10:48:08.155588+0000","last_active":"2024-09-21T10:50:28.191439+0000","last_peered":"2024-09-21T10:50:28.191439+0000","last_clean":"2024-09-21T10:50:28.191439+0000","last_became_active":"2024-09-21T10:48:08.153652+0000","last_became_peered":"2024-09-21T10:48:08.153652+0000","last_unstale":"2024-09-21T10:50:28.191439+0000","last_undegraded":"2024-09-21T10:50:28.191439+0000","last_fullsized":"2024-09-21T10:50:28.191439+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:58.415174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":536,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.185262+0000","last_change":"2024-09-21T10:48:08.181118+0000","last_active":"2024-09-21T10:50:28.185262+0000","last_peered":"2024-09-21T10:50:28.185262+0000","last_clean":"2024-09-21T10:50:28.185262+0000","last_became_active":"2024-09-21T10:48:08.180184+0000","last_became_peered":"2024-09-21T10:48:08.180184+0000","last_unstale":"2024-09-21T10:50:28.185262+0000","last_undegraded":"2024-09-21T10:50:28.185262+0000","last_fullsized":"2024-09-21T10:50:28.185262+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:03:18.370832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":94,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.184430+0000","last_change":"2024-09-21T10:46:30.264653+0000","last_active":"2024-09-21T10:50:28.184430+0000","last_peered":"2024-09-21T10:50:28.184430+0000","last_clean":"2024-09-21T10:50:28.184430+0000","last_became_active":"2024-09-21T10:46:30.264068+0000","last_became_peered":"2024-09-21T10:46:30.264068+0000","last_unstale":"2024-09-21T10:50:28.184430+0000","last_undegraded":"2024-09-21T10:50:28.184430+0000","last_fullsized":"2024-09-21T10:50:28.184430+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:25:53.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1433,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.195330+0000","last_change":"2024-09-21T10:48:08.171234+0000","last_active":"2024-09-21T10:50:28.195330+0000","last_peered":"2024-09-21T10:50:28.195330+0000","last_clean":"2024-09-21T10:50:28.195330+0000","last_became_active":"2024-09-21T10:48:08.170908+0000","last_became_peered":"2024-09-21T10:48:08.170908+0000","last_unstale":"2024-09-21T10:50:28.195330+0000","last_undegraded":"2024-09-21T10:50:28.195330+0000","last_fullsized":"2024-09-21T10:50:28.195330+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:28.112802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":144,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.204892+0000","last_change":"2024-09-21T10:48:08.215096+0000","last_active":"2024-09-21T10:50:28.204892+0000","last_peered":"2024-09-21T10:50:28.204892+0000","last_clean":"2024-09-21T10:50:28.204892+0000","last_became_active":"2024-09-21T10:48:08.145535+0000","last_became_peered":"2024-09-21T10:48:08.145535+0000","last_unstale":"2024-09-21T10:50:28.204892+0000","last_undegraded":"2024-09-21T10:50:28.204892+0000","last_fullsized":"2024-09-21T10:50:28.204892+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:55.688095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1427,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199024+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:50:28.199024+0000","last_peered":"2024-09-21T10:50:28.199024+0000","last_clean":"2024-09-21T10:50:28.199024+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:50:28.199024+0000","last_undegraded":"2024-09-21T10:50:28.199024+0000","last_fullsized":"2024-09-21T10:50:28.199024+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1419,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189427+0000","last_change":"2024-09-21T10:43:56.012844+0000","last_active":"2024-09-21T10:50:28.189427+0000","last_peered":"2024-09-21T10:50:28.189427+0000","last_clean":"2024-09-21T10:50:28.189427+0000","last_became_active":"2024-09-21T10:43:56.012504+0000","last_became_peered":"2024-09-21T10:43:56.012504+0000","last_unstale":"2024-09-21T10:50:28.189427+0000","last_undegraded":"2024-09-21T10:50:28.189427+0000","last_fullsized":"2024-09-21T10:50:28.189427+0000","mapping_epoch":1407,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:02.556342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1467,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191154+0000","last_change":"2024-09-21T10:48:08.153599+0000","last_active":"2024-09-21T10:50:28.191154+0000","last_peered":"2024-09-21T10:50:28.191154+0000","last_clean":"2024-09-21T10:50:28.191154+0000","last_became_active":"2024-09-21T10:48:08.153255+0000","last_became_peered":"2024-09-21T10:48:08.153255+0000","last_unstale":"2024-09-21T10:50:28.191154+0000","last_undegraded":"2024-09-21T10:50:28.191154+0000","last_fullsized":"2024-09-21T10:50:28.191154+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:04.469883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:04.469883+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:12:53.690175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1479,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192967+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:50:28.192967+0000","last_peered":"2024-09-21T10:50:28.192967+0000","last_clean":"2024-09-21T10:50:28.192967+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:50:28.192967+0000","last_undegraded":"2024-09-21T10:50:28.192967+0000","last_fullsized":"2024-09-21T10:50:28.192967+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":491,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.197154+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:50:28.197154+0000","last_peered":"2024-09-21T10:50:28.197154+0000","last_clean":"2024-09-21T10:50:28.197154+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:50:28.197154+0000","last_undegraded":"2024-09-21T10:50:28.197154+0000","last_fullsized":"2024-09-21T10:50:28.197154+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":443,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.193947+0000","last_change":"2024-09-21T10:46:08.467544+0000","last_active":"2024-09-21T10:50:28.193947+0000","last_peered":"2024-09-21T10:50:28.193947+0000","last_clean":"2024-09-21T10:50:28.193947+0000","last_became_active":"2024-09-21T10:45:59.695828+0000","last_became_peered":"2024-09-21T10:45:59.695828+0000","last_unstale":"2024-09-21T10:50:28.193947+0000","last_undegraded":"2024-09-21T10:50:28.193947+0000","last_fullsized":"2024-09-21T10:50:28.193947+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:08.467485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:08.467485+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:36.742481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2391,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199152+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:50:28.199152+0000","last_peered":"2024-09-21T10:50:28.199152+0000","last_clean":"2024-09-21T10:50:28.199152+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:50:28.199152+0000","last_undegraded":"2024-09-21T10:50:28.199152+0000","last_fullsized":"2024-09-21T10:50:28.199152+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1334,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192768+0000","last_change":"2024-09-21T10:48:08.166901+0000","last_active":"2024-09-21T10:50:28.192768+0000","last_peered":"2024-09-21T10:50:28.192768+0000","last_clean":"2024-09-21T10:50:28.192768+0000","last_became_active":"2024-09-21T10:48:08.166122+0000","last_became_peered":"2024-09-21T10:48:08.166122+0000","last_unstale":"2024-09-21T10:50:28.192768+0000","last_undegraded":"2024-09-21T10:50:28.192768+0000","last_fullsized":"2024-09-21T10:50:28.192768+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:18.037036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":526,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.203566+0000","last_change":"2024-09-21T10:48:08.158541+0000","last_active":"2024-09-21T10:50:28.203566+0000","last_peered":"2024-09-21T10:50:28.203566+0000","last_clean":"2024-09-21T10:50:28.203566+0000","last_became_active":"2024-09-21T10:48:08.158099+0000","last_became_peered":"2024-09-21T10:48:08.158099+0000","last_unstale":"2024-09-21T10:50:28.203566+0000","last_undegraded":"2024-09-21T10:50:28.203566+0000","last_fullsized":"2024-09-21T10:50:28.203566+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:44:31.818227+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1268,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189587+0000","last_change":"2024-09-21T10:48:08.154134+0000","last_active":"2024-09-21T10:50:28.189587+0000","last_peered":"2024-09-21T10:50:28.189587+0000","last_clean":"2024-09-21T10:50:28.189587+0000","last_became_active":"2024-09-21T10:48:08.153798+0000","last_became_peered":"2024-09-21T10:48:08.153798+0000","last_unstale":"2024-09-21T10:50:28.189587+0000","last_undegraded":"2024-09-21T10:50:28.189587+0000","last_fullsized":"2024-09-21T10:50:28.189587+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:59:30.113364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":1429,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192001+0000","last_change":"2024-09-21T10:48:08.170002+0000","last_active":"2024-09-21T10:50:28.192001+0000","last_peered":"2024-09-21T10:50:28.192001+0000","last_clean":"2024-09-21T10:50:28.192001+0000","last_became_active":"2024-09-21T10:48:08.152043+0000","last_became_peered":"2024-09-21T10:48:08.152043+0000","last_unstale":"2024-09-21T10:50:28.192001+0000","last_undegraded":"2024-09-21T10:50:28.192001+0000","last_fullsized":"2024-09-21T10:50:28.192001+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:31:55.281864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":530,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191956+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:50:28.191956+0000","last_peered":"2024-09-21T10:50:28.191956+0000","last_clean":"2024-09-21T10:50:28.191956+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:50:28.191956+0000","last_undegraded":"2024-09-21T10:50:28.191956+0000","last_fullsized":"2024-09-21T10:50:28.191956+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1461,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.185062+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:50:28.185062+0000","last_peered":"2024-09-21T10:50:28.185062+0000","last_clean":"2024-09-21T10:50:28.185062+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:50:28.185062+0000","last_undegraded":"2024-09-21T10:50:28.185062+0000","last_fullsized":"2024-09-21T10:50:28.185062+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":140,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.184656+0000","last_change":"2024-09-21T10:48:08.204361+0000","last_active":"2024-09-21T10:50:28.184656+0000","last_peered":"2024-09-21T10:50:28.184656+0000","last_clean":"2024-09-21T10:50:28.184656+0000","last_became_active":"2024-09-21T10:48:08.177064+0000","last_became_peered":"2024-09-21T10:48:08.177064+0000","last_unstale":"2024-09-21T10:50:28.184656+0000","last_undegraded":"2024-09-21T10:50:28.184656+0000","last_fullsized":"2024-09-21T10:50:28.184656+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:12.137967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":123,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.186111+0000","last_change":"2024-09-21T10:46:00.158554+0000","last_active":"2024-09-21T10:50:28.186111+0000","last_peered":"2024-09-21T10:50:28.186111+0000","last_clean":"2024-09-21T10:50:28.186111+0000","last_became_active":"2024-09-21T10:45:59.680546+0000","last_became_peered":"2024-09-21T10:45:59.680546+0000","last_unstale":"2024-09-21T10:50:28.186111+0000","last_undegraded":"2024-09-21T10:50:28.186111+0000","last_fullsized":"2024-09-21T10:50:28.186111+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:00.158500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:00.158500+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:39.581987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":230,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.191147+0000","last_change":"2024-09-21T10:48:08.158317+0000","last_active":"2024-09-21T10:50:28.191147+0000","last_peered":"2024-09-21T10:50:28.191147+0000","last_clean":"2024-09-21T10:50:28.191147+0000","last_became_active":"2024-09-21T10:48:08.154052+0000","last_became_peered":"2024-09-21T10:48:08.154052+0000","last_unstale":"2024-09-21T10:50:28.191147+0000","last_undegraded":"2024-09-21T10:50:28.191147+0000","last_fullsized":"2024-09-21T10:50:28.191147+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:25:28.045098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1497,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.196842+0000","last_change":"2024-09-21T10:48:08.213283+0000","last_active":"2024-09-21T10:50:28.196842+0000","last_peered":"2024-09-21T10:50:28.196842+0000","last_clean":"2024-09-21T10:50:28.196842+0000","last_became_active":"2024-09-21T10:48:08.166573+0000","last_became_peered":"2024-09-21T10:48:08.166573+0000","last_unstale":"2024-09-21T10:50:28.196842+0000","last_undegraded":"2024-09-21T10:50:28.196842+0000","last_fullsized":"2024-09-21T10:50:28.196842+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:43:51.213203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":71,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.189197+0000","last_change":"2024-09-21T10:48:08.843710+0000","last_active":"2024-09-21T10:50:28.189197+0000","last_peered":"2024-09-21T10:50:28.189197+0000","last_clean":"2024-09-21T10:50:28.189197+0000","last_became_active":"2024-09-21T10:48:08.162916+0000","last_became_peered":"2024-09-21T10:48:08.162916+0000","last_unstale":"2024-09-21T10:50:28.189197+0000","last_undegraded":"2024-09-21T10:50:28.189197+0000","last_fullsized":"2024-09-21T10:50:28.189197+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.843662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.843662+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:40:35.024010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1533,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199440+0000","last_change":"2024-09-21T10:48:08.193479+0000","last_active":"2024-09-21T10:50:28.199440+0000","last_peered":"2024-09-21T10:50:28.199440+0000","last_clean":"2024-09-21T10:50:28.199440+0000","last_became_active":"2024-09-21T10:48:08.149775+0000","last_became_peered":"2024-09-21T10:48:08.149775+0000","last_unstale":"2024-09-21T10:50:28.199440+0000","last_undegraded":"2024-09-21T10:50:28.199440+0000","last_fullsized":"2024-09-21T10:50:28.199440+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:03:00.302830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2260,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.203669+0000","last_change":"2024-09-21T10:45:59.696879+0000","last_active":"2024-09-21T10:50:28.203669+0000","last_peered":"2024-09-21T10:50:28.203669+0000","last_clean":"2024-09-21T10:50:28.203669+0000","last_became_active":"2024-09-21T10:45:59.696055+0000","last_became_peered":"2024-09-21T10:45:59.696055+0000","last_unstale":"2024-09-21T10:50:28.203669+0000","last_undegraded":"2024-09-21T10:50:28.203669+0000","last_fullsized":"2024-09-21T10:50:28.203669+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:10.189668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":535,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.185154+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:50:28.185154+0000","last_peered":"2024-09-21T10:50:28.185154+0000","last_clean":"2024-09-21T10:50:28.185154+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:50:28.185154+0000","last_undegraded":"2024-09-21T10:50:28.185154+0000","last_fullsized":"2024-09-21T10:50:28.185154+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1314,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.201016+0000","last_change":"2024-09-21T10:45:59.693730+0000","last_active":"2024-09-21T10:50:28.201016+0000","last_peered":"2024-09-21T10:50:28.201016+0000","last_clean":"2024-09-21T10:50:28.201016+0000","last_became_active":"2024-09-21T10:45:59.692582+0000","last_became_peered":"2024-09-21T10:45:59.692582+0000","last_unstale":"2024-09-21T10:50:28.201016+0000","last_undegraded":"2024-09-21T10:50:28.201016+0000","last_fullsized":"2024-09-21T10:50:28.201016+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:42:21.994503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2379,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.197199+0000","last_change":"2024-09-21T10:45:59.697081+0000","last_active":"2024-09-21T10:50:28.197199+0000","last_peered":"2024-09-21T10:50:28.197199+0000","last_clean":"2024-09-21T10:50:28.197199+0000","last_became_active":"2024-09-21T10:45:59.696414+0000","last_became_peered":"2024-09-21T10:45:59.696414+0000","last_unstale":"2024-09-21T10:50:28.197199+0000","last_undegraded":"2024-09-21T10:50:28.197199+0000","last_fullsized":"2024-09-21T10:50:28.197199+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:47.528914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":217,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.199386+0000","last_change":"2024-09-21T10:48:08.150768+0000","last_active":"2024-09-21T10:50:28.199386+0000","last_peered":"2024-09-21T10:50:28.199386+0000","last_clean":"2024-09-21T10:50:28.199386+0000","last_became_active":"2024-09-21T10:48:08.150235+0000","last_became_peered":"2024-09-21T10:48:08.150235+0000","last_unstale":"2024-09-21T10:50:28.199386+0000","last_undegraded":"2024-09-21T10:50:28.199386+0000","last_fullsized":"2024-09-21T10:50:28.199386+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:29.210811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:29.210811+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:21:58.640157+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2379,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.200038+0000","last_change":"2024-09-21T10:45:59.700672+0000","last_active":"2024-09-21T10:50:28.200038+0000","last_peered":"2024-09-21T10:50:28.200038+0000","last_clean":"2024-09-21T10:50:28.200038+0000","last_became_active":"2024-09-21T10:45:59.699332+0000","last_became_peered":"2024-09-21T10:45:59.699332+0000","last_unstale":"2024-09-21T10:50:28.200038+0000","last_undegraded":"2024-09-21T10:50:28.200038+0000","last_fullsized":"2024-09-21T10:50:28.200038+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:48.567053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":239,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.194995+0000","last_change":"2024-09-21T10:48:08.155880+0000","last_active":"2024-09-21T10:50:28.194995+0000","last_peered":"2024-09-21T10:50:28.194995+0000","last_clean":"2024-09-21T10:50:28.194995+0000","last_became_active":"2024-09-21T10:48:08.154917+0000","last_became_peered":"2024-09-21T10:48:08.154917+0000","last_unstale":"2024-09-21T10:50:28.194995+0000","last_undegraded":"2024-09-21T10:50:28.194995+0000","last_fullsized":"2024-09-21T10:50:28.194995+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:37:14.509661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":260,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.200931+0000","last_change":"2024-09-21T10:48:08.167083+0000","last_active":"2024-09-21T10:50:28.200931+0000","last_peered":"2024-09-21T10:50:28.200931+0000","last_clean":"2024-09-21T10:50:28.200931+0000","last_became_active":"2024-09-21T10:48:08.166441+0000","last_became_peered":"2024-09-21T10:48:08.166441+0000","last_unstale":"2024-09-21T10:50:28.200931+0000","last_undegraded":"2024-09-21T10:50:28.200931+0000","last_fullsized":"2024-09-21T10:50:28.200931+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:53:57.413991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":131,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.198430+0000","last_change":"2024-09-21T10:45:59.714397+0000","last_active":"2024-09-21T10:50:28.198430+0000","last_peered":"2024-09-21T10:50:28.198430+0000","last_clean":"2024-09-21T10:50:28.198430+0000","last_became_active":"2024-09-21T10:45:59.689648+0000","last_became_peered":"2024-09-21T10:45:59.689648+0000","last_unstale":"2024-09-21T10:50:28.198430+0000","last_undegraded":"2024-09-21T10:50:28.198430+0000","last_fullsized":"2024-09-21T10:50:28.198430+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:20.596738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2268,"reported_epoch":1467,"state":"active+clean","last_fresh":"2024-09-21T10:50:28.192697+0000","last_change":"2024-09-21T10:45:59.696141+0000","last_active":"2024-09-21T10:50:28.192697+0000","last_peered":"2024-09-21T10:50:28.192697+0000","last_clean":"2024-09-21T10:50:28.192697+0000","last_became_active":"2024-09-21T10:45:59.695761+0000","last_became_peered":"2024-09-21T10:45:59.695761+0000","last_unstale":"2024-09-21T10:50:28.192697+0000","last_undegraded":"2024-09-21T10:50:28.192697+0000","last_fullsized":"2024-09-21T10:50:28.192697+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:46.714846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":50,"num_pg":16,"stat_sum":{"num_bytes":184942616,"num_objects":2823,"num_object_clones":0,"num_object_copies":8469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":206327,"num_write_kb":6692737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30612,"num_bytes_recovered":2006122520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250439168,"data_stored":1250426952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75945,"ondisk_log_size":75945,"up":48,"acting":48,"num_store_stats":6},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":6806,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":26316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":210453398172,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049831,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601816,"kb_used_data":96720,"kb_used_omap":242,"kb_used_meta":1504845,"kb_avail":92143336,"statfs":{"total":95995035648,"available":94354776064,"internally_reserved":0,"allocated":99041280,"data_stored":94732578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":1540961503},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":158913790630,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644789,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1308556,"kb_used_data":203136,"kb_used_omap":67,"kb_used_meta":1105340,"kb_avail":92436596,"statfs":{"total":95995035648,"available":94655074304,"internally_reserved":0,"allocated":208011264,"data_stored":203713637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1131869021},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527807,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1275412,"kb_used_data":175624,"kb_used_omap":75,"kb_used_meta":1099508,"kb_avail":92469740,"statfs":{"total":95995035648,"available":94689013760,"internally_reserved":0,"allocated":179838976,"data_stored":175538339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":1125896346},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274960,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1189884,"kb_used_data":105268,"kb_used_omap":51,"kb_used_meta":1084492,"kb_avail":92555268,"statfs":{"total":95995035648,"available":94776594432,"internally_reserved":0,"allocated":107794432,"data_stored":103508384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1110520149},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354199,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015516,"kb_used_data":106644,"kb_used_omap":66,"kb_used_meta":908797,"kb_avail":92729636,"statfs":{"total":95995035648,"available":94955147264,"internally_reserved":0,"allocated":109203456,"data_stored":104890172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":930608452},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62853120,"data_stored":62849048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":83886080,"data_stored":83886080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":165023744,"data_stored":165019672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":191303680,"data_stored":191299608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":679149568,"data_stored":679149568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":50,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":68222976,"data_stored":68222976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:50:41.779 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:50:41.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:50:41.779 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T10:50:41.779 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 1.000000 2024-09-21T10:50:41.779 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T10:50:41.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:41.854 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:41.878 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:41.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:41.979 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:42.001 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:42.102 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:42.125 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:42.176 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is set 2024-09-21T10:50:42.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:42.247 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:42.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:42.360 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:42 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3645602259' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:42.361 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:42 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3645602259' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:42.370 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:42.471 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:42.492 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:42.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:42 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3645602259' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:42.594 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:42.615 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:42.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:42.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:42.738 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:42.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:42.840 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:42.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T10:50:42.861 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:42.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:42.984 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:42.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:42.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:43.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:43.109 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:43.116 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:43.116 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:43.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.234 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:43.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:43.335 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:43.357 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:43.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:43.458 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:43 smithi145 ceph-mon[89359]: pgmap v2691: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 577 op/s 2024-09-21T10:50:43.458 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:43 smithi145 ceph-mon[89359]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:43.459 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3645602259' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:43.459 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:43 smithi145 ceph-mon[89359]: osdmap e1468: 8 total, 8 up, 5 in 2024-09-21T10:50:43.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:43.480 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:43.582 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:43.604 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:43.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[105288]: pgmap v2691: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 577 op/s 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[105288]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3645602259' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[105288]: osdmap e1468: 8 total, 8 up, 5 in 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[101432]: pgmap v2691: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 577 op/s 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[101432]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3645602259' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:43.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:43 smithi119 ceph-mon[101432]: osdmap e1468: 8 total, 8 up, 5 in 2024-09-21T10:50:43.705 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:43.726 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:43.765 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:43.782 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T10:50:43.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:43.850 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:43.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:43.973 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:43.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:44.024 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:44.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:44.096 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:44.197 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:44.219 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:44.321 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:44.342 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:44.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:44.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:44.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:44.589 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:44.609 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:50:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:50:44] "GET /metrics HTTP/1.1" 200 36826 "" "Prometheus/2.51.0" 2024-09-21T10:50:44.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.712 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:44.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:44.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:44.835 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:44.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:44.958 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:44.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:44.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:45.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:45.081 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:45.183 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:45.205 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:45.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:45.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:50:45.256 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:45 smithi119 ceph-mon[105288]: pgmap v2693: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 282 op/s 2024-09-21T10:50:45.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:45 smithi119 ceph-mon[101432]: pgmap v2693: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 282 op/s 2024-09-21T10:50:45.307 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:45.329 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:45.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:45.459 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:45.460 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:45 smithi145 ceph-mon[89359]: pgmap v2693: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 282 op/s 2024-09-21T10:50:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:45.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:45.582 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:45.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:45.706 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:45.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:45.808 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:45.827 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:45.827 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:45.830 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:45.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:45.953 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:45.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:46.055 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:46.076 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:46.169 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:46.178 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:46.183 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T10:50:46.199 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:46.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:46.322 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:46.417 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:46.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:46.447 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:46.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:46.456 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2271948218' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:46.481 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T10:50:46.538 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2271948218' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:46.538 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2271948218' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:50:46.548 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:46.569 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:46.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:46.692 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:46.729 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:46.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:46.816 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:46.917 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:46.939 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:46.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:47.040 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:47.061 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:47.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:47.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.186 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:47.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:47.196 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'unique_pool_5' removed 2024-09-21T10:50:47.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.331 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:47.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[105288]: pgmap v2694: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 125 op/s 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2271948218' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[105288]: osdmap e1469: 8 total, 8 up, 5 in 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/129289659' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[101432]: pgmap v2694: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 125 op/s 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2271948218' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[101432]: osdmap e1469: 8 total, 8 up, 5 in 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/129289659' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:50:47.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:47 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:50:47.433 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:47.454 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:47 smithi145 ceph-mon[89359]: pgmap v2694: 109 pgs: 109 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 125 op/s 2024-09-21T10:50:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2271948218' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:50:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:47 smithi145 ceph-mon[89359]: osdmap e1469: 8 total, 8 up, 5 in 2024-09-21T10:50:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/129289659' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:50:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:47 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-21T10:50:47.556 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:47.578 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:47.680 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:47.702 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:47.803 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:47.824 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:47.926 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:47.948 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-21T10:50:47.948 INFO:teuthology.task.print:**** done end radosbench.yaml 2024-09-21T10:50:47.948 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-09-21T10:50:47.949 INFO:tasks.workunit:Pulling workunits from ref quincy 2024-09-21T10:50:47.949 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T10:50:47.949 DEBUG:teuthology.orchestra.run.smithi119:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:50:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.951 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:47.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T10:50:47.975 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T10:50:48.037 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T10:50:48.037 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T10:50:48.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:48.073 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:48.093 INFO:tasks.workunit:timeout=3h 2024-09-21T10:50:48.093 INFO:tasks.workunit:cleanup=True 2024-09-21T10:50:48.094 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:50:48.158 INFO:tasks.workunit.client.0.smithi119.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T10:50:48.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.197 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:48.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:48.298 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:48.319 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:48.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:48.421 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:48.422 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:48.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.445 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:48.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:48 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:50:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:48 smithi145 ceph-mon[89359]: osdmap e1470: 8 total, 8 up, 5 in 2024-09-21T10:50:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:48.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:48.567 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:48.589 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:48 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:50:48.589 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:48 smithi119 ceph-mon[101432]: osdmap e1470: 8 total, 8 up, 5 in 2024-09-21T10:50:48.589 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:48.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:48 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-21T10:50:48.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:48 smithi119 ceph-mon[105288]: osdmap e1470: 8 total, 8 up, 5 in 2024-09-21T10:50:48.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:50:48.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:48.690 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:48.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:48.814 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:48.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:48.940 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:48.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:49.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:49.054 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T10:50:49.062 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:49.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:49.185 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:49.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:49.293 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:49 smithi145 ceph-mon[89359]: pgmap v2697: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:49.299 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:49.309 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:49.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:49.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:49 smithi119 ceph-mon[101432]: pgmap v2697: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:49.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:49 smithi119 ceph-mon[105288]: pgmap v2697: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:49.410 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:49.432 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:49.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:49.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:49.556 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:49.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:49.679 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:49.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:49.780 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:49.802 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:49.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:49.904 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:49.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:49.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:50.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:50.048 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:50.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:50.172 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:50.202 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:50:50.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:50.295 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:50.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:50.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:50.418 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:50.428 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:50:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:50:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:50:50.429 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3786657257' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.429 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:50 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.510 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3786657257' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.510 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:50 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.510 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3786657257' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.511 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:50 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:50:50.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:50.541 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:50.643 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:50.664 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:50.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:50.787 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:50.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:50.909 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:50.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:50.973 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:50:50.973 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:50:51.011 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:51.033 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:51.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:51.155 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:51.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.279 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:51.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:51.286 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:51 smithi145 ceph-mon[89359]: pgmap v2698: 93 pgs: 93 active+clean; 395 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 4.2 KiB/s rd, 1 op/s 2024-09-21T10:50:51.286 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:51 smithi145 ceph-mon[89359]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:51.286 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:51 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:50:51.286 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:51 smithi145 ceph-mon[89359]: osdmap e1471: 8 total, 8 up, 5 in 2024-09-21T10:50:51.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[101432]: pgmap v2698: 93 pgs: 93 active+clean; 395 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 4.2 KiB/s rd, 1 op/s 2024-09-21T10:50:51.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[101432]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:51.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:50:51.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[101432]: osdmap e1471: 8 total, 8 up, 5 in 2024-09-21T10:50:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[105288]: pgmap v2698: 93 pgs: 93 active+clean; 395 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 4.2 KiB/s rd, 1 op/s 2024-09-21T10:50:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[105288]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:50:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:50:51.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:51 smithi119 ceph-mon[105288]: osdmap e1471: 8 total, 8 up, 5 in 2024-09-21T10:50:51.367 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:50:51.367 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:50:51.367 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:50:51.368 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:51.368 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:51.368 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:51.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:51.394 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:50:51.394 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:51.403 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:51.410 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:50:51.410 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:51.425 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:50:51.425 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:51.441 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:50:51.441 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:51.461 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:50:51.461 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:51.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:51.528 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:51.578 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T10:50:51.630 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:51.653 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:51.755 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:51.778 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:51.838 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:51.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:51.902 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:51.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:52.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:52.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:52.151 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:52.253 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:52.275 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:52.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:52.397 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:52.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:52.520 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:52.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1075604369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:52.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1075604369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:52.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1075604369' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:50:52.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:52.642 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:52.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:52.764 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:52.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:52.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:52.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:52.989 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.012 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:53.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:53.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:53.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:50:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.135 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:53.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:53.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:53.257 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:53.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:53.359 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:53.381 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:53.383 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:53.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:53.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:53 smithi119 ceph-mon[101432]: 1.10 scrub starts 2024-09-21T10:50:53.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:53 smithi119 ceph-mon[101432]: 1.10 scrub ok 2024-09-21T10:50:53.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:53 smithi119 ceph-mon[101432]: pgmap v2700: 93 pgs: 93 active+clean; 395 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 5.5 KiB/s rd, 1 op/s 2024-09-21T10:50:53.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:53 smithi119 ceph-mon[105288]: 1.10 scrub starts 2024-09-21T10:50:53.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:53 smithi119 ceph-mon[105288]: 1.10 scrub ok 2024-09-21T10:50:53.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:53 smithi119 ceph-mon[105288]: pgmap v2700: 93 pgs: 93 active+clean; 395 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 5.5 KiB/s rd, 1 op/s 2024-09-21T10:50:53.505 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:53.529 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:53 smithi145 ceph-mon[89359]: 1.10 scrub starts 2024-09-21T10:50:53.529 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:53 smithi145 ceph-mon[89359]: 1.10 scrub ok 2024-09-21T10:50:53.529 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:53 smithi145 ceph-mon[89359]: pgmap v2700: 93 pgs: 93 active+clean; 395 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 5.5 KiB/s rd, 1 op/s 2024-09-21T10:50:53.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:53.628 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:53.730 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:53.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:53.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:53.780 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:50:53.780 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:50:53.780 INFO:teuthology.orchestra.run.smithi145.stdout: "complaint_time": 30, 2024-09-21T10:50:53.781 INFO:teuthology.orchestra.run.smithi145.stdout: "num_blocked_ops": 0 2024-09-21T10:50:53.781 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:50:53.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:53.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:53.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:53.997 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:53.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:54.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:54.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:54.171 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T10:50:54.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:54.243 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:54.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:54.452 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T10:50:54.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:54.514 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:54.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:54.638 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:54.735 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:54.740 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:54.761 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:54.857 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T10:50:54.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:54.866 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:50:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:50:54] "GET /metrics HTTP/1.1" 200 34419 "" "Prometheus/2.51.0" 2024-09-21T10:50:54.886 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:54.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:54.903 INFO:tasks.workunit.client.0.smithi119.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T10:50:54.905 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T10:50:54.906 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T10:50:54.987 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:55.011 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:55.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:55.135 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:55.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:55.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:55.251 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6] out_osds: [4, 7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 7, 6, 4] 2024-09-21T10:50:55.251 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T10:50:55.252 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T10:50:55.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:50:55.258 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:55.287 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T10:50:55.360 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:55.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:55 smithi119 ceph-mon[101432]: pgmap v2701: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:55 smithi119 ceph-mon[105288]: pgmap v2701: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:55.383 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:55.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:55.457 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T10:50:55.485 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:55.501 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T10:50:55.502 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T10:50:55.503 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T10:50:55.505 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:55.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:55.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:55 smithi145 ceph-mon[89359]: pgmap v2701: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:55.547 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T10:50:55.567 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-21T10:50:55.567 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T10:50:55.593 INFO:tasks.workunit:Running workunits matching cls/test_cls_rbd.sh on client.0... 2024-09-21T10:50:55.594 INFO:tasks.workunit:Running workunit cls/test_cls_rbd.sh... 2024-09-21T10:50:55.594 DEBUG:teuthology.orchestra.run.smithi119:workunit test cls/test_cls_rbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2024-09-21T10:50:55.607 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:55.628 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:55.730 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:55.751 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:55.791 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:55.792 INFO:tasks.workunit.client.0.smithi119.stdout:Running main() from gmock_main.cc 2024-09-21T10:50:55.792 INFO:tasks.workunit.client.0.smithi119.stdout:[==========] Running 70 tests from 1 test suite. 2024-09-21T10:50:55.792 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] Global test environment set-up. 2024-09-21T10:50:55.792 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 70 tests from TestClsRbd 2024-09-21T10:50:55.830 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:50:55.853 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:55.874 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:55.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:55.996 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:55.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:56.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:56.119 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:56.216 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:50:56.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:56.249 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:56.351 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:56.372 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:56.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:56.495 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:56.512 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1452709236' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:56.512 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout: "size": 20, 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 600, 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [ 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_op(client.36978.0:57597 49.0 49:0fe7e42d:::benchmark_data_smithi119_301770_object57596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e1400)", 2024-09-21T10:50:56.517 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:42:57.380487+0000", 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 479.13384289300001, 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.29769361999999999, 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_info": { 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client": "client.36978", 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "client_addr": "172.21.15.119:0/1103834513", 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "tid": 57597 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.518 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380487+0000", 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380487+0000", 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380486+0000", 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.519 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380534+0000", 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8565e-05 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380545+0000", 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.077e-05 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380549+0000", 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9809999999999997e-06 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.520 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380586+0000", 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6424999999999998e-05 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380680+0000", 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.4672000000000004e-05 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "waiting for subops from 3,5", 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.380774+0000", 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.3926999999999997e-05 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.521 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "op_commit", 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.382469+0000", 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0016945420000000001 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.382759+0000", 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00029050000000000001 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "sub_op_commit_rec", 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.678119+0000", 2024-09-21T10:50:56.522 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.295359539 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.678155+0000", 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6708999999999997e-05 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:42:57.678181+0000", 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5190000000000001e-05 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.523 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:75765 50.3 e1440/1430 50:c927cfe2:::benchmark_data_smithi119_335063_object75764:head v 1440'4885, mlcod=1440'4883)", 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:47:45.342266+0000", 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 191.172064259, 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0085146719999999992, 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342266+0000", 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.524 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342266+0000", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342265+0000", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342295+0000", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0468999999999999e-05 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.525 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342310+0000", 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.4738e-05 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342318+0000", 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.7619999999999995e-06 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342369+0000", 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.1555999999999999e-05 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.526 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.342396+0000", 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.6359999999999998e-05 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.350732+0000", 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00833682 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.350780+0000", 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8050000000000002e-05 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.527 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:75766 50.3 e1440/1430 50:c2485188:::benchmark_data_smithi119_335063_object75765:head v 1440'4886, mlcod=1440'4883)", 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:47:45.343298+0000", 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 191.17103151200001, 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0075062760000000001, 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.343298+0000", 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.528 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.343298+0000", 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.343298+0000", 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.343386+0000", 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.8570000000000001e-05 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.529 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.343396+0000", 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.098e-06 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.343400+0000", 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.0339999999999998e-06 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.530 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.531 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.345190+0000", 2024-09-21T10:50:56.531 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0017907400000000001 2024-09-21T10:50:56.531 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.531 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.531 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.345212+0000", 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1784999999999999e-05 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.350785+0000", 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.005572512 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:45.350805+0000", 2024-09-21T10:50:56.532 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0142000000000001e-05 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:76720 50.9 e1440/1430 50:968dc8a9:::benchmark_data_smithi119_335063_object76719:head v 1440'4857, mlcod=1440'4855)", 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:47:46.741684+0000", 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 189.77264596800001, 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.011240470000000001, 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.533 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.741684+0000", 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.741684+0000", 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.741683+0000", 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.534 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.749024+0000", 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0073407439999999997 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.749031+0000", 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.041e-06 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.535 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.749035+0000", 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6119999999999999e-06 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.749084+0000", 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.8715999999999999e-05 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.749113+0000", 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9592000000000001e-05 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.536 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.752898+0000", 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037853650000000002 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:46.752924+0000", 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.5967999999999999e-05 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.537 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:82121 50.a e1440/1430 50:53e9b230:::benchmark_data_smithi119_335063_object82120:head v 1440'5086, mlcod=1440'5083)", 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:47:52.524851+0000", 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 183.98947899999999, 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083380529999999998, 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.538 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524851+0000", 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524851+0000", 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524850+0000", 2024-09-21T10:50:56.539 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524946+0000", 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.5365999999999995e-05 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524956+0000", 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0114e-05 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.540 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524960+0000", 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1930000000000003e-06 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.524999+0000", 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9172999999999999e-05 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.541 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.525020+0000", 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1186e-05 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.533149+0000", 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0081291169999999999 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.542 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:47:52.533189+0000", 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9675000000000002e-05 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:103744 50.8 e1445/1430 50:1ab8019d:::benchmark_data_smithi119_335063_object103743:head v 1445'6485, mlcod=1445'6480)", 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:48:21.187988+0000", 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 155.32634238200001, 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.008121329, 2024-09-21T10:50:56.543 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.187988+0000", 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.187988+0000", 2024-09-21T10:50:56.544 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.187987+0000", 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.189087+0000", 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0011002270000000001 2024-09-21T10:50:56.545 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.189100+0000", 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3062e-05 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.189107+0000", 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3899999999999998e-06 2024-09-21T10:50:56.546 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.189618+0000", 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.00051175299999999999 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.189641+0000", 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3040999999999998e-05 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.547 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.196049+0000", 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.006407739 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:48:21.196109+0000", 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.9800999999999999e-05 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.548 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:141544 50.8 e1450/1430 50:1f2d7af2:::benchmark_data_smithi119_335063_object36967:head v 1450'8806, mlcod=1450'8803)", 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:49:10.697372+0000", 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 105.816957583, 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0074417620000000002, 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697372+0000", 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697372+0000", 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697372+0000", 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697378+0000", 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.1419999999999998e-06 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.550 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697390+0000", 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.1907e-05 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697396+0000", 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.4749999999999998e-06 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697433+0000", 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6743999999999999e-05 2024-09-21T10:50:56.551 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.697453+0000", 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0579000000000001e-05 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.704781+0000", 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007327577 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:49:10.704814+0000", 2024-09-21T10:50:56.552 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.307e-05 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208545 50.c e1467/1430 50:3c1c1e42:::benchmark_data_smithi119_335063_object103968:head v 1467'13140, mlcod=1467'13138)", 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.263476+0000", 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.250853459999998, 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.037733442999999998, 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.553 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263476+0000", 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263476+0000", 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263475+0000", 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999981 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.554 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263487+0000", 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2173e-05 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263508+0000", 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1019999999999999e-05 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263518+0000", 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.7100000000000002e-06 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.555 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263586+0000", 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.8015000000000005e-05 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263618+0000", 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2283000000000002e-05 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.301168+0000", 2024-09-21T10:50:56.556 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.037549836000000003 2024-09-21T10:50:56.557 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.557 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.557 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.557 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.301210+0000", 2024-09-21T10:50:56.557 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.2089000000000003e-05 2024-09-21T10:50:56.557 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208543 50.8 e1467/1430 50:1f688fa1:::benchmark_data_smithi119_335063_object103966:head v 1467'13036, mlcod=1467'13034)", 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.263836+0000", 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.250493772999999, 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.043253332999999998, 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.558 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263836+0000", 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.559 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263836+0000", 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263835+0000", 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.560 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.263843+0000", 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2400000000000001e-06 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298615+0000", 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.034772714000000003 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298624+0000", 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.585e-06 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.561 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298659+0000", 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.5435999999999998e-05 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298678+0000", 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.844e-05 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307059+0000", 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0083810819999999998 2024-09-21T10:50:56.562 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307090+0000", 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.0522999999999999e-05 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208550 50.2 e1467/1463 50:491bb47c:::benchmark_data_smithi119_335063_object103973:head v 1467'12867, mlcod=1467'12865)", 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.264199+0000", 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.250130649999999, 2024-09-21T10:50:56.563 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.042177489999999998, 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.264199+0000", 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.264199+0000", 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.564 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.264199+0000", 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.264203+0000", 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.5220000000000001e-06 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298623+0000", 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.034419618999999999 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.565 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298630+0000", 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0090000000000001e-06 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298684+0000", 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3842999999999999e-05 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298704+0000", 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.9911e-05 2024-09-21T10:50:56.566 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306324+0000", 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076203119999999997 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306377+0000", 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.2710999999999997e-05 2024-09-21T10:50:56.567 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208547 50.8 e1467/1430 50:166952cd:::benchmark_data_smithi119_335063_object103970:head v 1467'13037, mlcod=1467'13034)", 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298639+0000", 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215690672000001, 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084724669999999992, 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.568 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298639+0000", 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298639+0000", 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298639+0000", 2024-09-21T10:50:56.569 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298645+0000", 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.5749999999999997e-06 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298653+0000", 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.8960000000000003e-06 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.570 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298657+0000", 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.1609999999999996e-06 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.301109+0000", 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.002451422 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.571 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.301126+0000", 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7646e-05 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307092+0000", 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0059652129999999996 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307112+0000", 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0183999999999999e-05 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.572 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208552 50.2 e1467/1463 50:42ac3025:::benchmark_data_smithi119_335063_object103975:head v 1467'12868, mlcod=1467'12865)", 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298660+0000", 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215669705, 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.007761591, 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.573 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.574 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.574 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.574 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.574 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298660+0000", 2024-09-21T10:50:56.574 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.574 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298660+0000", 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.575 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.576 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298660+0000", 2024-09-21T10:50:56.576 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.576 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.576 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298665+0000", 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7200000000000003e-06 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298674+0000", 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 8.3149999999999992e-06 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.577 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298678+0000", 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9899999999999999e-06 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.301017+0000", 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0023391620000000001 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.578 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.301035+0000", 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8291000000000002e-05 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306388+0000", 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0053527510000000002 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306422+0000", 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3853000000000003e-05 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.579 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208557 50.8 e1467/1430 50:102f4780:::benchmark_data_smithi119_335063_object103980:head v 1467'13038, mlcod=1467'13034)", 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298675+0000", 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215654961999999, 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084595539999999993, 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.580 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298675+0000", 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298675+0000", 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.581 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298675+0000", 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298679+0000", 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9330000000000003e-06 2024-09-21T10:50:56.582 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298684+0000", 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.7429999999999999e-06 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298689+0000", 2024-09-21T10:50:56.583 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.9389999999999996e-06 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.303179+0000", 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044902370000000002 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.303192+0000", 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.2493999999999999e-05 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.584 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307115+0000", 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039225429999999997 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307135+0000", 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0040000000000001e-05 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208560 50.2 e1467/1463 50:40e879f7:::benchmark_data_smithi119_335063_object103983:head v 1467'12869, mlcod=1467'12865)", 2024-09-21T10:50:56.585 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298683+0000", 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215647346000001, 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077634679999999999, 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298683+0000", 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.586 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298683+0000", 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298682+0000", 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298686+0000", 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.6880000000000001e-06 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.587 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298692+0000", 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.3099999999999997e-06 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298696+0000", 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.7809999999999999e-06 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.302493+0000", 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0037967669999999999 2024-09-21T10:50:56.588 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.302514+0000", 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.0967000000000001e-05 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306425+0000", 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039107530000000003 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306446+0000", 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.1539000000000002e-05 2024-09-21T10:50:56.589 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208558 50.0 e1467/1453 50:000af4d1:::benchmark_data_smithi119_335063_object103981:head v 1467'13088, mlcod=1467'13086)", 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298693+0000", 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215636477, 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084605159999999995, 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.590 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298693+0000", 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298693+0000", 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298693+0000", 2024-09-21T10:50:56.591 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298700+0000", 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.9310000000000002e-06 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298705+0000", 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.3129999999999998e-06 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.592 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298709+0000", 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3289999999999998e-06 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.303137+0000", 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0044286630000000002 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.303154+0000", 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.6742999999999999e-05 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.593 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307136+0000", 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039816399999999998 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307154+0000", 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8148e-05 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208561 50.0 e1467/1453 50:081840b8:::benchmark_data_smithi119_335063_object103984:head v 1467'13089, mlcod=1467'13086)", 2024-09-21T10:50:56.594 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298713+0000", 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215617414, 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0084908459999999998, 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298713+0000", 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.595 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298713+0000", 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298712+0000", 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967296 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.596 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298716+0000", 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4359999999999998e-06 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298721+0000", 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.7609999999999998e-06 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298724+0000", 2024-09-21T10:50:56.597 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.3589999999999999e-06 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.305291+0000", 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0065666120000000003 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.305304+0000", 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3183000000000001e-05 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.598 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307185+0000", 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.001881237 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.307203+0000", 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.8467000000000001e-05 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.599 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208553 50.a e1467/1430 50:5995b19c:::benchmark_data_smithi119_335063_object103976:head v 1467'12911, mlcod=1467'12909)", 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298795+0000", 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215534676000001, 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0076759339999999997, 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298795+0000", 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.600 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298795+0000", 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298795+0000", 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999995 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298798+0000", 2024-09-21T10:50:56.601 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.2270000000000002e-06 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298803+0000", 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.4190000000000004e-06 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298805+0000", 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.4049999999999998e-06 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.602 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.302478+0000", 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.003672809 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.302488+0000", 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0608000000000001e-05 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306448+0000", 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0039590830000000004 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.603 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306471+0000", 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3702000000000001e-05 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.604 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "osd_repop(client.37269.0:208567 50.a e1467/1430 50:543b6b51:::benchmark_data_smithi119_335063_object103990:head v 1467'12912, mlcod=1467'12909)", 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:34.298854+0000", 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 22.215475847, 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0077067330000000003, 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298854+0000", 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.605 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298854+0000", 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298854+0000", 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967296 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.606 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298857+0000", 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.9809999999999999e-06 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298862+0000", 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.0289999999999996e-06 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.298865+0000", 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.1489999999999998e-06 2024-09-21T10:50:56.607 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.304304+0000", 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0054391869999999998 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.304328+0000", 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3430000000000001e-05 2024-09-21T10:50:56.608 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "commit_sent", 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306537+0000", 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0.0022094079999999999 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.609 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:34.306561+0000", 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.3779000000000001e-05 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "MOSDScrubReserve(1.10 GRANT e1471)", 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:51.564981+0000", 2024-09-21T10:50:56.610 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.9493493669999999, 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.0729000000000005e-05, 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.564981+0000", 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.564981+0000", 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.611 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.564979+0000", 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.9999986 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.612 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.564986+0000", 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.0330000000000003e-06 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565001+0000", 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5063e-05 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.613 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565009+0000", 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 7.2339999999999997e-06 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565048+0000", 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.9541e-05 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.614 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565065+0000", 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.7178999999999999e-05 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565067+0000", 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.5540000000000001e-06 2024-09-21T10:50:56.615 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565071+0000", 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4.408e-06 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.616 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "description": "rep_scrubmap(1.10 e1469 from shard 2)", 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "initiated_at": "2024-09-21T10:50:51.565853+0000", 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "age": 4.9484765529999999, 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 9.1540999999999997e-05, 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "type_data": { 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "flag_point": "started", 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "events": [ 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "initiated", 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565853+0000", 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.617 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "throttled", 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565853+0000", 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 0 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "header_read", 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565853+0000", 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 4294967295.999999 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.618 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "all_read", 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565858+0000", 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 5.6380000000000001e-06 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "dispatched", 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565871+0000", 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.3305999999999999e-05 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "queued_for_pg", 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565878+0000", 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 6.0680000000000002e-06 2024-09-21T10:50:56.619 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "reached_pg", 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565912+0000", 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 3.4715000000000002e-05 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "started", 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565923+0000", 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 1.0607e-05 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: { 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "event": "done", 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "time": "2024-09-21T10:50:51.565945+0000", 2024-09-21T10:50:56.620 INFO:teuthology.orchestra.run.smithi145.stdout: "duration": 2.2089000000000001e-05 2024-09-21T10:50:56.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.621 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.621 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-21T10:50:56.621 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-09-21T10:50:56.621 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:50:56.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:56.622 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1452709236' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:56.622 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:56.622 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1452709236' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:56.622 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:50:56.642 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:56.744 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:50:56.765 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:50:56.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:56.888 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:56.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:56.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:57.011 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:57.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:57.136 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:57.218 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_all_features 2024-09-21T10:50:57.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:57.251 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_all_features (33 ms) 2024-09-21T10:50:57.251 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.copyup 2024-09-21T10:50:57.263 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:57.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:57.386 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:57.414 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T10:50:57.425 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.copyup (174 ms) 2024-09-21T10:50:57.425 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.sparse_copyup 2024-09-21T10:50:57.449 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.sparse_copyup (24 ms) 2024-09-21T10:50:57.449 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_and_set_id 2024-09-21T10:50:57.457 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_and_set_id (8 ms) 2024-09-21T10:50:57.457 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.add_remove_child 2024-09-21T10:50:57.468 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.add_remove_child (11 ms) 2024-09-21T10:50:57.468 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.directory_methods 2024-09-21T10:50:57.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:57.497 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.directory_methods (29 ms) 2024-09-21T10:50:57.497 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.create 2024-09-21T10:50:57.511 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.create (14 ms) 2024-09-21T10:50:57.512 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_features 2024-09-21T10:50:57.513 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.513 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_features (2 ms) 2024-09-21T10:50:57.513 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_object_prefix 2024-09-21T10:50:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:57.515 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_object_prefix (1 ms) 2024-09-21T10:50:57.515 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_create_timestamp 2024-09-21T10:50:57.516 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_create_timestamp (2 ms) 2024-09-21T10:50:57.516 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_access_timestamp 2024-09-21T10:50:57.517 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_access_timestamp (1 ms) 2024-09-21T10:50:57.517 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_modify_timestamp 2024-09-21T10:50:57.518 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_modify_timestamp (1 ms) 2024-09-21T10:50:57.519 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_data_pool 2024-09-21T10:50:57.522 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_data_pool (4 ms) 2024-09-21T10:50:57.522 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.get_size 2024-09-21T10:50:57.526 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.get_size (4 ms) 2024-09-21T10:50:57.526 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.set_size 2024-09-21T10:50:57.530 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.set_size (4 ms) 2024-09-21T10:50:57.530 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.protection_status 2024-09-21T10:50:57.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:57 smithi145 ceph-mon[89359]: pgmap v2702: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:57 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:50:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:57 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:50:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:57 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:57 smithi145 ceph-mon[89359]: osdmap e1472: 8 total, 8 up, 5 in 2024-09-21T10:50:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:57 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4139774503' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-361873-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:50:57.545 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.protection_status (15 ms) 2024-09-21T10:50:57.546 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.snapshot_limits 2024-09-21T10:50:57.552 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.snapshot_limits (7 ms) 2024-09-21T10:50:57.552 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.parents_v1 2024-09-21T10:50:57.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[105288]: pgmap v2702: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:57.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[105288]: osdmap e1472: 8 total, 8 up, 5 in 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4139774503' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-361873-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[101432]: pgmap v2702: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[101432]: osdmap e1472: 8 total, 8 up, 5 in 2024-09-21T10:50:57.583 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:57 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4139774503' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-361873-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-21T10:50:57.592 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.parents_v1 (40 ms) 2024-09-21T10:50:57.592 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.parents_v2 2024-09-21T10:50:57.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:57.621 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.parents_v2 (29 ms) 2024-09-21T10:50:57.621 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.snapshots 2024-09-21T10:50:57.641 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:57.646 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.snapshots (25 ms) 2024-09-21T10:50:57.647 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.snapid_race 2024-09-21T10:50:57.653 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.snapid_race (7 ms) 2024-09-21T10:50:57.653 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.stripingv2 2024-09-21T10:50:57.662 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.stripingv2 (9 ms) 2024-09-21T10:50:57.662 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.object_map_save 2024-09-21T10:50:57.663 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:50:57.666 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.object_map_save (4 ms) 2024-09-21T10:50:57.666 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.object_map_resize 2024-09-21T10:50:57.671 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.object_map_resize (5 ms) 2024-09-21T10:50:57.671 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.object_map_update 2024-09-21T10:50:57.678 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.object_map_update (7 ms) 2024-09-21T10:50:57.678 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.object_map_load_enoent 2024-09-21T10:50:57.678 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.object_map_load_enoent (0 ms) 2024-09-21T10:50:57.679 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.object_map_snap_add 2024-09-21T10:50:57.683 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.object_map_snap_add (5 ms) 2024-09-21T10:50:57.683 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.object_map_snap_remove 2024-09-21T10:50:57.692 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.object_map_snap_remove (9 ms) 2024-09-21T10:50:57.693 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.flags 2024-09-21T10:50:57.697 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.flags (5 ms) 2024-09-21T10:50:57.697 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.metadata 2024-09-21T10:50:57.719 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.metadata (22 ms) 2024-09-21T10:50:57.720 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.set_features 2024-09-21T10:50:57.726 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.set_features (7 ms) 2024-09-21T10:50:57.726 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.mirror 2024-09-21T10:50:57.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:57.772 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:57.773 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.mirror (47 ms) 2024-09-21T10:50:57.773 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.mirror_image 2024-09-21T10:50:57.791 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.mirror_image (18 ms) 2024-09-21T10:50:57.791 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.mirror_image_status 2024-09-21T10:50:57.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:50:57.895 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:57.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:57.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:50:57.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:58.017 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:58.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:58.119 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:58.140 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:58.242 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:58.268 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:58.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:58.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:58 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4139774503' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-361873-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:50:58.313 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:58 smithi145 ceph-mon[89359]: osdmap e1473: 8 total, 8 up, 5 in 2024-09-21T10:50:58.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:58.395 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:58.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:58.521 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:58.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:58 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4139774503' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-361873-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:50:58.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:58 smithi119 ceph-mon[105288]: osdmap e1473: 8 total, 8 up, 5 in 2024-09-21T10:50:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:58 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4139774503' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-361873-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-21T10:50:58.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:58 smithi119 ceph-mon[101432]: osdmap e1473: 8 total, 8 up, 5 in 2024-09-21T10:50:58.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:58.649 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:58.751 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:58.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:50:58.904 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:58.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:50:59.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:59.031 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:50:59.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.158 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:59.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.284 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.386 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:50:59.387 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:59 smithi145 ceph-mon[89359]: pgmap v2705: 125 pgs: 6 creating+peering, 26 unknown, 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:59.387 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:50:59 smithi145 ceph-mon[89359]: osdmap e1474: 8 total, 8 up, 5 in 2024-09-21T10:50:59.409 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:50:59.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T10:50:59.464 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:59 smithi119 ceph-mon[101432]: pgmap v2705: 125 pgs: 6 creating+peering, 26 unknown, 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:59.464 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:50:59 smithi119 ceph-mon[101432]: osdmap e1474: 8 total, 8 up, 5 in 2024-09-21T10:50:59.464 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:59 smithi119 ceph-mon[105288]: pgmap v2705: 125 pgs: 6 creating+peering, 26 unknown, 93 active+clean; 369 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail 2024-09-21T10:50:59.465 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:50:59 smithi119 ceph-mon[105288]: osdmap e1474: 8 total, 8 up, 5 in 2024-09-21T10:50:59.495 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-21T10:50:59.495 INFO:teuthology.orchestra.run.smithi145.stdout: "ops": [], 2024-09-21T10:50:59.495 INFO:teuthology.orchestra.run.smithi145.stdout: "num_ops": 0 2024-09-21T10:50:59.495 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-21T10:50:59.511 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:50:59.537 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:50:59.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:50:59.661 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:50:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:50:59.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:50:59.790 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.0: {} 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.1: {} 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.2: {} 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.3: {} 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.4: {} 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.5: {} 2024-09-21T10:50:59.872 INFO:teuthology.orchestra.run.smithi119.stdout:osd.6: {} 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T10:50:59.873 INFO:teuthology.orchestra.run.smithi119.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T10:50:59.887 INFO:teuthology.orchestra.run.smithi119.stdout:osd.7: {} 2024-09-21T10:50:59.887 INFO:teuthology.orchestra.run.smithi119.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T10:50:59.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:50:59.920 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:50:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:50:59.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:00.022 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:00.043 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:00.140 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T10:51:00.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:00.203 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:00.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:00.330 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:00.387 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:51:00.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:00.455 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:00.507 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3928695260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:00.508 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:51:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:51:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:51:00.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:00.562 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.mirror_image_status (2771 ms) 2024-09-21T10:51:00.562 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.mirror_image_map 2024-09-21T10:51:00.579 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:00.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3928695260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3928695260' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-21T10:51:00.646 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.mirror_image_map (84 ms) 2024-09-21T10:51:00.646 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.mirror_instances 2024-09-21T10:51:00.654 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.mirror_instances (8 ms) 2024-09-21T10:51:00.654 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.mirror_snapshot 2024-09-21T10:51:00.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.703 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:00.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:00.769 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.mirror_snapshot (114 ms) 2024-09-21T10:51:00.769 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_dir_list 2024-09-21T10:51:00.771 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_dir_list (2 ms) 2024-09-21T10:51:00.771 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_dir_add 2024-09-21T10:51:00.772 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_dir_add (2 ms) 2024-09-21T10:51:00.772 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.dir_add_already_existing 2024-09-21T10:51:00.775 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.dir_add_already_existing (3 ms) 2024-09-21T10:51:00.775 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_dir_rename 2024-09-21T10:51:00.780 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_dir_rename (5 ms) 2024-09-21T10:51:00.781 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_dir_remove 2024-09-21T10:51:00.783 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_dir_remove (3 ms) 2024-09-21T10:51:00.783 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_dir_remove_missing 2024-09-21T10:51:00.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_dir_remove_missing (4 ms) 2024-09-21T10:51:00.788 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_image_add 2024-09-21T10:51:00.789 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_image_add (2 ms) 2024-09-21T10:51:00.789 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_image_remove 2024-09-21T10:51:00.793 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_image_remove (3 ms) 2024-09-21T10:51:00.793 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_image_list 2024-09-21T10:51:00.794 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_image_list (2 ms) 2024-09-21T10:51:00.794 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_image_clean 2024-09-21T10:51:00.799 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_image_clean (4 ms) 2024-09-21T10:51:00.799 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.image_group_add 2024-09-21T10:51:00.801 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.image_group_add (2 ms) 2024-09-21T10:51:00.801 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.image_group_remove 2024-09-21T10:51:00.804 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5 2024-09-21T10:51:00.805 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.image_group_remove (2 ms) 2024-09-21T10:51:00.805 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.image_group_get 2024-09-21T10:51:00.807 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.image_group_get (2 ms) 2024-09-21T10:51:00.807 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_name 2024-09-21T10:51:00.809 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_set_empty_name (2 ms) 2024-09-21T10:51:00.809 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_set_empty_id 2024-09-21T10:51:00.810 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_set_empty_id (1 ms) 2024-09-21T10:51:00.810 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_id 2024-09-21T10:51:00.813 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_id (3 ms) 2024-09-21T10:51:00.813 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_set_duplicate_name 2024-09-21T10:51:00.815 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_set_duplicate_name (2 ms) 2024-09-21T10:51:00.815 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_set 2024-09-21T10:51:00.817 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_set (2 ms) 2024-09-21T10:51:00.817 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_list 2024-09-21T10:51:00.822 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_list (3 ms) 2024-09-21T10:51:00.822 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_list_max_return 2024-09-21T10:51:00.829 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.5' 2024-09-21T10:51:00.835 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_list_max_return (15 ms) 2024-09-21T10:51:00.835 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_list_max_read 2024-09-21T10:51:00.931 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:00.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_list_max_read (192 ms) 2024-09-21T10:51:01.028 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_remove 2024-09-21T10:51:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:/home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.7-1649-gfbd1a6ab/rpm/el9/BUILD/ceph-17.2.7-1649-gfbd1a6ab/src/test/cls_rbd/test_cls_rbd.cc:2792: Failure 2024-09-21T10:51:01.029 INFO:tasks.workunit.client.0.smithi119.stdout:Expected equality of these values: 2024-09-21T10:51:01.030 INFO:tasks.workunit.client.0.smithi119.stdout: 1U 2024-09-21T10:51:01.030 INFO:tasks.workunit.client.0.smithi119.stdout: Which is: 1 2024-09-21T10:51:01.030 INFO:tasks.workunit.client.0.smithi119.stdout: keys.size() 2024-09-21T10:51:01.030 INFO:tasks.workunit.client.0.smithi119.stdout: Which is: 3 2024-09-21T10:51:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:[ FAILED ] TestClsRbd.group_snap_remove (2 ms) 2024-09-21T10:51:01.030 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.group_snap_get_by_id 2024-09-21T10:51:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.group_snap_get_by_id (2 ms) 2024-09-21T10:51:01.031 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.trash_methods 2024-09-21T10:51:01.037 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.trash_methods (6 ms) 2024-09-21T10:51:01.037 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.op_features 2024-09-21T10:51:01.044 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.op_features (7 ms) 2024-09-21T10:51:01.044 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.clone_parent 2024-09-21T10:51:01.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:01.063 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.clone_parent (19 ms) 2024-09-21T10:51:01.064 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.clone_parent_ns 2024-09-21T10:51:01.071 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.clone_parent_ns (8 ms) 2024-09-21T10:51:01.071 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.clone_child 2024-09-21T10:51:01.079 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:01.082 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.clone_child (11 ms) 2024-09-21T10:51:01.082 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.namespace_methods 2024-09-21T10:51:01.088 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.namespace_methods (6 ms) 2024-09-21T10:51:01.088 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.migration 2024-09-21T10:51:01.098 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.migration (10 ms) 2024-09-21T10:51:01.098 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.migration_v1 2024-09-21T10:51:01.109 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.migration_v1 (11 ms) 2024-09-21T10:51:01.109 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.assert_snapc_seq 2024-09-21T10:51:01.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:01.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:01.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:01.325 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:01.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:01 smithi119 ceph-mon[101432]: pgmap v2707: 125 pgs: 6 creating+peering, 3 unknown, 116 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 307 KiB/s rd, 173 KiB/s wr, 497 op/s 2024-09-21T10:51:01.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:01 smithi119 ceph-mon[105288]: pgmap v2707: 125 pgs: 6 creating+peering, 3 unknown, 116 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 307 KiB/s rd, 173 KiB/s wr, 497 op/s 2024-09-21T10:51:01.426 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:01.448 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:01 smithi145 ceph-mon[89359]: pgmap v2707: 125 pgs: 6 creating+peering, 3 unknown, 116 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 307 KiB/s rd, 173 KiB/s wr, 497 op/s 2024-09-21T10:51:01.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2 2024-09-21T10:51:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.572 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:01.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.2' 2024-09-21T10:51:01.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:01.694 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:01.796 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:01.817 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:01.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:01.919 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:01.941 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:01.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:01.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:01.943 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:02.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:02.052 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:51:02.052 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:51:02.064 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:02.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:02.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:02.188 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:02.253 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.assert_snapc_seq (1144 ms) 2024-09-21T10:51:02.253 INFO:tasks.workunit.client.0.smithi119.stdout:[ RUN ] TestClsRbd.sparsify 2024-09-21T10:51:02.289 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:02.294 INFO:tasks.workunit.client.0.smithi119.stdout:[ OK ] TestClsRbd.sparsify (41 ms) 2024-09-21T10:51:02.311 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:02.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:02.434 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:02 smithi145 ceph-mon[89359]: osdmap e1475: 8 total, 8 up, 5 in 2024-09-21T10:51:02.435 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:02.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:02 smithi119 ceph-mon[105288]: osdmap e1475: 8 total, 8 up, 5 in 2024-09-21T10:51:02.504 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:02 smithi119 ceph-mon[101432]: osdmap e1475: 8 total, 8 up, 5 in 2024-09-21T10:51:02.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0 2024-09-21T10:51:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:02.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.0' 2024-09-21T10:51:02.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6 2024-09-21T10:51:02.682 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.6' 2024-09-21T10:51:02.721 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T10:51:02.783 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7 2024-09-21T10:51:02.826 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:02.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.7' 2024-09-21T10:51:02.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1 2024-09-21T10:51:02.949 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:02.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.1' 2024-09-21T10:51:03.007 INFO:teuthology.orchestra.run.smithi145.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T10:51:03.050 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:03.071 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:03.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:03.195 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] 70 tests from TestClsRbd (5076 ms total) 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout:[----------] Global test environment tear-down 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout:[==========] 70 tests from 1 test suite ran. (7468 ms total) 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout:[ PASSED ] 69 tests. 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout:[ FAILED ] 1 test, listed below: 2024-09-21T10:51:03.260 INFO:tasks.workunit.client.0.smithi119.stdout:[ FAILED ] TestClsRbd.group_snap_remove 2024-09-21T10:51:03.261 INFO:tasks.workunit.client.0.smithi119.stdout: 2024-09-21T10:51:03.261 INFO:tasks.workunit.client.0.smithi119.stdout: 1 FAILED TEST 2024-09-21T10:51:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.263 INFO:tasks.workunit:Stopping ['cls/test_cls_rbd.sh'] on client.0... 2024-09-21T10:51:03.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T10:51:03.296 DEBUG:teuthology.orchestra.run.smithi145:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4 2024-09-21T10:51:03.318 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: docker: command not found 2024-09-21T10:51:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi145 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.4' 2024-09-21T10:51:03.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3 2024-09-21T10:51:03.440 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: docker: command not found 2024-09-21T10:51:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T10:51:03.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi119 with status 1: 'sudo docker kill -s 1 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd.3' 2024-09-21T10:51:03.458 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:03 smithi145 ceph-mon[89359]: pgmap v2709: 125 pgs: 125 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 1.7 MiB/s rd, 883 KiB/s wr, 562 op/s 2024-09-21T10:51:03.458 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:03 smithi145 ceph-mon[89359]: osdmap e1476: 8 total, 8 up, 5 in 2024-09-21T10:51:03.458 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:03.458 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[105288]: pgmap v2709: 125 pgs: 125 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 1.7 MiB/s rd, 883 KiB/s wr, 562 op/s 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[105288]: osdmap e1476: 8 total, 8 up, 5 in 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[101432]: pgmap v2709: 125 pgs: 125 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 1.7 MiB/s rd, 883 KiB/s wr, 562 op/s 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[101432]: osdmap e1476: 8 total, 8 up, 5 in 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:03.469 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:03.470 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T10:51:04.269 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is set 2024-09-21T10:51:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-mon[101432]: osdmap e1477: 8 total, 8 up, 5 in 2024-09-21T10:51:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1942530091' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:04.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-mon[105288]: osdmap e1477: 8 total, 8 up, 5 in 2024-09-21T10:51:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1942530091' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:04.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:04.478 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:51:04.479 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:51:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:04 smithi145 ceph-mon[89359]: osdmap e1477: 8 total, 8 up, 5 in 2024-09-21T10:51:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1942530091' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:04 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-21T10:51:04.796 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:51:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:51:04] "GET /metrics HTTP/1.1" 200 36835 "" "Prometheus/2.51.0" 2024-09-21T10:51:04.834 INFO:teuthology.orchestra.run.smithi145.stdout:ERROR: (22) Invalid argument 2024-09-21T10:51:04.834 INFO:teuthology.orchestra.run.smithi145.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[105288]: pgmap v2712: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[105288]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[105288]: osdmap e1478: 8 total, 8 up, 5 in 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[105288]: from='client.47393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[101432]: pgmap v2712: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[101432]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:05.523 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:51:05.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[101432]: osdmap e1478: 8 total, 8 up, 5 in 2024-09-21T10:51:05.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[101432]: from='client.47393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:51:05.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:05 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:51:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:05 smithi145 ceph-mon[89359]: pgmap v2712: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:05 smithi145 ceph-mon[89359]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T10:51:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:05 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T10:51:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:05 smithi145 ceph-mon[89359]: osdmap e1478: 8 total, 8 up, 5 in 2024-09-21T10:51:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:05 smithi145 ceph-mon[89359]: from='client.47393 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:51:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:05 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:51:05.548 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2713,"stamp":"2024-09-21T10:51:04.262049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":377464,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":920008,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2158,"ondisk_log_size":2158,"up":279,"acting":279,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":365,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":5767068,"kb_used_data":49536,"kb_used_omap":504,"kb_used_meta":5716871,"kb_avail":462958692,"statfs":{"total":479975178240,"available":474069700608,"internally_reserved":0,"allocated":50724864,"data_stored":28824045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":516241,"internal_metadata":5854076783},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-294,"num_objects":-26,"num_object_clones":0,"num_object_copies":-78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-26,"num_whiteouts":0,"num_read":-1920,"num_read_kb":-1845,"num_write":-1066,"num_write_kb":-1037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.176589"},"pg_stats":[{"pgid":"3.1f","version":"64'4","reported_seq":254,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.251911+0000","last_change":"2024-09-21T10:48:08.207623+0000","last_active":"2024-09-21T10:51:02.251911+0000","last_peered":"2024-09-21T10:51:02.251911+0000","last_clean":"2024-09-21T10:51:02.251911+0000","last_became_active":"2024-09-21T10:48:08.152668+0000","last_became_peered":"2024-09-21T10:48:08.152668+0000","last_unstale":"2024-09-21T10:51:02.251911+0000","last_undegraded":"2024-09-21T10:51:02.251911+0000","last_fullsized":"2024-09-21T10:51:02.251911+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:55:46.388407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1710,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1318,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.243942+0000","last_change":"2024-09-21T10:50:47.242525+0000","last_active":"2024-09-21T10:51:01.243942+0000","last_peered":"2024-09-21T10:51:01.243942+0000","last_clean":"2024-09-21T10:51:01.243942+0000","last_became_active":"2024-09-21T10:50:47.242303+0000","last_became_peered":"2024-09-21T10:50:47.242303+0000","last_unstale":"2024-09-21T10:51:01.243942+0000","last_undegraded":"2024-09-21T10:51:01.243942+0000","last_fullsized":"2024-09-21T10:51:01.243942+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:15:52.394776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2274,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233814+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:50:58.233814+0000","last_peered":"2024-09-21T10:50:58.233814+0000","last_clean":"2024-09-21T10:50:58.233814+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:50:58.233814+0000","last_undegraded":"2024-09-21T10:50:58.233814+0000","last_fullsized":"2024-09-21T10:50:58.233814+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2292,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232283+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:50:58.232283+0000","last_peered":"2024-09-21T10:50:58.232283+0000","last_clean":"2024-09-21T10:50:58.232283+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:50:58.232283+0000","last_undegraded":"2024-09-21T10:50:58.232283+0000","last_fullsized":"2024-09-21T10:50:58.232283+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":54,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244796+0000","last_change":"2024-09-21T10:50:47.216222+0000","last_active":"2024-09-21T10:51:01.244796+0000","last_peered":"2024-09-21T10:51:01.244796+0000","last_clean":"2024-09-21T10:51:01.244796+0000","last_became_active":"2024-09-21T10:50:47.215600+0000","last_became_peered":"2024-09-21T10:50:47.215600+0000","last_unstale":"2024-09-21T10:51:01.244796+0000","last_undegraded":"2024-09-21T10:51:01.244796+0000","last_fullsized":"2024-09-21T10:51:01.244796+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:22:24.211405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1475,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232500+0000","last_change":"2024-09-21T10:36:21.850729+0000","last_active":"2024-09-21T10:50:58.232500+0000","last_peered":"2024-09-21T10:50:58.232500+0000","last_clean":"2024-09-21T10:50:58.232500+0000","last_became_active":"2024-09-21T10:36:21.849462+0000","last_became_peered":"2024-09-21T10:36:21.849462+0000","last_unstale":"2024-09-21T10:50:58.232500+0000","last_undegraded":"2024-09-21T10:50:58.232500+0000","last_fullsized":"2024-09-21T10:50:58.232500+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:55:24.361849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":113,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448654+0000","last_change":"2024-09-21T10:48:08.196786+0000","last_active":"2024-09-21T10:50:58.448654+0000","last_peered":"2024-09-21T10:50:58.448654+0000","last_clean":"2024-09-21T10:50:58.448654+0000","last_became_active":"2024-09-21T10:48:08.161165+0000","last_became_peered":"2024-09-21T10:48:08.161165+0000","last_unstale":"2024-09-21T10:50:58.448654+0000","last_undegraded":"2024-09-21T10:50:58.448654+0000","last_fullsized":"2024-09-21T10:50:58.448654+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:35:51.828935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1569,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233391+0000","last_change":"2024-09-21T10:48:08.171396+0000","last_active":"2024-09-21T10:50:58.233391+0000","last_peered":"2024-09-21T10:50:58.233391+0000","last_clean":"2024-09-21T10:50:58.233391+0000","last_became_active":"2024-09-21T10:48:08.165828+0000","last_became_peered":"2024-09-21T10:48:08.165828+0000","last_unstale":"2024-09-21T10:50:58.233391+0000","last_undegraded":"2024-09-21T10:50:58.233391+0000","last_fullsized":"2024-09-21T10:50:58.233391+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:17.505484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":101,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227096+0000","last_change":"2024-09-21T10:46:39.241906+0000","last_active":"2024-09-21T10:50:58.227096+0000","last_peered":"2024-09-21T10:50:58.227096+0000","last_clean":"2024-09-21T10:50:58.227096+0000","last_became_active":"2024-09-21T10:46:39.241499+0000","last_became_peered":"2024-09-21T10:46:39.241499+0000","last_unstale":"2024-09-21T10:50:58.227096+0000","last_undegraded":"2024-09-21T10:50:58.227096+0000","last_fullsized":"2024-09-21T10:50:58.227096+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:04:53.671658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":259,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227412+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:50:58.227412+0000","last_peered":"2024-09-21T10:50:58.227412+0000","last_clean":"2024-09-21T10:50:58.227412+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:50:58.227412+0000","last_undegraded":"2024-09-21T10:50:58.227412+0000","last_fullsized":"2024-09-21T10:50:58.227412+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":334,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448334+0000","last_change":"2024-09-21T10:36:21.858345+0000","last_active":"2024-09-21T10:50:58.448334+0000","last_peered":"2024-09-21T10:50:58.448334+0000","last_clean":"2024-09-21T10:50:58.448334+0000","last_became_active":"2024-09-21T10:36:21.857537+0000","last_became_peered":"2024-09-21T10:36:21.857537+0000","last_unstale":"2024-09-21T10:50:58.448334+0000","last_undegraded":"2024-09-21T10:50:58.448334+0000","last_fullsized":"2024-09-21T10:50:58.448334+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:12:05.428524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2281,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233233+0000","last_change":"2024-09-21T10:36:21.851122+0000","last_active":"2024-09-21T10:50:58.233233+0000","last_peered":"2024-09-21T10:50:58.233233+0000","last_clean":"2024-09-21T10:50:58.233233+0000","last_became_active":"2024-09-21T10:36:21.850376+0000","last_became_peered":"2024-09-21T10:36:21.850376+0000","last_unstale":"2024-09-21T10:50:58.233233+0000","last_undegraded":"2024-09-21T10:50:58.233233+0000","last_fullsized":"2024-09-21T10:50:58.233233+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:47:50.837843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":101,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.226811+0000","last_change":"2024-09-21T10:46:39.241669+0000","last_active":"2024-09-21T10:50:58.226811+0000","last_peered":"2024-09-21T10:50:58.226811+0000","last_clean":"2024-09-21T10:50:58.226811+0000","last_became_active":"2024-09-21T10:46:39.241358+0000","last_became_peered":"2024-09-21T10:46:39.241358+0000","last_unstale":"2024-09-21T10:50:58.226811+0000","last_undegraded":"2024-09-21T10:50:58.226811+0000","last_fullsized":"2024-09-21T10:50:58.226811+0000","mapping_epoch":1429,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1430,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:01:25.671265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2401,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.253361+0000","last_change":"2024-09-21T10:45:59.689522+0000","last_active":"2024-09-21T10:51:02.253361+0000","last_peered":"2024-09-21T10:51:02.253361+0000","last_clean":"2024-09-21T10:51:02.253361+0000","last_became_active":"2024-09-21T10:45:59.689148+0000","last_became_peered":"2024-09-21T10:45:59.689148+0000","last_unstale":"2024-09-21T10:51:02.253361+0000","last_undegraded":"2024-09-21T10:51:02.253361+0000","last_fullsized":"2024-09-21T10:51:02.253361+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:28.142643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2307,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232271+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:50:58.232271+0000","last_peered":"2024-09-21T10:50:58.232271+0000","last_clean":"2024-09-21T10:50:58.232271+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:50:58.232271+0000","last_undegraded":"2024-09-21T10:50:58.232271+0000","last_fullsized":"2024-09-21T10:50:58.232271+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1382,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.245017+0000","last_change":"2024-09-21T10:50:47.198976+0000","last_active":"2024-09-21T10:51:01.245017+0000","last_peered":"2024-09-21T10:51:01.245017+0000","last_clean":"2024-09-21T10:51:01.245017+0000","last_became_active":"2024-09-21T10:50:47.198719+0000","last_became_peered":"2024-09-21T10:50:47.198719+0000","last_unstale":"2024-09-21T10:51:01.245017+0000","last_undegraded":"2024-09-21T10:51:01.245017+0000","last_fullsized":"2024-09-21T10:51:01.245017+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:45:45.443412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2287,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232510+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:50:58.232510+0000","last_peered":"2024-09-21T10:50:58.232510+0000","last_clean":"2024-09-21T10:50:58.232510+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:50:58.232510+0000","last_undegraded":"2024-09-21T10:50:58.232510+0000","last_fullsized":"2024-09-21T10:50:58.232510+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2417,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.252165+0000","last_change":"2024-09-21T10:49:40.171098+0000","last_active":"2024-09-21T10:51:02.252165+0000","last_peered":"2024-09-21T10:51:02.252165+0000","last_clean":"2024-09-21T10:51:02.252165+0000","last_became_active":"2024-09-21T10:49:40.169461+0000","last_became_peered":"2024-09-21T10:49:40.169461+0000","last_unstale":"2024-09-21T10:51:02.252165+0000","last_undegraded":"2024-09-21T10:51:02.252165+0000","last_fullsized":"2024-09-21T10:51:02.252165+0000","mapping_epoch":1456,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1457,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:39:09.774760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":423,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227637+0000","last_change":"2024-09-21T10:45:59.680994+0000","last_active":"2024-09-21T10:50:58.227637+0000","last_peered":"2024-09-21T10:50:58.227637+0000","last_clean":"2024-09-21T10:50:58.227637+0000","last_became_active":"2024-09-21T10:45:59.680636+0000","last_became_peered":"2024-09-21T10:45:59.680636+0000","last_unstale":"2024-09-21T10:50:58.227637+0000","last_undegraded":"2024-09-21T10:50:58.227637+0000","last_fullsized":"2024-09-21T10:50:58.227637+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:42.488110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2284,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232516+0000","last_change":"2024-09-21T10:48:08.152845+0000","last_active":"2024-09-21T10:50:58.232516+0000","last_peered":"2024-09-21T10:50:58.232516+0000","last_clean":"2024-09-21T10:50:58.232516+0000","last_became_active":"2024-09-21T10:48:08.152542+0000","last_became_peered":"2024-09-21T10:48:08.152542+0000","last_unstale":"2024-09-21T10:50:58.232516+0000","last_undegraded":"2024-09-21T10:50:58.232516+0000","last_fullsized":"2024-09-21T10:50:58.232516+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:19:27.471208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":574,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448988+0000","last_change":"2024-09-21T10:48:08.177384+0000","last_active":"2024-09-21T10:50:58.448988+0000","last_peered":"2024-09-21T10:50:58.448988+0000","last_clean":"2024-09-21T10:50:58.448988+0000","last_became_active":"2024-09-21T10:48:08.176576+0000","last_became_peered":"2024-09-21T10:48:08.176576+0000","last_unstale":"2024-09-21T10:50:58.448988+0000","last_undegraded":"2024-09-21T10:50:58.448988+0000","last_fullsized":"2024-09-21T10:50:58.448988+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:26:35.528635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":169,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233044+0000","last_change":"2024-09-21T10:45:59.710479+0000","last_active":"2024-09-21T10:50:58.233044+0000","last_peered":"2024-09-21T10:50:58.233044+0000","last_clean":"2024-09-21T10:50:58.233044+0000","last_became_active":"2024-09-21T10:45:59.694474+0000","last_became_peered":"2024-09-21T10:45:59.694474+0000","last_unstale":"2024-09-21T10:50:58.233044+0000","last_undegraded":"2024-09-21T10:50:58.233044+0000","last_fullsized":"2024-09-21T10:50:58.233044+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:32.554219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":108,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448242+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:50:58.448242+0000","last_peered":"2024-09-21T10:50:58.448242+0000","last_clean":"2024-09-21T10:50:58.448242+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:50:58.448242+0000","last_undegraded":"2024-09-21T10:50:58.448242+0000","last_fullsized":"2024-09-21T10:50:58.448242+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":322,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244577+0000","last_change":"2024-09-21T10:50:47.220730+0000","last_active":"2024-09-21T10:51:01.244577+0000","last_peered":"2024-09-21T10:51:01.244577+0000","last_clean":"2024-09-21T10:51:01.244577+0000","last_became_active":"2024-09-21T10:50:47.218058+0000","last_became_peered":"2024-09-21T10:50:47.218058+0000","last_unstale":"2024-09-21T10:51:01.244577+0000","last_undegraded":"2024-09-21T10:51:01.244577+0000","last_fullsized":"2024-09-21T10:51:01.244577+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:42:26.937364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1283,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244372+0000","last_change":"2024-09-21T10:50:47.220720+0000","last_active":"2024-09-21T10:51:01.244372+0000","last_peered":"2024-09-21T10:51:01.244372+0000","last_clean":"2024-09-21T10:51:01.244372+0000","last_became_active":"2024-09-21T10:50:47.216475+0000","last_became_peered":"2024-09-21T10:50:47.216475+0000","last_unstale":"2024-09-21T10:51:01.244372+0000","last_undegraded":"2024-09-21T10:51:01.244372+0000","last_fullsized":"2024-09-21T10:51:01.244372+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:27:19.326942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":551,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232751+0000","last_change":"2024-09-21T10:48:08.167192+0000","last_active":"2024-09-21T10:50:58.232751+0000","last_peered":"2024-09-21T10:50:58.232751+0000","last_clean":"2024-09-21T10:50:58.232751+0000","last_became_active":"2024-09-21T10:48:08.166566+0000","last_became_peered":"2024-09-21T10:48:08.166566+0000","last_unstale":"2024-09-21T10:50:58.232751+0000","last_undegraded":"2024-09-21T10:50:58.232751+0000","last_fullsized":"2024-09-21T10:50:58.232751+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:19.985042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1235,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244212+0000","last_change":"2024-09-21T10:50:47.211913+0000","last_active":"2024-09-21T10:51:01.244212+0000","last_peered":"2024-09-21T10:51:01.244212+0000","last_clean":"2024-09-21T10:51:01.244212+0000","last_became_active":"2024-09-21T10:50:47.210320+0000","last_became_peered":"2024-09-21T10:50:47.210320+0000","last_unstale":"2024-09-21T10:51:01.244212+0000","last_undegraded":"2024-09-21T10:51:01.244212+0000","last_fullsized":"2024-09-21T10:51:01.244212+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:43:51.272367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1378,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244962+0000","last_change":"2024-09-21T10:50:47.237604+0000","last_active":"2024-09-21T10:51:01.244962+0000","last_peered":"2024-09-21T10:51:01.244962+0000","last_clean":"2024-09-21T10:51:01.244962+0000","last_became_active":"2024-09-21T10:50:47.234259+0000","last_became_peered":"2024-09-21T10:50:47.234259+0000","last_unstale":"2024-09-21T10:51:01.244962+0000","last_undegraded":"2024-09-21T10:51:01.244962+0000","last_fullsized":"2024-09-21T10:51:01.244962+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:58.703656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":290,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.253074+0000","last_change":"2024-09-21T10:48:08.210449+0000","last_active":"2024-09-21T10:51:02.253074+0000","last_peered":"2024-09-21T10:51:02.253074+0000","last_clean":"2024-09-21T10:51:02.253074+0000","last_became_active":"2024-09-21T10:48:08.162433+0000","last_became_peered":"2024-09-21T10:48:08.162433+0000","last_unstale":"2024-09-21T10:51:02.253074+0000","last_undegraded":"2024-09-21T10:51:02.253074+0000","last_fullsized":"2024-09-21T10:51:02.253074+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:26:55.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":325,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227387+0000","last_change":"2024-09-21T10:48:08.155548+0000","last_active":"2024-09-21T10:50:58.227387+0000","last_peered":"2024-09-21T10:50:58.227387+0000","last_clean":"2024-09-21T10:50:58.227387+0000","last_became_active":"2024-09-21T10:48:08.154428+0000","last_became_peered":"2024-09-21T10:48:08.154428+0000","last_unstale":"2024-09-21T10:50:58.227387+0000","last_undegraded":"2024-09-21T10:50:58.227387+0000","last_fullsized":"2024-09-21T10:50:58.227387+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:37.847953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:37.847953+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:23.699561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":75,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232004+0000","last_change":"2024-09-21T10:48:08.155960+0000","last_active":"2024-09-21T10:50:58.232004+0000","last_peered":"2024-09-21T10:50:58.232004+0000","last_clean":"2024-09-21T10:50:58.232004+0000","last_became_active":"2024-09-21T10:48:08.154801+0000","last_became_peered":"2024-09-21T10:48:08.154801+0000","last_unstale":"2024-09-21T10:50:58.232004+0000","last_undegraded":"2024-09-21T10:50:58.232004+0000","last_fullsized":"2024-09-21T10:50:58.232004+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:03:46.418044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2389,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.449059+0000","last_change":"2024-09-21T10:36:21.859688+0000","last_active":"2024-09-21T10:50:58.449059+0000","last_peered":"2024-09-21T10:50:58.449059+0000","last_clean":"2024-09-21T10:50:58.449059+0000","last_became_active":"2024-09-21T10:36:21.859175+0000","last_became_peered":"2024-09-21T10:36:21.859175+0000","last_unstale":"2024-09-21T10:50:58.449059+0000","last_undegraded":"2024-09-21T10:50:58.449059+0000","last_fullsized":"2024-09-21T10:50:58.449059+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.531550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":117,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.228166+0000","last_change":"2024-09-21T10:48:08.194127+0000","last_active":"2024-09-21T10:50:58.228166+0000","last_peered":"2024-09-21T10:50:58.228166+0000","last_clean":"2024-09-21T10:50:58.228166+0000","last_became_active":"2024-09-21T10:48:08.161824+0000","last_became_peered":"2024-09-21T10:48:08.161824+0000","last_unstale":"2024-09-21T10:50:58.228166+0000","last_undegraded":"2024-09-21T10:50:58.228166+0000","last_fullsized":"2024-09-21T10:50:58.228166+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:56:48.139417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":549,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244838+0000","last_change":"2024-09-21T10:50:47.230979+0000","last_active":"2024-09-21T10:51:01.244838+0000","last_peered":"2024-09-21T10:51:01.244838+0000","last_clean":"2024-09-21T10:51:01.244838+0000","last_became_active":"2024-09-21T10:50:47.226994+0000","last_became_peered":"2024-09-21T10:50:47.226994+0000","last_unstale":"2024-09-21T10:51:01.244838+0000","last_undegraded":"2024-09-21T10:51:01.244838+0000","last_fullsized":"2024-09-21T10:51:01.244838+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.555120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.555120+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:52.726165+0000","scrub_duration":0.0014665520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2410,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.251937+0000","last_change":"2024-09-21T10:48:08.162201+0000","last_active":"2024-09-21T10:51:02.251937+0000","last_peered":"2024-09-21T10:51:02.251937+0000","last_clean":"2024-09-21T10:51:02.251937+0000","last_became_active":"2024-09-21T10:48:08.161884+0000","last_became_peered":"2024-09-21T10:48:08.161884+0000","last_unstale":"2024-09-21T10:51:02.251937+0000","last_undegraded":"2024-09-21T10:51:02.251937+0000","last_fullsized":"2024-09-21T10:51:02.251937+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:58:08.759002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1494,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.250987+0000","last_change":"2024-09-21T10:45:59.694166+0000","last_active":"2024-09-21T10:51:02.250987+0000","last_peered":"2024-09-21T10:51:02.250987+0000","last_clean":"2024-09-21T10:51:02.250987+0000","last_became_active":"2024-09-21T10:45:59.693276+0000","last_became_peered":"2024-09-21T10:45:59.693276+0000","last_unstale":"2024-09-21T10:51:02.250987+0000","last_undegraded":"2024-09-21T10:51:02.250987+0000","last_fullsized":"2024-09-21T10:51:02.250987+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:12.784743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2384,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448662+0000","last_change":"2024-09-21T10:48:08.161699+0000","last_active":"2024-09-21T10:50:58.448662+0000","last_peered":"2024-09-21T10:50:58.448662+0000","last_clean":"2024-09-21T10:50:58.448662+0000","last_became_active":"2024-09-21T10:48:08.160423+0000","last_became_peered":"2024-09-21T10:48:08.160423+0000","last_unstale":"2024-09-21T10:50:58.448662+0000","last_undegraded":"2024-09-21T10:50:58.448662+0000","last_fullsized":"2024-09-21T10:50:58.448662+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:40:28.698737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":549,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244963+0000","last_change":"2024-09-21T10:50:47.238849+0000","last_active":"2024-09-21T10:51:01.244963+0000","last_peered":"2024-09-21T10:51:01.244963+0000","last_clean":"2024-09-21T10:51:01.244963+0000","last_became_active":"2024-09-21T10:50:47.234306+0000","last_became_peered":"2024-09-21T10:50:47.234306+0000","last_unstale":"2024-09-21T10:51:01.244963+0000","last_undegraded":"2024-09-21T10:51:01.244963+0000","last_fullsized":"2024-09-21T10:51:01.244963+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:43:58.698401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:43:58.698401+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:56:19.484276+0000","scrub_duration":0.001405536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2389,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448501+0000","last_change":"2024-09-21T10:48:08.161856+0000","last_active":"2024-09-21T10:50:58.448501+0000","last_peered":"2024-09-21T10:50:58.448501+0000","last_clean":"2024-09-21T10:50:58.448501+0000","last_became_active":"2024-09-21T10:48:08.160283+0000","last_became_peered":"2024-09-21T10:48:08.160283+0000","last_unstale":"2024-09-21T10:50:58.448501+0000","last_undegraded":"2024-09-21T10:50:58.448501+0000","last_fullsized":"2024-09-21T10:50:58.448501+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:45:09.474670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2384,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448789+0000","last_change":"2024-09-21T10:45:59.691645+0000","last_active":"2024-09-21T10:50:58.448789+0000","last_peered":"2024-09-21T10:50:58.448789+0000","last_clean":"2024-09-21T10:50:58.448789+0000","last_became_active":"2024-09-21T10:45:59.691334+0000","last_became_peered":"2024-09-21T10:45:59.691334+0000","last_unstale":"2024-09-21T10:50:58.448789+0000","last_undegraded":"2024-09-21T10:50:58.448789+0000","last_fullsized":"2024-09-21T10:50:58.448789+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:02:33.097343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":813,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233147+0000","last_change":"2024-09-21T10:48:08.144825+0000","last_active":"2024-09-21T10:50:58.233147+0000","last_peered":"2024-09-21T10:50:58.233147+0000","last_clean":"2024-09-21T10:50:58.233147+0000","last_became_active":"2024-09-21T10:48:08.142212+0000","last_became_peered":"2024-09-21T10:48:08.142212+0000","last_unstale":"2024-09-21T10:50:58.233147+0000","last_undegraded":"2024-09-21T10:50:58.233147+0000","last_fullsized":"2024-09-21T10:50:58.233147+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:42:46.899049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":472,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.250298+0000","last_change":"2024-09-21T10:48:08.960298+0000","last_active":"2024-09-21T10:51:02.250298+0000","last_peered":"2024-09-21T10:51:02.250298+0000","last_clean":"2024-09-21T10:51:02.250298+0000","last_became_active":"2024-09-21T10:48:08.151776+0000","last_became_peered":"2024-09-21T10:48:08.151776+0000","last_unstale":"2024-09-21T10:51:02.250298+0000","last_undegraded":"2024-09-21T10:51:02.250298+0000","last_fullsized":"2024-09-21T10:51:02.250298+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.960250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.960250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:24:25.581994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2374,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448765+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:50:58.448765+0000","last_peered":"2024-09-21T10:50:58.448765+0000","last_clean":"2024-09-21T10:50:58.448765+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:50:58.448765+0000","last_undegraded":"2024-09-21T10:50:58.448765+0000","last_fullsized":"2024-09-21T10:50:58.448765+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1499,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.251823+0000","last_change":"2024-09-21T10:36:21.860800+0000","last_active":"2024-09-21T10:51:02.251823+0000","last_peered":"2024-09-21T10:51:02.251823+0000","last_clean":"2024-09-21T10:51:02.251823+0000","last_became_active":"2024-09-21T10:36:21.860409+0000","last_became_peered":"2024-09-21T10:36:21.860409+0000","last_unstale":"2024-09-21T10:51:02.251823+0000","last_undegraded":"2024-09-21T10:51:02.251823+0000","last_fullsized":"2024-09-21T10:51:02.251823+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:08:07.343466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":213,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448413+0000","last_change":"2024-09-21T10:48:08.212451+0000","last_active":"2024-09-21T10:50:58.448413+0000","last_peered":"2024-09-21T10:50:58.448413+0000","last_clean":"2024-09-21T10:50:58.448413+0000","last_became_active":"2024-09-21T10:48:08.161435+0000","last_became_peered":"2024-09-21T10:48:08.161435+0000","last_unstale":"2024-09-21T10:50:58.448413+0000","last_undegraded":"2024-09-21T10:50:58.448413+0000","last_fullsized":"2024-09-21T10:50:58.448413+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:46:44.511679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":716,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.250151+0000","last_change":"2024-09-21T10:48:08.172490+0000","last_active":"2024-09-21T10:51:02.250151+0000","last_peered":"2024-09-21T10:51:02.250151+0000","last_clean":"2024-09-21T10:51:02.250151+0000","last_became_active":"2024-09-21T10:48:08.172117+0000","last_became_peered":"2024-09-21T10:48:08.172117+0000","last_unstale":"2024-09-21T10:51:02.250151+0000","last_undegraded":"2024-09-21T10:51:02.250151+0000","last_fullsized":"2024-09-21T10:51:02.250151+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:29:35.639980+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":116,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.449066+0000","last_change":"2024-09-21T10:48:08.182097+0000","last_active":"2024-09-21T10:50:58.449066+0000","last_peered":"2024-09-21T10:50:58.449066+0000","last_clean":"2024-09-21T10:50:58.449066+0000","last_became_active":"2024-09-21T10:48:08.181514+0000","last_became_peered":"2024-09-21T10:48:08.181514+0000","last_unstale":"2024-09-21T10:50:58.449066+0000","last_undegraded":"2024-09-21T10:50:58.449066+0000","last_fullsized":"2024-09-21T10:50:58.449066+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:59.851982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2375,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448128+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:50:58.448128+0000","last_peered":"2024-09-21T10:50:58.448128+0000","last_clean":"2024-09-21T10:50:58.448128+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:50:58.448128+0000","last_undegraded":"2024-09-21T10:50:58.448128+0000","last_fullsized":"2024-09-21T10:50:58.448128+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2306,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227136+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:50:58.227136+0000","last_peered":"2024-09-21T10:50:58.227136+0000","last_clean":"2024-09-21T10:50:58.227136+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:50:58.227136+0000","last_undegraded":"2024-09-21T10:50:58.227136+0000","last_fullsized":"2024-09-21T10:50:58.227136+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":566,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448760+0000","last_change":"2024-09-21T10:48:08.182699+0000","last_active":"2024-09-21T10:50:58.448760+0000","last_peered":"2024-09-21T10:50:58.448760+0000","last_clean":"2024-09-21T10:50:58.448760+0000","last_became_active":"2024-09-21T10:48:08.182308+0000","last_became_peered":"2024-09-21T10:48:08.182308+0000","last_unstale":"2024-09-21T10:50:58.448760+0000","last_undegraded":"2024-09-21T10:50:58.448760+0000","last_fullsized":"2024-09-21T10:50:58.448760+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:32.985403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":235,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.252945+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:51:02.252945+0000","last_peered":"2024-09-21T10:51:02.252945+0000","last_clean":"2024-09-21T10:51:02.252945+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:51:02.252945+0000","last_undegraded":"2024-09-21T10:51:02.252945+0000","last_fullsized":"2024-09-21T10:51:02.252945+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2418,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.249955+0000","last_change":"2024-09-21T10:48:08.166778+0000","last_active":"2024-09-21T10:51:02.249955+0000","last_peered":"2024-09-21T10:51:02.249955+0000","last_clean":"2024-09-21T10:51:02.249955+0000","last_became_active":"2024-09-21T10:48:08.166031+0000","last_became_peered":"2024-09-21T10:48:08.166031+0000","last_unstale":"2024-09-21T10:51:02.249955+0000","last_undegraded":"2024-09-21T10:51:02.249955+0000","last_fullsized":"2024-09-21T10:51:02.249955+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:13:24.713934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":652,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448263+0000","last_change":"2024-09-21T10:45:59.691940+0000","last_active":"2024-09-21T10:50:58.448263+0000","last_peered":"2024-09-21T10:50:58.448263+0000","last_clean":"2024-09-21T10:50:58.448263+0000","last_became_active":"2024-09-21T10:45:59.691610+0000","last_became_peered":"2024-09-21T10:45:59.691610+0000","last_unstale":"2024-09-21T10:50:58.448263+0000","last_undegraded":"2024-09-21T10:50:58.448263+0000","last_fullsized":"2024-09-21T10:50:58.448263+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:14:57.766019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1384,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244670+0000","last_change":"2024-09-21T10:50:47.208183+0000","last_active":"2024-09-21T10:51:01.244670+0000","last_peered":"2024-09-21T10:51:01.244670+0000","last_clean":"2024-09-21T10:51:01.244670+0000","last_became_active":"2024-09-21T10:50:47.207517+0000","last_became_peered":"2024-09-21T10:50:47.207517+0000","last_unstale":"2024-09-21T10:51:01.244670+0000","last_undegraded":"2024-09-21T10:51:01.244670+0000","last_fullsized":"2024-09-21T10:51:01.244670+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:55.366585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2299,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232610+0000","last_change":"2024-09-21T10:48:08.158793+0000","last_active":"2024-09-21T10:50:58.232610+0000","last_peered":"2024-09-21T10:50:58.232610+0000","last_clean":"2024-09-21T10:50:58.232610+0000","last_became_active":"2024-09-21T10:48:08.158487+0000","last_became_peered":"2024-09-21T10:48:08.158487+0000","last_unstale":"2024-09-21T10:50:58.232610+0000","last_undegraded":"2024-09-21T10:50:58.232610+0000","last_fullsized":"2024-09-21T10:50:58.232610+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:53.537691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":211,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227155+0000","last_change":"2024-09-21T10:48:08.214131+0000","last_active":"2024-09-21T10:50:58.227155+0000","last_peered":"2024-09-21T10:50:58.227155+0000","last_clean":"2024-09-21T10:50:58.227155+0000","last_became_active":"2024-09-21T10:48:08.157618+0000","last_became_peered":"2024-09-21T10:48:08.157618+0000","last_unstale":"2024-09-21T10:50:58.227155+0000","last_undegraded":"2024-09-21T10:50:58.227155+0000","last_fullsized":"2024-09-21T10:50:58.227155+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:13:36.574106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":1647,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.248189+0000","last_change":"2024-09-21T10:45:59.692592+0000","last_active":"2024-09-21T10:51:02.248189+0000","last_peered":"2024-09-21T10:51:02.248189+0000","last_clean":"2024-09-21T10:51:02.248189+0000","last_became_active":"2024-09-21T10:45:59.692232+0000","last_became_peered":"2024-09-21T10:45:59.692232+0000","last_unstale":"2024-09-21T10:51:02.248189+0000","last_undegraded":"2024-09-21T10:51:02.248189+0000","last_fullsized":"2024-09-21T10:51:02.248189+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:26.920879+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":143,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227857+0000","last_change":"2024-09-21T10:48:08.148499+0000","last_active":"2024-09-21T10:50:58.227857+0000","last_peered":"2024-09-21T10:50:58.227857+0000","last_clean":"2024-09-21T10:50:58.227857+0000","last_became_active":"2024-09-21T10:48:08.148141+0000","last_became_peered":"2024-09-21T10:48:08.148141+0000","last_unstale":"2024-09-21T10:50:58.227857+0000","last_undegraded":"2024-09-21T10:50:58.227857+0000","last_fullsized":"2024-09-21T10:50:58.227857+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.293583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.293583+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:54:35.860469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2393,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.449155+0000","last_change":"2024-09-21T10:48:08.171805+0000","last_active":"2024-09-21T10:50:58.449155+0000","last_peered":"2024-09-21T10:50:58.449155+0000","last_clean":"2024-09-21T10:50:58.449155+0000","last_became_active":"2024-09-21T10:48:08.170840+0000","last_became_peered":"2024-09-21T10:48:08.170840+0000","last_unstale":"2024-09-21T10:50:58.449155+0000","last_undegraded":"2024-09-21T10:50:58.449155+0000","last_fullsized":"2024-09-21T10:50:58.449155+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:30:18.802870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2292,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232325+0000","last_change":"2024-09-21T10:48:08.155588+0000","last_active":"2024-09-21T10:50:58.232325+0000","last_peered":"2024-09-21T10:50:58.232325+0000","last_clean":"2024-09-21T10:50:58.232325+0000","last_became_active":"2024-09-21T10:48:08.153652+0000","last_became_peered":"2024-09-21T10:48:08.153652+0000","last_unstale":"2024-09-21T10:50:58.232325+0000","last_undegraded":"2024-09-21T10:50:58.232325+0000","last_fullsized":"2024-09-21T10:50:58.232325+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:30:58.415174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":550,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448662+0000","last_change":"2024-09-21T10:48:08.181118+0000","last_active":"2024-09-21T10:50:58.448662+0000","last_peered":"2024-09-21T10:50:58.448662+0000","last_clean":"2024-09-21T10:50:58.448662+0000","last_became_active":"2024-09-21T10:48:08.180184+0000","last_became_peered":"2024-09-21T10:48:08.180184+0000","last_unstale":"2024-09-21T10:50:58.448662+0000","last_undegraded":"2024-09-21T10:50:58.448662+0000","last_fullsized":"2024-09-21T10:50:58.448662+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.629744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.629744+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:03:18.370832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1293,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244855+0000","last_change":"2024-09-21T10:50:47.236521+0000","last_active":"2024-09-21T10:51:01.244855+0000","last_peered":"2024-09-21T10:51:01.244855+0000","last_clean":"2024-09-21T10:51:01.244855+0000","last_became_active":"2024-09-21T10:50:47.229692+0000","last_became_peered":"2024-09-21T10:50:47.229692+0000","last_unstale":"2024-09-21T10:51:01.244855+0000","last_undegraded":"2024-09-21T10:51:01.244855+0000","last_fullsized":"2024-09-21T10:51:01.244855+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:31:05.511387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1447,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227264+0000","last_change":"2024-09-21T10:48:08.171234+0000","last_active":"2024-09-21T10:50:58.227264+0000","last_peered":"2024-09-21T10:50:58.227264+0000","last_clean":"2024-09-21T10:50:58.227264+0000","last_became_active":"2024-09-21T10:48:08.170908+0000","last_became_peered":"2024-09-21T10:48:08.170908+0000","last_unstale":"2024-09-21T10:50:58.227264+0000","last_undegraded":"2024-09-21T10:50:58.227264+0000","last_fullsized":"2024-09-21T10:50:58.227264+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:28.112802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":158,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.226919+0000","last_change":"2024-09-21T10:48:08.215096+0000","last_active":"2024-09-21T10:50:58.226919+0000","last_peered":"2024-09-21T10:50:58.226919+0000","last_clean":"2024-09-21T10:50:58.226919+0000","last_became_active":"2024-09-21T10:48:08.145535+0000","last_became_peered":"2024-09-21T10:48:08.145535+0000","last_unstale":"2024-09-21T10:50:58.226919+0000","last_undegraded":"2024-09-21T10:50:58.226919+0000","last_fullsized":"2024-09-21T10:50:58.226919+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:24.352339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:24.352339+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:55.688095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1441,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227482+0000","last_change":"2024-09-21T10:36:21.852442+0000","last_active":"2024-09-21T10:50:58.227482+0000","last_peered":"2024-09-21T10:50:58.227482+0000","last_clean":"2024-09-21T10:50:58.227482+0000","last_became_active":"2024-09-21T10:36:21.852004+0000","last_became_peered":"2024-09-21T10:36:21.852004+0000","last_unstale":"2024-09-21T10:50:58.227482+0000","last_undegraded":"2024-09-21T10:50:58.227482+0000","last_fullsized":"2024-09-21T10:50:58.227482+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:48:31.203584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1390,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244358+0000","last_change":"2024-09-21T10:50:47.218312+0000","last_active":"2024-09-21T10:51:01.244358+0000","last_peered":"2024-09-21T10:51:01.244358+0000","last_clean":"2024-09-21T10:51:01.244358+0000","last_became_active":"2024-09-21T10:50:47.215551+0000","last_became_peered":"2024-09-21T10:50:47.215551+0000","last_unstale":"2024-09-21T10:51:01.244358+0000","last_undegraded":"2024-09-21T10:51:01.244358+0000","last_fullsized":"2024-09-21T10:51:01.244358+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:32:39.614778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":1481,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232175+0000","last_change":"2024-09-21T10:48:08.153599+0000","last_active":"2024-09-21T10:50:58.232175+0000","last_peered":"2024-09-21T10:50:58.232175+0000","last_clean":"2024-09-21T10:50:58.232175+0000","last_became_active":"2024-09-21T10:48:08.153255+0000","last_became_peered":"2024-09-21T10:48:08.153255+0000","last_unstale":"2024-09-21T10:50:58.232175+0000","last_undegraded":"2024-09-21T10:50:58.232175+0000","last_fullsized":"2024-09-21T10:50:58.232175+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:04.469883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:04.469883+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:12:53.690175+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1493,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233014+0000","last_change":"2024-09-21T10:36:21.851195+0000","last_active":"2024-09-21T10:50:58.233014+0000","last_peered":"2024-09-21T10:50:58.233014+0000","last_clean":"2024-09-21T10:50:58.233014+0000","last_became_active":"2024-09-21T10:36:21.850900+0000","last_became_peered":"2024-09-21T10:36:21.850900+0000","last_unstale":"2024-09-21T10:50:58.233014+0000","last_undegraded":"2024-09-21T10:50:58.233014+0000","last_fullsized":"2024-09-21T10:50:58.233014+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:51:03.191578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":505,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233285+0000","last_change":"2024-09-21T10:38:34.893842+0000","last_active":"2024-09-21T10:50:58.233285+0000","last_peered":"2024-09-21T10:50:58.233285+0000","last_clean":"2024-09-21T10:50:58.233285+0000","last_became_active":"2024-09-21T10:38:34.886177+0000","last_became_peered":"2024-09-21T10:38:34.886177+0000","last_unstale":"2024-09-21T10:50:58.233285+0000","last_undegraded":"2024-09-21T10:50:58.233285+0000","last_fullsized":"2024-09-21T10:50:58.233285+0000","mapping_epoch":1361,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1362,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:31:55.977224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":458,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233678+0000","last_change":"2024-09-21T10:46:08.467544+0000","last_active":"2024-09-21T10:50:58.233678+0000","last_peered":"2024-09-21T10:50:58.233678+0000","last_clean":"2024-09-21T10:50:58.233678+0000","last_became_active":"2024-09-21T10:45:59.695828+0000","last_became_peered":"2024-09-21T10:45:59.695828+0000","last_unstale":"2024-09-21T10:50:58.233678+0000","last_undegraded":"2024-09-21T10:50:58.233678+0000","last_fullsized":"2024-09-21T10:50:58.233678+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:08.467485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:08.467485+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:36.742481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2409,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.251814+0000","last_change":"2024-09-21T10:36:21.861399+0000","last_active":"2024-09-21T10:51:02.251814+0000","last_peered":"2024-09-21T10:51:02.251814+0000","last_clean":"2024-09-21T10:51:02.251814+0000","last_became_active":"2024-09-21T10:36:21.860629+0000","last_became_peered":"2024-09-21T10:36:21.860629+0000","last_unstale":"2024-09-21T10:51:02.251814+0000","last_undegraded":"2024-09-21T10:51:02.251814+0000","last_fullsized":"2024-09-21T10:51:02.251814+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:20.808344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1348,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233011+0000","last_change":"2024-09-21T10:48:08.166901+0000","last_active":"2024-09-21T10:50:58.233011+0000","last_peered":"2024-09-21T10:50:58.233011+0000","last_clean":"2024-09-21T10:50:58.233011+0000","last_became_active":"2024-09-21T10:48:08.166122+0000","last_became_peered":"2024-09-21T10:48:08.166122+0000","last_unstale":"2024-09-21T10:50:58.233011+0000","last_undegraded":"2024-09-21T10:50:58.233011+0000","last_fullsized":"2024-09-21T10:50:58.233011+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:04:18.037036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":568,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.245016+0000","last_change":"2024-09-21T10:50:47.238907+0000","last_active":"2024-09-21T10:51:01.245016+0000","last_peered":"2024-09-21T10:51:01.245016+0000","last_clean":"2024-09-21T10:51:01.245016+0000","last_became_active":"2024-09-21T10:50:47.237733+0000","last_became_peered":"2024-09-21T10:50:47.237733+0000","last_unstale":"2024-09-21T10:51:01.245016+0000","last_undegraded":"2024-09-21T10:51:01.245016+0000","last_fullsized":"2024-09-21T10:51:01.245016+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:19:37.353696+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1362,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244269+0000","last_change":"2024-09-21T10:50:47.221057+0000","last_active":"2024-09-21T10:51:01.244269+0000","last_peered":"2024-09-21T10:51:01.244269+0000","last_clean":"2024-09-21T10:51:01.244269+0000","last_became_active":"2024-09-21T10:50:47.216283+0000","last_became_peered":"2024-09-21T10:50:47.216283+0000","last_unstale":"2024-09-21T10:51:01.244269+0000","last_undegraded":"2024-09-21T10:51:01.244269+0000","last_fullsized":"2024-09-21T10:51:01.244269+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:56.608899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":30,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244728+0000","last_change":"2024-09-21T10:50:47.208266+0000","last_active":"2024-09-21T10:51:01.244728+0000","last_peered":"2024-09-21T10:51:01.244728+0000","last_clean":"2024-09-21T10:51:01.244728+0000","last_became_active":"2024-09-21T10:50:47.207559+0000","last_became_peered":"2024-09-21T10:50:47.207559+0000","last_unstale":"2024-09-21T10:51:01.244728+0000","last_undegraded":"2024-09-21T10:51:01.244728+0000","last_fullsized":"2024-09-21T10:51:01.244728+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:11:50.891292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":544,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232664+0000","last_change":"2024-09-21T10:34:30.143679+0000","last_active":"2024-09-21T10:50:58.232664+0000","last_peered":"2024-09-21T10:50:58.232664+0000","last_clean":"2024-09-21T10:50:58.232664+0000","last_became_active":"2024-09-21T10:33:21.763467+0000","last_became_peered":"2024-09-21T10:33:21.763467+0000","last_unstale":"2024-09-21T10:50:58.232664+0000","last_undegraded":"2024-09-21T10:50:58.232664+0000","last_fullsized":"2024-09-21T10:50:58.232664+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1283,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:28:21.154137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:28:21.154137+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic deep scrub scheduled @ 2024-09-28T21:24:44.206811+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1475,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448186+0000","last_change":"2024-09-21T10:36:21.859616+0000","last_active":"2024-09-21T10:50:58.448186+0000","last_peered":"2024-09-21T10:50:58.448186+0000","last_clean":"2024-09-21T10:50:58.448186+0000","last_became_active":"2024-09-21T10:36:21.858954+0000","last_became_peered":"2024-09-21T10:36:21.858954+0000","last_unstale":"2024-09-21T10:50:58.448186+0000","last_undegraded":"2024-09-21T10:50:58.448186+0000","last_fullsized":"2024-09-21T10:50:58.448186+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:43:05.240363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":154,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448944+0000","last_change":"2024-09-21T10:48:08.204361+0000","last_active":"2024-09-21T10:50:58.448944+0000","last_peered":"2024-09-21T10:50:58.448944+0000","last_clean":"2024-09-21T10:50:58.448944+0000","last_became_active":"2024-09-21T10:48:08.177064+0000","last_became_peered":"2024-09-21T10:48:08.177064+0000","last_unstale":"2024-09-21T10:50:58.448944+0000","last_undegraded":"2024-09-21T10:50:58.448944+0000","last_fullsized":"2024-09-21T10:50:58.448944+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:12.137967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":137,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448944+0000","last_change":"2024-09-21T10:46:00.158554+0000","last_active":"2024-09-21T10:50:58.448944+0000","last_peered":"2024-09-21T10:50:58.448944+0000","last_clean":"2024-09-21T10:50:58.448944+0000","last_became_active":"2024-09-21T10:45:59.680546+0000","last_became_peered":"2024-09-21T10:45:59.680546+0000","last_unstale":"2024-09-21T10:50:58.448944+0000","last_undegraded":"2024-09-21T10:50:58.448944+0000","last_fullsized":"2024-09-21T10:50:58.448944+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:00.158500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:00.158500+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:24:39.581987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1182,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244118+0000","last_change":"2024-09-21T10:50:47.224598+0000","last_active":"2024-09-21T10:51:01.244118+0000","last_peered":"2024-09-21T10:51:01.244118+0000","last_clean":"2024-09-21T10:51:01.244118+0000","last_became_active":"2024-09-21T10:50:47.211766+0000","last_became_peered":"2024-09-21T10:50:47.211766+0000","last_unstale":"2024-09-21T10:51:01.244118+0000","last_undegraded":"2024-09-21T10:51:01.244118+0000","last_fullsized":"2024-09-21T10:51:01.244118+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:41:40.503392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":1515,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.252021+0000","last_change":"2024-09-21T10:48:08.213283+0000","last_active":"2024-09-21T10:51:02.252021+0000","last_peered":"2024-09-21T10:51:02.252021+0000","last_clean":"2024-09-21T10:51:02.252021+0000","last_became_active":"2024-09-21T10:48:08.166573+0000","last_became_peered":"2024-09-21T10:48:08.166573+0000","last_unstale":"2024-09-21T10:51:02.252021+0000","last_undegraded":"2024-09-21T10:51:02.252021+0000","last_fullsized":"2024-09-21T10:51:02.252021+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:43:51.213203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":88,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":85,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448314+0000","last_change":"2024-09-21T10:48:08.843710+0000","last_active":"2024-09-21T10:50:58.448314+0000","last_peered":"2024-09-21T10:50:58.448314+0000","last_clean":"2024-09-21T10:50:58.448314+0000","last_became_active":"2024-09-21T10:48:08.162916+0000","last_became_peered":"2024-09-21T10:48:08.162916+0000","last_unstale":"2024-09-21T10:50:58.448314+0000","last_undegraded":"2024-09-21T10:50:58.448314+0000","last_fullsized":"2024-09-21T10:50:58.448314+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.843662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.843662+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:40:35.024010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1459,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244213+0000","last_change":"2024-09-21T10:50:47.225368+0000","last_active":"2024-09-21T10:51:01.244213+0000","last_peered":"2024-09-21T10:51:01.244213+0000","last_clean":"2024-09-21T10:51:01.244213+0000","last_became_active":"2024-09-21T10:50:47.216658+0000","last_became_peered":"2024-09-21T10:50:47.216658+0000","last_unstale":"2024-09-21T10:51:01.244213+0000","last_undegraded":"2024-09-21T10:51:01.244213+0000","last_fullsized":"2024-09-21T10:51:01.244213+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:18.997227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2274,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.233785+0000","last_change":"2024-09-21T10:45:59.696879+0000","last_active":"2024-09-21T10:50:58.233785+0000","last_peered":"2024-09-21T10:50:58.233785+0000","last_clean":"2024-09-21T10:50:58.233785+0000","last_became_active":"2024-09-21T10:45:59.696055+0000","last_became_peered":"2024-09-21T10:45:59.696055+0000","last_unstale":"2024-09-21T10:50:58.233785+0000","last_undegraded":"2024-09-21T10:50:58.233785+0000","last_fullsized":"2024-09-21T10:50:58.233785+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:55:10.189668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":550,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.448598+0000","last_change":"2024-09-21T10:37:34.983958+0000","last_active":"2024-09-21T10:50:58.448598+0000","last_peered":"2024-09-21T10:50:58.448598+0000","last_clean":"2024-09-21T10:50:58.448598+0000","last_became_active":"2024-09-21T10:36:21.856390+0000","last_became_peered":"2024-09-21T10:36:21.856390+0000","last_unstale":"2024-09-21T10:50:58.448598+0000","last_undegraded":"2024-09-21T10:50:58.448598+0000","last_fullsized":"2024-09-21T10:50:58.448598+0000","mapping_epoch":1328,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1329,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:44:20.259839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1278,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.244388+0000","last_change":"2024-09-21T10:50:47.267338+0000","last_active":"2024-09-21T10:51:01.244388+0000","last_peered":"2024-09-21T10:51:01.244388+0000","last_clean":"2024-09-21T10:51:01.244388+0000","last_became_active":"2024-09-21T10:50:47.267055+0000","last_became_peered":"2024-09-21T10:50:47.267055+0000","last_unstale":"2024-09-21T10:51:01.244388+0000","last_undegraded":"2024-09-21T10:51:01.244388+0000","last_fullsized":"2024-09-21T10:51:01.244388+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:50:46.964590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2397,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.253286+0000","last_change":"2024-09-21T10:45:59.697081+0000","last_active":"2024-09-21T10:51:02.253286+0000","last_peered":"2024-09-21T10:51:02.253286+0000","last_clean":"2024-09-21T10:51:02.253286+0000","last_became_active":"2024-09-21T10:45:59.696414+0000","last_became_peered":"2024-09-21T10:45:59.696414+0000","last_unstale":"2024-09-21T10:51:02.253286+0000","last_undegraded":"2024-09-21T10:51:02.253286+0000","last_fullsized":"2024-09-21T10:51:02.253286+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:32:47.528914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":346,"reported_epoch":1475,"state":"active+clean","last_fresh":"2024-09-21T10:51:01.243902+0000","last_change":"2024-09-21T10:50:51.566364+0000","last_active":"2024-09-21T10:51:01.243902+0000","last_peered":"2024-09-21T10:51:01.243902+0000","last_clean":"2024-09-21T10:51:01.243902+0000","last_became_active":"2024-09-21T10:50:47.242360+0000","last_became_peered":"2024-09-21T10:50:47.242360+0000","last_unstale":"2024-09-21T10:51:01.243902+0000","last_undegraded":"2024-09-21T10:51:01.243902+0000","last_fullsized":"2024-09-21T10:51:01.243902+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1470,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:50:51.566260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:50:51.566260+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:01:49.425461+0000","scrub_duration":0.0018756650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2397,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.250249+0000","last_change":"2024-09-21T10:45:59.700672+0000","last_active":"2024-09-21T10:51:02.250249+0000","last_peered":"2024-09-21T10:51:02.250249+0000","last_clean":"2024-09-21T10:51:02.250249+0000","last_became_active":"2024-09-21T10:45:59.699332+0000","last_became_peered":"2024-09-21T10:45:59.699332+0000","last_unstale":"2024-09-21T10:51:02.250249+0000","last_undegraded":"2024-09-21T10:51:02.250249+0000","last_fullsized":"2024-09-21T10:51:02.250249+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T09:58:48.567053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":254,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227182+0000","last_change":"2024-09-21T10:48:08.155880+0000","last_active":"2024-09-21T10:50:58.227182+0000","last_peered":"2024-09-21T10:50:58.227182+0000","last_clean":"2024-09-21T10:50:58.227182+0000","last_became_active":"2024-09-21T10:48:08.154917+0000","last_became_peered":"2024-09-21T10:48:08.154917+0000","last_unstale":"2024-09-21T10:50:58.227182+0000","last_undegraded":"2024-09-21T10:50:58.227182+0000","last_fullsized":"2024-09-21T10:50:58.227182+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:37:14.509661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":278,"reported_epoch":1476,"state":"active+clean","last_fresh":"2024-09-21T10:51:02.249714+0000","last_change":"2024-09-21T10:48:08.167083+0000","last_active":"2024-09-21T10:51:02.249714+0000","last_peered":"2024-09-21T10:51:02.249714+0000","last_clean":"2024-09-21T10:51:02.249714+0000","last_became_active":"2024-09-21T10:48:08.166441+0000","last_became_peered":"2024-09-21T10:48:08.166441+0000","last_unstale":"2024-09-21T10:51:02.249714+0000","last_undegraded":"2024-09-21T10:51:02.249714+0000","last_fullsized":"2024-09-21T10:51:02.249714+0000","mapping_epoch":1442,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1443,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:53:57.413991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":146,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.227200+0000","last_change":"2024-09-21T10:45:59.714397+0000","last_active":"2024-09-21T10:50:58.227200+0000","last_peered":"2024-09-21T10:50:58.227200+0000","last_clean":"2024-09-21T10:50:58.227200+0000","last_became_active":"2024-09-21T10:45:59.689648+0000","last_became_peered":"2024-09-21T10:45:59.689648+0000","last_unstale":"2024-09-21T10:50:58.227200+0000","last_undegraded":"2024-09-21T10:50:58.227200+0000","last_fullsized":"2024-09-21T10:50:58.227200+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:19:20.596738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2282,"reported_epoch":1474,"state":"active+clean","last_fresh":"2024-09-21T10:50:58.232778+0000","last_change":"2024-09-21T10:45:59.696141+0000","last_active":"2024-09-21T10:50:58.232778+0000","last_peered":"2024-09-21T10:50:58.232778+0000","last_clean":"2024-09-21T10:50:58.232778+0000","last_became_active":"2024-09-21T10:45:59.695761+0000","last_became_peered":"2024-09-21T10:45:59.695761+0000","last_unstale":"2024-09-21T10:50:58.232778+0000","last_undegraded":"2024-09-21T10:50:58.232778+0000","last_fullsized":"2024-09-21T10:50:58.232778+0000","mapping_epoch":1416,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1417,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:03:46.714846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":371915,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":918048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2100,"ondisk_log_size":2100,"up":63,"acting":63,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":1922,"num_keys_recovered":10,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":143360,"data_stored":26316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":210453398178,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049837,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506992,"kb_used_data":10164,"kb_used_omap":242,"kb_used_meta":1496525,"kb_avail":92238160,"statfs":{"total":95995035648,"available":94451875840,"internally_reserved":0,"allocated":10407936,"data_stored":6014312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":1532441823},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":158913790636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":5201205395742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644795,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1133584,"kb_used_data":10116,"kb_used_omap":67,"kb_used_meta":1123388,"kb_avail":92611568,"statfs":{"total":95995035648,"available":94834245632,"internally_reserved":0,"allocated":10358784,"data_stored":5990330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1150350173},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527813,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1109392,"kb_used_data":9796,"kb_used_omap":75,"kb_used_meta":1099508,"kb_avail":92635760,"statfs":{"total":95995035648,"available":94859018240,"internally_reserved":0,"allocated":10031104,"data_stored":5667616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":1125896346},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274966,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1098248,"kb_used_data":9536,"kb_used_omap":51,"kb_used_meta":1088652,"kb_avail":92646904,"statfs":{"total":95995035648,"available":94870429696,"internally_reserved":0,"allocated":9764864,"data_stored":5399851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1114779989},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354205,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":918852,"kb_used_data":9924,"kb_used_omap":66,"kb_used_meta":908797,"kb_avail":92826300,"statfs":{"total":95995035648,"available":95054131200,"internally_reserved":0,"allocated":10162176,"data_stored":5751936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":930608452},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":5348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":4856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":5508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:51:05.552 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T10:51:05.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 31 2024-09-21T10:51:06.141 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:07.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:07 smithi119 ceph-mon[105288]: pgmap v2714: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:07.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:07 smithi119 ceph-mon[101432]: pgmap v2714: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:07 smithi145 ceph-mon[89359]: pgmap v2714: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:08.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T10:51:08.470 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:09.279 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 1 pg_num to 31 2024-09-21T10:51:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:09 smithi145 ceph-mon[89359]: pgmap v2715: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2596853453' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T10:51:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:09 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:09 smithi119 ceph-mon[105288]: pgmap v2715: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2596853453' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T10:51:09.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:09 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:09 smithi119 ceph-mon[101432]: pgmap v2715: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2596853453' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-21T10:51:09.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:09 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.538 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2596853453' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[105288]: osdmap e1479: 8 total, 8 up, 5 in 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2596853453' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[101432]: osdmap e1479: 8 total, 8 up, 5 in 2024-09-21T10:51:10.539 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.540 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.540 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:10 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T10:51:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:51:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:51:10.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:51:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:10 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2596853453' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-21T10:51:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:10 smithi145 ceph-mon[89359]: osdmap e1479: 8 total, 8 up, 5 in 2024-09-21T10:51:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:10.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:10 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-21T10:51:11.645 INFO:teuthology.orchestra.run.smithi119.stderr:noscrub is unset 2024-09-21T10:51:11.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[105288]: pgmap v2717: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:11.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-21T10:51:11.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[105288]: osdmap e1480: 8 total, 8 up, 5 in 2024-09-21T10:51:11.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1909825493' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:51:11.981 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:51:11.982 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[101432]: pgmap v2717: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:11.982 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-21T10:51:11.982 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[101432]: osdmap e1480: 8 total, 8 up, 5 in 2024-09-21T10:51:11.982 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1909825493' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:51:11.982 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:51:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:11 smithi145 ceph-mon[89359]: pgmap v2717: 93 pgs: 93 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:11 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-21T10:51:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:11 smithi145 ceph-mon[89359]: osdmap e1480: 8 total, 8 up, 5 in 2024-09-21T10:51:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1909825493' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:51:12.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-21T10:51:12.416 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T10:51:12.645 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[101432]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-21T10:51:12.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:51:12.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:51:12.646 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[101432]: osdmap e1481: 8 total, 8 up, 5 in 2024-09-21T10:51:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:12 smithi145 ceph-mon[89359]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-21T10:51:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:12 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:51:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:12 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:51:13.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:12 smithi145 ceph-mon[89359]: osdmap e1481: 8 total, 8 up, 5 in 2024-09-21T10:51:13.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[105288]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-21T10:51:13.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:51:13.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T10:51:13.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:12 smithi119 ceph-mon[105288]: osdmap e1481: 8 total, 8 up, 5 in 2024-09-21T10:51:13.188 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: pgmap v2720: 103 pgs: 8 unknown, 95 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:14.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: 1.b scrub starts 2024-09-21T10:51:14.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: 1.b scrub ok 2024-09-21T10:51:14.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: 1.c scrub starts 2024-09-21T10:51:14.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: 1.c scrub ok 2024-09-21T10:51:14.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: 1.7 scrub starts 2024-09-21T10:51:14.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:13 smithi145 ceph-mon[89359]: 1.7 scrub ok 2024-09-21T10:51:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: pgmap v2720: 103 pgs: 8 unknown, 95 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: 1.b scrub starts 2024-09-21T10:51:14.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: 1.b scrub ok 2024-09-21T10:51:14.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: 1.c scrub starts 2024-09-21T10:51:14.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: 1.c scrub ok 2024-09-21T10:51:14.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: 1.7 scrub starts 2024-09-21T10:51:14.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[101432]: 1.7 scrub ok 2024-09-21T10:51:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: pgmap v2720: 103 pgs: 8 unknown, 95 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:14.118 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: 1.b scrub starts 2024-09-21T10:51:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: 1.b scrub ok 2024-09-21T10:51:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: 1.c scrub starts 2024-09-21T10:51:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: 1.c scrub ok 2024-09-21T10:51:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: 1.7 scrub starts 2024-09-21T10:51:14.119 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:13 smithi119 ceph-mon[105288]: 1.7 scrub ok 2024-09-21T10:51:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:51:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:51:14] "GET /metrics HTTP/1.1" 200 36835 "" "Prometheus/2.51.0" 2024-09-21T10:51:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:14 smithi119 ceph-mon[101432]: 1.6 scrub starts 2024-09-21T10:51:14.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:14 smithi119 ceph-mon[101432]: 1.6 scrub ok 2024-09-21T10:51:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:14 smithi119 ceph-mon[105288]: 1.6 scrub starts 2024-09-21T10:51:14.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:14 smithi119 ceph-mon[105288]: 1.6 scrub ok 2024-09-21T10:51:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:14 smithi145 ceph-mon[89359]: 1.6 scrub starts 2024-09-21T10:51:15.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:14 smithi145 ceph-mon[89359]: 1.6 scrub ok 2024-09-21T10:51:15.563 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T10:51:15.563 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T10:51:15.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T10:51:15.827 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:15.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: pgmap v2721: 103 pgs: 6 unknown, 97 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:15.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: 1.d scrub starts 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: 1.d scrub ok 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: 1.1c scrub starts 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: 1.1c scrub ok 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: 1.16 scrub starts 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: 1.16 scrub ok 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: pgmap v2721: 103 pgs: 6 unknown, 97 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: 1.d scrub starts 2024-09-21T10:51:15.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: 1.d scrub ok 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: 1.1c scrub starts 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: 1.1c scrub ok 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: 1.16 scrub starts 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: 1.16 scrub ok 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:51:15.949 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:15 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: pgmap v2721: 103 pgs: 6 unknown, 97 active+clean; 369 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: 1.d scrub starts 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: 1.d scrub ok 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: 1.1c scrub starts 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: 1.1c scrub ok 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: 1.16 scrub starts 2024-09-21T10:51:16.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: 1.16 scrub ok 2024-09-21T10:51:16.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:16.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:16.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:51:16.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:51:16.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:15 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:16.666 INFO:teuthology.orchestra.run.smithi119.stderr:nodeep-scrub is unset 2024-09-21T10:51:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:16 smithi145 ceph-mon[89359]: 1.9 scrub starts 2024-09-21T10:51:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:16 smithi145 ceph-mon[89359]: 1.9 scrub ok 2024-09-21T10:51:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:16 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3520861945' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T10:51:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:16 smithi145 ceph-mon[89359]: 1.5 scrub starts 2024-09-21T10:51:17.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:16 smithi145 ceph-mon[89359]: 1.5 scrub ok 2024-09-21T10:51:17.078 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[105288]: 1.9 scrub starts 2024-09-21T10:51:17.078 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[105288]: 1.9 scrub ok 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3520861945' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[105288]: 1.5 scrub starts 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[105288]: 1.5 scrub ok 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[101432]: 1.9 scrub starts 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[101432]: 1.9 scrub ok 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3520861945' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[101432]: 1.5 scrub starts 2024-09-21T10:51:17.079 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:16 smithi119 ceph-mon[101432]: 1.5 scrub ok 2024-09-21T10:51:17.927 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: pgmap v2722: 103 pgs: 103 active+clean; 582 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:17.927 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: 1.17 deep-scrub starts 2024-09-21T10:51:17.927 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: 1.17 deep-scrub ok 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3520861945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: osdmap e1482: 8 total, 8 up, 5 in 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: 1.8 scrub starts 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[101432]: 1.8 scrub ok 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: pgmap v2722: 103 pgs: 103 active+clean; 582 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: 1.17 deep-scrub starts 2024-09-21T10:51:17.928 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: 1.17 deep-scrub ok 2024-09-21T10:51:17.929 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3520861945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:17.929 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-21T10:51:17.929 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: osdmap e1482: 8 total, 8 up, 5 in 2024-09-21T10:51:17.929 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: 1.8 scrub starts 2024-09-21T10:51:17.929 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-mon[105288]: 1.8 scrub ok 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: pgmap v2722: 103 pgs: 103 active+clean; 582 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: 1.17 deep-scrub starts 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: 1.17 deep-scrub ok 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3520861945' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: osdmap e1482: 8 total, 8 up, 5 in 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: 1.8 scrub starts 2024-09-21T10:51:18.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:17 smithi145 ceph-mon[89359]: 1.8 scrub ok 2024-09-21T10:51:18.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:51:17.924+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1483 definitely_dead 0 2024-09-21T10:51:18.696 INFO:teuthology.orchestra.run.smithi119.stderr:marked in osd.4. 2024-09-21T10:51:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:18 smithi145 ceph-mon[89359]: osdmap e1483: 8 total, 8 up, 5 in 2024-09-21T10:51:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4058760769' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:51:19.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:18 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[105288]: osdmap e1483: 8 total, 8 up, 5 in 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4058760769' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[101432]: osdmap e1483: 8 total, 8 up, 5 in 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:19.065 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4058760769' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:51:19.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:18 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-21T10:51:19.468 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T10:51:19.468 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T10:51:19.468 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-09-21T10:51:19.469 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd in 5 2024-09-21T10:51:19.694 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: pgmap v2725: 103 pgs: 6 peering, 97 active+clean; 582 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: 1.e scrub starts 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: 1.e scrub ok 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: 1.1c scrub starts 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: 1.1c scrub ok 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:51:19.695 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[101432]: osdmap e1484: 8 total, 8 up, 6 in 2024-09-21T10:51:19.734 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: pgmap v2725: 103 pgs: 6 peering, 97 active+clean; 582 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: 1.e scrub starts 2024-09-21T10:51:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: 1.e scrub ok 2024-09-21T10:51:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: 1.1c scrub starts 2024-09-21T10:51:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: 1.1c scrub ok 2024-09-21T10:51:20.037 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:51:20.038 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:51:20.038 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:19 smithi119 ceph-mon[105288]: osdmap e1484: 8 total, 8 up, 6 in 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: pgmap v2725: 103 pgs: 6 peering, 97 active+clean; 582 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: 1.e scrub starts 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: 1.e scrub ok 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: 1.1c scrub starts 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: 1.1c scrub ok 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T10:51:20.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:19 smithi145 ceph-mon[89359]: osdmap e1484: 8 total, 8 up, 6 in 2024-09-21T10:51:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:51:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:51:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:51:20.813 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:20 smithi145 ceph-mon[89359]: 1.16 scrub starts 2024-09-21T10:51:20.813 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:20 smithi145 ceph-mon[89359]: 1.16 scrub ok 2024-09-21T10:51:20.813 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:20 smithi145 ceph-mon[89359]: osdmap e1485: 8 total, 8 up, 6 in 2024-09-21T10:51:21.076 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:20 smithi119 ceph-mon[101432]: 1.16 scrub starts 2024-09-21T10:51:21.076 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:20 smithi119 ceph-mon[101432]: 1.16 scrub ok 2024-09-21T10:51:21.076 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:20 smithi119 ceph-mon[101432]: osdmap e1485: 8 total, 8 up, 6 in 2024-09-21T10:51:21.076 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:20 smithi119 ceph-mon[105288]: 1.16 scrub starts 2024-09-21T10:51:21.076 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:20 smithi119 ceph-mon[105288]: 1.16 scrub ok 2024-09-21T10:51:21.076 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:20 smithi119 ceph-mon[105288]: osdmap e1485: 8 total, 8 up, 6 in 2024-09-21T10:51:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:21 smithi145 ceph-mon[89359]: pgmap v2728: 103 pgs: 1 active+recovery_wait+degraded, 6 peering, 96 active+clean; 635 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 2/81 objects degraded (2.469%); 40 KiB/s, 0 objects/s recovering 2024-09-21T10:51:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:21 smithi145 ceph-mon[89359]: 1.19 scrub starts 2024-09-21T10:51:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:21 smithi145 ceph-mon[89359]: 1.19 scrub ok 2024-09-21T10:51:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:21 smithi145 ceph-mon[89359]: Health check failed: Degraded data redundancy: 2/81 objects degraded (2.469%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[105288]: pgmap v2728: 103 pgs: 1 active+recovery_wait+degraded, 6 peering, 96 active+clean; 635 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 2/81 objects degraded (2.469%); 40 KiB/s, 0 objects/s recovering 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[105288]: 1.19 scrub starts 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[105288]: 1.19 scrub ok 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[105288]: Health check failed: Degraded data redundancy: 2/81 objects degraded (2.469%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[101432]: pgmap v2728: 103 pgs: 1 active+recovery_wait+degraded, 6 peering, 96 active+clean; 635 KiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 2/81 objects degraded (2.469%); 40 KiB/s, 0 objects/s recovering 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[101432]: 1.19 scrub starts 2024-09-21T10:51:22.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[101432]: 1.19 scrub ok 2024-09-21T10:51:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-mon[101432]: Health check failed: Degraded data redundancy: 2/81 objects degraded (2.469%), 1 pg degraded (PG_DEGRADED) 2024-09-21T10:51:22.118 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:21 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:51:21.761+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1485 definitely_dead 0 2024-09-21T10:51:22.703 INFO:teuthology.orchestra.run.smithi119.stderr:marked in osd.5. 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.15 scrub starts 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.15 scrub ok 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.1b scrub starts 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.1b scrub ok 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.17 scrub starts 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.17 scrub ok 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2109451659' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.18 scrub starts 2024-09-21T10:51:23.044 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[101432]: 1.18 scrub ok 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.15 scrub starts 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.15 scrub ok 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.1b scrub starts 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.1b scrub ok 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.17 scrub starts 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.17 scrub ok 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2109451659' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.18 scrub starts 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:22 smithi119 ceph-mon[105288]: 1.18 scrub ok 2024-09-21T10:51:23.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.15 scrub starts 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.15 scrub ok 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.1b scrub starts 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.1b scrub ok 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.17 scrub starts 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.17 scrub ok 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2109451659' entity='client.admin' cmd={"prefix": "osd in", "ids": ["5"]} : dispatch 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.18 scrub starts 2024-09-21T10:51:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:22 smithi145 ceph-mon[89359]: 1.18 scrub ok 2024-09-21T10:51:23.446 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-09-21T10:51:23.447 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T10:51:23.447 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-21T10:51:23.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd in 7 2024-09-21T10:51:23.697 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:23.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[101432]: pgmap v2729: 103 pgs: 1 active+recovery_wait+degraded, 3 peering, 99 active+clean; 507 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 2/81 objects degraded (2.469%); 53 KiB/s, 0 objects/s recovering 2024-09-21T10:51:23.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2109451659' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T10:51:23.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[101432]: osdmap e1486: 8 total, 8 up, 7 in 2024-09-21T10:51:23.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:23.975 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[105288]: pgmap v2729: 103 pgs: 1 active+recovery_wait+degraded, 3 peering, 99 active+clean; 507 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 2/81 objects degraded (2.469%); 53 KiB/s, 0 objects/s recovering 2024-09-21T10:51:23.976 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2109451659' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T10:51:23.976 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[105288]: osdmap e1486: 8 total, 8 up, 7 in 2024-09-21T10:51:23.976 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:23 smithi145 ceph-mon[89359]: pgmap v2729: 103 pgs: 1 active+recovery_wait+degraded, 3 peering, 99 active+clean; 507 KiB data, 5.5 GiB used, 441 GiB / 447 GiB avail; 2/81 objects degraded (2.469%); 53 KiB/s, 0 objects/s recovering 2024-09-21T10:51:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:23 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2109451659' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-09-21T10:51:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:23 smithi145 ceph-mon[89359]: osdmap e1486: 8 total, 8 up, 7 in 2024-09-21T10:51:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:51:24] "GET /metrics HTTP/1.1" 200 34419 "" "Prometheus/2.51.0" 2024-09-21T10:51:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[101432]: 1.5 deep-scrub starts 2024-09-21T10:51:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[101432]: 1.5 deep-scrub ok 2024-09-21T10:51:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[101432]: osdmap e1487: 8 total, 8 up, 7 in 2024-09-21T10:51:24.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[101432]: pgmap v2732: 103 pgs: 1 active+recovery_wait+degraded, 8 peering, 94 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 9.4 KiB/s rd, 21 KiB/s wr, 5 op/s; 2/81 objects degraded (2.469%); 2.3 KiB/s, 2 objects/s recovering 2024-09-21T10:51:24.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[105288]: 1.5 deep-scrub starts 2024-09-21T10:51:24.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[105288]: 1.5 deep-scrub ok 2024-09-21T10:51:24.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[105288]: osdmap e1487: 8 total, 8 up, 7 in 2024-09-21T10:51:24.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:24 smithi119 ceph-mon[105288]: pgmap v2732: 103 pgs: 1 active+recovery_wait+degraded, 8 peering, 94 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 9.4 KiB/s rd, 21 KiB/s wr, 5 op/s; 2/81 objects degraded (2.469%); 2.3 KiB/s, 2 objects/s recovering 2024-09-21T10:51:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:24 smithi145 ceph-mon[89359]: 1.5 deep-scrub starts 2024-09-21T10:51:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:24 smithi145 ceph-mon[89359]: 1.5 deep-scrub ok 2024-09-21T10:51:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:24 smithi145 ceph-mon[89359]: osdmap e1487: 8 total, 8 up, 7 in 2024-09-21T10:51:25.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:24 smithi145 ceph-mon[89359]: pgmap v2732: 103 pgs: 1 active+recovery_wait+degraded, 8 peering, 94 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 9.4 KiB/s rd, 21 KiB/s wr, 5 op/s; 2/81 objects degraded (2.469%); 2.3 KiB/s, 2 objects/s recovering 2024-09-21T10:51:25.777 INFO:teuthology.orchestra.run.smithi119.stderr:marked in osd.7. 2024-09-21T10:51:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:25 smithi119 ceph-mon[105288]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:51:25.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:25 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1568659525' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:51:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:25 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:51:25.720+0000 7f312c1e8640 -1 mon.a@0(leader).osd e1487 definitely_dead 0 2024-09-21T10:51:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:25 smithi119 ceph-mon[101432]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:51:25.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:25 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1568659525' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:51:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:25 smithi145 ceph-mon[89359]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-21T10:51:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:25 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1568659525' entity='client.admin' cmd={"prefix": "osd in", "ids": ["7"]} : dispatch 2024-09-21T10:51:26.737 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-21T10:51:26.737 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T10:51:26.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 31 2024-09-21T10:51:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:27.023 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1568659525' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[101432]: osdmap e1488: 8 total, 8 up, 8 in 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[101432]: 1.1b scrub starts 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[101432]: 1.1b scrub ok 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[101432]: pgmap v2734: 103 pgs: 8 peering, 95 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 5 op/s; 2.1 KiB/s, 2 objects/s recovering 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1568659525' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[105288]: osdmap e1488: 8 total, 8 up, 8 in 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[105288]: 1.1b scrub starts 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[105288]: 1.1b scrub ok 2024-09-21T10:51:27.024 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:26 smithi119 ceph-mon[105288]: pgmap v2734: 103 pgs: 8 peering, 95 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 5 op/s; 2.1 KiB/s, 2 objects/s recovering 2024-09-21T10:51:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:26 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1568659525' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-21T10:51:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:26 smithi145 ceph-mon[89359]: osdmap e1488: 8 total, 8 up, 8 in 2024-09-21T10:51:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:26 smithi145 ceph-mon[89359]: 1.1b scrub starts 2024-09-21T10:51:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:26 smithi145 ceph-mon[89359]: 1.1b scrub ok 2024-09-21T10:51:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:26 smithi145 ceph-mon[89359]: pgmap v2734: 103 pgs: 8 peering, 95 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 8.5 KiB/s rd, 18 KiB/s wr, 5 op/s; 2.1 KiB/s, 2 objects/s recovering 2024-09-21T10:51:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:27 smithi119 ceph-mon[105288]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/81 objects degraded (2.469%), 1 pg degraded) 2024-09-21T10:51:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:27 smithi119 ceph-mon[105288]: Cluster is now healthy 2024-09-21T10:51:27.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:27 smithi119 ceph-mon[105288]: osdmap e1489: 8 total, 8 up, 8 in 2024-09-21T10:51:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:27 smithi119 ceph-mon[101432]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/81 objects degraded (2.469%), 1 pg degraded) 2024-09-21T10:51:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:27 smithi119 ceph-mon[101432]: Cluster is now healthy 2024-09-21T10:51:27.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:27 smithi119 ceph-mon[101432]: osdmap e1489: 8 total, 8 up, 8 in 2024-09-21T10:51:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:27 smithi145 ceph-mon[89359]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/81 objects degraded (2.469%), 1 pg degraded) 2024-09-21T10:51:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:27 smithi145 ceph-mon[89359]: Cluster is now healthy 2024-09-21T10:51:28.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:27 smithi145 ceph-mon[89359]: osdmap e1489: 8 total, 8 up, 8 in 2024-09-21T10:51:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: 1.1d scrub starts 2024-09-21T10:51:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: 1.1d scrub ok 2024-09-21T10:51:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: 1.1a scrub starts 2024-09-21T10:51:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: 1.1a scrub ok 2024-09-21T10:51:28.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: 1.a deep-scrub starts 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: 1.a deep-scrub ok 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[105288]: pgmap v2736: 103 pgs: 9 peering, 94 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 9.5 KiB/s rd, 21 KiB/s wr, 5 op/s; 0 B/s, 3 objects/s recovering 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: 1.1d scrub starts 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: 1.1d scrub ok 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: 1.1a scrub starts 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: 1.1a scrub ok 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: 1.a deep-scrub starts 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: 1.a deep-scrub ok 2024-09-21T10:51:28.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:28 smithi119 ceph-mon[101432]: pgmap v2736: 103 pgs: 9 peering, 94 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 9.5 KiB/s rd, 21 KiB/s wr, 5 op/s; 0 B/s, 3 objects/s recovering 2024-09-21T10:51:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: 1.1d scrub starts 2024-09-21T10:51:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: 1.1d scrub ok 2024-09-21T10:51:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: 1.1a scrub starts 2024-09-21T10:51:29.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: 1.1a scrub ok 2024-09-21T10:51:29.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: 1.a deep-scrub starts 2024-09-21T10:51:29.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: 1.a deep-scrub ok 2024-09-21T10:51:29.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:28 smithi145 ceph-mon[89359]: pgmap v2736: 103 pgs: 9 peering, 94 active+clean; 493 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 9.5 KiB/s rd, 21 KiB/s wr, 5 op/s; 0 B/s, 3 objects/s recovering 2024-09-21T10:51:29.798 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 1 pgp_num to 31 2024-09-21T10:51:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:29 smithi145 ceph-mon[89359]: 1.12 scrub starts 2024-09-21T10:51:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:29 smithi145 ceph-mon[89359]: 1.12 scrub ok 2024-09-21T10:51:30.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1251259756' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-21T10:51:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:29 smithi119 ceph-mon[105288]: 1.12 scrub starts 2024-09-21T10:51:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:29 smithi119 ceph-mon[105288]: 1.12 scrub ok 2024-09-21T10:51:30.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1251259756' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-21T10:51:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:29 smithi119 ceph-mon[101432]: 1.12 scrub starts 2024-09-21T10:51:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:29 smithi119 ceph-mon[101432]: 1.12 scrub ok 2024-09-21T10:51:30.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1251259756' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-21T10:51:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:51:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:51:30.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:51:30.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T10:51:30.808 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:31.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[101432]: 1.e scrub starts 2024-09-21T10:51:31.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[101432]: 1.e scrub ok 2024-09-21T10:51:31.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1251259756' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-21T10:51:31.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[101432]: osdmap e1490: 8 total, 8 up, 8 in 2024-09-21T10:51:31.053 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[101432]: pgmap v2738: 103 pgs: 1 peering, 102 active+clean; 573 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 75 KiB/s, 1 keys/s, 3 objects/s recovering 2024-09-21T10:51:31.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[105288]: 1.e scrub starts 2024-09-21T10:51:31.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[105288]: 1.e scrub ok 2024-09-21T10:51:31.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1251259756' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-21T10:51:31.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[105288]: osdmap e1490: 8 total, 8 up, 8 in 2024-09-21T10:51:31.054 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:30 smithi119 ceph-mon[105288]: pgmap v2738: 103 pgs: 1 peering, 102 active+clean; 573 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 75 KiB/s, 1 keys/s, 3 objects/s recovering 2024-09-21T10:51:31.276 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:30 smithi145 ceph-mon[89359]: 1.e scrub starts 2024-09-21T10:51:31.276 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:30 smithi145 ceph-mon[89359]: 1.e scrub ok 2024-09-21T10:51:31.276 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:30 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1251259756' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-21T10:51:31.276 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:30 smithi145 ceph-mon[89359]: osdmap e1490: 8 total, 8 up, 8 in 2024-09-21T10:51:31.276 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:30 smithi145 ceph-mon[89359]: pgmap v2738: 103 pgs: 1 peering, 102 active+clean; 573 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 5.7 KiB/s rd, 12 KiB/s wr, 3 op/s; 75 KiB/s, 1 keys/s, 3 objects/s recovering 2024-09-21T10:51:32.636 INFO:teuthology.orchestra.run.smithi119.stdout:mon.a: {} 2024-09-21T10:51:32.636 INFO:teuthology.orchestra.run.smithi119.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:51:32.645 INFO:teuthology.orchestra.run.smithi119.stdout:mon.b: {} 2024-09-21T10:51:32.645 INFO:teuthology.orchestra.run.smithi119.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:51:32.653 INFO:teuthology.orchestra.run.smithi119.stdout:mon.c: {} 2024-09-21T10:51:32.653 INFO:teuthology.orchestra.run.smithi119.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[105288]: pgmap v2739: 103 pgs: 103 active+clean; 573 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 5.4 KiB/s rd, 12 KiB/s wr, 3 op/s; 71 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[105288]: 1.1e scrub starts 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[105288]: 1.1e scrub ok 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2178685268' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[101432]: pgmap v2739: 103 pgs: 103 active+clean; 573 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 5.4 KiB/s rd, 12 KiB/s wr, 3 op/s; 71 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:51:33.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[101432]: 1.1e scrub starts 2024-09-21T10:51:33.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[101432]: 1.1e scrub ok 2024-09-21T10:51:33.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2178685268' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:51:33.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:33.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 0 1 2024-09-21T10:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:33 smithi145 ceph-mon[89359]: pgmap v2739: 103 pgs: 103 active+clean; 573 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 5.4 KiB/s rd, 12 KiB/s wr, 3 op/s; 71 KiB/s, 1 keys/s, 2 objects/s recovering 2024-09-21T10:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:33 smithi145 ceph-mon[89359]: 1.1e scrub starts 2024-09-21T10:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:33 smithi145 ceph-mon[89359]: 1.1e scrub ok 2024-09-21T10:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:33 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2178685268' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-21T10:51:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:33.748 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-mon[105288]: 1.18 scrub starts 2024-09-21T10:51:34.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-mon[105288]: 1.18 scrub ok 2024-09-21T10:51:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-mon[101432]: 1.18 scrub starts 2024-09-21T10:51:34.868 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-mon[101432]: 1.18 scrub ok 2024-09-21T10:51:34.868 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:51:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:51:34] "GET /metrics HTTP/1.1" 200 34444 "" "Prometheus/2.51.0" 2024-09-21T10:51:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:34 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:51:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:34 smithi145 ceph-mon[89359]: 1.18 scrub starts 2024-09-21T10:51:35.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:34 smithi145 ceph-mon[89359]: 1.18 scrub ok 2024-09-21T10:51:36.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:35 smithi145 ceph-mon[89359]: pgmap v2740: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:51:36.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:35 smithi119 ceph-mon[101432]: pgmap v2740: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:51:36.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:35 smithi119 ceph-mon[105288]: pgmap v2740: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:51:36.639 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T10:51:36.933 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:36 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1401813521' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:51:36.933 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:36 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1401813521' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:51:37.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:36 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1401813521' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:51:37.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-21T10:51:37.570 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:37.986 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:37 smithi119 ceph-mon[101432]: pgmap v2741: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:51:37.987 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:37 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1401813521' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:51:37.987 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:37 smithi119 ceph-mon[101432]: osdmap e1491: 8 total, 8 up, 8 in 2024-09-21T10:51:37.987 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:37 smithi119 ceph-mon[105288]: pgmap v2741: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:51:37.987 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:37 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1401813521' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:51:37.987 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:37 smithi119 ceph-mon[105288]: osdmap e1491: 8 total, 8 up, 8 in 2024-09-21T10:51:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:37 smithi145 ceph-mon[89359]: pgmap v2741: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 0 B/s, 1 keys/s, 1 objects/s recovering 2024-09-21T10:51:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:37 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1401813521' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:51:38.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:37 smithi145 ceph-mon[89359]: osdmap e1491: 8 total, 8 up, 8 in 2024-09-21T10:51:39.642 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[105288]: pgmap v2743: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[105288]: 1.1c scrub starts 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[105288]: 1.1c scrub ok 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2584735535' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[101432]: pgmap v2743: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[101432]: 1.1c scrub starts 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[101432]: 1.1c scrub ok 2024-09-21T10:51:39.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:39 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2584735535' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:51:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:39 smithi145 ceph-mon[89359]: pgmap v2743: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:39 smithi145 ceph-mon[89359]: 1.1c scrub starts 2024-09-21T10:51:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:39 smithi145 ceph-mon[89359]: 1.1c scrub ok 2024-09-21T10:51:40.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:39 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2584735535' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-21T10:51:40.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T10:51:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:51:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:51:40.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:51:40.705 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:40.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2584735535' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:51:40.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[105288]: osdmap e1492: 8 total, 8 up, 8 in 2024-09-21T10:51:40.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[105288]: 1.9 scrub starts 2024-09-21T10:51:40.826 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[105288]: 1.9 scrub ok 2024-09-21T10:51:40.826 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2584735535' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:51:40.827 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[101432]: osdmap e1492: 8 total, 8 up, 8 in 2024-09-21T10:51:40.827 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[101432]: 1.9 scrub starts 2024-09-21T10:51:40.827 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:40 smithi119 ceph-mon[101432]: 1.9 scrub ok 2024-09-21T10:51:41.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:40 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2584735535' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-21T10:51:41.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:40 smithi145 ceph-mon[89359]: osdmap e1492: 8 total, 8 up, 8 in 2024-09-21T10:51:41.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:40 smithi145 ceph-mon[89359]: 1.9 scrub starts 2024-09-21T10:51:41.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:40 smithi145 ceph-mon[89359]: 1.9 scrub ok 2024-09-21T10:51:41.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:41 smithi119 ceph-mon[101432]: pgmap v2745: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:41.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:41 smithi119 ceph-mon[105288]: pgmap v2745: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:42.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:41 smithi145 ceph-mon[89359]: pgmap v2745: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:43.663 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T10:51:43.993 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:43 smithi119 ceph-mon[105288]: pgmap v2746: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:43.993 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:43 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1939052496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:43.993 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:43 smithi119 ceph-mon[101432]: pgmap v2746: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:43.993 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:43 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1939052496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:43 smithi145 ceph-mon[89359]: pgmap v2746: 103 pgs: 103 active+clean; 661 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:44.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:43 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1939052496' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:44.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T10:51:44.668 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:51:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:51:44] "GET /metrics HTTP/1.1" 200 34444 "" "Prometheus/2.51.0" 2024-09-21T10:51:44.672 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:44.935 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:44 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1939052496' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:44.935 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:44 smithi119 ceph-mon[105288]: osdmap e1493: 8 total, 8 up, 8 in 2024-09-21T10:51:44.936 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:44 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1939052496' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:44.936 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:44 smithi119 ceph-mon[101432]: osdmap e1493: 8 total, 8 up, 8 in 2024-09-21T10:51:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:44 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1939052496' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:45.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:44 smithi145 ceph-mon[89359]: osdmap e1493: 8 total, 8 up, 8 in 2024-09-21T10:51:45.842 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:45 smithi119 ceph-mon[105288]: pgmap v2748: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:45.843 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:45 smithi119 ceph-mon[101432]: pgmap v2748: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:46.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:45 smithi145 ceph-mon[89359]: pgmap v2748: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:46.676 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T10:51:46.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:46 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/24662403' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:46.974 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:46 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/24662403' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:47.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:46 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/24662403' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-21T10:51:47.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T10:51:47.520 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:47.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:47 smithi119 ceph-mon[101432]: pgmap v2749: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:47.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:47 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/24662403' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:47.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:47 smithi119 ceph-mon[101432]: osdmap e1494: 8 total, 8 up, 8 in 2024-09-21T10:51:47.946 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:47 smithi119 ceph-mon[105288]: pgmap v2749: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:47.946 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:47 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/24662403' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:47.946 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:47 smithi119 ceph-mon[105288]: osdmap e1494: 8 total, 8 up, 8 in 2024-09-21T10:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:47 smithi145 ceph-mon[89359]: pgmap v2749: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:47 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/24662403' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T10:51:48.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:47 smithi145 ceph-mon[89359]: osdmap e1494: 8 total, 8 up, 8 in 2024-09-21T10:51:48.825 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:48.825 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:49.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:51:49.686 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T10:51:50.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:49 smithi119 ceph-mon[105288]: pgmap v2751: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:50.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:49 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/67847036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:51:50.007 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:49 smithi119 ceph-mon[101432]: pgmap v2751: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:50.007 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:49 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/67847036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:51:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:49 smithi145 ceph-mon[89359]: pgmap v2751: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:50.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:49 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/67847036' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:51:50.425 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T10:51:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:51:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:51:50.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:51:50.670 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:50.944 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:50 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/67847036' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:51:50.944 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:50 smithi119 ceph-mon[105288]: osdmap e1495: 8 total, 8 up, 8 in 2024-09-21T10:51:50.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:50 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/67847036' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:51:50.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:50 smithi119 ceph-mon[101432]: osdmap e1495: 8 total, 8 up, 8 in 2024-09-21T10:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:50 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/67847036' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:51:51.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:50 smithi145 ceph-mon[89359]: osdmap e1495: 8 total, 8 up, 8 in 2024-09-21T10:51:51.791 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:51 smithi119 ceph-mon[101432]: pgmap v2753: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:51.791 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:51 smithi119 ceph-mon[105288]: pgmap v2753: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:51.817 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:51 smithi145 ceph-mon[89359]: pgmap v2753: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T10:51:53.007 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:52 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3631158702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:51:53.007 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:52 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3631158702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:51:53.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:52 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3631158702' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-21T10:51:53.356 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 3 1 2024-09-21T10:51:53.600 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:54.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:53 smithi119 ceph-mon[101432]: pgmap v2754: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:54.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:53 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3631158702' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:51:54.002 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:53 smithi119 ceph-mon[101432]: osdmap e1496: 8 total, 8 up, 8 in 2024-09-21T10:51:54.002 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:53 smithi119 ceph-mon[105288]: pgmap v2754: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:54.003 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:53 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3631158702' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:51:54.003 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:53 smithi119 ceph-mon[105288]: osdmap e1496: 8 total, 8 up, 8 in 2024-09-21T10:51:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:53 smithi145 ceph-mon[89359]: pgmap v2754: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:53 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3631158702' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T10:51:54.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:53 smithi145 ceph-mon[89359]: osdmap e1496: 8 total, 8 up, 8 in 2024-09-21T10:51:54.790 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:51:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:51:54] "GET /metrics HTTP/1.1" 200 34419 "" "Prometheus/2.51.0" 2024-09-21T10:51:55.709 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T10:51:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:55 smithi145 ceph-mon[89359]: pgmap v2756: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:55 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/424721470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:56.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:55 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:56.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:55 smithi119 ceph-mon[101432]: pgmap v2756: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:56.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:55 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/424721470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:56.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:55 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:56.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:55 smithi119 ceph-mon[105288]: pgmap v2756: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:56.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:55 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/424721470' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:56.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:55 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:56.435 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-21T10:51:56.695 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:57.004 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:56 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:51:57.005 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:56 smithi119 ceph-mon[105288]: osdmap e1497: 8 total, 8 up, 8 in 2024-09-21T10:51:57.005 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:56 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:51:57.005 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:56 smithi119 ceph-mon[101432]: osdmap e1497: 8 total, 8 up, 8 in 2024-09-21T10:51:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:56 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:51:57.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:56 smithi145 ceph-mon[89359]: osdmap e1497: 8 total, 8 up, 8 in 2024-09-21T10:51:57.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:57 smithi119 ceph-mon[101432]: pgmap v2758: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:57.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:57 smithi119 ceph-mon[105288]: pgmap v2758: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:58.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:57 smithi145 ceph-mon[89359]: pgmap v2758: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:58.731 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T10:51:58.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:58 smithi119 ceph-mon[105288]: pgmap v2759: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:58.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:58 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/414360349' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:58 smithi119 ceph-mon[101432]: pgmap v2759: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:58.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:58 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/414360349' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:58 smithi145 ceph-mon[89359]: pgmap v2759: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:51:59.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:58 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/414360349' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-21T10:51:59.505 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T10:51:59.754 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:51:59.997 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:59 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/414360349' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:51:59.997 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:51:59 smithi119 ceph-mon[105288]: osdmap e1498: 8 total, 8 up, 8 in 2024-09-21T10:51:59.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:59 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/414360349' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:51:59.997 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:51:59 smithi119 ceph-mon[101432]: osdmap e1498: 8 total, 8 up, 8 in 2024-09-21T10:52:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:59 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/414360349' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-21T10:52:00.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:51:59 smithi145 ceph-mon[89359]: osdmap e1498: 8 total, 8 up, 8 in 2024-09-21T10:52:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:52:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:52:00.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:52:00.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:00 smithi119 ceph-mon[101432]: pgmap v2761: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:00.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:00 smithi119 ceph-mon[105288]: pgmap v2761: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:01.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:00 smithi145 ceph-mon[89359]: pgmap v2761: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:01.748 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T10:52:02.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:01 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2119840472' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:52:02.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:01 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2119840472' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:52:02.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:01 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2119840472' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:52:02.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-21T10:52:02.701 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:03.012 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:02 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2119840472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:52:03.012 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:02 smithi119 ceph-mon[101432]: osdmap e1499: 8 total, 8 up, 8 in 2024-09-21T10:52:03.012 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:02 smithi119 ceph-mon[101432]: pgmap v2763: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:03.013 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:02 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2119840472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:52:03.013 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:02 smithi119 ceph-mon[105288]: osdmap e1499: 8 total, 8 up, 8 in 2024-09-21T10:52:03.013 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:02 smithi119 ceph-mon[105288]: pgmap v2763: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:02 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2119840472' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:52:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:02 smithi145 ceph-mon[89359]: osdmap e1499: 8 total, 8 up, 8 in 2024-09-21T10:52:03.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:02 smithi145 ceph-mon[89359]: pgmap v2763: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:03.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:03.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:04.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:04.762 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T10:52:04.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:52:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:52:04] "GET /metrics HTTP/1.1" 200 34421 "" "Prometheus/2.51.0" 2024-09-21T10:52:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:04 smithi119 ceph-mon[101432]: pgmap v2764: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:04.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3907911795' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:52:04.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:04 smithi119 ceph-mon[105288]: pgmap v2764: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:04.868 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3907911795' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:52:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:04 smithi145 ceph-mon[89359]: pgmap v2764: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:05.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3907911795' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-21T10:52:05.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T10:52:05.673 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:06.039 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:05 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3907911795' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:52:06.039 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:05 smithi119 ceph-mon[105288]: osdmap e1500: 8 total, 8 up, 8 in 2024-09-21T10:52:06.040 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:05 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3907911795' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:52:06.040 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:05 smithi119 ceph-mon[101432]: osdmap e1500: 8 total, 8 up, 8 in 2024-09-21T10:52:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:05 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3907911795' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-21T10:52:06.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:05 smithi145 ceph-mon[89359]: osdmap e1500: 8 total, 8 up, 8 in 2024-09-21T10:52:06.813 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:06 smithi119 ceph-mon[101432]: pgmap v2766: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:06.813 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:06 smithi119 ceph-mon[105288]: pgmap v2766: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:07.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:06 smithi145 ceph-mon[89359]: pgmap v2766: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:07.780 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T10:52:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3884799541' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:08.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:07 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3884799541' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:08.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:07 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3884799541' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:08.117 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:07 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:08.525 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-21T10:52:08.769 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:08 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:08 smithi145 ceph-mon[89359]: osdmap e1501: 8 total, 8 up, 8 in 2024-09-21T10:52:09.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:08 smithi145 ceph-mon[89359]: pgmap v2768: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:09.051 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:08 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:09.051 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:08 smithi119 ceph-mon[101432]: osdmap e1501: 8 total, 8 up, 8 in 2024-09-21T10:52:09.051 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:08 smithi119 ceph-mon[101432]: pgmap v2768: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:09.051 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:08 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:09.051 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:08 smithi119 ceph-mon[105288]: osdmap e1501: 8 total, 8 up, 8 in 2024-09-21T10:52:09.051 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:08 smithi119 ceph-mon[105288]: pgmap v2768: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:52:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:52:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:52:11.167 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T10:52:11.499 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:11 smithi119 ceph-mon[105288]: pgmap v2769: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:11.500 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:11 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3777978470' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:11.500 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:11 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:11.500 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:11 smithi119 ceph-mon[101432]: pgmap v2769: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:11.500 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:11 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3777978470' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:11.500 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:11 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:11 smithi145 ceph-mon[89359]: pgmap v2769: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:11 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3777978470' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:11 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-21T10:52:12.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 6 1 2024-09-21T10:52:12.264 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:12 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:12 smithi119 ceph-mon[101432]: osdmap e1502: 8 total, 8 up, 8 in 2024-09-21T10:52:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:12 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:12 smithi119 ceph-mon[105288]: osdmap e1502: 8 total, 8 up, 8 in 2024-09-21T10:52:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:12 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-21T10:52:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:12 smithi145 ceph-mon[89359]: osdmap e1502: 8 total, 8 up, 8 in 2024-09-21T10:52:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:13 smithi119 ceph-mon[101432]: pgmap v2771: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:13.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:13 smithi119 ceph-mon[101432]: osdmap e1503: 8 total, 8 up, 8 in 2024-09-21T10:52:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:13 smithi119 ceph-mon[105288]: pgmap v2771: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:13.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:13 smithi119 ceph-mon[105288]: osdmap e1503: 8 total, 8 up, 8 in 2024-09-21T10:52:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:13 smithi145 ceph-mon[89359]: pgmap v2771: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:13 smithi145 ceph-mon[89359]: osdmap e1503: 8 total, 8 up, 8 in 2024-09-21T10:52:14.175 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T10:52:14.542 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:14 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3174007704' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.542 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:14 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3174007704' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:14 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3174007704' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:14.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:52:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:52:14] "GET /metrics HTTP/1.1" 200 34421 "" "Prometheus/2.51.0" 2024-09-21T10:52:14.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-21T10:52:15.086 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:15.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:15 smithi119 ceph-mon[101432]: pgmap v2773: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:15.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:15 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3174007704' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:15.365 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:15 smithi119 ceph-mon[101432]: osdmap e1504: 8 total, 8 up, 8 in 2024-09-21T10:52:15.365 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:15 smithi119 ceph-mon[105288]: pgmap v2773: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:15.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:15 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3174007704' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:15.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:15 smithi119 ceph-mon[105288]: osdmap e1504: 8 total, 8 up, 8 in 2024-09-21T10:52:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:15 smithi145 ceph-mon[89359]: pgmap v2773: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:15 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3174007704' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:15.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:15 smithi145 ceph-mon[89359]: osdmap e1504: 8 total, 8 up, 8 in 2024-09-21T10:52:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:52:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:16.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:16 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:52:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:16 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-21T10:52:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:16.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:16 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:17.456 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:17 smithi119 ceph-mon[105288]: pgmap v2775: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:17.457 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:17 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3630432021' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:17.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:17 smithi119 ceph-mon[101432]: pgmap v2775: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:17.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:17 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3630432021' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:17 smithi145 ceph-mon[89359]: pgmap v2775: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:17.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:17 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3630432021' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-21T10:52:17.898 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T10:52:18.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:18 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:18.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:18 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3630432021' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:18.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:18 smithi145 ceph-mon[89359]: osdmap e1505: 8 total, 8 up, 8 in 2024-09-21T10:52:18.612 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:18 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:18.612 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:18 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3630432021' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:18.612 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:18 smithi119 ceph-mon[105288]: osdmap e1505: 8 total, 8 up, 8 in 2024-09-21T10:52:18.613 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:18 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:18.613 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:18 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3630432021' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-21T10:52:18.613 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:18 smithi119 ceph-mon[101432]: osdmap e1505: 8 total, 8 up, 8 in 2024-09-21T10:52:18.637 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T10:52:19.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:19 smithi119 ceph-mon[105288]: pgmap v2777: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:19.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:19 smithi119 ceph-mon[101432]: pgmap v2777: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:19.398 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:19.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:19 smithi145 ceph-mon[89359]: pgmap v2777: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:20.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:52:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:52:20.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:52:20.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:20.810 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:20 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:20.810 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:20.811 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:20 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:21.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:20 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:21.867 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:21 smithi119 ceph-mon[105288]: pgmap v2778: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:21.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:21 smithi119 ceph-mon[101432]: pgmap v2778: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:22.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:21 smithi145 ceph-mon[89359]: pgmap v2778: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:22.721 INFO:teuthology.orchestra.run.smithi119.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T10:52:23.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:22 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/4092173555' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:23.045 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:22 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:22 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.046 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:22 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.098 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/4092173555' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/4092173555' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.099 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:22 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-21T10:52:23.685 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:23.997 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[105288]: pgmap v2779: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[105288]: osdmap e1506: 8 total, 8 up, 8 in 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[101432]: pgmap v2779: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[101432]: osdmap e1506: 8 total, 8 up, 8 in 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:52:23.998 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:23 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:23 smithi145 ceph-mon[89359]: pgmap v2779: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:23 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:23 smithi145 ceph-mon[89359]: osdmap e1506: 8 total, 8 up, 8 in 2024-09-21T10:52:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-21T10:52:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-21T10:52:24.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:23 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' 2024-09-21T10:52:24.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:52:24 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:52:24] "GET /metrics HTTP/1.1" 200 34422 "" "Prometheus/2.51.0" 2024-09-21T10:52:25.725 INFO:teuthology.orchestra.run.smithi119.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T10:52:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:25 smithi145 ceph-mon[89359]: pgmap v2781: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:25 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2862813529' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:26.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:25 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:26.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:25 smithi119 ceph-mon[105288]: pgmap v2781: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:26.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:25 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2862813529' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:26.082 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:25 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:26.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:25 smithi119 ceph-mon[101432]: pgmap v2781: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:26.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:25 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2862813529' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:26.082 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:25 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-21T10:52:26.544 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T10:52:26.544 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T10:52:26.544 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T10:52:26.545 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T10:52:26.545 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T10:52:26.545 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:52:26.802 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:26 smithi119 ceph-mon[105288]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:26 smithi119 ceph-mon[105288]: osdmap e1507: 8 total, 8 up, 8 in 2024-09-21T10:52:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:26 smithi119 ceph-mon[105288]: pgmap v2783: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:26.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:26 smithi119 ceph-mon[101432]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:26.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:26 smithi119 ceph-mon[101432]: osdmap e1507: 8 total, 8 up, 8 in 2024-09-21T10:52:26.821 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:26 smithi119 ceph-mon[101432]: pgmap v2783: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:26 smithi145 ceph-mon[89359]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-21T10:52:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:26 smithi145 ceph-mon[89359]: osdmap e1507: 8 total, 8 up, 8 in 2024-09-21T10:52:27.044 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:26 smithi145 ceph-mon[89359]: pgmap v2783: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:28.619 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:52:28.619 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":1507,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T10:52:25.718036+0000","last_up_change":"2024-09-21T10:27:18.842900+0000","last_in_change":"2024-09-21T10:51:25.772388+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":51,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1490","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1480","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:56:58.923308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:56:59.224014+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":496,"up_from":509,"up_thru":1488,"down_at":497,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6803","nonce":2607764328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6805","nonce":2607764328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6809","nonce":2607764328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6807","nonce":2607764328}]},"public_addr":"172.21.15.119:6803/2607764328","cluster_addr":"172.21.15.119:6805/2607764328","heartbeat_back_addr":"172.21.15.119:6809/2607764328","heartbeat_front_addr":"172.21.15.119:6807/2607764328","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":529,"up_from":542,"up_thru":1488,"down_at":530,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6811","nonce":2769715805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6813","nonce":2769715805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6817","nonce":2769715805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6815","nonce":2769715805}]},"public_addr":"172.21.15.119:6811/2769715805","cluster_addr":"172.21.15.119:6813/2769715805","heartbeat_back_addr":"172.21.15.119:6817/2769715805","heartbeat_front_addr":"172.21.15.119:6815/2769715805","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":613,"up_from":626,"up_thru":1488,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6819","nonce":3806024048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6821","nonce":3806024048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6825","nonce":3806024048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6823","nonce":3806024048}]},"public_addr":"172.21.15.119:6819/3806024048","cluster_addr":"172.21.15.119:6821/3806024048","heartbeat_back_addr":"172.21.15.119:6825/3806024048","heartbeat_front_addr":"172.21.15.119:6823/3806024048","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":644,"up_from":653,"up_thru":1488,"down_at":644,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6827","nonce":1249776352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6829","nonce":1249776352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6833","nonce":1249776352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6831","nonce":1249776352}]},"public_addr":"172.21.15.119:6827/1249776352","cluster_addr":"172.21.15.119:6829/1249776352","heartbeat_back_addr":"172.21.15.119:6833/1249776352","heartbeat_front_addr":"172.21.15.119:6831/1249776352","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":1199,"up_from":1211,"up_thru":1488,"down_at":1200,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6801","nonce":842981597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6803","nonce":842981597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6807","nonce":842981597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6805","nonce":842981597}]},"public_addr":"172.21.15.145:6801/842981597","cluster_addr":"172.21.15.145:6803/842981597","heartbeat_back_addr":"172.21.15.145:6807/842981597","heartbeat_front_addr":"172.21.15.145:6805/842981597","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1488,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":43,"last_clean_end":1184,"up_from":1191,"up_thru":1488,"down_at":1185,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6817","nonce":2712241400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6819","nonce":2712241400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6823","nonce":2712241400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6821","nonce":2712241400}]},"public_addr":"172.21.15.145:6817/2712241400","cluster_addr":"172.21.15.145:6819/2712241400","heartbeat_back_addr":"172.21.15.145:6823/2712241400","heartbeat_front_addr":"172.21.15.145:6821/2712241400","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1502,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":496},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":529},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":613},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":644},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":1199},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":1184},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.17","mappings":[{"from":3,"to":2}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/1139856442":"2024-09-21T11:11:38.888722+0000","172.21.15.119:0/2535372813":"2024-09-21T11:06:31.109355+0000","172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.145:0/2808539890":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/49791021":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/576401189":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3992357764":"2024-09-21T11:06:32.113588+0000","172.21.15.119:0/194249182":"2024-09-22T10:01:32.364993+0000","172.21.15.145:6835/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/2781454362":"2024-09-22T09:59:43.810176+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1608923788":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.145:0/974232973":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/547636249":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/460015937":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.145:0/2571529276":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.145:6834/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2566533967":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/2715091572":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/923321224":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/391955164":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/3401809855":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/3153987795":"2024-09-22T10:01:32.364993+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":4,"length":5}]},{"pool":16,"snaps":[{"begin":30,"length":8},{"begin":40,"length":8}]},{"pool":17,"snaps":[{"begin":17,"length":2},{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":43,"snaps":[{"begin":104,"length":1}]},{"pool":44,"snaps":[{"begin":104,"length":1}]},{"pool":51,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:52:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:29 smithi119 ceph-mon[105288]: pgmap v2784: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:29.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:29 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2815056365' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:52:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:29 smithi119 ceph-mon[101432]: pgmap v2784: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:29.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:29 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2815056365' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:52:29.417 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T10:52:29.417 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T10:52:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:29 smithi145 ceph-mon[89359]: pgmap v2784: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:29.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:29 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2815056365' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:52:29.662 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:30.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:52:30 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:52:30.212Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:52:31.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:31 smithi119 ceph-mon[101432]: pgmap v2785: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:31.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:31 smithi119 ceph-mon[105288]: pgmap v2785: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:31.502 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:52:31.503 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":1507,"fsid":"c96f302a-77fe-11ef-baf6-efdc52797490","created":"2024-09-21T09:50:08.133736+0000","modified":"2024-09-21T10:52:25.718036+0000","last_up_change":"2024-09-21T10:27:18.842900+0000","last_in_change":"2024-09-21T10:51:25.772388+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":24,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":51,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T09:53:12.866105+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1490","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1480","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.059999942779541,"score_stable":2.059999942779541,"optimal_score":1,"raw_score_acting":2.059999942779541,"raw_score_stable":2.059999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T09:55:40.420041+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":58,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs.foo.meta","create_time":"2024-09-21T09:56:58.923308+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs.foo.data","create_time":"2024-09-21T09:56:59.224014+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"foo"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cac46a83-240b-41e9-9cbe-6b685ebbb92b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":496,"up_from":509,"up_thru":1488,"down_at":497,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6803","nonce":2607764328}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6805","nonce":2607764328}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6809","nonce":2607764328}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2607764328},{"type":"v1","addr":"172.21.15.119:6807","nonce":2607764328}]},"public_addr":"172.21.15.119:6803/2607764328","cluster_addr":"172.21.15.119:6805/2607764328","heartbeat_back_addr":"172.21.15.119:6809/2607764328","heartbeat_front_addr":"172.21.15.119:6807/2607764328","state":["exists","up"]},{"osd":1,"uuid":"fe034449-410d-4164-9895-411181da04f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":529,"up_from":542,"up_thru":1488,"down_at":530,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6811","nonce":2769715805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6813","nonce":2769715805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6817","nonce":2769715805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":2769715805},{"type":"v1","addr":"172.21.15.119:6815","nonce":2769715805}]},"public_addr":"172.21.15.119:6811/2769715805","cluster_addr":"172.21.15.119:6813/2769715805","heartbeat_back_addr":"172.21.15.119:6817/2769715805","heartbeat_front_addr":"172.21.15.119:6815/2769715805","state":["exists","up"]},{"osd":2,"uuid":"db36e41e-f53c-4033-85fe-11dd420f633b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":613,"up_from":626,"up_thru":1488,"down_at":614,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6819","nonce":3806024048}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6821","nonce":3806024048}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6825","nonce":3806024048}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":3806024048},{"type":"v1","addr":"172.21.15.119:6823","nonce":3806024048}]},"public_addr":"172.21.15.119:6819/3806024048","cluster_addr":"172.21.15.119:6821/3806024048","heartbeat_back_addr":"172.21.15.119:6825/3806024048","heartbeat_front_addr":"172.21.15.119:6823/3806024048","state":["exists","up"]},{"osd":3,"uuid":"2f44491e-2550-4886-88cc-5d1c790f8783","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":644,"up_from":653,"up_thru":1488,"down_at":644,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6827","nonce":1249776352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6829","nonce":1249776352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6833","nonce":1249776352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":1249776352},{"type":"v1","addr":"172.21.15.119:6831","nonce":1249776352}]},"public_addr":"172.21.15.119:6827/1249776352","cluster_addr":"172.21.15.119:6829/1249776352","heartbeat_back_addr":"172.21.15.119:6833/1249776352","heartbeat_front_addr":"172.21.15.119:6831/1249776352","state":["exists","up"]},{"osd":4,"uuid":"2315f8d6-fbe6-4590-8842-d8aba506a0b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":31,"last_clean_end":1199,"up_from":1211,"up_thru":1488,"down_at":1200,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6801","nonce":842981597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6803","nonce":842981597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6807","nonce":842981597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":842981597},{"type":"v1","addr":"172.21.15.145:6805","nonce":842981597}]},"public_addr":"172.21.15.145:6801/842981597","cluster_addr":"172.21.15.145:6803/842981597","heartbeat_back_addr":"172.21.15.145:6807/842981597","heartbeat_front_addr":"172.21.15.145:6805/842981597","state":["exists","up"]},{"osd":5,"uuid":"e02fdc71-b568-4361-ab0f-1569f55f6034","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":1488,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6809","nonce":340169320}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6811","nonce":340169320}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6815","nonce":340169320}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":340169320},{"type":"v1","addr":"172.21.15.145:6813","nonce":340169320}]},"public_addr":"172.21.15.145:6809/340169320","cluster_addr":"172.21.15.145:6811/340169320","heartbeat_back_addr":"172.21.15.145:6815/340169320","heartbeat_front_addr":"172.21.15.145:6813/340169320","state":["exists","up"]},{"osd":6,"uuid":"e5bc8e69-00be-456e-b6fd-d6e4e43029a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":43,"last_clean_end":1184,"up_from":1191,"up_thru":1488,"down_at":1185,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6817","nonce":2712241400}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6819","nonce":2712241400}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6823","nonce":2712241400}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":2712241400},{"type":"v1","addr":"172.21.15.145:6821","nonce":2712241400}]},"public_addr":"172.21.15.145:6817/2712241400","cluster_addr":"172.21.15.145:6819/2712241400","heartbeat_back_addr":"172.21.15.145:6823/2712241400","heartbeat_front_addr":"172.21.15.145:6821/2712241400","state":["exists","up"]},{"osd":7,"uuid":"cfc1495b-a36e-4f6f-8491-781460970f42","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":1502,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6825","nonce":3509158741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6827","nonce":3509158741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6831","nonce":3509158741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":3509158741},{"type":"v1","addr":"172.21.15.145:6829","nonce":3509158741}]},"public_addr":"172.21.15.145:6825/3509158741","cluster_addr":"172.21.15.145:6827/3509158741","heartbeat_back_addr":"172.21.15.145:6831/3509158741","heartbeat_front_addr":"172.21.15.145:6829/3509158741","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:20.691221+0000","dead_epoch":496},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:52:44.810119+0000","dead_epoch":529},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:09.395744+0000","dead_epoch":613},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:34.572776+0000","dead_epoch":644},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:53:56.486994+0000","dead_epoch":1199},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:20.666347+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:54:45.541031+0000","dead_epoch":1184},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T09:55:09.778122+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.17","mappings":[{"from":3,"to":2}]},{"pgid":"3.19","mappings":[{"from":1,"to":2}]},{"pgid":"3.1b","mappings":[{"from":0,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/1139856442":"2024-09-21T11:11:38.888722+0000","172.21.15.119:0/2535372813":"2024-09-21T11:06:31.109355+0000","172.21.15.119:6801/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/391955164":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/3153987795":"2024-09-22T10:01:32.364993+0000","172.21.15.119:6800/2722863607":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/923321224":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/760350456":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/2566533967":"2024-09-22T09:59:43.810176+0000","172.21.15.145:6834/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/49791021":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/576401189":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/3992357764":"2024-09-21T11:06:32.113588+0000","172.21.15.119:0/194249182":"2024-09-22T10:01:32.364993+0000","172.21.15.145:6835/2195859977":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/1608923788":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/2211199843":"2024-09-22T09:50:36.170901+0000","172.21.15.145:0/974232973":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/460015937":"2024-09-22T10:01:32.364993+0000","172.21.15.119:0/547636249":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/2571529276":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/277848857":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/2781454362":"2024-09-22T09:59:43.810176+0000","172.21.15.119:6800/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.119:0/893592105":"2024-09-22T09:50:58.568079+0000","172.21.15.119:0/1497467670":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6800/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6801/704164355":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6801/1747871475":"2024-09-22T09:55:43.105417+0000","172.21.15.145:0/2715091572":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/23023314":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6800/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:6801/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.119:0/3980060910":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/1218540943":"2024-09-22T09:50:58.568079+0000","172.21.15.119:6800/1780938388":"2024-09-22T09:59:43.810176+0000","172.21.15.145:0/3401809855":"2024-09-22T10:00:49.315248+0000","172.21.15.119:0/2905946815":"2024-09-22T09:55:43.105417+0000","172.21.15.119:6801/4205960579":"2024-09-22T09:50:36.170901+0000","172.21.15.119:0/4255735773":"2024-09-22T09:50:58.568079+0000","172.21.15.145:0/2808539890":"2024-09-22T10:00:49.315248+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":1}]},{"pool":6,"snaps":[{"begin":2,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":1}]},{"pool":8,"snaps":[{"begin":2,"length":1}]},{"pool":9,"snaps":[{"begin":2,"length":1}]},{"pool":10,"snaps":[{"begin":2,"length":1}]},{"pool":11,"snaps":[{"begin":2,"length":1}]},{"pool":14,"snaps":[{"begin":4,"length":5}]},{"pool":16,"snaps":[{"begin":30,"length":8},{"begin":40,"length":8}]},{"pool":17,"snaps":[{"begin":17,"length":2},{"begin":22,"length":1}]},{"pool":18,"snaps":[{"begin":2,"length":1}]},{"pool":43,"snaps":[{"begin":104,"length":1}]},{"pool":44,"snaps":[{"begin":104,"length":1}]},{"pool":51,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T10:52:31.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:31 smithi145 ceph-mon[89359]: pgmap v2785: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:32.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T10:52:32.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T10:52:32.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T10:52:32.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T10:52:32.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T10:52:32.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T10:52:32.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T10:52:32.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T10:52:32.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:32 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2562008565' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:52:32.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:32 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2562008565' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:52:32.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:32 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2562008565' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-21T10:52:32.728 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.735 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.736 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.737 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:32.739 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:33.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:33 smithi119 ceph-mon[101432]: pgmap v2786: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:33.459 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:33 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:33.459 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:33 smithi119 ceph-mon[105288]: pgmap v2786: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:33.459 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:33 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:33 smithi145 ceph-mon[89359]: pgmap v2786: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:33.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:33 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:34.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:52:34 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:52:34] "GET /metrics HTTP/1.1" 200 34419 "" "Prometheus/2.51.0" 2024-09-21T10:52:35.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:35 smithi145 ceph-mon[89359]: pgmap v2787: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:35.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:35 smithi119 ceph-mon[101432]: pgmap v2787: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:35.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:35 smithi119 ceph-mon[105288]: pgmap v2787: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:37.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:37 smithi119 ceph-mon[105288]: pgmap v2788: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:37.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:37 smithi119 ceph-mon[101432]: pgmap v2788: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:37.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:37 smithi145 ceph-mon[89359]: pgmap v2788: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:39.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:39 smithi145 ceph-mon[89359]: pgmap v2789: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:39.555 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:39 smithi119 ceph-mon[105288]: pgmap v2789: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:39.555 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:39 smithi119 ceph-mon[101432]: pgmap v2789: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:40.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:52:40 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:52:40.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:52:41.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:41 smithi145 ceph-mon[89359]: pgmap v2790: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:41.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:41 smithi119 ceph-mon[101432]: pgmap v2790: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:41.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:41 smithi119 ceph-mon[105288]: pgmap v2790: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:43.342 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:43 smithi119 ceph-mon[105288]: pgmap v2791: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:43.342 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:43 smithi119 ceph-mon[101432]: pgmap v2791: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:43.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:43 smithi145 ceph-mon[89359]: pgmap v2791: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:43.767 INFO:teuthology.orchestra.run.smithi119.stdout:158913790656 2024-09-21T10:52:43.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T10:52:44.844 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:52:44 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:52:44] "GET /metrics HTTP/1.1" 200 34419 "" "Prometheus/2.51.0" 2024-09-21T10:52:44.868 INFO:teuthology.orchestra.run.smithi119.stdout:2327872274987 2024-09-21T10:52:44.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T10:52:45.302 INFO:teuthology.orchestra.run.smithi119.stdout:5201205395762 2024-09-21T10:52:45.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T10:52:45.506 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:45 smithi119 ceph-mon[101432]: pgmap v2792: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:45.507 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:45 smithi119 ceph-mon[105288]: pgmap v2792: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:45.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:45 smithi145 ceph-mon[89359]: pgmap v2792: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:46.663 INFO:teuthology.orchestra.run.smithi119.stdout:5115306049857 2024-09-21T10:52:46.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T10:52:47.077 INFO:teuthology.orchestra.run.smithi119.stdout:2804613644816 2024-09-21T10:52:47.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T10:52:47.336 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:47 smithi119 ceph-mon[101432]: pgmap v2793: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:47.336 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:47 smithi119 ceph-mon[105288]: pgmap v2793: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:47.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:47 smithi145 ceph-mon[89359]: pgmap v2793: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:48.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:48 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:48.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:48 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:48.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:48 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:52:48.846 INFO:teuthology.orchestra.run.smithi119.stdout:2688649527835 2024-09-21T10:52:48.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T10:52:49.257 INFO:teuthology.orchestra.run.smithi119.stdout:2186138354227 2024-09-21T10:52:49.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T10:52:49.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:49 smithi145 ceph-mon[89359]: pgmap v2794: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:49.608 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:49 smithi119 ceph-mon[105288]: pgmap v2794: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:49.608 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:49 smithi119 ceph-mon[101432]: pgmap v2794: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:49.616 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:49.616 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:49.617 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:49.642 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:49.683 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:49.683 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:50.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:52:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:52:50.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:52:51.346 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:51 smithi119 ceph-mon[101432]: pgmap v2795: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:51.347 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:51 smithi119 ceph-mon[105288]: pgmap v2795: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:51.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:51 smithi145 ceph-mon[89359]: pgmap v2795: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:53.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:53 smithi119 ceph-mon[105288]: pgmap v2796: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:53.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:53 smithi119 ceph-mon[101432]: pgmap v2796: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:53.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:53 smithi145 ceph-mon[89359]: pgmap v2796: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:54.793 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:52:54 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:52:54] "GET /metrics HTTP/1.1" 200 34421 "" "Prometheus/2.51.0" 2024-09-21T10:52:55.240 INFO:teuthology.orchestra.run.smithi119.stdout:158913790659 2024-09-21T10:52:55.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:55 smithi119 ceph-mon[101432]: pgmap v2797: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:55.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:55 smithi119 ceph-mon[105288]: pgmap v2797: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:55.450 INFO:teuthology.orchestra.run.smithi119.stdout:2804613644818 2024-09-21T10:52:55.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:55 smithi145 ceph-mon[89359]: pgmap v2797: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:56.339 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/264084021' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:52:56.339 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:56 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2560365607' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:52:56.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/264084021' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:52:56.339 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:56 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2560365607' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:52:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/264084021' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-21T10:52:56.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:56 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2560365607' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-21T10:52:57.497 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:57 smithi119 ceph-mon[105288]: pgmap v2798: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:57.498 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:57 smithi119 ceph-mon[101432]: pgmap v2798: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:57.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:57 smithi145 ceph-mon[89359]: pgmap v2798: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:57.892 INFO:teuthology.orchestra.run.smithi119.stdout:210453398201 2024-09-21T10:52:57.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T10:52:59.035 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:52:59.054 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2804613644816 got 2804613644818 for osd.3 2024-09-21T10:52:59.054 DEBUG:teuthology.parallel:result is None 2024-09-21T10:52:59.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:52:59 smithi119 ceph-mon[101432]: pgmap v2799: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:59.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:52:59 smithi119 ceph-mon[105288]: pgmap v2799: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:52:59.439 INFO:teuthology.orchestra.run.smithi119.stdout:5201205395765 2024-09-21T10:52:59.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:52:59 smithi145 ceph-mon[89359]: pgmap v2799: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:00.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:53:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:53:00.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:53:00.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:00 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3921108113' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:53:00.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:00 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3921108113' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:53:00.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:00 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3921108113' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-21T10:53:00.693 INFO:tasks.cephadm.ceph_manager.ceph:need seq 5201205395762 got 5201205395765 for osd.4 2024-09-21T10:53:00.693 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:01.131 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790656 got 158913790659 for osd.5 2024-09-21T10:53:01.132 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:01.361 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:01 smithi119 ceph-mon[101432]: pgmap v2800: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:01.361 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:01 smithi119 ceph-mon[105288]: pgmap v2800: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:01.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:01 smithi145 ceph-mon[89359]: pgmap v2800: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:02.415 INFO:teuthology.orchestra.run.smithi119.stdout:2327872274991 2024-09-21T10:53:03.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2327872274987 got 2327872274991 for osd.1 2024-09-21T10:53:03.162 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:03.538 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:03 smithi119 ceph-mon[105288]: pgmap v2801: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:03.539 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:03 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2669588963' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:53:03.539 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:03 smithi119 ceph-mon[105288]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:03.539 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:03 smithi119 ceph-mon[101432]: pgmap v2801: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:03.539 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:03 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2669588963' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:53:03.539 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:03 smithi119 ceph-mon[101432]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:03 smithi145 ceph-mon[89359]: pgmap v2801: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:03 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2669588963' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-21T10:53:03.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:03 smithi145 ceph-mon[89359]: from='mgr.34104 172.21.15.119:0/4058245371' entity='mgr.y' cmd={"prefix": "osd blocklist ls", "format": "json"} : dispatch 2024-09-21T10:53:03.967 INFO:teuthology.orchestra.run.smithi119.stdout:5115306049861 2024-09-21T10:53:04.311 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:04 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/2122997237' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:53:04.311 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:04 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/2122997237' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:53:04.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:04 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/2122997237' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-21T10:53:04.580 INFO:teuthology.orchestra.run.smithi119.stdout:2186138354230 2024-09-21T10:53:04.580 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:04 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:53:04] "GET /metrics HTTP/1.1" 200 34421 "" "Prometheus/2.51.0" 2024-09-21T10:53:04.848 INFO:tasks.cephadm.ceph_manager.ceph:need seq 5115306049857 got 5115306049861 for osd.6 2024-09-21T10:53:04.848 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:05 smithi119 ceph-mon[105288]: pgmap v2802: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:05.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:05 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3947561059' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:53:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:05 smithi119 ceph-mon[101432]: pgmap v2802: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:05.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:05 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3947561059' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:53:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:05 smithi145 ceph-mon[89359]: pgmap v2802: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:05.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:05 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3947561059' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-21T10:53:05.697 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:05.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2186138354227 got 2186138354230 for osd.0 2024-09-21T10:53:05.698 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:06.345 INFO:teuthology.orchestra.run.smithi119.stdout:2688649527839 2024-09-21T10:53:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:07 smithi145 ceph-mon[89359]: pgmap v2803: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:07.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:07 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/3317530657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:53:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:07 smithi119 ceph-mon[101432]: pgmap v2803: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:07.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:07 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/3317530657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:53:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:07 smithi119 ceph-mon[105288]: pgmap v2803: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:07.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:07 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/3317530657' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-21T10:53:08.187 INFO:tasks.cephadm.ceph_manager.ceph:need seq 2688649527835 got 2688649527839 for osd.2 2024-09-21T10:53:08.187 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:08.840 INFO:teuthology.orchestra.run.smithi119.stdout:210453398203 2024-09-21T10:53:09.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:09 smithi119 ceph-mon[105288]: pgmap v2804: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:09.518 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:09 smithi119 ceph-mon[105288]: from='client.? 172.21.15.119:0/1960571149' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:53:09.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:09 smithi119 ceph-mon[101432]: pgmap v2804: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:09.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:09 smithi119 ceph-mon[101432]: from='client.? 172.21.15.119:0/1960571149' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:53:09.542 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453398201 got 210453398203 for osd.7 2024-09-21T10:53:09.542 DEBUG:teuthology.parallel:result is None 2024-09-21T10:53:09.543 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T10:53:09.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:53:09.543 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:09 smithi145 ceph-mon[89359]: pgmap v2804: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:09.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:09 smithi145 ceph-mon[89359]: from='client.? 172.21.15.119:0/1960571149' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-21T10:53:09.786 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:10.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:53:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:53:10.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:53:11.307 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:11 smithi119 ceph-mon[105288]: pgmap v2805: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:11.308 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:11 smithi119 ceph-mon[101432]: pgmap v2805: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:11.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:11 smithi145 ceph-mon[89359]: pgmap v2805: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:11.706 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:53:11.707 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:53:12.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:12 smithi119 ceph-mon[105288]: from='client.37746 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:12.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:12 smithi119 ceph-mon[101432]: from='client.37746 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:12.425 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2805,"stamp":"2024-09-21T10:53:10.118559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":12374,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3158,"ondisk_log_size":3158,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7932408,"kb_used_data":82136,"kb_used_omap":731,"kb_used_meta":7849444,"kb_avail":742028808,"statfs":{"total":767960285184,"available":759837499392,"internally_reserved":0,"allocated":84107264,"data_stored":47333463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":749144,"internal_metadata":8037831080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002899"},"pg_stats":[{"pgid":"1.1d","version":"256'100","reported_seq":234,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732801+0000","last_change":"2024-09-21T10:51:27.068248+0000","last_active":"2024-09-21T10:52:25.732801+0000","last_peered":"2024-09-21T10:52:25.732801+0000","last_clean":"2024-09-21T10:52:25.732801+0000","last_became_active":"2024-09-21T10:51:19.718918+0000","last_became_peered":"2024-09-21T10:51:19.718918+0000","last_unstale":"2024-09-21T10:52:25.732801+0000","last_undegraded":"2024-09-21T10:52:25.732801+0000","last_fullsized":"2024-09-21T10:52:25.732801+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:27.068193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:27.068193+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:33.048016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":330,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731651+0000","last_change":"2024-09-21T10:51:23.738777+0000","last_active":"2024-09-21T10:52:25.731651+0000","last_peered":"2024-09-21T10:52:25.731651+0000","last_clean":"2024-09-21T10:52:25.731651+0000","last_became_active":"2024-09-21T10:51:23.729563+0000","last_became_peered":"2024-09-21T10:51:23.729563+0000","last_unstale":"2024-09-21T10:52:25.731651+0000","last_undegraded":"2024-09-21T10:52:25.731651+0000","last_fullsized":"2024-09-21T10:52:25.731651+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:41.467046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859948+0000","last_change":"2024-09-21T10:51:19.699841+0000","last_active":"2024-09-21T10:52:22.859948+0000","last_peered":"2024-09-21T10:52:22.859948+0000","last_clean":"2024-09-21T10:52:22.859948+0000","last_became_active":"2024-09-21T10:51:19.699493+0000","last_became_peered":"2024-09-21T10:51:19.699493+0000","last_unstale":"2024-09-21T10:52:22.859948+0000","last_undegraded":"2024-09-21T10:52:22.859948+0000","last_fullsized":"2024-09-21T10:52:22.859948+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:48.873414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"256'100","reported_seq":38,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860439+0000","last_change":"2024-09-21T10:51:39.018702+0000","last_active":"2024-09-21T10:52:22.860439+0000","last_peered":"2024-09-21T10:52:22.860439+0000","last_clean":"2024-09-21T10:52:22.860439+0000","last_became_active":"2024-09-21T10:51:26.804861+0000","last_became_peered":"2024-09-21T10:51:26.804861+0000","last_unstale":"2024-09-21T10:52:22.860439+0000","last_undegraded":"2024-09-21T10:52:22.860439+0000","last_fullsized":"2024-09-21T10:52:22.860439+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:39.018624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:39.018624+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:14.282200+0000","scrub_duration":0.0016991790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2342,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722616+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:52:25.722616+0000","last_peered":"2024-09-21T10:52:25.722616+0000","last_clean":"2024-09-21T10:52:25.722616+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:52:25.722616+0000","last_undegraded":"2024-09-21T10:52:25.722616+0000","last_fullsized":"2024-09-21T10:52:25.722616+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2359,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722335+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:52:25.722335+0000","last_peered":"2024-09-21T10:52:25.722335+0000","last_clean":"2024-09-21T10:52:25.722335+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:52:25.722335+0000","last_undegraded":"2024-09-21T10:52:25.722335+0000","last_fullsized":"2024-09-21T10:52:25.722335+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859720+0000","last_change":"2024-09-21T10:51:23.739774+0000","last_active":"2024-09-21T10:52:22.859720+0000","last_peered":"2024-09-21T10:52:22.859720+0000","last_clean":"2024-09-21T10:52:22.859720+0000","last_became_active":"2024-09-21T10:51:23.730493+0000","last_became_peered":"2024-09-21T10:51:23.730493+0000","last_unstale":"2024-09-21T10:52:22.859720+0000","last_undegraded":"2024-09-21T10:52:22.859720+0000","last_fullsized":"2024-09-21T10:52:22.859720+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:51.690849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860252+0000","last_change":"2024-09-21T10:51:19.710501+0000","last_active":"2024-09-21T10:52:22.860252+0000","last_peered":"2024-09-21T10:52:22.860252+0000","last_clean":"2024-09-21T10:52:22.860252+0000","last_became_active":"2024-09-21T10:51:19.710202+0000","last_became_peered":"2024-09-21T10:51:19.710202+0000","last_unstale":"2024-09-21T10:52:22.860252+0000","last_undegraded":"2024-09-21T10:52:22.860252+0000","last_fullsized":"2024-09-21T10:52:22.860252+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:11.158284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"256'100","reported_seq":562,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722286+0000","last_change":"2024-09-21T10:51:32.463718+0000","last_active":"2024-09-21T10:52:25.722286+0000","last_peered":"2024-09-21T10:52:25.722286+0000","last_clean":"2024-09-21T10:52:25.722286+0000","last_became_active":"2024-09-21T10:51:26.800500+0000","last_became_peered":"2024-09-21T10:51:26.800500+0000","last_unstale":"2024-09-21T10:52:25.722286+0000","last_undegraded":"2024-09-21T10:52:25.722286+0000","last_fullsized":"2024-09-21T10:52:25.722286+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:32.463666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:32.463666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:03.241851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860372+0000","last_change":"2024-09-21T10:51:23.735436+0000","last_active":"2024-09-21T10:52:22.860372+0000","last_peered":"2024-09-21T10:52:22.860372+0000","last_clean":"2024-09-21T10:52:22.860372+0000","last_became_active":"2024-09-21T10:51:23.724032+0000","last_became_peered":"2024-09-21T10:51:23.724032+0000","last_unstale":"2024-09-21T10:52:22.860372+0000","last_undegraded":"2024-09-21T10:52:22.860372+0000","last_fullsized":"2024-09-21T10:52:22.860372+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:45.454930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859597+0000","last_change":"2024-09-21T10:51:19.714313+0000","last_active":"2024-09-21T10:52:22.859597+0000","last_peered":"2024-09-21T10:52:22.859597+0000","last_clean":"2024-09-21T10:52:22.859597+0000","last_became_active":"2024-09-21T10:51:19.713999+0000","last_became_peered":"2024-09-21T10:51:19.713999+0000","last_unstale":"2024-09-21T10:52:22.859597+0000","last_undegraded":"2024-09-21T10:52:22.859597+0000","last_fullsized":"2024-09-21T10:52:22.859597+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:38.289820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"256'100","reported_seq":1571,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722006+0000","last_change":"2024-09-21T10:51:20.305009+0000","last_active":"2024-09-21T10:52:25.722006+0000","last_peered":"2024-09-21T10:52:25.722006+0000","last_clean":"2024-09-21T10:52:25.722006+0000","last_became_active":"2024-09-21T10:51:17.680713+0000","last_became_peered":"2024-09-21T10:51:17.680713+0000","last_unstale":"2024-09-21T10:52:25.722006+0000","last_undegraded":"2024-09-21T10:52:25.722006+0000","last_fullsized":"2024-09-21T10:52:25.722006+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1483,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:20.304962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:20.304962+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:00.486399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":182,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731903+0000","last_change":"2024-09-21T10:51:26.794168+0000","last_active":"2024-09-21T10:52:25.731903+0000","last_peered":"2024-09-21T10:52:25.731903+0000","last_clean":"2024-09-21T10:52:25.731903+0000","last_became_active":"2024-09-21T10:51:26.793759+0000","last_became_peered":"2024-09-21T10:51:26.793759+0000","last_unstale":"2024-09-21T10:52:25.731903+0000","last_undegraded":"2024-09-21T10:52:25.731903+0000","last_fullsized":"2024-09-21T10:52:25.731903+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:03.050183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":326,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732079+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:52:25.732079+0000","last_peered":"2024-09-21T10:52:25.732079+0000","last_clean":"2024-09-21T10:52:25.732079+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:52:25.732079+0000","last_undegraded":"2024-09-21T10:52:25.732079+0000","last_fullsized":"2024-09-21T10:52:25.732079+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"256'100","reported_seq":58,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859307+0000","last_change":"2024-09-21T10:51:33.806344+0000","last_active":"2024-09-21T10:52:22.859307+0000","last_peered":"2024-09-21T10:52:22.859307+0000","last_clean":"2024-09-21T10:52:22.859307+0000","last_became_active":"2024-09-21T10:51:26.813092+0000","last_became_peered":"2024-09-21T10:51:26.813092+0000","last_unstale":"2024-09-21T10:52:22.859307+0000","last_undegraded":"2024-09-21T10:52:22.859307+0000","last_fullsized":"2024-09-21T10:52:22.859307+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:33.806271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:33.806271+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:47:54.793994+0000","scrub_duration":0.0018767289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860169+0000","last_change":"2024-09-21T10:51:19.718380+0000","last_active":"2024-09-21T10:52:22.860169+0000","last_peered":"2024-09-21T10:52:22.860169+0000","last_clean":"2024-09-21T10:52:22.860169+0000","last_became_active":"2024-09-21T10:51:19.717955+0000","last_became_peered":"2024-09-21T10:51:19.717955+0000","last_unstale":"2024-09-21T10:52:22.860169+0000","last_undegraded":"2024-09-21T10:52:22.860169+0000","last_fullsized":"2024-09-21T10:52:22.860169+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:31.240897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2355,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722776+0000","last_change":"2024-09-21T10:51:19.706879+0000","last_active":"2024-09-21T10:52:25.722776+0000","last_peered":"2024-09-21T10:52:25.722776+0000","last_clean":"2024-09-21T10:52:25.722776+0000","last_became_active":"2024-09-21T10:51:19.706103+0000","last_became_peered":"2024-09-21T10:51:19.706103+0000","last_unstale":"2024-09-21T10:52:25.722776+0000","last_undegraded":"2024-09-21T10:52:25.722776+0000","last_fullsized":"2024-09-21T10:52:25.722776+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:43.017006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859364+0000","last_change":"2024-09-21T10:51:25.937690+0000","last_active":"2024-09-21T10:52:22.859364+0000","last_peered":"2024-09-21T10:52:22.859364+0000","last_clean":"2024-09-21T10:52:22.859364+0000","last_became_active":"2024-09-21T10:51:23.721779+0000","last_became_peered":"2024-09-21T10:51:23.721779+0000","last_unstale":"2024-09-21T10:52:22.859364+0000","last_undegraded":"2024-09-21T10:52:22.859364+0000","last_fullsized":"2024-09-21T10:52:22.859364+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:25.937626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:25.937626+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:37.783185+0000","scrub_duration":0.0016995199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":175,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731693+0000","last_change":"2024-09-21T10:51:19.704667+0000","last_active":"2024-09-21T10:52:25.731693+0000","last_peered":"2024-09-21T10:52:25.731693+0000","last_clean":"2024-09-21T10:52:25.731693+0000","last_became_active":"2024-09-21T10:51:19.704366+0000","last_became_peered":"2024-09-21T10:51:19.704366+0000","last_unstale":"2024-09-21T10:52:25.731693+0000","last_undegraded":"2024-09-21T10:52:25.731693+0000","last_fullsized":"2024-09-21T10:52:25.731693+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:48.376605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2471,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731921+0000","last_change":"2024-09-21T10:51:26.807345+0000","last_active":"2024-09-21T10:52:25.731921+0000","last_peered":"2024-09-21T10:52:25.731921+0000","last_clean":"2024-09-21T10:52:25.731921+0000","last_became_active":"2024-09-21T10:51:26.806887+0000","last_became_peered":"2024-09-21T10:51:26.806887+0000","last_unstale":"2024-09-21T10:52:25.731921+0000","last_undegraded":"2024-09-21T10:52:25.731921+0000","last_fullsized":"2024-09-21T10:52:25.731921+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:44.878329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"256'100","reported_seq":62,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860264+0000","last_change":"2024-09-21T10:51:27.399993+0000","last_active":"2024-09-21T10:52:22.860264+0000","last_peered":"2024-09-21T10:52:22.860264+0000","last_clean":"2024-09-21T10:52:22.860264+0000","last_became_active":"2024-09-21T10:51:26.796440+0000","last_became_peered":"2024-09-21T10:51:26.796440+0000","last_unstale":"2024-09-21T10:52:22.860264+0000","last_undegraded":"2024-09-21T10:52:22.860264+0000","last_fullsized":"2024-09-21T10:52:22.860264+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:27.399930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:27.399930+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:02:30.699583+0000","scrub_duration":0.0018178720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2374,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722090+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:52:25.722090+0000","last_peered":"2024-09-21T10:52:25.722090+0000","last_clean":"2024-09-21T10:52:25.722090+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:52:25.722090+0000","last_undegraded":"2024-09-21T10:52:25.722090+0000","last_fullsized":"2024-09-21T10:52:25.722090+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1423,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718185+0000","last_change":"2024-09-21T10:51:23.733462+0000","last_active":"2024-09-21T10:52:22.718185+0000","last_peered":"2024-09-21T10:52:22.718185+0000","last_clean":"2024-09-21T10:52:22.718185+0000","last_became_active":"2024-09-21T10:51:23.733150+0000","last_became_peered":"2024-09-21T10:51:23.733150+0000","last_unstale":"2024-09-21T10:52:22.718185+0000","last_undegraded":"2024-09-21T10:52:22.718185+0000","last_fullsized":"2024-09-21T10:52:22.718185+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:48.510994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"256'100","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861397+0000","last_change":"2024-09-21T10:51:26.809999+0000","last_active":"2024-09-21T10:52:22.861397+0000","last_peered":"2024-09-21T10:52:22.861397+0000","last_clean":"2024-09-21T10:52:22.861397+0000","last_became_active":"2024-09-21T10:51:26.808485+0000","last_became_peered":"2024-09-21T10:51:26.808485+0000","last_unstale":"2024-09-21T10:52:22.861397+0000","last_undegraded":"2024-09-21T10:52:22.861397+0000","last_fullsized":"2024-09-21T10:52:22.861397+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:21.142696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:21.142696+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:19.266752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2353,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722157+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:52:25.722157+0000","last_peered":"2024-09-21T10:52:25.722157+0000","last_clean":"2024-09-21T10:52:25.722157+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:52:25.722157+0000","last_undegraded":"2024-09-21T10:52:25.722157+0000","last_fullsized":"2024-09-21T10:52:25.722157+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2487,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731874+0000","last_change":"2024-09-21T10:51:23.720734+0000","last_active":"2024-09-21T10:52:25.731874+0000","last_peered":"2024-09-21T10:52:25.731874+0000","last_clean":"2024-09-21T10:52:25.731874+0000","last_became_active":"2024-09-21T10:51:23.720442+0000","last_became_peered":"2024-09-21T10:51:23.720442+0000","last_unstale":"2024-09-21T10:52:25.731874+0000","last_undegraded":"2024-09-21T10:52:25.731874+0000","last_fullsized":"2024-09-21T10:52:25.731874+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:12.592262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":508,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732053+0000","last_change":"2024-09-21T10:51:26.794751+0000","last_active":"2024-09-21T10:52:25.732053+0000","last_peered":"2024-09-21T10:52:25.732053+0000","last_clean":"2024-09-21T10:52:25.732053+0000","last_became_active":"2024-09-21T10:51:26.794375+0000","last_became_peered":"2024-09-21T10:51:26.794375+0000","last_unstale":"2024-09-21T10:52:25.732053+0000","last_undegraded":"2024-09-21T10:52:25.732053+0000","last_fullsized":"2024-09-21T10:52:25.732053+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:53.843188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2364,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722408+0000","last_change":"2024-09-21T10:51:26.809260+0000","last_active":"2024-09-21T10:52:25.722408+0000","last_peered":"2024-09-21T10:52:25.722408+0000","last_clean":"2024-09-21T10:52:25.722408+0000","last_became_active":"2024-09-21T10:51:26.808907+0000","last_became_peered":"2024-09-21T10:51:26.808907+0000","last_unstale":"2024-09-21T10:52:25.722408+0000","last_undegraded":"2024-09-21T10:52:25.722408+0000","last_fullsized":"2024-09-21T10:52:25.722408+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.645807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859754+0000","last_change":"2024-09-21T10:51:26.809002+0000","last_active":"2024-09-21T10:52:22.859754+0000","last_peered":"2024-09-21T10:52:22.859754+0000","last_clean":"2024-09-21T10:52:22.859754+0000","last_became_active":"2024-09-21T10:51:26.808657+0000","last_became_peered":"2024-09-21T10:51:26.808657+0000","last_unstale":"2024-09-21T10:52:22.859754+0000","last_undegraded":"2024-09-21T10:52:22.859754+0000","last_fullsized":"2024-09-21T10:52:22.859754+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:18.104217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"256'100","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860135+0000","last_change":"2024-09-21T10:51:23.726165+0000","last_active":"2024-09-21T10:52:22.860135+0000","last_peered":"2024-09-21T10:52:22.860135+0000","last_clean":"2024-09-21T10:52:22.860135+0000","last_became_active":"2024-09-21T10:51:23.725783+0000","last_became_peered":"2024-09-21T10:51:23.725783+0000","last_unstale":"2024-09-21T10:52:22.860135+0000","last_undegraded":"2024-09-21T10:52:22.860135+0000","last_fullsized":"2024-09-21T10:52:22.860135+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:21.730413+0000","last_deep_scrub":"256'100","last_deep_scrub_stamp":"2024-09-21T10:51:16.536490+0000","last_clean_scrub_stamp":"2024-09-21T10:51:21.730413+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:17.270019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861348+0000","last_change":"2024-09-21T10:51:26.820653+0000","last_active":"2024-09-21T10:52:22.861348+0000","last_peered":"2024-09-21T10:52:22.861348+0000","last_clean":"2024-09-21T10:52:22.861348+0000","last_became_active":"2024-09-21T10:51:26.799521+0000","last_became_peered":"2024-09-21T10:51:26.799521+0000","last_unstale":"2024-09-21T10:52:22.861348+0000","last_undegraded":"2024-09-21T10:52:22.861348+0000","last_fullsized":"2024-09-21T10:52:22.861348+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:42.996013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":175,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731739+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:52:25.731739+0000","last_peered":"2024-09-21T10:52:25.731739+0000","last_clean":"2024-09-21T10:52:25.731739+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:52:25.731739+0000","last_undegraded":"2024-09-21T10:52:25.731739+0000","last_fullsized":"2024-09-21T10:52:25.731739+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"256'100","reported_seq":69,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731834+0000","last_change":"2024-09-21T10:51:19.154684+0000","last_active":"2024-09-21T10:52:25.731834+0000","last_peered":"2024-09-21T10:52:25.731834+0000","last_clean":"2024-09-21T10:52:25.731834+0000","last_became_active":"2024-09-21T10:51:17.681580+0000","last_became_peered":"2024-09-21T10:51:17.681580+0000","last_unstale":"2024-09-21T10:52:25.731834+0000","last_undegraded":"2024-09-21T10:52:25.731834+0000","last_fullsized":"2024-09-21T10:52:25.731834+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1483,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:19.154633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:19.154633+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:31.453042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":62,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859929+0000","last_change":"2024-09-21T10:51:26.815362+0000","last_active":"2024-09-21T10:52:22.859929+0000","last_peered":"2024-09-21T10:52:22.859929+0000","last_clean":"2024-09-21T10:52:22.859929+0000","last_became_active":"2024-09-21T10:51:26.797456+0000","last_became_peered":"2024-09-21T10:51:26.797456+0000","last_unstale":"2024-09-21T10:52:22.859929+0000","last_undegraded":"2024-09-21T10:52:22.859929+0000","last_fullsized":"2024-09-21T10:52:22.859929+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:07.587157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859840+0000","last_change":"2024-09-21T10:51:19.718455+0000","last_active":"2024-09-21T10:52:22.859840+0000","last_peered":"2024-09-21T10:52:22.859840+0000","last_clean":"2024-09-21T10:52:22.859840+0000","last_became_active":"2024-09-21T10:51:19.717807+0000","last_became_peered":"2024-09-21T10:51:19.717807+0000","last_unstale":"2024-09-21T10:52:22.859840+0000","last_undegraded":"2024-09-21T10:52:22.859840+0000","last_fullsized":"2024-09-21T10:52:22.859840+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:38.313963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":643,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722475+0000","last_change":"2024-09-21T10:51:26.801568+0000","last_active":"2024-09-21T10:52:25.722475+0000","last_peered":"2024-09-21T10:52:25.722475+0000","last_clean":"2024-09-21T10:52:25.722475+0000","last_became_active":"2024-09-21T10:51:26.801292+0000","last_became_peered":"2024-09-21T10:51:26.801292+0000","last_unstale":"2024-09-21T10:52:25.722475+0000","last_undegraded":"2024-09-21T10:52:25.722475+0000","last_fullsized":"2024-09-21T10:52:25.722475+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:14.972988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":42,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859417+0000","last_change":"2024-09-21T10:51:23.720921+0000","last_active":"2024-09-21T10:52:22.859417+0000","last_peered":"2024-09-21T10:52:22.859417+0000","last_clean":"2024-09-21T10:52:22.859417+0000","last_became_active":"2024-09-21T10:51:23.720575+0000","last_became_peered":"2024-09-21T10:51:23.720575+0000","last_unstale":"2024-09-21T10:52:22.859417+0000","last_undegraded":"2024-09-21T10:52:22.859417+0000","last_fullsized":"2024-09-21T10:52:22.859417+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:40.539488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1419,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718111+0000","last_change":"2024-09-21T10:51:26.802469+0000","last_active":"2024-09-21T10:52:22.718111+0000","last_peered":"2024-09-21T10:52:22.718111+0000","last_clean":"2024-09-21T10:52:22.718111+0000","last_became_active":"2024-09-21T10:51:26.802135+0000","last_became_peered":"2024-09-21T10:51:26.802135+0000","last_unstale":"2024-09-21T10:52:22.718111+0000","last_undegraded":"2024-09-21T10:52:22.718111+0000","last_fullsized":"2024-09-21T10:52:22.718111+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:11.432966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":364,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731736+0000","last_change":"2024-09-21T10:51:19.727865+0000","last_active":"2024-09-21T10:52:25.731736+0000","last_peered":"2024-09-21T10:52:25.731736+0000","last_clean":"2024-09-21T10:52:25.731736+0000","last_became_active":"2024-09-21T10:51:19.715076+0000","last_became_peered":"2024-09-21T10:51:19.715076+0000","last_unstale":"2024-09-21T10:52:25.731736+0000","last_undegraded":"2024-09-21T10:52:25.731736+0000","last_fullsized":"2024-09-21T10:52:25.731736+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:10.385979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":42,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860412+0000","last_change":"2024-09-21T10:51:23.423247+0000","last_active":"2024-09-21T10:52:22.860412+0000","last_peered":"2024-09-21T10:52:22.860412+0000","last_clean":"2024-09-21T10:52:22.860412+0000","last_became_active":"2024-09-21T10:51:19.699614+0000","last_became_peered":"2024-09-21T10:51:19.699614+0000","last_unstale":"2024-09-21T10:52:22.860412+0000","last_undegraded":"2024-09-21T10:52:22.860412+0000","last_fullsized":"2024-09-21T10:52:22.860412+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:23.423180+0000","last_deep_scrub":"256'100","last_deep_scrub_stamp":"2024-09-21T10:51:23.423180+0000","last_clean_scrub_stamp":"2024-09-21T10:51:23.423180+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:37.289679+0000","scrub_duration":0.0016804439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":149,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.721966+0000","last_change":"2024-09-21T10:51:26.801503+0000","last_active":"2024-09-21T10:52:25.721966+0000","last_peered":"2024-09-21T10:52:25.721966+0000","last_clean":"2024-09-21T10:52:25.721966+0000","last_became_active":"2024-09-21T10:51:26.801140+0000","last_became_peered":"2024-09-21T10:51:26.801140+0000","last_unstale":"2024-09-21T10:52:25.721966+0000","last_undegraded":"2024-09-21T10:52:25.721966+0000","last_fullsized":"2024-09-21T10:52:25.721966+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:18.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2463,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732195+0000","last_change":"2024-09-21T10:51:19.718247+0000","last_active":"2024-09-21T10:52:25.732195+0000","last_peered":"2024-09-21T10:52:25.732195+0000","last_clean":"2024-09-21T10:52:25.732195+0000","last_became_active":"2024-09-21T10:51:19.717892+0000","last_became_peered":"2024-09-21T10:51:19.717892+0000","last_unstale":"2024-09-21T10:52:25.732195+0000","last_undegraded":"2024-09-21T10:52:25.732195+0000","last_fullsized":"2024-09-21T10:52:25.732195+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:41.786269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":216,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732299+0000","last_change":"2024-09-21T10:51:26.817216+0000","last_active":"2024-09-21T10:52:25.732299+0000","last_peered":"2024-09-21T10:52:25.732299+0000","last_clean":"2024-09-21T10:52:25.732299+0000","last_became_active":"2024-09-21T10:51:26.797691+0000","last_became_peered":"2024-09-21T10:51:26.797691+0000","last_unstale":"2024-09-21T10:52:25.732299+0000","last_undegraded":"2024-09-21T10:52:25.732299+0000","last_fullsized":"2024-09-21T10:52:25.732299+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:34.583770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":600,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718055+0000","last_change":"2024-09-21T10:51:13.523081+0000","last_active":"2024-09-21T10:52:22.718055+0000","last_peered":"2024-09-21T10:52:22.718055+0000","last_clean":"2024-09-21T10:52:22.718055+0000","last_became_active":"2024-09-21T10:51:11.652771+0000","last_became_peered":"2024-09-21T10:51:11.652771+0000","last_unstale":"2024-09-21T10:52:22.718055+0000","last_undegraded":"2024-09-21T10:52:22.718055+0000","last_fullsized":"2024-09-21T10:52:22.718055+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1481,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:13.523014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:13.523014+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:25:29.983664+0000","scrub_duration":0.0019017459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2487,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732810+0000","last_change":"2024-09-21T10:51:26.804052+0000","last_active":"2024-09-21T10:52:25.732810+0000","last_peered":"2024-09-21T10:52:25.732810+0000","last_clean":"2024-09-21T10:52:25.732810+0000","last_became_active":"2024-09-21T10:51:26.803711+0000","last_became_peered":"2024-09-21T10:51:26.803711+0000","last_unstale":"2024-09-21T10:52:25.732810+0000","last_undegraded":"2024-09-21T10:52:25.732810+0000","last_fullsized":"2024-09-21T10:52:25.732810+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:34.399518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860614+0000","last_change":"2024-09-21T10:51:26.801966+0000","last_active":"2024-09-21T10:52:22.860614+0000","last_peered":"2024-09-21T10:52:22.860614+0000","last_clean":"2024-09-21T10:52:22.860614+0000","last_became_active":"2024-09-21T10:51:26.801652+0000","last_became_peered":"2024-09-21T10:51:26.801652+0000","last_unstale":"2024-09-21T10:52:22.860614+0000","last_undegraded":"2024-09-21T10:52:22.860614+0000","last_fullsized":"2024-09-21T10:52:22.860614+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:39.935044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2458,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733090+0000","last_change":"2024-09-21T10:51:23.724830+0000","last_active":"2024-09-21T10:52:25.733090+0000","last_peered":"2024-09-21T10:52:25.733090+0000","last_clean":"2024-09-21T10:52:25.733090+0000","last_became_active":"2024-09-21T10:51:23.724517+0000","last_became_peered":"2024-09-21T10:51:23.724517+0000","last_unstale":"2024-09-21T10:52:25.733090+0000","last_undegraded":"2024-09-21T10:52:25.733090+0000","last_fullsized":"2024-09-21T10:52:25.733090+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:36.847627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":620,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718088+0000","last_change":"2024-09-21T10:51:26.803015+0000","last_active":"2024-09-21T10:52:22.718088+0000","last_peered":"2024-09-21T10:52:22.718088+0000","last_clean":"2024-09-21T10:52:22.718088+0000","last_became_active":"2024-09-21T10:51:26.802635+0000","last_became_peered":"2024-09-21T10:51:26.802635+0000","last_unstale":"2024-09-21T10:52:22.718088+0000","last_undegraded":"2024-09-21T10:52:22.718088+0000","last_fullsized":"2024-09-21T10:52:22.718088+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:12.553831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:12.553831+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:30:16.492365+0000","scrub_duration":0.001520109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2470,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732109+0000","last_change":"2024-09-21T10:51:23.722218+0000","last_active":"2024-09-21T10:52:25.732109+0000","last_peered":"2024-09-21T10:52:25.732109+0000","last_clean":"2024-09-21T10:52:25.732109+0000","last_became_active":"2024-09-21T10:51:23.721890+0000","last_became_peered":"2024-09-21T10:51:23.721890+0000","last_unstale":"2024-09-21T10:52:25.732109+0000","last_undegraded":"2024-09-21T10:52:25.732109+0000","last_fullsized":"2024-09-21T10:52:25.732109+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:18:51.358682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2458,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733159+0000","last_change":"2024-09-21T10:51:26.809612+0000","last_active":"2024-09-21T10:52:25.733159+0000","last_peered":"2024-09-21T10:52:25.733159+0000","last_clean":"2024-09-21T10:52:25.733159+0000","last_became_active":"2024-09-21T10:51:26.808796+0000","last_became_peered":"2024-09-21T10:51:26.808796+0000","last_unstale":"2024-09-21T10:52:25.733159+0000","last_undegraded":"2024-09-21T10:52:25.733159+0000","last_fullsized":"2024-09-21T10:52:25.733159+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:31.891319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859609+0000","last_change":"2024-09-21T10:51:23.718391+0000","last_active":"2024-09-21T10:52:22.859609+0000","last_peered":"2024-09-21T10:52:22.859609+0000","last_clean":"2024-09-21T10:52:22.859609+0000","last_became_active":"2024-09-21T10:51:23.717920+0000","last_became_peered":"2024-09-21T10:51:23.717920+0000","last_unstale":"2024-09-21T10:52:22.859609+0000","last_undegraded":"2024-09-21T10:52:22.859609+0000","last_fullsized":"2024-09-21T10:52:22.859609+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:39.336218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859240+0000","last_change":"2024-09-21T10:51:23.722108+0000","last_active":"2024-09-21T10:52:22.859240+0000","last_peered":"2024-09-21T10:52:22.859240+0000","last_clean":"2024-09-21T10:52:22.859240+0000","last_became_active":"2024-09-21T10:51:23.721548+0000","last_became_peered":"2024-09-21T10:51:23.721548+0000","last_unstale":"2024-09-21T10:52:22.859240+0000","last_undegraded":"2024-09-21T10:52:22.859240+0000","last_fullsized":"2024-09-21T10:52:22.859240+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.960250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.960250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:13:49.816964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2441,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733102+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:52:25.733102+0000","last_peered":"2024-09-21T10:52:25.733102+0000","last_clean":"2024-09-21T10:52:25.733102+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:52:25.733102+0000","last_undegraded":"2024-09-21T10:52:25.733102+0000","last_fullsized":"2024-09-21T10:52:25.733102+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859754+0000","last_change":"2024-09-21T10:51:19.712241+0000","last_active":"2024-09-21T10:52:22.859754+0000","last_peered":"2024-09-21T10:52:22.859754+0000","last_clean":"2024-09-21T10:52:22.859754+0000","last_became_active":"2024-09-21T10:51:19.711943+0000","last_became_peered":"2024-09-21T10:51:19.711943+0000","last_unstale":"2024-09-21T10:52:22.859754+0000","last_undegraded":"2024-09-21T10:52:22.859754+0000","last_fullsized":"2024-09-21T10:52:22.859754+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:51.003446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859486+0000","last_change":"2024-09-21T10:51:23.745775+0000","last_active":"2024-09-21T10:52:22.859486+0000","last_peered":"2024-09-21T10:52:22.859486+0000","last_clean":"2024-09-21T10:52:22.859486+0000","last_became_active":"2024-09-21T10:51:23.724595+0000","last_became_peered":"2024-09-21T10:51:23.724595+0000","last_unstale":"2024-09-21T10:52:22.859486+0000","last_undegraded":"2024-09-21T10:52:22.859486+0000","last_fullsized":"2024-09-21T10:52:22.859486+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:59.664737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":766,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860947+0000","last_change":"2024-09-21T10:52:12.175206+0000","last_active":"2024-09-21T10:52:22.860947+0000","last_peered":"2024-09-21T10:52:22.860947+0000","last_clean":"2024-09-21T10:52:22.860947+0000","last_became_active":"2024-09-21T10:52:12.174363+0000","last_became_peered":"2024-09-21T10:52:12.174363+0000","last_unstale":"2024-09-21T10:52:22.860947+0000","last_undegraded":"2024-09-21T10:52:22.860947+0000","last_fullsized":"2024-09-21T10:52:22.860947+0000","mapping_epoch":1502,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1503,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:21.873271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":190,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733044+0000","last_change":"2024-09-21T10:51:23.727021+0000","last_active":"2024-09-21T10:52:25.733044+0000","last_peered":"2024-09-21T10:52:25.733044+0000","last_clean":"2024-09-21T10:52:25.733044+0000","last_became_active":"2024-09-21T10:51:23.726633+0000","last_became_peered":"2024-09-21T10:51:23.726633+0000","last_unstale":"2024-09-21T10:52:25.733044+0000","last_undegraded":"2024-09-21T10:52:25.733044+0000","last_fullsized":"2024-09-21T10:52:25.733044+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:00.390334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2442,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732817+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:52:25.732817+0000","last_peered":"2024-09-21T10:52:25.732817+0000","last_clean":"2024-09-21T10:52:25.732817+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:52:25.732817+0000","last_undegraded":"2024-09-21T10:52:25.732817+0000","last_fullsized":"2024-09-21T10:52:25.732817+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2373,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732453+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:52:25.732453+0000","last_peered":"2024-09-21T10:52:25.732453+0000","last_clean":"2024-09-21T10:52:25.732453+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:52:25.732453+0000","last_undegraded":"2024-09-21T10:52:25.732453+0000","last_fullsized":"2024-09-21T10:52:25.732453+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":644,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732457+0000","last_change":"2024-09-21T10:51:19.719681+0000","last_active":"2024-09-21T10:52:25.732457+0000","last_peered":"2024-09-21T10:52:25.732457+0000","last_clean":"2024-09-21T10:52:25.732457+0000","last_became_active":"2024-09-21T10:51:19.718591+0000","last_became_peered":"2024-09-21T10:51:19.718591+0000","last_unstale":"2024-09-21T10:52:25.732457+0000","last_undegraded":"2024-09-21T10:52:25.732457+0000","last_fullsized":"2024-09-21T10:52:25.732457+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:39.734484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":298,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731603+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:52:25.731603+0000","last_peered":"2024-09-21T10:52:25.731603+0000","last_clean":"2024-09-21T10:52:25.731603+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:52:25.731603+0000","last_undegraded":"2024-09-21T10:52:25.731603+0000","last_fullsized":"2024-09-21T10:52:25.731603+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2488,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731588+0000","last_change":"2024-09-21T10:51:19.718291+0000","last_active":"2024-09-21T10:52:25.731588+0000","last_peered":"2024-09-21T10:52:25.731588+0000","last_clean":"2024-09-21T10:52:25.731588+0000","last_became_active":"2024-09-21T10:51:19.716870+0000","last_became_peered":"2024-09-21T10:51:19.716870+0000","last_unstale":"2024-09-21T10:52:25.731588+0000","last_undegraded":"2024-09-21T10:52:25.731588+0000","last_fullsized":"2024-09-21T10:52:25.731588+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:21.632408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.858925+0000","last_change":"2024-09-21T10:51:26.803803+0000","last_active":"2024-09-21T10:52:22.858925+0000","last_peered":"2024-09-21T10:52:22.858925+0000","last_clean":"2024-09-21T10:52:22.858925+0000","last_became_active":"2024-09-21T10:51:26.803376+0000","last_became_peered":"2024-09-21T10:51:26.803376+0000","last_unstale":"2024-09-21T10:52:22.858925+0000","last_undegraded":"2024-09-21T10:52:22.858925+0000","last_fullsized":"2024-09-21T10:52:22.858925+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:49.706791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1435,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717952+0000","last_change":"2024-09-21T10:51:26.803612+0000","last_active":"2024-09-21T10:52:22.717952+0000","last_peered":"2024-09-21T10:52:22.717952+0000","last_clean":"2024-09-21T10:52:22.717952+0000","last_became_active":"2024-09-21T10:51:26.803307+0000","last_became_peered":"2024-09-21T10:51:26.803307+0000","last_unstale":"2024-09-21T10:52:22.717952+0000","last_undegraded":"2024-09-21T10:52:22.717952+0000","last_fullsized":"2024-09-21T10:52:22.717952+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:52.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2380,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722232+0000","last_change":"2024-09-21T10:51:23.720296+0000","last_active":"2024-09-21T10:52:25.722232+0000","last_peered":"2024-09-21T10:52:25.722232+0000","last_clean":"2024-09-21T10:52:25.722232+0000","last_became_active":"2024-09-21T10:51:23.719982+0000","last_became_peered":"2024-09-21T10:51:23.719982+0000","last_unstale":"2024-09-21T10:52:25.722232+0000","last_undegraded":"2024-09-21T10:52:25.722232+0000","last_fullsized":"2024-09-21T10:52:25.722232+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.519160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":48,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859108+0000","last_change":"2024-09-21T10:51:26.820358+0000","last_active":"2024-09-21T10:52:22.859108+0000","last_peered":"2024-09-21T10:52:22.859108+0000","last_clean":"2024-09-21T10:52:22.859108+0000","last_became_active":"2024-09-21T10:51:26.809379+0000","last_became_peered":"2024-09-21T10:51:26.809379+0000","last_unstale":"2024-09-21T10:52:22.859108+0000","last_undegraded":"2024-09-21T10:52:22.859108+0000","last_fullsized":"2024-09-21T10:52:22.859108+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:48.562692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859053+0000","last_change":"2024-09-21T10:51:26.800633+0000","last_active":"2024-09-21T10:52:22.859053+0000","last_peered":"2024-09-21T10:52:22.859053+0000","last_clean":"2024-09-21T10:52:22.859053+0000","last_became_active":"2024-09-21T10:51:26.799809+0000","last_became_peered":"2024-09-21T10:51:26.799809+0000","last_unstale":"2024-09-21T10:52:22.859053+0000","last_undegraded":"2024-09-21T10:52:22.859053+0000","last_fullsized":"2024-09-21T10:52:22.859053+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:32.421023+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859654+0000","last_change":"2024-09-21T10:51:27.865788+0000","last_active":"2024-09-21T10:52:22.859654+0000","last_peered":"2024-09-21T10:52:22.859654+0000","last_clean":"2024-09-21T10:52:22.859654+0000","last_became_active":"2024-09-21T10:51:23.726524+0000","last_became_peered":"2024-09-21T10:51:23.726524+0000","last_unstale":"2024-09-21T10:52:22.859654+0000","last_undegraded":"2024-09-21T10:52:22.859654+0000","last_fullsized":"2024-09-21T10:52:22.859654+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:27.865723+0000","last_deep_scrub":"256'100","last_deep_scrub_stamp":"2024-09-21T10:51:27.865723+0000","last_clean_scrub_stamp":"2024-09-21T10:51:27.865723+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:18.975843+0000","scrub_duration":0.001524596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2467,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733015+0000","last_change":"2024-09-21T10:51:19.719114+0000","last_active":"2024-09-21T10:52:25.733015+0000","last_peered":"2024-09-21T10:52:25.733015+0000","last_clean":"2024-09-21T10:52:25.733015+0000","last_became_active":"2024-09-21T10:51:19.718769+0000","last_became_peered":"2024-09-21T10:51:19.718769+0000","last_unstale":"2024-09-21T10:52:25.733015+0000","last_undegraded":"2024-09-21T10:52:25.733015+0000","last_fullsized":"2024-09-21T10:52:25.733015+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:03.655730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2373,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722071+0000","last_change":"2024-09-21T10:51:26.801151+0000","last_active":"2024-09-21T10:52:25.722071+0000","last_peered":"2024-09-21T10:52:25.722071+0000","last_clean":"2024-09-21T10:52:25.722071+0000","last_became_active":"2024-09-21T10:51:26.800792+0000","last_became_peered":"2024-09-21T10:51:26.800792+0000","last_unstale":"2024-09-21T10:52:25.722071+0000","last_undegraded":"2024-09-21T10:52:25.722071+0000","last_fullsized":"2024-09-21T10:52:25.722071+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:20:10.166740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":642,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732409+0000","last_change":"2024-09-21T10:51:23.728196+0000","last_active":"2024-09-21T10:52:25.732409+0000","last_peered":"2024-09-21T10:52:25.732409+0000","last_clean":"2024-09-21T10:52:25.732409+0000","last_became_active":"2024-09-21T10:51:23.727811+0000","last_became_peered":"2024-09-21T10:51:23.727811+0000","last_unstale":"2024-09-21T10:52:25.732409+0000","last_undegraded":"2024-09-21T10:52:25.732409+0000","last_fullsized":"2024-09-21T10:52:25.732409+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:12.134349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:12.134349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:55.537453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860493+0000","last_change":"2024-09-21T10:51:19.716570+0000","last_active":"2024-09-21T10:52:22.860493+0000","last_peered":"2024-09-21T10:52:22.860493+0000","last_clean":"2024-09-21T10:52:22.860493+0000","last_became_active":"2024-09-21T10:51:19.715324+0000","last_became_peered":"2024-09-21T10:51:19.715324+0000","last_unstale":"2024-09-21T10:52:22.860493+0000","last_undegraded":"2024-09-21T10:52:22.860493+0000","last_fullsized":"2024-09-21T10:52:22.860493+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:12.926642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859164+0000","last_change":"2024-09-21T10:51:26.802544+0000","last_active":"2024-09-21T10:52:22.859164+0000","last_peered":"2024-09-21T10:52:22.859164+0000","last_clean":"2024-09-21T10:52:22.859164+0000","last_became_active":"2024-09-21T10:51:26.802078+0000","last_became_peered":"2024-09-21T10:51:26.802078+0000","last_unstale":"2024-09-21T10:52:22.859164+0000","last_undegraded":"2024-09-21T10:52:22.859164+0000","last_fullsized":"2024-09-21T10:52:22.859164+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:44.846468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859429+0000","last_change":"2024-09-21T10:51:23.719026+0000","last_active":"2024-09-21T10:52:22.859429+0000","last_peered":"2024-09-21T10:52:22.859429+0000","last_clean":"2024-09-21T10:52:22.859429+0000","last_became_active":"2024-09-21T10:51:23.718679+0000","last_became_peered":"2024-09-21T10:51:23.718679+0000","last_unstale":"2024-09-21T10:52:22.859429+0000","last_undegraded":"2024-09-21T10:52:22.859429+0000","last_fullsized":"2024-09-21T10:52:22.859429+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:17.397101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:17.397101+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:37.045469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860679+0000","last_change":"2024-09-21T10:51:19.715047+0000","last_active":"2024-09-21T10:52:22.860679+0000","last_peered":"2024-09-21T10:52:22.860679+0000","last_clean":"2024-09-21T10:52:22.860679+0000","last_became_active":"2024-09-21T10:51:19.714491+0000","last_became_peered":"2024-09-21T10:51:19.714491+0000","last_unstale":"2024-09-21T10:52:22.860679+0000","last_undegraded":"2024-09-21T10:52:22.860679+0000","last_fullsized":"2024-09-21T10:52:22.860679+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:06.088284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1431,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717836+0000","last_change":"2024-09-21T10:51:19.707777+0000","last_active":"2024-09-21T10:52:22.717836+0000","last_peered":"2024-09-21T10:52:22.717836+0000","last_clean":"2024-09-21T10:52:22.717836+0000","last_became_active":"2024-09-21T10:51:19.707407+0000","last_became_peered":"2024-09-21T10:51:19.707407+0000","last_unstale":"2024-09-21T10:52:22.717836+0000","last_undegraded":"2024-09-21T10:52:22.717836+0000","last_fullsized":"2024-09-21T10:52:22.717836+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:16.297430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":38,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859669+0000","last_change":"2024-09-21T10:51:40.034320+0000","last_active":"2024-09-21T10:52:22.859669+0000","last_peered":"2024-09-21T10:52:22.859669+0000","last_clean":"2024-09-21T10:52:22.859669+0000","last_became_active":"2024-09-21T10:51:26.800552+0000","last_became_peered":"2024-09-21T10:51:26.800552+0000","last_unstale":"2024-09-21T10:52:22.859669+0000","last_undegraded":"2024-09-21T10:52:22.859669+0000","last_fullsized":"2024-09-21T10:52:22.859669+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:40.034250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:40.034250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:45.167441+0000","scrub_duration":0.0017901530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860123+0000","last_change":"2024-09-21T10:51:19.709533+0000","last_active":"2024-09-21T10:52:22.860123+0000","last_peered":"2024-09-21T10:52:22.860123+0000","last_clean":"2024-09-21T10:52:22.860123+0000","last_became_active":"2024-09-21T10:51:19.709211+0000","last_became_peered":"2024-09-21T10:51:19.709211+0000","last_unstale":"2024-09-21T10:52:22.860123+0000","last_undegraded":"2024-09-21T10:52:22.860123+0000","last_fullsized":"2024-09-21T10:52:22.860123+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:33.467064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1493,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732548+0000","last_change":"2024-09-21T10:51:19.729909+0000","last_active":"2024-09-21T10:52:25.732548+0000","last_peered":"2024-09-21T10:52:25.732548+0000","last_clean":"2024-09-21T10:52:25.732548+0000","last_became_active":"2024-09-21T10:51:19.719252+0000","last_became_peered":"2024-09-21T10:51:19.719252+0000","last_unstale":"2024-09-21T10:52:25.732548+0000","last_undegraded":"2024-09-21T10:52:25.732548+0000","last_fullsized":"2024-09-21T10:52:25.732548+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:44.050500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":52,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860504+0000","last_change":"2024-09-21T10:51:29.423375+0000","last_active":"2024-09-21T10:52:22.860504+0000","last_peered":"2024-09-21T10:52:22.860504+0000","last_clean":"2024-09-21T10:52:22.860504+0000","last_became_active":"2024-09-21T10:51:26.797880+0000","last_became_peered":"2024-09-21T10:51:26.797880+0000","last_unstale":"2024-09-21T10:52:22.860504+0000","last_undegraded":"2024-09-21T10:52:22.860504+0000","last_fullsized":"2024-09-21T10:52:22.860504+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:29.423304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:29.423304+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:03.797459+0000","scrub_duration":0.0018586259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2479,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732748+0000","last_change":"2024-09-21T10:51:19.716770+0000","last_active":"2024-09-21T10:52:25.732748+0000","last_peered":"2024-09-21T10:52:25.732748+0000","last_clean":"2024-09-21T10:52:25.732748+0000","last_became_active":"2024-09-21T10:51:19.716197+0000","last_became_peered":"2024-09-21T10:51:19.716197+0000","last_unstale":"2024-09-21T10:52:25.732748+0000","last_undegraded":"2024-09-21T10:52:25.732748+0000","last_fullsized":"2024-09-21T10:52:25.732748+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:18.215947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860373+0000","last_change":"2024-09-21T10:51:23.725538+0000","last_active":"2024-09-21T10:52:22.860373+0000","last_peered":"2024-09-21T10:52:22.860373+0000","last_clean":"2024-09-21T10:52:22.860373+0000","last_became_active":"2024-09-21T10:51:23.725198+0000","last_became_peered":"2024-09-21T10:51:23.725198+0000","last_unstale":"2024-09-21T10:52:22.860373+0000","last_undegraded":"2024-09-21T10:52:22.860373+0000","last_fullsized":"2024-09-21T10:52:22.860373+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:11.636890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":619,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718183+0000","last_change":"2024-09-21T10:51:23.731927+0000","last_active":"2024-09-21T10:52:22.718183+0000","last_peered":"2024-09-21T10:52:22.718183+0000","last_clean":"2024-09-21T10:52:22.718183+0000","last_became_active":"2024-09-21T10:51:23.731572+0000","last_became_peered":"2024-09-21T10:51:23.731572+0000","last_unstale":"2024-09-21T10:52:22.718183+0000","last_undegraded":"2024-09-21T10:52:22.718183+0000","last_fullsized":"2024-09-21T10:52:22.718183+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:41.649690+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1413,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717723+0000","last_change":"2024-09-21T10:51:26.802738+0000","last_active":"2024-09-21T10:52:22.717723+0000","last_peered":"2024-09-21T10:52:22.717723+0000","last_clean":"2024-09-21T10:52:22.717723+0000","last_became_active":"2024-09-21T10:51:26.802271+0000","last_became_peered":"2024-09-21T10:51:26.802271+0000","last_unstale":"2024-09-21T10:52:22.717723+0000","last_undegraded":"2024-09-21T10:52:22.717723+0000","last_fullsized":"2024-09-21T10:52:22.717723+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:48.191078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861219+0000","last_change":"2024-09-21T10:52:12.176363+0000","last_active":"2024-09-21T10:52:22.861219+0000","last_peered":"2024-09-21T10:52:22.861219+0000","last_clean":"2024-09-21T10:52:22.861219+0000","last_became_active":"2024-09-21T10:52:12.175968+0000","last_became_peered":"2024-09-21T10:52:12.175968+0000","last_unstale":"2024-09-21T10:52:22.861219+0000","last_undegraded":"2024-09-21T10:52:22.861219+0000","last_fullsized":"2024-09-21T10:52:22.861219+0000","mapping_epoch":1502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:34.589176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":630,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722464+0000","last_change":"2024-09-21T10:51:12.411398+0000","last_active":"2024-09-21T10:52:25.722464+0000","last_peered":"2024-09-21T10:52:25.722464+0000","last_clean":"2024-09-21T10:52:25.722464+0000","last_became_active":"2024-09-21T10:51:11.649055+0000","last_became_peered":"2024-09-21T10:51:11.649055+0000","last_unstale":"2024-09-21T10:52:25.722464+0000","last_undegraded":"2024-09-21T10:52:25.722464+0000","last_fullsized":"2024-09-21T10:52:25.722464+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1481,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:12.411343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:12.411343+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:18.585319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859553+0000","last_change":"2024-09-21T10:51:19.716483+0000","last_active":"2024-09-21T10:52:22.859553+0000","last_peered":"2024-09-21T10:52:22.859553+0000","last_clean":"2024-09-21T10:52:22.859553+0000","last_became_active":"2024-09-21T10:51:19.716109+0000","last_became_peered":"2024-09-21T10:51:19.716109+0000","last_unstale":"2024-09-21T10:52:22.859553+0000","last_undegraded":"2024-09-21T10:52:22.859553+0000","last_fullsized":"2024-09-21T10:52:22.859553+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:31.056931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861453+0000","last_change":"2024-09-21T10:51:26.824514+0000","last_active":"2024-09-21T10:52:22.861453+0000","last_peered":"2024-09-21T10:52:22.861453+0000","last_clean":"2024-09-21T10:52:22.861453+0000","last_became_active":"2024-09-21T10:51:26.804254+0000","last_became_peered":"2024-09-21T10:51:26.804254+0000","last_unstale":"2024-09-21T10:52:22.861453+0000","last_undegraded":"2024-09-21T10:52:22.861453+0000","last_fullsized":"2024-09-21T10:52:22.861453+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:30.501261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861310+0000","last_change":"2024-09-21T10:51:26.804746+0000","last_active":"2024-09-21T10:52:22.861310+0000","last_peered":"2024-09-21T10:52:22.861310+0000","last_clean":"2024-09-21T10:52:22.861310+0000","last_became_active":"2024-09-21T10:51:26.804415+0000","last_became_peered":"2024-09-21T10:51:26.804415+0000","last_unstale":"2024-09-21T10:52:22.861310+0000","last_undegraded":"2024-09-21T10:52:22.861310+0000","last_fullsized":"2024-09-21T10:52:22.861310+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:14.151556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:14.151556+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:51.665464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859242+0000","last_change":"2024-09-21T10:51:26.809449+0000","last_active":"2024-09-21T10:52:22.859242+0000","last_peered":"2024-09-21T10:52:22.859242+0000","last_clean":"2024-09-21T10:52:22.859242+0000","last_became_active":"2024-09-21T10:51:26.808895+0000","last_became_peered":"2024-09-21T10:51:26.808895+0000","last_unstale":"2024-09-21T10:52:22.859242+0000","last_undegraded":"2024-09-21T10:52:22.859242+0000","last_fullsized":"2024-09-21T10:52:22.859242+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:40.151479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860688+0000","last_change":"2024-09-21T10:51:26.822830+0000","last_active":"2024-09-21T10:52:22.860688+0000","last_peered":"2024-09-21T10:52:22.860688+0000","last_clean":"2024-09-21T10:52:22.860688+0000","last_became_active":"2024-09-21T10:51:26.803697+0000","last_became_peered":"2024-09-21T10:51:26.803697+0000","last_unstale":"2024-09-21T10:52:22.860688+0000","last_undegraded":"2024-09-21T10:52:22.860688+0000","last_fullsized":"2024-09-21T10:52:22.860688+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:06.677964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859570+0000","last_change":"2024-09-21T10:51:28.893933+0000","last_active":"2024-09-21T10:52:22.859570+0000","last_peered":"2024-09-21T10:52:22.859570+0000","last_clean":"2024-09-21T10:52:22.859570+0000","last_became_active":"2024-09-21T10:51:23.724692+0000","last_became_peered":"2024-09-21T10:51:23.724692+0000","last_unstale":"2024-09-21T10:52:22.859570+0000","last_undegraded":"2024-09-21T10:52:22.859570+0000","last_fullsized":"2024-09-21T10:52:22.859570+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:28.893864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:28.893864+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:39.628862+0000","scrub_duration":0.001659862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1510,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717693+0000","last_change":"2024-09-21T10:51:23.743383+0000","last_active":"2024-09-21T10:52:22.717693+0000","last_peered":"2024-09-21T10:52:22.717693+0000","last_clean":"2024-09-21T10:52:22.717693+0000","last_became_active":"2024-09-21T10:51:23.730143+0000","last_became_peered":"2024-09-21T10:51:23.730143+0000","last_unstale":"2024-09-21T10:52:22.717693+0000","last_undegraded":"2024-09-21T10:52:22.717693+0000","last_fullsized":"2024-09-21T10:52:22.717693+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:38.077413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2348,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722332+0000","last_change":"2024-09-21T10:51:26.800999+0000","last_active":"2024-09-21T10:52:25.722332+0000","last_peered":"2024-09-21T10:52:25.722332+0000","last_clean":"2024-09-21T10:52:25.722332+0000","last_became_active":"2024-09-21T10:51:26.800445+0000","last_became_peered":"2024-09-21T10:51:26.800445+0000","last_unstale":"2024-09-21T10:52:25.722332+0000","last_undegraded":"2024-09-21T10:52:25.722332+0000","last_fullsized":"2024-09-21T10:52:25.722332+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:39:32.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":627,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732276+0000","last_change":"2024-09-21T10:51:19.719964+0000","last_active":"2024-09-21T10:52:25.732276+0000","last_peered":"2024-09-21T10:52:25.732276+0000","last_clean":"2024-09-21T10:52:25.732276+0000","last_became_active":"2024-09-21T10:51:19.719430+0000","last_became_peered":"2024-09-21T10:51:19.719430+0000","last_unstale":"2024-09-21T10:52:25.732276+0000","last_undegraded":"2024-09-21T10:52:25.732276+0000","last_fullsized":"2024-09-21T10:52:25.732276+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:56.282017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861077+0000","last_change":"2024-09-21T10:52:12.176229+0000","last_active":"2024-09-21T10:52:22.861077+0000","last_peered":"2024-09-21T10:52:22.861077+0000","last_clean":"2024-09-21T10:52:22.861077+0000","last_became_active":"2024-09-21T10:52:12.175790+0000","last_became_peered":"2024-09-21T10:52:12.175790+0000","last_unstale":"2024-09-21T10:52:22.861077+0000","last_undegraded":"2024-09-21T10:52:22.861077+0000","last_fullsized":"2024-09-21T10:52:22.861077+0000","mapping_epoch":1502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:12.844163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2467,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731836+0000","last_change":"2024-09-21T10:51:26.805572+0000","last_active":"2024-09-21T10:52:25.731836+0000","last_peered":"2024-09-21T10:52:25.731836+0000","last_clean":"2024-09-21T10:52:25.731836+0000","last_became_active":"2024-09-21T10:51:26.805206+0000","last_became_peered":"2024-09-21T10:51:26.805206+0000","last_unstale":"2024-09-21T10:52:25.731836+0000","last_undegraded":"2024-09-21T10:52:25.731836+0000","last_fullsized":"2024-09-21T10:52:25.731836+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:23.513847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.858971+0000","last_change":"2024-09-21T10:51:26.813501+0000","last_active":"2024-09-21T10:52:22.858971+0000","last_peered":"2024-09-21T10:52:22.858971+0000","last_clean":"2024-09-21T10:52:22.858971+0000","last_became_active":"2024-09-21T10:51:26.813010+0000","last_became_peered":"2024-09-21T10:51:26.813010+0000","last_unstale":"2024-09-21T10:52:22.858971+0000","last_undegraded":"2024-09-21T10:52:22.858971+0000","last_fullsized":"2024-09-21T10:52:22.858971+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:50:51.566260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:50:51.566260+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:03.992303+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2467,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732592+0000","last_change":"2024-09-21T10:51:26.804171+0000","last_active":"2024-09-21T10:52:25.732592+0000","last_peered":"2024-09-21T10:52:25.732592+0000","last_clean":"2024-09-21T10:52:25.732592+0000","last_became_active":"2024-09-21T10:51:26.803792+0000","last_became_peered":"2024-09-21T10:51:26.803792+0000","last_unstale":"2024-09-21T10:52:25.732592+0000","last_undegraded":"2024-09-21T10:52:25.732592+0000","last_fullsized":"2024-09-21T10:52:25.732592+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:36.654413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859436+0000","last_change":"2024-09-21T10:51:26.809443+0000","last_active":"2024-09-21T10:52:22.859436+0000","last_peered":"2024-09-21T10:52:22.859436+0000","last_clean":"2024-09-21T10:52:22.859436+0000","last_became_active":"2024-09-21T10:51:26.808907+0000","last_became_peered":"2024-09-21T10:51:26.808907+0000","last_unstale":"2024-09-21T10:52:22.859436+0000","last_undegraded":"2024-09-21T10:52:22.859436+0000","last_fullsized":"2024-09-21T10:52:22.859436+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:05.922402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":352,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731421+0000","last_change":"2024-09-21T10:51:23.725035+0000","last_active":"2024-09-21T10:52:25.731421+0000","last_peered":"2024-09-21T10:52:25.731421+0000","last_clean":"2024-09-21T10:52:25.731421+0000","last_became_active":"2024-09-21T10:51:23.724574+0000","last_became_peered":"2024-09-21T10:51:23.724574+0000","last_unstale":"2024-09-21T10:52:25.731421+0000","last_undegraded":"2024-09-21T10:52:25.731421+0000","last_fullsized":"2024-09-21T10:52:25.731421+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:13.916198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860075+0000","last_change":"2024-09-21T10:51:26.817029+0000","last_active":"2024-09-21T10:52:22.860075+0000","last_peered":"2024-09-21T10:52:22.860075+0000","last_clean":"2024-09-21T10:52:22.860075+0000","last_became_active":"2024-09-21T10:51:26.794038+0000","last_became_peered":"2024-09-21T10:51:26.794038+0000","last_unstale":"2024-09-21T10:52:22.860075+0000","last_undegraded":"2024-09-21T10:52:22.860075+0000","last_fullsized":"2024-09-21T10:52:22.860075+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:54.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2356,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722512+0000","last_change":"2024-09-21T10:51:26.801631+0000","last_active":"2024-09-21T10:52:25.722512+0000","last_peered":"2024-09-21T10:52:25.722512+0000","last_clean":"2024-09-21T10:52:25.722512+0000","last_became_active":"2024-09-21T10:51:26.800926+0000","last_became_peered":"2024-09-21T10:51:26.800926+0000","last_unstale":"2024-09-21T10:52:25.722512+0000","last_undegraded":"2024-09-21T10:52:25.722512+0000","last_fullsized":"2024-09-21T10:52:25.722512+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:33.338777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":12336,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398204,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905952,"kb_used_data":10324,"kb_used_omap":1,"kb_used_meta":895614,"kb_avail":92839200,"statfs":{"total":95995035648,"available":95067340800,"internally_reserved":0,"allocated":10571776,"data_stored":5960854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917109080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049863,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511316,"kb_used_data":10376,"kb_used_omap":242,"kb_used_meta":1500685,"kb_avail":92233836,"statfs":{"total":95995035648,"available":94447448064,"internally_reserved":0,"allocated":10625024,"data_stored":6025719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":1536701663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790662,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907344,"kb_used_data":10308,"kb_used_omap":1,"kb_used_meta":897022,"kb_avail":92837808,"statfs":{"total":95995035648,"available":95065915392,"internally_reserved":0,"allocated":10555392,"data_stored":5956553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918550872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395768,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333592,"kb_used_data":10060,"kb_used_omap":224,"kb_used_meta":323295,"kb_avail":93411560,"statfs":{"total":95995035648,"available":95653437440,"internally_reserved":0,"allocated":10301440,"data_stored":5698096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":331054985},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644822,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137988,"kb_used_data":10360,"kb_used_omap":67,"kb_used_meta":1127548,"kb_avail":92607164,"statfs":{"total":95995035648,"available":94829735936,"internally_reserved":0,"allocated":10608640,"data_stored":6020988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1154610013},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527839,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1114060,"kb_used_data":10304,"kb_used_omap":75,"kb_used_meta":1103668,"kb_avail":92631092,"statfs":{"total":95995035648,"available":94854238208,"internally_reserved":0,"allocated":10551296,"data_stored":5956526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":1130156186},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274992,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102916,"kb_used_data":10040,"kb_used_omap":51,"kb_used_meta":1092812,"kb_avail":92642236,"statfs":{"total":95995035648,"available":94865649664,"internally_reserved":0,"allocated":10280960,"data_stored":5693237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1119039829},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354231,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919240,"kb_used_data":10364,"kb_used_omap":66,"kb_used_meta":908797,"kb_avail":92825912,"statfs":{"total":95995035648,"available":95053733888,"internally_reserved":0,"allocated":10612736,"data_stored":6021490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":930608452},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:53:12.430 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid c96f302a-77fe-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T10:53:12.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:12 smithi145 ceph-mon[89359]: from='client.37746 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:12.674 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T10:53:13.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:13 smithi145 ceph-mon[89359]: pgmap v2806: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:13.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:13 smithi119 ceph-mon[105288]: pgmap v2806: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:13.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:13 smithi119 ceph-mon[101432]: pgmap v2806: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:14.631 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-21T10:53:14.632 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-21T10:53:14.867 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:14 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: ::ffff:172.21.15.145 - - [21/Sep/2024:10:53:14] "GET /metrics HTTP/1.1" 200 34421 "" "Prometheus/2.51.0" 2024-09-21T10:53:15.391 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":2807,"stamp":"2024-09-21T10:53:14.119762+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595917,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":80,"num_bytes_recovered":12374,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3158,"ondisk_log_size":3158,"up":309,"acting":309,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":309,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7932408,"kb_used_data":82136,"kb_used_omap":731,"kb_used_meta":7849444,"kb_avail":742028808,"statfs":{"total":767960285184,"available":759837499392,"internally_reserved":0,"allocated":84107264,"data_stored":47333463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":749144,"internal_metadata":8037831080},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"1.1d","version":"256'100","reported_seq":234,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732801+0000","last_change":"2024-09-21T10:51:27.068248+0000","last_active":"2024-09-21T10:52:25.732801+0000","last_peered":"2024-09-21T10:52:25.732801+0000","last_clean":"2024-09-21T10:52:25.732801+0000","last_became_active":"2024-09-21T10:51:19.718918+0000","last_became_peered":"2024-09-21T10:51:19.718918+0000","last_unstale":"2024-09-21T10:52:25.732801+0000","last_undegraded":"2024-09-21T10:52:25.732801+0000","last_fullsized":"2024-09-21T10:52:25.732801+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:27.068193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:27.068193+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:49:33.048016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"64'4","reported_seq":330,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731651+0000","last_change":"2024-09-21T10:51:23.738777+0000","last_active":"2024-09-21T10:52:25.731651+0000","last_peered":"2024-09-21T10:52:25.731651+0000","last_clean":"2024-09-21T10:52:25.731651+0000","last_became_active":"2024-09-21T10:51:23.729563+0000","last_became_peered":"2024-09-21T10:51:23.729563+0000","last_unstale":"2024-09-21T10:52:25.731651+0000","last_undegraded":"2024-09-21T10:52:25.731651+0000","last_fullsized":"2024-09-21T10:52:25.731651+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:39:41.467046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":570,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2280,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859948+0000","last_change":"2024-09-21T10:51:19.699841+0000","last_active":"2024-09-21T10:52:22.859948+0000","last_peered":"2024-09-21T10:52:22.859948+0000","last_clean":"2024-09-21T10:52:22.859948+0000","last_became_active":"2024-09-21T10:51:19.699493+0000","last_became_peered":"2024-09-21T10:51:19.699493+0000","last_unstale":"2024-09-21T10:52:22.859948+0000","last_undegraded":"2024-09-21T10:52:22.859948+0000","last_fullsized":"2024-09-21T10:52:22.859948+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:53:48.873414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1c","version":"256'100","reported_seq":38,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860439+0000","last_change":"2024-09-21T10:51:39.018702+0000","last_active":"2024-09-21T10:52:22.860439+0000","last_peered":"2024-09-21T10:52:22.860439+0000","last_clean":"2024-09-21T10:52:22.860439+0000","last_became_active":"2024-09-21T10:51:26.804861+0000","last_became_peered":"2024-09-21T10:51:26.804861+0000","last_unstale":"2024-09-21T10:52:22.860439+0000","last_undegraded":"2024-09-21T10:52:22.860439+0000","last_fullsized":"2024-09-21T10:52:22.860439+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:39.018624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:39.018624+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:00:14.282200+0000","scrub_duration":0.0016991790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"64'1","reported_seq":2342,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722616+0000","last_change":"2024-09-21T10:26:04.193172+0000","last_active":"2024-09-21T10:52:25.722616+0000","last_peered":"2024-09-21T10:52:25.722616+0000","last_clean":"2024-09-21T10:52:25.722616+0000","last_became_active":"2024-09-21T10:26:04.192785+0000","last_became_peered":"2024-09-21T10:26:04.192785+0000","last_unstale":"2024-09-21T10:52:25.722616+0000","last_undegraded":"2024-09-21T10:52:25.722616+0000","last_fullsized":"2024-09-21T10:52:25.722616+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:38:19.145105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":2359,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722335+0000","last_change":"2024-09-21T10:08:55.136375+0000","last_active":"2024-09-21T10:52:25.722335+0000","last_peered":"2024-09-21T10:52:25.722335+0000","last_clean":"2024-09-21T10:52:25.722335+0000","last_became_active":"2024-09-21T10:08:55.135718+0000","last_became_peered":"2024-09-21T10:08:55.135718+0000","last_unstale":"2024-09-21T10:52:25.722335+0000","last_undegraded":"2024-09-21T10:52:25.722335+0000","last_fullsized":"2024-09-21T10:52:25.722335+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:23:08.851884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"64'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859720+0000","last_change":"2024-09-21T10:51:23.739774+0000","last_active":"2024-09-21T10:52:22.859720+0000","last_peered":"2024-09-21T10:52:22.859720+0000","last_clean":"2024-09-21T10:52:22.859720+0000","last_became_active":"2024-09-21T10:51:23.730493+0000","last_became_peered":"2024-09-21T10:51:23.730493+0000","last_unstale":"2024-09-21T10:52:22.859720+0000","last_undegraded":"2024-09-21T10:52:22.859720+0000","last_fullsized":"2024-09-21T10:52:22.859720+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:51.690849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860252+0000","last_change":"2024-09-21T10:51:19.710501+0000","last_active":"2024-09-21T10:52:22.860252+0000","last_peered":"2024-09-21T10:52:22.860252+0000","last_clean":"2024-09-21T10:52:22.860252+0000","last_became_active":"2024-09-21T10:51:19.710202+0000","last_became_peered":"2024-09-21T10:51:19.710202+0000","last_unstale":"2024-09-21T10:52:22.860252+0000","last_undegraded":"2024-09-21T10:52:22.860252+0000","last_fullsized":"2024-09-21T10:52:22.860252+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:54:11.158284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1e","version":"256'100","reported_seq":562,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722286+0000","last_change":"2024-09-21T10:51:32.463718+0000","last_active":"2024-09-21T10:52:25.722286+0000","last_peered":"2024-09-21T10:52:25.722286+0000","last_clean":"2024-09-21T10:52:25.722286+0000","last_became_active":"2024-09-21T10:51:26.800500+0000","last_became_peered":"2024-09-21T10:51:26.800500+0000","last_unstale":"2024-09-21T10:52:25.722286+0000","last_undegraded":"2024-09-21T10:52:25.722286+0000","last_fullsized":"2024-09-21T10:52:25.722286+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:32.463666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:32.463666+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:03.241851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"64'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860372+0000","last_change":"2024-09-21T10:51:23.735436+0000","last_active":"2024-09-21T10:52:22.860372+0000","last_peered":"2024-09-21T10:52:22.860372+0000","last_clean":"2024-09-21T10:52:22.860372+0000","last_became_active":"2024-09-21T10:51:23.724032+0000","last_became_peered":"2024-09-21T10:51:23.724032+0000","last_unstale":"2024-09-21T10:52:22.860372+0000","last_undegraded":"2024-09-21T10:52:22.860372+0000","last_fullsized":"2024-09-21T10:52:22.860372+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:12:45.454930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859597+0000","last_change":"2024-09-21T10:51:19.714313+0000","last_active":"2024-09-21T10:52:22.859597+0000","last_peered":"2024-09-21T10:52:22.859597+0000","last_clean":"2024-09-21T10:52:22.859597+0000","last_became_active":"2024-09-21T10:51:19.713999+0000","last_became_peered":"2024-09-21T10:51:19.713999+0000","last_unstale":"2024-09-21T10:52:22.859597+0000","last_undegraded":"2024-09-21T10:52:22.859597+0000","last_fullsized":"2024-09-21T10:52:22.859597+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:23:38.289820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"256'100","reported_seq":1571,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722006+0000","last_change":"2024-09-21T10:51:20.305009+0000","last_active":"2024-09-21T10:52:25.722006+0000","last_peered":"2024-09-21T10:52:25.722006+0000","last_clean":"2024-09-21T10:52:25.722006+0000","last_became_active":"2024-09-21T10:51:17.680713+0000","last_became_peered":"2024-09-21T10:51:17.680713+0000","last_unstale":"2024-09-21T10:52:25.722006+0000","last_undegraded":"2024-09-21T10:52:25.722006+0000","last_fullsized":"2024-09-21T10:52:25.722006+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1483,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:20.304962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:20.304962+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:01:00.486399+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"64'1","reported_seq":182,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731903+0000","last_change":"2024-09-21T10:51:26.794168+0000","last_active":"2024-09-21T10:52:25.731903+0000","last_peered":"2024-09-21T10:52:25.731903+0000","last_clean":"2024-09-21T10:52:25.731903+0000","last_became_active":"2024-09-21T10:51:26.793759+0000","last_became_peered":"2024-09-21T10:51:26.793759+0000","last_unstale":"2024-09-21T10:52:25.731903+0000","last_undegraded":"2024-09-21T10:52:25.731903+0000","last_fullsized":"2024-09-21T10:52:25.731903+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:21:03.050183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":326,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732079+0000","last_change":"2024-09-21T10:37:41.925633+0000","last_active":"2024-09-21T10:52:25.732079+0000","last_peered":"2024-09-21T10:52:25.732079+0000","last_clean":"2024-09-21T10:52:25.732079+0000","last_became_active":"2024-09-21T10:37:41.925131+0000","last_became_peered":"2024-09-21T10:37:41.925131+0000","last_unstale":"2024-09-21T10:52:25.732079+0000","last_undegraded":"2024-09-21T10:52:25.732079+0000","last_fullsized":"2024-09-21T10:52:25.732079+0000","mapping_epoch":1352,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1353,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:50:50.445412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.18","version":"256'100","reported_seq":58,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859307+0000","last_change":"2024-09-21T10:51:33.806344+0000","last_active":"2024-09-21T10:52:22.859307+0000","last_peered":"2024-09-21T10:52:22.859307+0000","last_clean":"2024-09-21T10:52:22.859307+0000","last_became_active":"2024-09-21T10:51:26.813092+0000","last_became_peered":"2024-09-21T10:51:26.813092+0000","last_unstale":"2024-09-21T10:52:22.859307+0000","last_undegraded":"2024-09-21T10:52:22.859307+0000","last_fullsized":"2024-09-21T10:52:22.859307+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:33.806271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:33.806271+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:47:54.793994+0000","scrub_duration":0.0018767289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"64'1","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860169+0000","last_change":"2024-09-21T10:51:19.718380+0000","last_active":"2024-09-21T10:52:22.860169+0000","last_peered":"2024-09-21T10:52:22.860169+0000","last_clean":"2024-09-21T10:52:22.860169+0000","last_became_active":"2024-09-21T10:51:19.717955+0000","last_became_peered":"2024-09-21T10:51:19.717955+0000","last_unstale":"2024-09-21T10:52:22.860169+0000","last_undegraded":"2024-09-21T10:52:22.860169+0000","last_fullsized":"2024-09-21T10:52:22.860169+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:27:31.240897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":2355,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722776+0000","last_change":"2024-09-21T10:51:19.706879+0000","last_active":"2024-09-21T10:52:25.722776+0000","last_peered":"2024-09-21T10:52:25.722776+0000","last_clean":"2024-09-21T10:52:25.722776+0000","last_became_active":"2024-09-21T10:51:19.706103+0000","last_became_peered":"2024-09-21T10:51:19.706103+0000","last_unstale":"2024-09-21T10:52:25.722776+0000","last_undegraded":"2024-09-21T10:52:25.722776+0000","last_fullsized":"2024-09-21T10:52:25.722776+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:43.017006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859364+0000","last_change":"2024-09-21T10:51:25.937690+0000","last_active":"2024-09-21T10:52:22.859364+0000","last_peered":"2024-09-21T10:52:22.859364+0000","last_clean":"2024-09-21T10:52:22.859364+0000","last_became_active":"2024-09-21T10:51:23.721779+0000","last_became_peered":"2024-09-21T10:51:23.721779+0000","last_unstale":"2024-09-21T10:52:22.859364+0000","last_undegraded":"2024-09-21T10:52:22.859364+0000","last_fullsized":"2024-09-21T10:52:22.859364+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:25.937626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:25.937626+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:27:37.783185+0000","scrub_duration":0.0016995199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":175,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731693+0000","last_change":"2024-09-21T10:51:19.704667+0000","last_active":"2024-09-21T10:52:25.731693+0000","last_peered":"2024-09-21T10:52:25.731693+0000","last_clean":"2024-09-21T10:52:25.731693+0000","last_became_active":"2024-09-21T10:51:19.704366+0000","last_became_peered":"2024-09-21T10:51:19.704366+0000","last_unstale":"2024-09-21T10:52:25.731693+0000","last_undegraded":"2024-09-21T10:52:25.731693+0000","last_fullsized":"2024-09-21T10:52:25.731693+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:46:48.376605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":2471,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731921+0000","last_change":"2024-09-21T10:51:26.807345+0000","last_active":"2024-09-21T10:52:25.731921+0000","last_peered":"2024-09-21T10:52:25.731921+0000","last_clean":"2024-09-21T10:52:25.731921+0000","last_became_active":"2024-09-21T10:51:26.806887+0000","last_became_peered":"2024-09-21T10:51:26.806887+0000","last_unstale":"2024-09-21T10:52:25.731921+0000","last_undegraded":"2024-09-21T10:52:25.731921+0000","last_fullsized":"2024-09-21T10:52:25.731921+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:18:44.878329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1a","version":"256'100","reported_seq":62,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860264+0000","last_change":"2024-09-21T10:51:27.399993+0000","last_active":"2024-09-21T10:52:22.860264+0000","last_peered":"2024-09-21T10:52:22.860264+0000","last_clean":"2024-09-21T10:52:22.860264+0000","last_became_active":"2024-09-21T10:51:26.796440+0000","last_became_peered":"2024-09-21T10:51:26.796440+0000","last_unstale":"2024-09-21T10:52:22.860264+0000","last_undegraded":"2024-09-21T10:52:22.860264+0000","last_fullsized":"2024-09-21T10:52:22.860264+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:27.399930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:27.399930+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:02:30.699583+0000","scrub_duration":0.0018178720000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":2374,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722090+0000","last_change":"2024-09-21T10:39:00.849346+0000","last_active":"2024-09-21T10:52:25.722090+0000","last_peered":"2024-09-21T10:52:25.722090+0000","last_clean":"2024-09-21T10:52:25.722090+0000","last_became_active":"2024-09-21T10:39:00.848060+0000","last_became_peered":"2024-09-21T10:39:00.848060+0000","last_unstale":"2024-09-21T10:52:25.722090+0000","last_undegraded":"2024-09-21T10:52:25.722090+0000","last_fullsized":"2024-09-21T10:52:25.722090+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:49:24.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1423,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718185+0000","last_change":"2024-09-21T10:51:23.733462+0000","last_active":"2024-09-21T10:52:22.718185+0000","last_peered":"2024-09-21T10:52:22.718185+0000","last_clean":"2024-09-21T10:52:22.718185+0000","last_became_active":"2024-09-21T10:51:23.733150+0000","last_became_peered":"2024-09-21T10:51:23.733150+0000","last_unstale":"2024-09-21T10:52:22.718185+0000","last_undegraded":"2024-09-21T10:52:22.718185+0000","last_fullsized":"2024-09-21T10:52:22.718185+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:09:48.510994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.15","version":"256'100","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861397+0000","last_change":"2024-09-21T10:51:26.809999+0000","last_active":"2024-09-21T10:52:22.861397+0000","last_peered":"2024-09-21T10:52:22.861397+0000","last_clean":"2024-09-21T10:52:22.861397+0000","last_became_active":"2024-09-21T10:51:26.808485+0000","last_became_peered":"2024-09-21T10:51:26.808485+0000","last_unstale":"2024-09-21T10:52:22.861397+0000","last_undegraded":"2024-09-21T10:52:22.861397+0000","last_fullsized":"2024-09-21T10:52:22.861397+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:21.142696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:21.142696+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:19.266752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":2353,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722157+0000","last_change":"2024-09-21T10:26:04.193368+0000","last_active":"2024-09-21T10:52:25.722157+0000","last_peered":"2024-09-21T10:52:25.722157+0000","last_clean":"2024-09-21T10:52:25.722157+0000","last_became_active":"2024-09-21T10:26:04.193028+0000","last_became_peered":"2024-09-21T10:26:04.193028+0000","last_unstale":"2024-09-21T10:52:25.722157+0000","last_undegraded":"2024-09-21T10:52:25.722157+0000","last_fullsized":"2024-09-21T10:52:25.722157+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:13:56.517417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":2487,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731874+0000","last_change":"2024-09-21T10:51:23.720734+0000","last_active":"2024-09-21T10:52:25.731874+0000","last_peered":"2024-09-21T10:52:25.731874+0000","last_clean":"2024-09-21T10:52:25.731874+0000","last_became_active":"2024-09-21T10:51:23.720442+0000","last_became_peered":"2024-09-21T10:51:23.720442+0000","last_unstale":"2024-09-21T10:52:25.731874+0000","last_undegraded":"2024-09-21T10:52:25.731874+0000","last_fullsized":"2024-09-21T10:52:25.731874+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:40:12.592262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"256'100","reported_seq":508,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732053+0000","last_change":"2024-09-21T10:51:26.794751+0000","last_active":"2024-09-21T10:52:25.732053+0000","last_peered":"2024-09-21T10:52:25.732053+0000","last_clean":"2024-09-21T10:52:25.732053+0000","last_became_active":"2024-09-21T10:51:26.794375+0000","last_became_peered":"2024-09-21T10:51:26.794375+0000","last_unstale":"2024-09-21T10:52:25.732053+0000","last_undegraded":"2024-09-21T10:52:25.732053+0000","last_fullsized":"2024-09-21T10:52:25.732053+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:47.788595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:47.788595+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:59:53.843188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":2364,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722408+0000","last_change":"2024-09-21T10:51:26.809260+0000","last_active":"2024-09-21T10:52:25.722408+0000","last_peered":"2024-09-21T10:52:25.722408+0000","last_clean":"2024-09-21T10:52:25.722408+0000","last_became_active":"2024-09-21T10:51:26.808907+0000","last_became_peered":"2024-09-21T10:51:26.808907+0000","last_unstale":"2024-09-21T10:52:25.722408+0000","last_undegraded":"2024-09-21T10:52:25.722408+0000","last_fullsized":"2024-09-21T10:52:25.722408+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:17:32.645807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859754+0000","last_change":"2024-09-21T10:51:26.809002+0000","last_active":"2024-09-21T10:52:22.859754+0000","last_peered":"2024-09-21T10:52:22.859754+0000","last_clean":"2024-09-21T10:52:22.859754+0000","last_became_active":"2024-09-21T10:51:26.808657+0000","last_became_peered":"2024-09-21T10:51:26.808657+0000","last_unstale":"2024-09-21T10:52:22.859754+0000","last_undegraded":"2024-09-21T10:52:22.859754+0000","last_fullsized":"2024-09-21T10:52:22.859754+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:52:18.104217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"256'100","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860135+0000","last_change":"2024-09-21T10:51:23.726165+0000","last_active":"2024-09-21T10:52:22.860135+0000","last_peered":"2024-09-21T10:52:22.860135+0000","last_clean":"2024-09-21T10:52:22.860135+0000","last_became_active":"2024-09-21T10:51:23.725783+0000","last_became_peered":"2024-09-21T10:51:23.725783+0000","last_unstale":"2024-09-21T10:52:22.860135+0000","last_undegraded":"2024-09-21T10:52:22.860135+0000","last_fullsized":"2024-09-21T10:52:22.860135+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:21.730413+0000","last_deep_scrub":"256'100","last_deep_scrub_stamp":"2024-09-21T10:51:16.536490+0000","last_clean_scrub_stamp":"2024-09-21T10:51:21.730413+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:14:17.270019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"64'3","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861348+0000","last_change":"2024-09-21T10:51:26.820653+0000","last_active":"2024-09-21T10:52:22.861348+0000","last_peered":"2024-09-21T10:52:22.861348+0000","last_clean":"2024-09-21T10:52:22.861348+0000","last_became_active":"2024-09-21T10:51:26.799521+0000","last_became_peered":"2024-09-21T10:51:26.799521+0000","last_unstale":"2024-09-21T10:52:22.861348+0000","last_undegraded":"2024-09-21T10:52:22.861348+0000","last_fullsized":"2024-09-21T10:52:22.861348+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:42.996013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":180,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":175,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731739+0000","last_change":"2024-09-21T10:46:30.264483+0000","last_active":"2024-09-21T10:52:25.731739+0000","last_peered":"2024-09-21T10:52:25.731739+0000","last_clean":"2024-09-21T10:52:25.731739+0000","last_became_active":"2024-09-21T10:46:30.264079+0000","last_became_peered":"2024-09-21T10:46:30.264079+0000","last_unstale":"2024-09-21T10:52:25.731739+0000","last_undegraded":"2024-09-21T10:52:25.731739+0000","last_fullsized":"2024-09-21T10:52:25.731739+0000","mapping_epoch":1425,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:26:18.173626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.16","version":"256'100","reported_seq":69,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731834+0000","last_change":"2024-09-21T10:51:19.154684+0000","last_active":"2024-09-21T10:52:25.731834+0000","last_peered":"2024-09-21T10:52:25.731834+0000","last_clean":"2024-09-21T10:52:25.731834+0000","last_became_active":"2024-09-21T10:51:17.681580+0000","last_became_peered":"2024-09-21T10:51:17.681580+0000","last_unstale":"2024-09-21T10:52:25.731834+0000","last_undegraded":"2024-09-21T10:52:25.731834+0000","last_fullsized":"2024-09-21T10:52:25.731834+0000","mapping_epoch":1482,"log_start":"0'0","ondisk_log_start":"0'0","created":1480,"last_epoch_clean":1483,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:19.154633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:19.154633+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:34:31.453042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"338'22","reported_seq":62,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859929+0000","last_change":"2024-09-21T10:51:26.815362+0000","last_active":"2024-09-21T10:52:22.859929+0000","last_peered":"2024-09-21T10:52:22.859929+0000","last_clean":"2024-09-21T10:52:22.859929+0000","last_became_active":"2024-09-21T10:51:26.797456+0000","last_became_peered":"2024-09-21T10:51:26.797456+0000","last_unstale":"2024-09-21T10:52:22.859929+0000","last_undegraded":"2024-09-21T10:52:22.859929+0000","last_fullsized":"2024-09-21T10:52:22.859929+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:57:07.587157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4744,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":9488,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859840+0000","last_change":"2024-09-21T10:51:19.718455+0000","last_active":"2024-09-21T10:52:22.859840+0000","last_peered":"2024-09-21T10:52:22.859840+0000","last_clean":"2024-09-21T10:52:22.859840+0000","last_became_active":"2024-09-21T10:51:19.717807+0000","last_became_peered":"2024-09-21T10:51:19.717807+0000","last_unstale":"2024-09-21T10:52:22.859840+0000","last_undegraded":"2024-09-21T10:52:22.859840+0000","last_fullsized":"2024-09-21T10:52:22.859840+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:35:38.313963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"256'100","reported_seq":643,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722475+0000","last_change":"2024-09-21T10:51:26.801568+0000","last_active":"2024-09-21T10:52:25.722475+0000","last_peered":"2024-09-21T10:52:25.722475+0000","last_clean":"2024-09-21T10:52:25.722475+0000","last_became_active":"2024-09-21T10:51:26.801292+0000","last_became_peered":"2024-09-21T10:51:26.801292+0000","last_unstale":"2024-09-21T10:52:25.722475+0000","last_undegraded":"2024-09-21T10:52:25.722475+0000","last_fullsized":"2024-09-21T10:52:25.722475+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:46:05.453529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:46:05.453529+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:36:14.972988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":42,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859417+0000","last_change":"2024-09-21T10:51:23.720921+0000","last_active":"2024-09-21T10:52:22.859417+0000","last_peered":"2024-09-21T10:52:22.859417+0000","last_clean":"2024-09-21T10:52:22.859417+0000","last_became_active":"2024-09-21T10:51:23.720575+0000","last_became_peered":"2024-09-21T10:51:23.720575+0000","last_unstale":"2024-09-21T10:52:22.859417+0000","last_undegraded":"2024-09-21T10:52:22.859417+0000","last_fullsized":"2024-09-21T10:52:22.859417+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:51:40.539488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1419,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718111+0000","last_change":"2024-09-21T10:51:26.802469+0000","last_active":"2024-09-21T10:52:22.718111+0000","last_peered":"2024-09-21T10:52:22.718111+0000","last_clean":"2024-09-21T10:52:22.718111+0000","last_became_active":"2024-09-21T10:51:26.802135+0000","last_became_peered":"2024-09-21T10:51:26.802135+0000","last_unstale":"2024-09-21T10:52:22.718111+0000","last_undegraded":"2024-09-21T10:52:22.718111+0000","last_fullsized":"2024-09-21T10:52:22.718111+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:55:11.432966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00021671,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"64'1","reported_seq":364,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731736+0000","last_change":"2024-09-21T10:51:19.727865+0000","last_active":"2024-09-21T10:52:25.731736+0000","last_peered":"2024-09-21T10:52:25.731736+0000","last_clean":"2024-09-21T10:52:25.731736+0000","last_became_active":"2024-09-21T10:51:19.715076+0000","last_became_peered":"2024-09-21T10:51:19.715076+0000","last_unstale":"2024-09-21T10:52:25.731736+0000","last_undegraded":"2024-09-21T10:52:25.731736+0000","last_fullsized":"2024-09-21T10:52:25.731736+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:19:10.385979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"256'100","reported_seq":42,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860412+0000","last_change":"2024-09-21T10:51:23.423247+0000","last_active":"2024-09-21T10:52:22.860412+0000","last_peered":"2024-09-21T10:52:22.860412+0000","last_clean":"2024-09-21T10:52:22.860412+0000","last_became_active":"2024-09-21T10:51:19.699614+0000","last_became_peered":"2024-09-21T10:51:19.699614+0000","last_unstale":"2024-09-21T10:52:22.860412+0000","last_undegraded":"2024-09-21T10:52:22.860412+0000","last_fullsized":"2024-09-21T10:52:22.860412+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:23.423180+0000","last_deep_scrub":"256'100","last_deep_scrub_stamp":"2024-09-21T10:51:23.423180+0000","last_clean_scrub_stamp":"2024-09-21T10:51:23.423180+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:34:37.289679+0000","scrub_duration":0.0016804439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":149,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.721966+0000","last_change":"2024-09-21T10:51:26.801503+0000","last_active":"2024-09-21T10:52:25.721966+0000","last_peered":"2024-09-21T10:52:25.721966+0000","last_clean":"2024-09-21T10:52:25.721966+0000","last_became_active":"2024-09-21T10:51:26.801140+0000","last_became_peered":"2024-09-21T10:51:26.801140+0000","last_unstale":"2024-09-21T10:52:25.721966+0000","last_undegraded":"2024-09-21T10:52:25.721966+0000","last_fullsized":"2024-09-21T10:52:25.721966+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:09:18.528231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":2463,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732195+0000","last_change":"2024-09-21T10:51:19.718247+0000","last_active":"2024-09-21T10:52:25.732195+0000","last_peered":"2024-09-21T10:52:25.732195+0000","last_clean":"2024-09-21T10:52:25.732195+0000","last_became_active":"2024-09-21T10:51:19.717892+0000","last_became_peered":"2024-09-21T10:51:19.717892+0000","last_unstale":"2024-09-21T10:52:25.732195+0000","last_undegraded":"2024-09-21T10:52:25.732195+0000","last_fullsized":"2024-09-21T10:52:25.732195+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:25:41.786269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"64'2","reported_seq":217,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732350+0000","last_change":"2024-09-21T10:51:26.817216+0000","last_active":"2024-09-21T10:52:25.732350+0000","last_peered":"2024-09-21T10:52:25.732350+0000","last_clean":"2024-09-21T10:52:25.732350+0000","last_became_active":"2024-09-21T10:51:26.797691+0000","last_became_peered":"2024-09-21T10:51:26.797691+0000","last_unstale":"2024-09-21T10:52:25.732350+0000","last_undegraded":"2024-09-21T10:52:25.732350+0000","last_fullsized":"2024-09-21T10:52:25.732350+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:34.583770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"256'100","reported_seq":600,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718055+0000","last_change":"2024-09-21T10:51:13.523081+0000","last_active":"2024-09-21T10:52:22.718055+0000","last_peered":"2024-09-21T10:52:22.718055+0000","last_clean":"2024-09-21T10:52:22.718055+0000","last_became_active":"2024-09-21T10:51:11.652771+0000","last_became_peered":"2024-09-21T10:51:11.652771+0000","last_unstale":"2024-09-21T10:52:22.718055+0000","last_undegraded":"2024-09-21T10:52:22.718055+0000","last_fullsized":"2024-09-21T10:52:22.718055+0000","mapping_epoch":1469,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1481,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:13.523014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:13.523014+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:25:29.983664+0000","scrub_duration":0.0019017459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":2487,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732810+0000","last_change":"2024-09-21T10:51:26.804052+0000","last_active":"2024-09-21T10:52:25.732810+0000","last_peered":"2024-09-21T10:52:25.732810+0000","last_clean":"2024-09-21T10:52:25.732810+0000","last_became_active":"2024-09-21T10:51:26.803711+0000","last_became_peered":"2024-09-21T10:51:26.803711+0000","last_unstale":"2024-09-21T10:52:25.732810+0000","last_undegraded":"2024-09-21T10:52:25.732810+0000","last_fullsized":"2024-09-21T10:52:25.732810+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:22:34.399518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860614+0000","last_change":"2024-09-21T10:51:26.801966+0000","last_active":"2024-09-21T10:52:22.860614+0000","last_peered":"2024-09-21T10:52:22.860614+0000","last_clean":"2024-09-21T10:52:22.860614+0000","last_became_active":"2024-09-21T10:51:26.801652+0000","last_became_peered":"2024-09-21T10:51:26.801652+0000","last_unstale":"2024-09-21T10:52:22.860614+0000","last_undegraded":"2024-09-21T10:52:22.860614+0000","last_fullsized":"2024-09-21T10:52:22.860614+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:39.935044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00028741299999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":2458,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733090+0000","last_change":"2024-09-21T10:51:23.724830+0000","last_active":"2024-09-21T10:52:25.733090+0000","last_peered":"2024-09-21T10:52:25.733090+0000","last_clean":"2024-09-21T10:52:25.733090+0000","last_became_active":"2024-09-21T10:51:23.724517+0000","last_became_peered":"2024-09-21T10:51:23.724517+0000","last_unstale":"2024-09-21T10:52:25.733090+0000","last_undegraded":"2024-09-21T10:52:25.733090+0000","last_fullsized":"2024-09-21T10:52:25.733090+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:58:36.847627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"256'100","reported_seq":620,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718088+0000","last_change":"2024-09-21T10:51:26.803015+0000","last_active":"2024-09-21T10:52:22.718088+0000","last_peered":"2024-09-21T10:52:22.718088+0000","last_clean":"2024-09-21T10:52:22.718088+0000","last_became_active":"2024-09-21T10:51:26.802635+0000","last_became_peered":"2024-09-21T10:51:26.802635+0000","last_unstale":"2024-09-21T10:52:22.718088+0000","last_undegraded":"2024-09-21T10:52:22.718088+0000","last_fullsized":"2024-09-21T10:52:22.718088+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:12.553831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:12.553831+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:30:16.492365+0000","scrub_duration":0.001520109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":2470,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732109+0000","last_change":"2024-09-21T10:51:23.722218+0000","last_active":"2024-09-21T10:52:25.732109+0000","last_peered":"2024-09-21T10:52:25.732109+0000","last_clean":"2024-09-21T10:52:25.732109+0000","last_became_active":"2024-09-21T10:51:23.721890+0000","last_became_peered":"2024-09-21T10:51:23.721890+0000","last_unstale":"2024-09-21T10:52:25.732109+0000","last_undegraded":"2024-09-21T10:52:25.732109+0000","last_fullsized":"2024-09-21T10:52:25.732109+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:18:51.358682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":2458,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733159+0000","last_change":"2024-09-21T10:51:26.809612+0000","last_active":"2024-09-21T10:52:25.733159+0000","last_peered":"2024-09-21T10:52:25.733159+0000","last_clean":"2024-09-21T10:52:25.733159+0000","last_became_active":"2024-09-21T10:51:26.808796+0000","last_became_peered":"2024-09-21T10:51:26.808796+0000","last_unstale":"2024-09-21T10:52:25.733159+0000","last_undegraded":"2024-09-21T10:52:25.733159+0000","last_fullsized":"2024-09-21T10:52:25.733159+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:29:31.891319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026438399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859609+0000","last_change":"2024-09-21T10:51:23.718391+0000","last_active":"2024-09-21T10:52:22.859609+0000","last_peered":"2024-09-21T10:52:22.859609+0000","last_clean":"2024-09-21T10:52:22.859609+0000","last_became_active":"2024-09-21T10:51:23.717920+0000","last_became_peered":"2024-09-21T10:51:23.717920+0000","last_unstale":"2024-09-21T10:52:22.859609+0000","last_undegraded":"2024-09-21T10:52:22.859609+0000","last_fullsized":"2024-09-21T10:52:22.859609+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:34:39.336218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"256'100","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859240+0000","last_change":"2024-09-21T10:51:23.722108+0000","last_active":"2024-09-21T10:52:22.859240+0000","last_peered":"2024-09-21T10:52:22.859240+0000","last_clean":"2024-09-21T10:52:22.859240+0000","last_became_active":"2024-09-21T10:51:23.721548+0000","last_became_peered":"2024-09-21T10:51:23.721548+0000","last_unstale":"2024-09-21T10:52:22.859240+0000","last_undegraded":"2024-09-21T10:52:22.859240+0000","last_fullsized":"2024-09-21T10:52:22.859240+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:48:08.960250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:48:08.960250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:13:49.816964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":2441,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733102+0000","last_change":"2024-09-21T10:08:55.119751+0000","last_active":"2024-09-21T10:52:25.733102+0000","last_peered":"2024-09-21T10:52:25.733102+0000","last_clean":"2024-09-21T10:52:25.733102+0000","last_became_active":"2024-09-21T10:08:55.118624+0000","last_became_peered":"2024-09-21T10:08:55.118624+0000","last_unstale":"2024-09-21T10:52:25.733102+0000","last_undegraded":"2024-09-21T10:52:25.733102+0000","last_fullsized":"2024-09-21T10:52:25.733102+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:35.397028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859754+0000","last_change":"2024-09-21T10:51:19.712241+0000","last_active":"2024-09-21T10:52:22.859754+0000","last_peered":"2024-09-21T10:52:22.859754+0000","last_clean":"2024-09-21T10:52:22.859754+0000","last_became_active":"2024-09-21T10:51:19.711943+0000","last_became_peered":"2024-09-21T10:51:19.711943+0000","last_unstale":"2024-09-21T10:52:22.859754+0000","last_undegraded":"2024-09-21T10:52:22.859754+0000","last_fullsized":"2024-09-21T10:52:22.859754+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:41:51.003446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"58'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859486+0000","last_change":"2024-09-21T10:51:23.745775+0000","last_active":"2024-09-21T10:52:22.859486+0000","last_peered":"2024-09-21T10:52:22.859486+0000","last_clean":"2024-09-21T10:52:22.859486+0000","last_became_active":"2024-09-21T10:51:23.724595+0000","last_became_peered":"2024-09-21T10:51:23.724595+0000","last_unstale":"2024-09-21T10:52:22.859486+0000","last_undegraded":"2024-09-21T10:52:22.859486+0000","last_fullsized":"2024-09-21T10:52:22.859486+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:37:59.664737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"256'100","reported_seq":766,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860947+0000","last_change":"2024-09-21T10:52:12.175206+0000","last_active":"2024-09-21T10:52:22.860947+0000","last_peered":"2024-09-21T10:52:22.860947+0000","last_clean":"2024-09-21T10:52:22.860947+0000","last_became_active":"2024-09-21T10:52:12.174363+0000","last_became_peered":"2024-09-21T10:52:12.174363+0000","last_unstale":"2024-09-21T10:52:22.860947+0000","last_undegraded":"2024-09-21T10:52:22.860947+0000","last_fullsized":"2024-09-21T10:52:22.860947+0000","mapping_epoch":1502,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1503,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:38.606297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:38.606297+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:21.873271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":190,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733044+0000","last_change":"2024-09-21T10:51:23.727021+0000","last_active":"2024-09-21T10:52:25.733044+0000","last_peered":"2024-09-21T10:52:25.733044+0000","last_clean":"2024-09-21T10:52:25.733044+0000","last_became_active":"2024-09-21T10:51:23.726633+0000","last_became_peered":"2024-09-21T10:51:23.726633+0000","last_unstale":"2024-09-21T10:52:25.733044+0000","last_undegraded":"2024-09-21T10:52:25.733044+0000","last_fullsized":"2024-09-21T10:52:25.733044+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:07:00.390334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":2442,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732817+0000","last_change":"2024-09-21T10:26:04.180619+0000","last_active":"2024-09-21T10:52:25.732817+0000","last_peered":"2024-09-21T10:52:25.732817+0000","last_clean":"2024-09-21T10:52:25.732817+0000","last_became_active":"2024-09-21T10:26:04.179881+0000","last_became_peered":"2024-09-21T10:26:04.179881+0000","last_unstale":"2024-09-21T10:52:25.732817+0000","last_undegraded":"2024-09-21T10:52:25.732817+0000","last_fullsized":"2024-09-21T10:52:25.732817+0000","mapping_epoch":1191,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:35.669321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":2373,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732453+0000","last_change":"2024-09-21T10:08:55.114454+0000","last_active":"2024-09-21T10:52:25.732453+0000","last_peered":"2024-09-21T10:52:25.732453+0000","last_clean":"2024-09-21T10:52:25.732453+0000","last_became_active":"2024-09-21T10:08:55.114123+0000","last_became_peered":"2024-09-21T10:08:55.114123+0000","last_unstale":"2024-09-21T10:52:25.732453+0000","last_undegraded":"2024-09-21T10:52:25.732453+0000","last_fullsized":"2024-09-21T10:52:25.732453+0000","mapping_epoch":653,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":654,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:04:26.165708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00029014500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"256'100","reported_seq":644,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732457+0000","last_change":"2024-09-21T10:51:19.719681+0000","last_active":"2024-09-21T10:52:25.732457+0000","last_peered":"2024-09-21T10:52:25.732457+0000","last_clean":"2024-09-21T10:52:25.732457+0000","last_became_active":"2024-09-21T10:51:19.718591+0000","last_became_peered":"2024-09-21T10:51:19.718591+0000","last_unstale":"2024-09-21T10:52:25.732457+0000","last_undegraded":"2024-09-21T10:52:25.732457+0000","last_fullsized":"2024-09-21T10:52:25.732457+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":3,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:36.007121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:36.007121+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:14:39.734484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":298,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731603+0000","last_change":"2024-09-21T10:39:00.846217+0000","last_active":"2024-09-21T10:52:25.731603+0000","last_peered":"2024-09-21T10:52:25.731603+0000","last_clean":"2024-09-21T10:52:25.731603+0000","last_became_active":"2024-09-21T10:39:00.845874+0000","last_became_peered":"2024-09-21T10:39:00.845874+0000","last_unstale":"2024-09-21T10:52:25.731603+0000","last_undegraded":"2024-09-21T10:52:25.731603+0000","last_fullsized":"2024-09-21T10:52:25.731603+0000","mapping_epoch":1366,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1367,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:41.221850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2488,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731588+0000","last_change":"2024-09-21T10:51:19.718291+0000","last_active":"2024-09-21T10:52:25.731588+0000","last_peered":"2024-09-21T10:52:25.731588+0000","last_clean":"2024-09-21T10:52:25.731588+0000","last_became_active":"2024-09-21T10:51:19.716870+0000","last_became_peered":"2024-09-21T10:51:19.716870+0000","last_unstale":"2024-09-21T10:52:25.731588+0000","last_undegraded":"2024-09-21T10:52:25.731588+0000","last_fullsized":"2024-09-21T10:52:25.731588+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:35:21.632408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.858925+0000","last_change":"2024-09-21T10:51:26.803803+0000","last_active":"2024-09-21T10:52:22.858925+0000","last_peered":"2024-09-21T10:52:22.858925+0000","last_clean":"2024-09-21T10:52:22.858925+0000","last_became_active":"2024-09-21T10:51:26.803376+0000","last_became_peered":"2024-09-21T10:51:26.803376+0000","last_unstale":"2024-09-21T10:52:22.858925+0000","last_undegraded":"2024-09-21T10:52:22.858925+0000","last_fullsized":"2024-09-21T10:52:22.858925+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:23:49.706791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026208600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1435,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717952+0000","last_change":"2024-09-21T10:51:26.803612+0000","last_active":"2024-09-21T10:52:22.717952+0000","last_peered":"2024-09-21T10:52:22.717952+0000","last_clean":"2024-09-21T10:52:22.717952+0000","last_became_active":"2024-09-21T10:51:26.803307+0000","last_became_peered":"2024-09-21T10:51:26.803307+0000","last_unstale":"2024-09-21T10:52:22.717952+0000","last_undegraded":"2024-09-21T10:52:22.717952+0000","last_fullsized":"2024-09-21T10:52:22.717952+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:56:52.231058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2380,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722232+0000","last_change":"2024-09-21T10:51:23.720296+0000","last_active":"2024-09-21T10:52:25.722232+0000","last_peered":"2024-09-21T10:52:25.722232+0000","last_clean":"2024-09-21T10:52:25.722232+0000","last_became_active":"2024-09-21T10:51:23.719982+0000","last_became_peered":"2024-09-21T10:51:23.719982+0000","last_unstale":"2024-09-21T10:52:25.722232+0000","last_undegraded":"2024-09-21T10:52:25.722232+0000","last_fullsized":"2024-09-21T10:52:25.722232+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:06:57.519160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"56'1","reported_seq":48,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859108+0000","last_change":"2024-09-21T10:51:26.820358+0000","last_active":"2024-09-21T10:52:22.859108+0000","last_peered":"2024-09-21T10:52:22.859108+0000","last_clean":"2024-09-21T10:52:22.859108+0000","last_became_active":"2024-09-21T10:51:26.809379+0000","last_became_peered":"2024-09-21T10:51:26.809379+0000","last_unstale":"2024-09-21T10:52:22.859108+0000","last_undegraded":"2024-09-21T10:52:22.859108+0000","last_fullsized":"2024-09-21T10:52:22.859108+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:55:40.627784+0000","last_clean_scrub_stamp":"2024-09-21T09:55:40.627784+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:01:48.562692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"256'100","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859053+0000","last_change":"2024-09-21T10:51:26.800633+0000","last_active":"2024-09-21T10:52:22.859053+0000","last_peered":"2024-09-21T10:52:22.859053+0000","last_clean":"2024-09-21T10:52:22.859053+0000","last_became_active":"2024-09-21T10:51:26.799809+0000","last_became_peered":"2024-09-21T10:51:26.799809+0000","last_unstale":"2024-09-21T10:52:22.859053+0000","last_undegraded":"2024-09-21T10:52:22.859053+0000","last_fullsized":"2024-09-21T10:52:22.859053+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:33:25.551305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:33:25.551305+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:27:32.421023+0000","scrub_duration":0.0016971849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859654+0000","last_change":"2024-09-21T10:51:27.865788+0000","last_active":"2024-09-21T10:52:22.859654+0000","last_peered":"2024-09-21T10:52:22.859654+0000","last_clean":"2024-09-21T10:52:22.859654+0000","last_became_active":"2024-09-21T10:51:23.726524+0000","last_became_peered":"2024-09-21T10:51:23.726524+0000","last_unstale":"2024-09-21T10:52:22.859654+0000","last_undegraded":"2024-09-21T10:52:22.859654+0000","last_fullsized":"2024-09-21T10:52:22.859654+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:27.865723+0000","last_deep_scrub":"256'100","last_deep_scrub_stamp":"2024-09-21T10:51:27.865723+0000","last_clean_scrub_stamp":"2024-09-21T10:51:27.865723+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:58:18.975843+0000","scrub_duration":0.001524596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":2467,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.733015+0000","last_change":"2024-09-21T10:51:19.719114+0000","last_active":"2024-09-21T10:52:25.733015+0000","last_peered":"2024-09-21T10:52:25.733015+0000","last_clean":"2024-09-21T10:52:25.733015+0000","last_became_active":"2024-09-21T10:51:19.718769+0000","last_became_peered":"2024-09-21T10:51:19.718769+0000","last_unstale":"2024-09-21T10:52:25.733015+0000","last_undegraded":"2024-09-21T10:52:25.733015+0000","last_fullsized":"2024-09-21T10:52:25.733015+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:10:03.655730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2373,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722071+0000","last_change":"2024-09-21T10:51:26.801151+0000","last_active":"2024-09-21T10:52:25.722071+0000","last_peered":"2024-09-21T10:52:25.722071+0000","last_clean":"2024-09-21T10:52:25.722071+0000","last_became_active":"2024-09-21T10:51:26.800792+0000","last_became_peered":"2024-09-21T10:51:26.800792+0000","last_unstale":"2024-09-21T10:52:25.722071+0000","last_undegraded":"2024-09-21T10:52:25.722071+0000","last_fullsized":"2024-09-21T10:52:25.722071+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:20:10.166740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"256'100","reported_seq":642,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732409+0000","last_change":"2024-09-21T10:51:23.728196+0000","last_active":"2024-09-21T10:52:25.732409+0000","last_peered":"2024-09-21T10:52:25.732409+0000","last_clean":"2024-09-21T10:52:25.732409+0000","last_became_active":"2024-09-21T10:51:23.727811+0000","last_became_peered":"2024-09-21T10:51:23.727811+0000","last_unstale":"2024-09-21T10:52:25.732409+0000","last_undegraded":"2024-09-21T10:52:25.732409+0000","last_fullsized":"2024-09-21T10:52:25.732409+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:12.134349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:12.134349+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:08:55.537453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860493+0000","last_change":"2024-09-21T10:51:19.716570+0000","last_active":"2024-09-21T10:52:22.860493+0000","last_peered":"2024-09-21T10:52:22.860493+0000","last_clean":"2024-09-21T10:52:22.860493+0000","last_became_active":"2024-09-21T10:51:19.715324+0000","last_became_peered":"2024-09-21T10:51:19.715324+0000","last_unstale":"2024-09-21T10:52:22.860493+0000","last_undegraded":"2024-09-21T10:52:22.860493+0000","last_fullsized":"2024-09-21T10:52:22.860493+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:32:12.926642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":42,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859164+0000","last_change":"2024-09-21T10:51:26.802544+0000","last_active":"2024-09-21T10:52:22.859164+0000","last_peered":"2024-09-21T10:52:22.859164+0000","last_clean":"2024-09-21T10:52:22.859164+0000","last_became_active":"2024-09-21T10:51:26.802078+0000","last_became_peered":"2024-09-21T10:51:26.802078+0000","last_unstale":"2024-09-21T10:52:22.859164+0000","last_undegraded":"2024-09-21T10:52:22.859164+0000","last_fullsized":"2024-09-21T10:52:22.859164+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:17:44.846468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"256'100","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859429+0000","last_change":"2024-09-21T10:51:23.719026+0000","last_active":"2024-09-21T10:52:22.859429+0000","last_peered":"2024-09-21T10:52:22.859429+0000","last_clean":"2024-09-21T10:52:22.859429+0000","last_became_active":"2024-09-21T10:51:23.718679+0000","last_became_peered":"2024-09-21T10:51:23.718679+0000","last_unstale":"2024-09-21T10:52:22.859429+0000","last_undegraded":"2024-09-21T10:52:22.859429+0000","last_fullsized":"2024-09-21T10:52:22.859429+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:17.397101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:17.397101+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:06:37.045469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860679+0000","last_change":"2024-09-21T10:51:19.715047+0000","last_active":"2024-09-21T10:52:22.860679+0000","last_peered":"2024-09-21T10:52:22.860679+0000","last_clean":"2024-09-21T10:52:22.860679+0000","last_became_active":"2024-09-21T10:51:19.714491+0000","last_became_peered":"2024-09-21T10:51:19.714491+0000","last_unstale":"2024-09-21T10:52:22.860679+0000","last_undegraded":"2024-09-21T10:52:22.860679+0000","last_fullsized":"2024-09-21T10:52:22.860679+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:30:06.088284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1431,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717836+0000","last_change":"2024-09-21T10:51:19.707777+0000","last_active":"2024-09-21T10:52:22.717836+0000","last_peered":"2024-09-21T10:52:22.717836+0000","last_clean":"2024-09-21T10:52:22.717836+0000","last_became_active":"2024-09-21T10:51:19.707407+0000","last_became_peered":"2024-09-21T10:51:19.707407+0000","last_unstale":"2024-09-21T10:52:22.717836+0000","last_undegraded":"2024-09-21T10:52:22.717836+0000","last_fullsized":"2024-09-21T10:52:22.717836+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:20:16.297430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"256'100","reported_seq":38,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859669+0000","last_change":"2024-09-21T10:51:40.034320+0000","last_active":"2024-09-21T10:52:22.859669+0000","last_peered":"2024-09-21T10:52:22.859669+0000","last_clean":"2024-09-21T10:52:22.859669+0000","last_became_active":"2024-09-21T10:51:26.800552+0000","last_became_peered":"2024-09-21T10:51:26.800552+0000","last_unstale":"2024-09-21T10:52:22.859669+0000","last_undegraded":"2024-09-21T10:52:22.859669+0000","last_fullsized":"2024-09-21T10:52:22.859669+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1232,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:40.034250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:40.034250+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:15:45.167441+0000","scrub_duration":0.0017901530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860123+0000","last_change":"2024-09-21T10:51:19.709533+0000","last_active":"2024-09-21T10:52:22.860123+0000","last_peered":"2024-09-21T10:52:22.860123+0000","last_clean":"2024-09-21T10:52:22.860123+0000","last_became_active":"2024-09-21T10:51:19.709211+0000","last_became_peered":"2024-09-21T10:51:19.709211+0000","last_unstale":"2024-09-21T10:52:22.860123+0000","last_undegraded":"2024-09-21T10:52:22.860123+0000","last_fullsized":"2024-09-21T10:52:22.860123+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:02:33.467064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"64'2","reported_seq":1493,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732548+0000","last_change":"2024-09-21T10:51:19.729909+0000","last_active":"2024-09-21T10:52:25.732548+0000","last_peered":"2024-09-21T10:52:25.732548+0000","last_clean":"2024-09-21T10:52:25.732548+0000","last_became_active":"2024-09-21T10:51:19.719252+0000","last_became_peered":"2024-09-21T10:51:19.719252+0000","last_unstale":"2024-09-21T10:52:25.732548+0000","last_undegraded":"2024-09-21T10:52:25.732548+0000","last_fullsized":"2024-09-21T10:52:25.732548+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:19:44.050500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"256'100","reported_seq":52,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860504+0000","last_change":"2024-09-21T10:51:29.423375+0000","last_active":"2024-09-21T10:52:22.860504+0000","last_peered":"2024-09-21T10:52:22.860504+0000","last_clean":"2024-09-21T10:52:22.860504+0000","last_became_active":"2024-09-21T10:51:26.797880+0000","last_became_peered":"2024-09-21T10:51:26.797880+0000","last_unstale":"2024-09-21T10:52:22.860504+0000","last_undegraded":"2024-09-21T10:52:22.860504+0000","last_fullsized":"2024-09-21T10:52:22.860504+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:29.423304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:29.423304+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:24:03.797459+0000","scrub_duration":0.0018586259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":2479,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732748+0000","last_change":"2024-09-21T10:51:19.716770+0000","last_active":"2024-09-21T10:52:25.732748+0000","last_peered":"2024-09-21T10:52:25.732748+0000","last_clean":"2024-09-21T10:52:25.732748+0000","last_became_active":"2024-09-21T10:51:19.716197+0000","last_became_peered":"2024-09-21T10:51:19.716197+0000","last_unstale":"2024-09-21T10:52:25.732748+0000","last_undegraded":"2024-09-21T10:52:25.732748+0000","last_fullsized":"2024-09-21T10:52:25.732748+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:37:18.215947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":30,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860373+0000","last_change":"2024-09-21T10:51:23.725538+0000","last_active":"2024-09-21T10:52:22.860373+0000","last_peered":"2024-09-21T10:52:22.860373+0000","last_clean":"2024-09-21T10:52:22.860373+0000","last_became_active":"2024-09-21T10:51:23.725198+0000","last_became_peered":"2024-09-21T10:51:23.725198+0000","last_unstale":"2024-09-21T10:52:22.860373+0000","last_undegraded":"2024-09-21T10:52:22.860373+0000","last_fullsized":"2024-09-21T10:52:22.860373+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:11.636890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.f","version":"256'100","reported_seq":619,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.718183+0000","last_change":"2024-09-21T10:51:23.731927+0000","last_active":"2024-09-21T10:52:22.718183+0000","last_peered":"2024-09-21T10:52:22.718183+0000","last_clean":"2024-09-21T10:52:22.718183+0000","last_became_active":"2024-09-21T10:51:23.731572+0000","last_became_peered":"2024-09-21T10:51:23.731572+0000","last_unstale":"2024-09-21T10:52:22.718183+0000","last_undegraded":"2024-09-21T10:52:22.718183+0000","last_fullsized":"2024-09-21T10:52:22.718183+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:44:00.317241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:44:00.317241+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:37:41.649690+0000","scrub_duration":0.001848176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1413,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717723+0000","last_change":"2024-09-21T10:51:26.802738+0000","last_active":"2024-09-21T10:52:22.717723+0000","last_peered":"2024-09-21T10:52:22.717723+0000","last_clean":"2024-09-21T10:52:22.717723+0000","last_became_active":"2024-09-21T10:51:26.802271+0000","last_became_peered":"2024-09-21T10:51:26.802271+0000","last_unstale":"2024-09-21T10:52:22.717723+0000","last_undegraded":"2024-09-21T10:52:22.717723+0000","last_fullsized":"2024-09-21T10:52:22.717723+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:56:48.191078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"64'1","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861219+0000","last_change":"2024-09-21T10:52:12.176363+0000","last_active":"2024-09-21T10:52:22.861219+0000","last_peered":"2024-09-21T10:52:22.861219+0000","last_clean":"2024-09-21T10:52:22.861219+0000","last_became_active":"2024-09-21T10:52:12.175968+0000","last_became_peered":"2024-09-21T10:52:12.175968+0000","last_unstale":"2024-09-21T10:52:22.861219+0000","last_undegraded":"2024-09-21T10:52:22.861219+0000","last_fullsized":"2024-09-21T10:52:22.861219+0000","mapping_epoch":1502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T13:28:34.589176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"256'100","reported_seq":630,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722464+0000","last_change":"2024-09-21T10:51:12.411398+0000","last_active":"2024-09-21T10:52:25.722464+0000","last_peered":"2024-09-21T10:52:25.722464+0000","last_clean":"2024-09-21T10:52:25.722464+0000","last_became_active":"2024-09-21T10:51:11.649055+0000","last_became_peered":"2024-09-21T10:51:11.649055+0000","last_unstale":"2024-09-21T10:52:25.722464+0000","last_undegraded":"2024-09-21T10:52:25.722464+0000","last_fullsized":"2024-09-21T10:52:25.722464+0000","mapping_epoch":1282,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1481,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:12.411343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:12.411343+0000","objects_scrubbed":1,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:35:18.585319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859553+0000","last_change":"2024-09-21T10:51:19.716483+0000","last_active":"2024-09-21T10:52:22.859553+0000","last_peered":"2024-09-21T10:52:22.859553+0000","last_clean":"2024-09-21T10:52:22.859553+0000","last_became_active":"2024-09-21T10:51:19.716109+0000","last_became_peered":"2024-09-21T10:51:19.716109+0000","last_unstale":"2024-09-21T10:52:22.859553+0000","last_undegraded":"2024-09-21T10:52:22.859553+0000","last_fullsized":"2024-09-21T10:52:22.859553+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:53:31.056931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"64'2","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861453+0000","last_change":"2024-09-21T10:51:26.824514+0000","last_active":"2024-09-21T10:52:22.861453+0000","last_peered":"2024-09-21T10:52:22.861453+0000","last_clean":"2024-09-21T10:52:22.861453+0000","last_became_active":"2024-09-21T10:51:26.804254+0000","last_became_peered":"2024-09-21T10:51:26.804254+0000","last_unstale":"2024-09-21T10:52:22.861453+0000","last_undegraded":"2024-09-21T10:52:22.861453+0000","last_fullsized":"2024-09-21T10:52:22.861453+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:38:30.501261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"256'100","reported_seq":28,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861310+0000","last_change":"2024-09-21T10:51:26.804746+0000","last_active":"2024-09-21T10:52:22.861310+0000","last_peered":"2024-09-21T10:52:22.861310+0000","last_clean":"2024-09-21T10:52:22.861310+0000","last_became_active":"2024-09-21T10:51:26.804415+0000","last_became_peered":"2024-09-21T10:51:26.804415+0000","last_unstale":"2024-09-21T10:52:22.861310+0000","last_undegraded":"2024-09-21T10:52:22.861310+0000","last_fullsized":"2024-09-21T10:52:22.861310+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":4,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:14.151556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:14.151556+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T14:52:51.665464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859242+0000","last_change":"2024-09-21T10:51:26.809449+0000","last_active":"2024-09-21T10:52:22.859242+0000","last_peered":"2024-09-21T10:52:22.859242+0000","last_clean":"2024-09-21T10:52:22.859242+0000","last_became_active":"2024-09-21T10:51:26.808895+0000","last_became_peered":"2024-09-21T10:51:26.808895+0000","last_unstale":"2024-09-21T10:52:22.859242+0000","last_undegraded":"2024-09-21T10:52:22.859242+0000","last_fullsized":"2024-09-21T10:52:22.859242+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:03:40.151479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"64'1","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860688+0000","last_change":"2024-09-21T10:51:26.822830+0000","last_active":"2024-09-21T10:52:22.860688+0000","last_peered":"2024-09-21T10:52:22.860688+0000","last_clean":"2024-09-21T10:52:22.860688+0000","last_became_active":"2024-09-21T10:51:26.803697+0000","last_became_peered":"2024-09-21T10:51:26.803697+0000","last_unstale":"2024-09-21T10:52:22.860688+0000","last_undegraded":"2024-09-21T10:52:22.860688+0000","last_fullsized":"2024-09-21T10:52:22.860688+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:54:06.677964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859570+0000","last_change":"2024-09-21T10:51:28.893933+0000","last_active":"2024-09-21T10:52:22.859570+0000","last_peered":"2024-09-21T10:52:22.859570+0000","last_clean":"2024-09-21T10:52:22.859570+0000","last_became_active":"2024-09-21T10:51:23.724692+0000","last_became_peered":"2024-09-21T10:51:23.724692+0000","last_unstale":"2024-09-21T10:52:22.859570+0000","last_undegraded":"2024-09-21T10:52:22.859570+0000","last_fullsized":"2024-09-21T10:52:22.859570+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:51:28.893864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:51:28.893864+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:23:39.628862+0000","scrub_duration":0.001659862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"64'8","reported_seq":1510,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.717693+0000","last_change":"2024-09-21T10:51:23.743383+0000","last_active":"2024-09-21T10:52:22.717693+0000","last_peered":"2024-09-21T10:52:22.717693+0000","last_clean":"2024-09-21T10:52:22.717693+0000","last_became_active":"2024-09-21T10:51:23.730143+0000","last_became_peered":"2024-09-21T10:51:23.730143+0000","last_unstale":"2024-09-21T10:52:22.717693+0000","last_undegraded":"2024-09-21T10:52:22.717693+0000","last_fullsized":"2024-09-21T10:52:22.717693+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T10:14:38.077413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":2348,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722332+0000","last_change":"2024-09-21T10:51:26.800999+0000","last_active":"2024-09-21T10:52:25.722332+0000","last_peered":"2024-09-21T10:52:25.722332+0000","last_clean":"2024-09-21T10:52:25.722332+0000","last_became_active":"2024-09-21T10:51:26.800445+0000","last_became_peered":"2024-09-21T10:51:26.800445+0000","last_unstale":"2024-09-21T10:52:25.722332+0000","last_undegraded":"2024-09-21T10:52:25.722332+0000","last_fullsized":"2024-09-21T10:52:25.722332+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:39:32.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.13","version":"256'100","reported_seq":627,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732276+0000","last_change":"2024-09-21T10:51:19.719964+0000","last_active":"2024-09-21T10:52:25.732276+0000","last_peered":"2024-09-21T10:52:25.732276+0000","last_clean":"2024-09-21T10:52:25.732276+0000","last_became_active":"2024-09-21T10:51:19.719430+0000","last_became_peered":"2024-09-21T10:51:19.719430+0000","last_unstale":"2024-09-21T10:52:25.732276+0000","last_undegraded":"2024-09-21T10:52:25.732276+0000","last_fullsized":"2024-09-21T10:52:25.732276+0000","mapping_epoch":1484,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1485,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:34.983887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:34.983887+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:56.282017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":32,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.861077+0000","last_change":"2024-09-21T10:52:12.176229+0000","last_active":"2024-09-21T10:52:22.861077+0000","last_peered":"2024-09-21T10:52:22.861077+0000","last_clean":"2024-09-21T10:52:22.861077+0000","last_became_active":"2024-09-21T10:52:12.175790+0000","last_became_peered":"2024-09-21T10:52:12.175790+0000","last_unstale":"2024-09-21T10:52:22.861077+0000","last_undegraded":"2024-09-21T10:52:22.861077+0000","last_fullsized":"2024-09-21T10:52:22.861077+0000","mapping_epoch":1502,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:24:12.844163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":2467,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731836+0000","last_change":"2024-09-21T10:51:26.805572+0000","last_active":"2024-09-21T10:52:25.731836+0000","last_peered":"2024-09-21T10:52:25.731836+0000","last_clean":"2024-09-21T10:52:25.731836+0000","last_became_active":"2024-09-21T10:51:26.805206+0000","last_became_peered":"2024-09-21T10:51:26.805206+0000","last_unstale":"2024-09-21T10:52:25.731836+0000","last_undegraded":"2024-09-21T10:52:25.731836+0000","last_fullsized":"2024-09-21T10:52:25.731836+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:29:23.513847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"256'100","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.858971+0000","last_change":"2024-09-21T10:51:26.813501+0000","last_active":"2024-09-21T10:52:22.858971+0000","last_peered":"2024-09-21T10:52:22.858971+0000","last_clean":"2024-09-21T10:52:22.858971+0000","last_became_active":"2024-09-21T10:51:26.813010+0000","last_became_peered":"2024-09-21T10:51:26.813010+0000","last_unstale":"2024-09-21T10:52:22.858971+0000","last_undegraded":"2024-09-21T10:52:22.858971+0000","last_fullsized":"2024-09-21T10:52:22.858971+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:50:51.566260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:50:51.566260+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:48:03.992303+0000","scrub_duration":0.0013071770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":2467,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.732592+0000","last_change":"2024-09-21T10:51:26.804171+0000","last_active":"2024-09-21T10:52:25.732592+0000","last_peered":"2024-09-21T10:52:25.732592+0000","last_clean":"2024-09-21T10:52:25.732592+0000","last_became_active":"2024-09-21T10:51:26.803792+0000","last_became_peered":"2024-09-21T10:51:26.803792+0000","last_unstale":"2024-09-21T10:52:25.732592+0000","last_undegraded":"2024-09-21T10:52:25.732592+0000","last_fullsized":"2024-09-21T10:52:25.732592+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:29:36.654413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":40,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.859436+0000","last_change":"2024-09-21T10:51:26.809443+0000","last_active":"2024-09-21T10:52:22.859436+0000","last_peered":"2024-09-21T10:52:22.859436+0000","last_clean":"2024-09-21T10:52:22.859436+0000","last_became_active":"2024-09-21T10:51:26.808907+0000","last_became_peered":"2024-09-21T10:51:26.808907+0000","last_unstale":"2024-09-21T10:52:22.859436+0000","last_undegraded":"2024-09-21T10:52:22.859436+0000","last_fullsized":"2024-09-21T10:52:22.859436+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T11:18:05.922402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"256'100","reported_seq":352,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.731421+0000","last_change":"2024-09-21T10:51:23.725035+0000","last_active":"2024-09-21T10:52:25.731421+0000","last_peered":"2024-09-21T10:52:25.731421+0000","last_clean":"2024-09-21T10:52:25.731421+0000","last_became_active":"2024-09-21T10:51:23.724574+0000","last_became_peered":"2024-09-21T10:51:23.724574+0000","last_unstale":"2024-09-21T10:52:25.731421+0000","last_undegraded":"2024-09-21T10:52:25.731421+0000","last_fullsized":"2024-09-21T10:52:25.731421+0000","mapping_epoch":1486,"log_start":"0'0","ondisk_log_start":"0'0","created":1257,"last_epoch_clean":1487,"parent":"0.0","parent_split_bits":5,"last_scrub":"256'100","last_scrub_stamp":"2024-09-21T10:37:49.672228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:53:13.106674+0000","last_clean_scrub_stamp":"2024-09-21T10:37:49.672228+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T15:54:13.916198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"64'2","reported_seq":34,"reported_epoch":1506,"state":"active+clean","last_fresh":"2024-09-21T10:52:22.860075+0000","last_change":"2024-09-21T10:51:26.817029+0000","last_active":"2024-09-21T10:52:22.860075+0000","last_peered":"2024-09-21T10:52:22.860075+0000","last_clean":"2024-09-21T10:52:22.860075+0000","last_became_active":"2024-09-21T10:51:26.794038+0000","last_became_peered":"2024-09-21T10:51:26.794038+0000","last_unstale":"2024-09-21T10:52:22.860075+0000","last_undegraded":"2024-09-21T10:52:22.860075+0000","last_fullsized":"2024-09-21T10:52:22.860075+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:58.973441+0000","last_clean_scrub_stamp":"2024-09-21T09:56:58.973441+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:36:54.821094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":68,"num_keys_recovered":20,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":2356,"reported_epoch":1507,"state":"active+clean","last_fresh":"2024-09-21T10:52:25.722512+0000","last_change":"2024-09-21T10:51:26.801631+0000","last_active":"2024-09-21T10:52:25.722512+0000","last_peered":"2024-09-21T10:52:25.722512+0000","last_clean":"2024-09-21T10:52:25.722512+0000","last_became_active":"2024-09-21T10:51:26.800926+0000","last_became_peered":"2024-09-21T10:51:26.800926+0000","last_unstale":"2024-09-21T10:52:25.722512+0000","last_undegraded":"2024-09-21T10:52:25.722512+0000","last_fullsized":"2024-09-21T10:52:25.722512+0000","mapping_epoch":1488,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":1489,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T09:56:59.977499+0000","last_clean_scrub_stamp":"2024-09-21T09:56:59.977499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T12:01:33.338777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3100,"ondisk_log_size":3100,"up":93,"acting":93,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":5530,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":61,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":12336,"num_keys_recovered":20,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":16590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2326,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":49,"seq":210453398205,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905952,"kb_used_data":10324,"kb_used_omap":1,"kb_used_meta":895614,"kb_avail":92839200,"statfs":{"total":95995035648,"available":95067340800,"internally_reserved":0,"allocated":10571776,"data_stored":5960854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917109080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":1191,"seq":5115306049864,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1511316,"kb_used_data":10376,"kb_used_omap":242,"kb_used_meta":1500685,"kb_avail":92233836,"statfs":{"total":95995035648,"available":94447448064,"internally_reserved":0,"allocated":10625024,"data_stored":6025719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":248609,"internal_metadata":1536701663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790663,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907344,"kb_used_data":10308,"kb_used_omap":1,"kb_used_meta":897022,"kb_avail":92837808,"statfs":{"total":95995035648,"available":95065915392,"internally_reserved":0,"allocated":10555392,"data_stored":5956553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918550872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":1211,"seq":5201205395769,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333592,"kb_used_data":10060,"kb_used_omap":224,"kb_used_meta":323295,"kb_avail":93411560,"statfs":{"total":95995035648,"available":95653437440,"internally_reserved":0,"allocated":10301440,"data_stored":5698096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":229495,"internal_metadata":331054985},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":653,"seq":2804613644822,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137988,"kb_used_data":10360,"kb_used_omap":67,"kb_used_meta":1127548,"kb_avail":92607164,"statfs":{"total":95995035648,"available":94829735936,"internally_reserved":0,"allocated":10608640,"data_stored":6020988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68771,"internal_metadata":1154610013},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":626,"seq":2688649527840,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1114060,"kb_used_data":10304,"kb_used_omap":75,"kb_used_meta":1103668,"kb_avail":92631092,"statfs":{"total":95995035648,"available":94854238208,"internally_reserved":0,"allocated":10551296,"data_stored":5956526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":77670,"internal_metadata":1130156186},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":542,"seq":2327872274993,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1102916,"kb_used_data":10040,"kb_used_omap":51,"kb_used_meta":1092812,"kb_avail":92642236,"statfs":{"total":95995035648,"available":94865649664,"internally_reserved":0,"allocated":10280960,"data_stored":5693237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52907,"internal_metadata":1119039829},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":509,"seq":2186138354232,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":919240,"kb_used_data":10364,"kb_used_omap":66,"kb_used_meta":908797,"kb_avail":92825912,"statfs":{"total":95995035648,"available":95053733888,"internally_reserved":0,"allocated":10612736,"data_stored":6021490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68284,"internal_metadata":930608452},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1454,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":872,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":4814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T10:53:15.396 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-21T10:53:15.396 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T10:53:15.569 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=408b126d58474820a3b2efa8dedbde51 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 48, in task with parallel.parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_62da54b4b83a96ca9976b7e1d0c8272a564a2208/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cls/test_cls_rbd.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2024-09-21T10:53:15.572 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T10:53:15.583 INFO:tasks.cephadm:Teardown begin 2024-09-21T10:53:15.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:53:15.589 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:15 smithi145 ceph-mon[89359]: pgmap v2807: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:15.589 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:15 smithi145 ceph-mon[89359]: from='client.37752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:15.612 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:15 smithi119 ceph-mon[105288]: pgmap v2807: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:15.612 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:15 smithi119 ceph-mon[105288]: from='client.37752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:15.612 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:15 smithi119 ceph-mon[101432]: pgmap v2807: 103 pgs: 103 active+clean; 582 KiB data, 7.6 GiB used, 708 GiB / 715 GiB avail 2024-09-21T10:53:15.612 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:15 smithi119 ceph-mon[101432]: from='client.37752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T10:53:15.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:53:15.664 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T10:53:15.664 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T10:53:15.683 DEBUG:teuthology.orchestra.run.smithi145:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T10:53:15.723 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T10:53:15.723 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T10:53:15.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a 2024-09-21T10:53:15.867 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:15 smithi119 systemd[1]: Stopping Ceph mon.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:16.367 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:53:16.020+0000 7f31319f3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a[101409]: 2024-09-21T10:53:16.020+0000 7f31319f3640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-21T10:53:16.368 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:16 smithi119 podman[373532]: 2024-09-21 10:53:16.092584279 +0000 UTC m=+0.174968551 container died 030d68226d6c3648ebdf56480c5b35f2ad9aad20c680ccecab0008a66565038e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=main-62da54b) 2024-09-21T10:53:16.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:16] ENGINE Bus STOPPING 2024-09-21T10:53:16.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:53:16.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:16] ENGINE Bus STOPPED 2024-09-21T10:53:16.368 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:16] ENGINE Bus STARTING 2024-09-21T10:53:16.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:16] ENGINE Serving on http://:::9283 2024-09-21T10:53:16.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:16 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:16] ENGINE Bus STARTED 2024-09-21T10:53:16.806 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:16 smithi119 podman[373532]: 2024-09-21 10:53:16.577897151 +0000 UTC m=+0.660281420 container cleanup 030d68226d6c3648ebdf56480c5b35f2ad9aad20c680ccecab0008a66565038e (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T10:53:16.806 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 21 10:53:16 smithi119 bash[373532]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-a 2024-09-21T10:53:16.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T10:53:16.981 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:16.981 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T10:53:16.981 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T10:53:16.982 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.c 2024-09-21T10:53:17.117 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:17 smithi119 systemd[1]: Stopping Ceph mon.c for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:17.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:17] ENGINE Bus STOPPING 2024-09-21T10:53:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c[105265]: 2024-09-21T10:53:17.267+0000 7faf87204640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c[105265]: 2024-09-21T10:53:17.267+0000 7faf87204640 -1 mon.c@2(peon) e4 *** Got Signal Terminated *** 2024-09-21T10:53:17.617 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:17 smithi119 podman[373675]: 2024-09-21 10:53:17.462340293 +0000 UTC m=+0.300836729 container died d747bb0bb662b2a376e2b674aa205950dd7db2fe872038d168b993d4d5a0eef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-62da54b) 2024-09-21T10:53:18.094 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:17 smithi119 podman[373675]: 2024-09-21 10:53:17.841059207 +0000 UTC m=+0.679555644 container cleanup d747bb0bb662b2a376e2b674aa205950dd7db2fe872038d168b993d4d5a0eef0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T10:53:18.094 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 21 10:53:17 smithi119 bash[373675]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-c 2024-09-21T10:53:18.094 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T10:53:18.094 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:17] ENGINE Bus STOPPED 2024-09-21T10:53:18.095 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:17] ENGINE Bus STARTING 2024-09-21T10:53:18.095 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:17] ENGINE Serving on http://:::9283 2024-09-21T10:53:18.095 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:17 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: [21/Sep/2024:10:53:17] ENGINE Bus STARTED 2024-09-21T10:53:18.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T10:53:18.251 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:18.252 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T10:53:18.252 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T10:53:18.252 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.b 2024-09-21T10:53:18.544 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:18 smithi145 systemd[1]: Stopping Ceph mon.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:18.962 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:18 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b[89336]: 2024-09-21T10:53:18.585+0000 7f4bd6cf1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:18.963 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:18 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b[89336]: 2024-09-21T10:53:18.585+0000 7f4bd6cf1640 -1 mon.b@1(peon) e4 *** Got Signal Terminated *** 2024-09-21T10:53:18.963 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:18 smithi145 podman[245491]: 2024-09-21 10:53:18.687618156 +0000 UTC m=+0.208178260 container died 550baecb68e011536364fe641567a51415adcd8cd4f9aa34b5b4313c6c508c09 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:53:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:19 smithi145 podman[245491]: 2024-09-21 10:53:19.150314455 +0000 UTC m=+0.670874555 container cleanup 550baecb68e011536364fe641567a51415adcd8cd4f9aa34b5b4313c6c508c09 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T10:53:19.294 INFO:journalctl@ceph.mon.b.smithi145.stdout:Sep 21 10:53:19 smithi145 bash[245491]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mon-b 2024-09-21T10:53:19.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T10:53:19.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:19.671 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T10:53:19.672 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T10:53:19.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y 2024-09-21T10:53:19.957 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:19 smithi119 systemd[1]: Stopping Ceph mgr.y for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:20.264 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:19 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:53:19.955+0000 7fc11c965640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:20.264 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:19 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y[92478]: 2024-09-21T10:53:19.955+0000 7fc11c965640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T10:53:20.265 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:19 smithi119 podman[373815]: 2024-09-21 10:53:19.998011703 +0000 UTC m=+0.138722971 container died d7bcb26ddb6215674e498dd628dbd05731e2974e4761a9f7bb9804afec4307b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-21T10:53:20.294 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:53:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:53:20.213Z caller=group.go:483 level=warn name=CephPoolGrowthWarning index=0 component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on (pool_id, instance)\n group_right () ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: Pool '{{ $labels.name }}' will be full in less than 5 days assuming\n the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed capacity\n" err="found duplicate series for the match group {instance=\"ceph_cluster\", pool_id=\"1\"} on the left hand-side of the operation: [{cluster=\"c96f302a-77fe-11ef-baf6-efdc52797490\", instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}, {instance=\"ceph_cluster\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-21T10:53:20.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:20 smithi119 podman[373815]: 2024-09-21 10:53:20.463002527 +0000 UTC m=+0.603713836 container cleanup d7bcb26ddb6215674e498dd628dbd05731e2974e4761a9f7bb9804afec4307b6 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:53:20.617 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 21 10:53:20 smithi119 bash[373815]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-y 2024-09-21T10:53:20.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T10:53:20.932 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:20.932 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T10:53:20.933 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T10:53:20.933 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x 2024-09-21T10:53:21.281 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:53:20 smithi145 systemd[1]: Stopping Ceph mgr.x for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:21.544 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:53:21 smithi145 podman[245635]: 2024-09-21 10:53:21.27941998 +0000 UTC m=+0.158268919 container died 1346375dd74716f801cfc6cb378cdd27c2af9787385fc074fc7afae957fcf1f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:53:21.974 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:53:21 smithi145 podman[245635]: 2024-09-21 10:53:21.721841845 +0000 UTC m=+0.600690784 container cleanup 1346375dd74716f801cfc6cb378cdd27c2af9787385fc074fc7afae957fcf1f0 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:53:21.974 INFO:journalctl@ceph.mgr.x.smithi145.stdout:Sep 21 10:53:21 smithi145 bash[245635]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-mgr-x 2024-09-21T10:53:22.113 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T10:53:22.165 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:22.165 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T10:53:22.165 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T10:53:22.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.0 2024-09-21T10:53:22.470 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:22 smithi119 systemd[1]: Stopping Ceph osd.0 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:22.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[114899]: 2024-09-21T10:53:22.467+0000 7f438af66640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:22.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[114899]: 2024-09-21T10:53:22.467+0000 7f438af66640 -1 osd.0 1507 *** Got signal Terminated *** 2024-09-21T10:53:22.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:22 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0[114899]: 2024-09-21T10:53:22.467+0000 7f438af66640 -1 osd.0 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:53:28.273 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:27 smithi119 podman[373947]: 2024-09-21 10:53:27.956883414 +0000 UTC m=+5.603465083 container died 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-21T10:53:28.828 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:28 smithi119 podman[373947]: 2024-09-21 10:53:28.575573633 +0000 UTC m=+6.222155300 container cleanup 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T10:53:28.829 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:28 smithi119 bash[373947]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0 2024-09-21T10:53:29.116 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:28 smithi119 podman[373968]: 2024-09-21 10:53:28.827350721 +0000 UTC m=+0.860151096 container remove 55bebfd490f2626bce20659940d0017cc01e498e0f53f466cdb8ae3cfb69ae7b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0, RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T10:53:29.507 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:29 smithi119 podman[374070]: 2024-09-21 10:53:29.081203743 +0000 UTC m=+0.016383907 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:53:29.508 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:29 smithi119 podman[374070]: 2024-09-21 10:53:29.245148489 +0000 UTC m=+0.180328637 container create 22ce8ab1b1584031b2debb33bbe6cb1d3a138aa1afb3eeea2052cfeeac1984a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T10:53:29.508 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:29 smithi119 podman[374070]: 2024-09-21 10:53:29.506124546 +0000 UTC m=+0.441304712 container init 22ce8ab1b1584031b2debb33bbe6cb1d3a138aa1afb3eeea2052cfeeac1984a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T10:53:29.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:29 smithi119 podman[374070]: 2024-09-21 10:53:29.511421042 +0000 UTC m=+0.446601189 container start 22ce8ab1b1584031b2debb33bbe6cb1d3a138aa1afb3eeea2052cfeeac1984a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T10:53:29.867 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:29 smithi119 podman[374070]: 2024-09-21 10:53:29.622662181 +0000 UTC m=+0.557842330 container attach 22ce8ab1b1584031b2debb33bbe6cb1d3a138aa1afb3eeea2052cfeeac1984a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-62da54b, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:53:30.727 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:30 smithi119 podman[374070]: 2024-09-21 10:53:30.432553488 +0000 UTC m=+1.367733649 container died 22ce8ab1b1584031b2debb33bbe6cb1d3a138aa1afb3eeea2052cfeeac1984a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-62da54b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:53:30.980 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 21 10:53:30 smithi119 podman[374120]: 2024-09-21 10:53:30.959618671 +0000 UTC m=+0.518173875 container remove 22ce8ab1b1584031b2debb33bbe6cb1d3a138aa1afb3eeea2052cfeeac1984a5 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:53:30.986 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T10:53:31.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:31.024 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T10:53:31.025 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T10:53:31.025 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.1 2024-09-21T10:53:31.357 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:31 smithi119 systemd[1]: Stopping Ceph osd.1 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:31.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[120311]: 2024-09-21T10:53:31.355+0000 7fe2c7dfb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:31.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[120311]: 2024-09-21T10:53:31.355+0000 7fe2c7dfb640 -1 osd.1 1507 *** Got signal Terminated *** 2024-09-21T10:53:31.617 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:31 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1[120311]: 2024-09-21T10:53:31.355+0000 7fe2c7dfb640 -1 osd.1 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:53:37.187 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:36 smithi119 podman[374204]: 2024-09-21 10:53:36.884346468 +0000 UTC m=+5.639105351 container died e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T10:53:37.523 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:37 smithi119 podman[374204]: 2024-09-21 10:53:37.26223132 +0000 UTC m=+6.016990192 container cleanup e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T10:53:37.523 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:37 smithi119 bash[374204]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1 2024-09-21T10:53:37.832 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:37 smithi119 podman[374224]: 2024-09-21 10:53:37.52185785 +0000 UTC m=+0.639263720 container remove e679cf598e41f28db82407444588160c836125106f0e938b3fbe6bddf5a13c6b (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T10:53:38.117 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:37 smithi119 podman[374319]: 2024-09-21 10:53:37.733411953 +0000 UTC m=+0.015905188 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:53:38.117 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:37 smithi119 podman[374319]: 2024-09-21 10:53:37.830929655 +0000 UTC m=+0.113422877 container create e2a6e294445fe8ac289a040805f5a30f560e62b6903cc3c2055bc93ac670291a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T10:53:38.117 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:38 smithi119 podman[374319]: 2024-09-21 10:53:38.066528502 +0000 UTC m=+0.349021724 container init e2a6e294445fe8ac289a040805f5a30f560e62b6903cc3c2055bc93ac670291a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=main-62da54b, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-21T10:53:38.117 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:38 smithi119 podman[374319]: 2024-09-21 10:53:38.071707196 +0000 UTC m=+0.354200428 container start e2a6e294445fe8ac289a040805f5a30f560e62b6903cc3c2055bc93ac670291a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T10:53:38.555 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:38 smithi119 podman[374319]: 2024-09-21 10:53:38.208692469 +0000 UTC m=+0.491185701 container attach e2a6e294445fe8ac289a040805f5a30f560e62b6903cc3c2055bc93ac670291a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T10:53:38.555 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 21 10:53:38 smithi119 podman[374319]: 2024-09-21 10:53:38.257216842 +0000 UTC m=+0.539710065 container died e2a6e294445fe8ac289a040805f5a30f560e62b6903cc3c2055bc93ac670291a (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2) 2024-09-21T10:53:38.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T10:53:38.902 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:38.903 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T10:53:38.903 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T10:53:38.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.2 2024-09-21T10:53:39.109 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:38 smithi119 systemd[1]: Stopping Ceph osd.2 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:39.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[127814]: 2024-09-21T10:53:39.225+0000 7f912698f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:39.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[127814]: 2024-09-21T10:53:39.225+0000 7f912698f640 -1 osd.2 1507 *** Got signal Terminated *** 2024-09-21T10:53:39.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:39 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2[127814]: 2024-09-21T10:53:39.225+0000 7f912698f640 -1 osd.2 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:53:45.075 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:44 smithi119 podman[374447]: 2024-09-21 10:53:44.782999568 +0000 UTC m=+5.659911207 container died 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:53:45.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:45 smithi119 podman[374447]: 2024-09-21 10:53:45.256931928 +0000 UTC m=+6.133843591 container cleanup 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=main-62da54b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-09-21T10:53:45.367 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:45 smithi119 bash[374447]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2 2024-09-21T10:53:45.835 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:45 smithi119 podman[374467]: 2024-09-21 10:53:45.541793786 +0000 UTC m=+0.759894272 container remove 3019e7bd2405d40254764aca8e80980c8918ffd2392acb62184342ef8b26be90 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-62da54b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T10:53:46.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:45 smithi119 podman[374564]: 2024-09-21 10:53:45.834476792 +0000 UTC m=+0.097863548 container create b2d04885f311864783f046192b300ce92ecd58081f2f9f662b49f81b1ace8b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=main-62da54b, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T10:53:46.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:45 smithi119 podman[374564]: 2024-09-21 10:53:45.752280574 +0000 UTC m=+0.015667299 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:53:46.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:46 smithi119 podman[374564]: 2024-09-21 10:53:46.070080933 +0000 UTC m=+0.333467655 container init b2d04885f311864783f046192b300ce92ecd58081f2f9f662b49f81b1ace8b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, RELEASE=main-62da54b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-21T10:53:46.117 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:46 smithi119 podman[374564]: 2024-09-21 10:53:46.075223471 +0000 UTC m=+0.338610181 container start b2d04885f311864783f046192b300ce92ecd58081f2f9f662b49f81b1ace8b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:53:46.607 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:46 smithi119 podman[374564]: 2024-09-21 10:53:46.161748946 +0000 UTC m=+0.425135659 container attach b2d04885f311864783f046192b300ce92ecd58081f2f9f662b49f81b1ace8b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T10:53:46.608 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 21 10:53:46 smithi119 podman[374564]: 2024-09-21 10:53:46.262861902 +0000 UTC m=+0.526248623 container died b2d04885f311864783f046192b300ce92ecd58081f2f9f662b49f81b1ace8b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=main-62da54b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:53:46.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T10:53:46.957 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:46.957 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T10:53:46.957 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T10:53:46.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.3 2024-09-21T10:53:47.280 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:47 smithi119 systemd[1]: Stopping Ceph osd.3 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:47.617 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:53:47.277+0000 7f55005af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:47.617 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:53:47.277+0000 7f55005af640 -1 osd.3 1507 *** Got signal Terminated *** 2024-09-21T10:53:47.617 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:47 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:53:47.277+0000 7f55005af640 -1 osd.3 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:53:49.794 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:49 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:49.357+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:50.654 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:50.356+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:50.926 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:50 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:50.651+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:51.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:51 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:51.310+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:51.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:51 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:51.266+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:52.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:51 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:51.676+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:52.367 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:51 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3[134086]: 2024-09-21T10:53:51.874+0000 7f54f8bbc640 -1 osd.3 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:21.846199+0000 front 2024-09-21T10:53:21.846274+0000 (oldest deadline 2024-09-21T10:53:51.846837+0000) 2024-09-21T10:53:52.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:52.326+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:52.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:52.254+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:52.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:52.311+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:53.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:52 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:52.640+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:53.212 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:52 smithi119 podman[374691]: 2024-09-21 10:53:52.927145959 +0000 UTC m=+5.746459945 container died 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-62da54b, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-21T10:53:53.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:53.323+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:53.544 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:53.273+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:53.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:53.292+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:53.617 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:53 smithi119 podman[374691]: 2024-09-21 10:53:53.409201758 +0000 UTC m=+6.228515731 container cleanup 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T10:53:53.617 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:53 smithi119 bash[374691]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3 2024-09-21T10:53:54.021 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:53 smithi119 podman[374716]: 2024-09-21 10:53:53.710579601 +0000 UTC m=+0.785872767 container remove 52358544d24ec1adbe346dbe7747c08a42c1e84b1d12d8f7f9f54a0882c97336 (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-62da54b) 2024-09-21T10:53:54.043 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:53 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:53.657+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:54.306 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:54 smithi119 podman[374805]: 2024-09-21 10:53:54.019713558 +0000 UTC m=+0.109302737 container create 7523772a5d44413c486a806fab6665a80857ef204182ded5155f079e456e0fef (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-62da54b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2) 2024-09-21T10:53:54.307 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:54 smithi119 podman[374805]: 2024-09-21 10:53:53.926278918 +0000 UTC m=+0.015868061 image pull a8ebb3d62dd1a69d83c65ce6bcc14c50b43d220d13e8424b0cbe758f609711f0 quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47 2024-09-21T10:53:54.307 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:54 smithi119 podman[374805]: 2024-09-21 10:53:54.230345475 +0000 UTC m=+0.319934626 container init 7523772a5d44413c486a806fab6665a80857ef204182ded5155f079e456e0fef (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-62da54b, org.label-schema.build-date=20240731) 2024-09-21T10:53:54.307 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:54 smithi119 podman[374805]: 2024-09-21 10:53:54.235373769 +0000 UTC m=+0.324962902 container start 7523772a5d44413c486a806fab6665a80857ef204182ded5155f079e456e0fef (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-62da54b) 2024-09-21T10:53:54.307 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:54 smithi119 podman[374805]: 2024-09-21 10:53:54.305065549 +0000 UTC m=+0.394654683 container attach 7523772a5d44413c486a806fab6665a80857ef204182ded5155f079e456e0fef (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=main-62da54b) 2024-09-21T10:53:54.617 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:54 smithi119 podman[374805]: 2024-09-21 10:53:54.426999979 +0000 UTC m=+0.516589120 container died 7523772a5d44413c486a806fab6665a80857ef204182ded5155f079e456e0fef (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, RELEASE=main-62da54b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:53:54.641 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:54.639+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:54.642 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:54.295+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:54.642 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:54.322+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:54.643 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:54 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:54.316+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:55.128 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 21 10:53:55 smithi119 podman[374848]: 2024-09-21 10:53:55.108408046 +0000 UTC m=+0.673834471 container remove 7523772a5d44413c486a806fab6665a80857ef204182ded5155f079e456e0fef (image=quay.ceph.io/ceph-ci/ceph@sha256:1283e169fa2a012678b44d89aba6a8833eba9b66e43c608c048a2ff9c690cf47, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-62da54b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:53:55.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T10:53:55.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:53:55.173 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T10:53:55.173 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T10:53:55.173 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4 2024-09-21T10:53:55.476 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:55 smithi145 systemd[1]: Stopping Ceph osd.4 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:53:55.476 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:55.356+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:55.477 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:55.278+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:55.477 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:55.345+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:55.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:55.473+0000 7fbe59cfb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:53:55.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:55.473+0000 7fbe59cfb640 -1 osd.4 1507 *** Got signal Terminated *** 2024-09-21T10:53:55.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:55.473+0000 7fbe59cfb640 -1 osd.4 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:53:55.794 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:55 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:55.682+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:56.730 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:56.305+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:56.731 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:56.343+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:56.731 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:56.367+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:57.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:56 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:56.728+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:57.544 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:57.270+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:57.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:57.298+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:57.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:57.351+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:57.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:57.351+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:53:58.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:57 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:57.752+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:58.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:58.298+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:58.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:58.298+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:53:58.544 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:58.259+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:58.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:58.259+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:53:58.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:58.372+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:58.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:58.372+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:53:59.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:58 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:58.755+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:53:59.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:59.322+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:53:59.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:53:59.322+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:53:59.544 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:59.263+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:53:59.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:53:59.263+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:53:59.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:59.336+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:53:59.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:53:59.336+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:00.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:53:59 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4[123885]: 2024-09-21T10:53:59.767+0000 7fbe57315640 -1 osd.4 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.238510+0000 front 2024-09-21T10:53:25.238693+0000 (oldest deadline 2024-09-21T10:53:50.538424+0000) 2024-09-21T10:54:00.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:00.312+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:00.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:00.312+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:00.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:00.291+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:00.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:00.291+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:00.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:00.370+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:00.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:00 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:00.370+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:01.294 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:01 smithi145 podman[245782]: 2024-09-21 10:54:01.037189223 +0000 UTC m=+5.671769149 container died 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T10:54:01.742 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:01 smithi145 podman[245782]: 2024-09-21 10:54:01.447762776 +0000 UTC m=+6.082342769 container cleanup 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a) 2024-09-21T10:54:01.742 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:01 smithi145 bash[245782]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4 2024-09-21T10:54:01.743 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:01.367+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:01.743 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:01.367+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:01.743 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:01.325+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:01.743 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:01.325+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:01.743 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:01.294+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:01.744 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:01 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:01.294+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:02.044 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:01 smithi145 podman[245803]: 2024-09-21 10:54:01.740707225 +0000 UTC m=+0.703601188 container remove 5b54c89df0f6c8fc243c17f2ae5448b5202704985a24eb0bfb61b833ddc6f271 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-21T10:54:02.313 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:02.290+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:02.313 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:02.290+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:02.313 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:02 smithi145 podman[245897]: 2024-09-21 10:54:01.956867644 +0000 UTC m=+0.016995526 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:54:02.313 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:02 smithi145 podman[245897]: 2024-09-21 10:54:02.058612805 +0000 UTC m=+0.118740679 container create 9d8af4b0c1670fb7a1549ed6e6427862214ac151369ef366d451a484dd54a308 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T10:54:02.314 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:02 smithi145 podman[245897]: 2024-09-21 10:54:02.26100287 +0000 UTC m=+0.321130750 container init 9d8af4b0c1670fb7a1549ed6e6427862214ac151369ef366d451a484dd54a308 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:54:02.314 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:02 smithi145 podman[245897]: 2024-09-21 10:54:02.266163617 +0000 UTC m=+0.326291495 container start 9d8af4b0c1670fb7a1549ed6e6427862214ac151369ef366d451a484dd54a308 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-21T10:54:02.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:02.309+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:02.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:02.309+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:02.795 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:02 smithi145 podman[245897]: 2024-09-21 10:54:02.386070993 +0000 UTC m=+0.446198865 container attach 9d8af4b0c1670fb7a1549ed6e6427862214ac151369ef366d451a484dd54a308 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T10:54:02.795 INFO:journalctl@ceph.osd.4.smithi145.stdout:Sep 21 10:54:02 smithi145 podman[245897]: 2024-09-21 10:54:02.517253932 +0000 UTC m=+0.577381804 container died 9d8af4b0c1670fb7a1549ed6e6427862214ac151369ef366d451a484dd54a308 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-21T10:54:02.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:02.393+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:02.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:02 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:02.393+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:03.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T10:54:03.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:54:03.146 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T10:54:03.146 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T10:54:03.146 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.5 2024-09-21T10:54:03.354 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:03 smithi145 systemd[1]: Stopping Ceph osd.5 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:03.354 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:03.260+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:03.354 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:03.260+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:03.794 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:03.485+0000 7f70e8455640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:54:03.794 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:03.485+0000 7f70e8455640 -1 osd.5 1507 *** Got signal Terminated *** 2024-09-21T10:54:03.794 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:03.485+0000 7f70e8455640 -1 osd.5 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:54:03.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:03.359+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:03.795 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:03.359+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:03.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:03.352+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:03.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:03 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:03.352+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:04.531 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:04.225+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:04.532 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:04.225+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:04.532 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:04.318+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:04.532 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:04.318+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:04.532 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:04.303+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:04.533 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:04.303+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:04.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.530Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:04.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.531Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:04.794 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.533Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.535Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.535Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.535Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:04.795 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:04 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:04.732Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.119:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.119:8765: connect: connection refused" 2024-09-21T10:54:05.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:05.254+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:05.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:05.254+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:05.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:05.290+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:05.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:05.291+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:05.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:05.277+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:05.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:05.277+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:05.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:05 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:05.277+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:06.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:06.264+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:06.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:06.264+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:06.544 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:06.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:06.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:06.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:06.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:06.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:06.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:06.319+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:06.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:06.319+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:06.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:06 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:06.319+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:07.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:07.247+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:07.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:07.247+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:07.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:07.345+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:07.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:07.345+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:07.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:07.345+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:07.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:07.322+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:07.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:07.322+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:07.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:07 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:07.322+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:08.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:08.209+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:25.714071+0000 front 2024-09-21T10:53:25.714084+0000 (oldest deadline 2024-09-21T10:53:49.213988+0000) 2024-09-21T10:54:08.544 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5[44894]: 2024-09-21T10:54:08.209+0000 7f70e1a67640 -1 osd.5 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.914953+0000 front 2024-09-21T10:53:33.914890+0000 (oldest deadline 2024-09-21T10:53:58.014911+0000) 2024-09-21T10:54:08.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:08.368+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:08.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:08.368+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:08.545 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:08.368+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:08.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:08.287+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:08.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:08.287+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:08.545 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:08 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:08.287+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:09.356 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:09 smithi145 podman[246023]: 2024-09-21 10:54:09.073696449 +0000 UTC m=+5.701184507 container died 90ab6d4f85712192662763857a64c0c8111cfcd9373e7e29aef0a05706da72f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:54:09.357 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:09 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:09.244+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:09.357 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:09 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:09.244+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:09.357 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:09 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:09.244+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:09.745 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:09 smithi145 podman[246023]: 2024-09-21 10:54:09.475493517 +0000 UTC m=+6.102981576 container cleanup 90ab6d4f85712192662763857a64c0c8111cfcd9373e7e29aef0a05706da72f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:54:09.745 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:09 smithi145 bash[246023]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5 2024-09-21T10:54:09.745 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:09 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:09.402+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:09.745 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:09 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:09.402+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:09.745 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:09 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:09.402+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:10.044 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:09 smithi145 podman[246047]: 2024-09-21 10:54:09.743451462 +0000 UTC m=+0.655582520 container remove 90ab6d4f85712192662763857a64c0c8111cfcd9373e7e29aef0a05706da72f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True) 2024-09-21T10:54:10.323 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:10.264+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:10.323 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:10.264+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:10.323 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:10.264+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:10.323 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:10 smithi145 podman[246140]: 2024-09-21 10:54:09.95259215 +0000 UTC m=+0.015639173 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:54:10.324 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:10 smithi145 podman[246140]: 2024-09-21 10:54:10.102556751 +0000 UTC m=+0.165603757 container create 4da079583947c84067f29f79853dc3ea8812c999e51784b95b5cf3f03c5923e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-21T10:54:10.324 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:10 smithi145 podman[246140]: 2024-09-21 10:54:10.321685131 +0000 UTC m=+0.384732139 container init 4da079583947c84067f29f79853dc3ea8812c999e51784b95b5cf3f03c5923e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:54:10.792 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:10.424+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:10.793 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:10.424+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:10.793 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:10 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:10.424+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:10.793 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:10 smithi145 podman[246140]: 2024-09-21 10:54:10.326597674 +0000 UTC m=+0.389644689 container start 4da079583947c84067f29f79853dc3ea8812c999e51784b95b5cf3f03c5923e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:54:10.793 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:10 smithi145 podman[246140]: 2024-09-21 10:54:10.404751042 +0000 UTC m=+0.467798068 container attach 4da079583947c84067f29f79853dc3ea8812c999e51784b95b5cf3f03c5923e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a) 2024-09-21T10:54:10.794 INFO:journalctl@ceph.osd.5.smithi145.stdout:Sep 21 10:54:10 smithi145 podman[246140]: 2024-09-21 10:54:10.505025714 +0000 UTC m=+0.568072722 container died 4da079583947c84067f29f79853dc3ea8812c999e51784b95b5cf3f03c5923e3 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2) 2024-09-21T10:54:11.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T10:54:11.131 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:54:11.132 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T10:54:11.132 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T10:54:11.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6 2024-09-21T10:54:11.392 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 systemd[1]: Stopping Ceph osd.6 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:11.393 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:11.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:11.393 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:11.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:11.393 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:11.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:11.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:11.390+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:11.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:11.390+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:11.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:11.390+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:11.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:11.479+0000 7fde22620640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:54:11.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:11.479+0000 7fde22620640 -1 osd.6 1507 *** Got signal Terminated *** 2024-09-21T10:54:11.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:11 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:11.479+0000 7fde22620640 -1 osd.6 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:54:12.544 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:12.424+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:12.546 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:12.424+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:12.546 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:12.424+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:12.546 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:12.281+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:12.547 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:12.281+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:12.547 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:12 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:12.281+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:13.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:13.421+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:13.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:13.421+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:13.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:13.421+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:13.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:13.310+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:13.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:13.310+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:13.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:13 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:13.310+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:14.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:14.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:14.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:14.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:14.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:14.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:14.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:14.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:14.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:14.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:14.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:14.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:14.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:14 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:14.334+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.841632+0000 front 2024-09-21T10:53:50.841644+0000 (oldest deadline 2024-09-21T10:54:13.741614+0000) 2024-09-21T10:54:15.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:15.371+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:15.840 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:15.371+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:15.841 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:15.371+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:15.841 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:15.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:15.841 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:15.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:15.841 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:15.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:15.841 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:15 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:15.297+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.841632+0000 front 2024-09-21T10:53:50.841644+0000 (oldest deadline 2024-09-21T10:54:13.741614+0000) 2024-09-21T10:54:16.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:16.391+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:16.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:16.391+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:16.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:16.391+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:16.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:16.391+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:16.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:16.299+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:27.137997+0000 front 2024-09-21T10:53:27.138019+0000 (oldest deadline 2024-09-21T10:53:50.637905+0000) 2024-09-21T10:54:16.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:16.299+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:32.938637+0000 front 2024-09-21T10:53:32.938721+0000 (oldest deadline 2024-09-21T10:53:57.038591+0000) 2024-09-21T10:54:16.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:16.299+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:42.939626+0000 front 2024-09-21T10:53:42.939677+0000 (oldest deadline 2024-09-21T10:54:05.839587+0000) 2024-09-21T10:54:16.795 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:16 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6[116394]: 2024-09-21T10:54:16.299+0000 7fde1fc3a640 -1 osd.6 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.841632+0000 front 2024-09-21T10:53:50.841644+0000 (oldest deadline 2024-09-21T10:54:13.741614+0000) 2024-09-21T10:54:17.404 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:17 smithi145 podman[246267]: 2024-09-21 10:54:17.107180719 +0000 UTC m=+5.755917658 container died e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T10:54:17.763 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:17 smithi145 podman[246267]: 2024-09-21 10:54:17.510612987 +0000 UTC m=+6.159349913 container cleanup e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T10:54:17.763 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:17 smithi145 bash[246267]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6 2024-09-21T10:54:17.763 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:17 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:17.434+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:17.764 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:17 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:17.434+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:17.764 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:17 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:17.434+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:17.764 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:17 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:17.434+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:18.044 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:17 smithi145 podman[246294]: 2024-09-21 10:54:17.761754871 +0000 UTC m=+0.648558384 container remove e6c42785413961da6dc3d9565664e3d813ee59cc6730477cda615f32bffc9c34 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T10:54:18.358 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:18 smithi145 podman[246384]: 2024-09-21 10:54:18.062917424 +0000 UTC m=+0.110181654 container create f1df5560a99a20205f6e1911e8bafee5dcd288ed938c6b1b4b62e959a529ba65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=quincy-fbd1a6a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:54:18.358 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:18 smithi145 podman[246384]: 2024-09-21 10:54:17.968152067 +0000 UTC m=+0.015416296 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:54:18.358 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:18 smithi145 podman[246384]: 2024-09-21 10:54:18.265141637 +0000 UTC m=+0.312405870 container init f1df5560a99a20205f6e1911e8bafee5dcd288ed938c6b1b4b62e959a529ba65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-21T10:54:18.358 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:18 smithi145 podman[246384]: 2024-09-21 10:54:18.269854539 +0000 UTC m=+0.317118771 container start f1df5560a99a20205f6e1911e8bafee5dcd288ed938c6b1b4b62e959a529ba65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-fbd1a6a, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T10:54:18.359 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:18 smithi145 podman[246384]: 2024-09-21 10:54:18.356634209 +0000 UTC m=+0.403898439 container attach f1df5560a99a20205f6e1911e8bafee5dcd288ed938c6b1b4b62e959a529ba65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T10:54:18.735 INFO:journalctl@ceph.osd.6.smithi145.stdout:Sep 21 10:54:18 smithi145 podman[246384]: 2024-09-21 10:54:18.445615212 +0000 UTC m=+0.492879452 container died f1df5560a99a20205f6e1911e8bafee5dcd288ed938c6b1b4b62e959a529ba65 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=quincy-fbd1a6a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T10:54:18.736 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:18 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:18.425+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:18.736 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:18 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:18.425+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:18.736 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:18 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:18.425+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:18.736 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:18 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:18.425+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:19.046 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T10:54:19.083 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:54:19.084 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T10:54:19.084 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T10:54:19.084 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.7 2024-09-21T10:54:19.293 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 systemd[1]: Stopping Ceph osd.7 for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:19.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.405+0000 7f14f0c5a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T10:54:19.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.405+0000 7f14f0c5a640 -1 osd.7 1507 *** Got signal Terminated *** 2024-09-21T10:54:19.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.405+0000 7f14f0c5a640 -1 osd.7 1507 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T10:54:19.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.451+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:19.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.451+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:19.795 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.451+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:19.795 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:19 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:19.451+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:20.701 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:20.414+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:20.701 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:20.414+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:20.701 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:20.414+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:20.701 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:20 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:20.414+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:21.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:21 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:21.383+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:21.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:21 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:21.383+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:21.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:21 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:21.383+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:21.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:21 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:21.383+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:22.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:22.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:22.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:22.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:22.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:22.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:22.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:22.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:22.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:22 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:22.394+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.145:6804 osd.4 since back 2024-09-21T10:53:57.826380+0000 front 2024-09-21T10:53:57.826464+0000 (oldest deadline 2024-09-21T10:54:21.926400+0000) 2024-09-21T10:54:23.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:23.427+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6806 osd.0 since back 2024-09-21T10:53:26.021455+0000 front 2024-09-21T10:53:26.021330+0000 (oldest deadline 2024-09-21T10:53:51.921224+0000) 2024-09-21T10:54:23.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:23.427+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6814 osd.1 since back 2024-09-21T10:53:33.022430+0000 front 2024-09-21T10:53:33.022443+0000 (oldest deadline 2024-09-21T10:53:58.222548+0000) 2024-09-21T10:54:23.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:23.427+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6822 osd.2 since back 2024-09-21T10:53:40.924446+0000 front 2024-09-21T10:53:40.924422+0000 (oldest deadline 2024-09-21T10:54:05.024318+0000) 2024-09-21T10:54:23.795 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:23.427+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.119:6830 osd.3 since back 2024-09-21T10:53:50.825695+0000 front 2024-09-21T10:53:50.825649+0000 (oldest deadline 2024-09-21T10:54:16.125650+0000) 2024-09-21T10:54:23.795 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:23 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7[56896]: 2024-09-21T10:54:23.427+0000 7f14ea26c640 -1 osd.7 1507 heartbeat_check: no reply from 172.21.15.145:6804 osd.4 since back 2024-09-21T10:53:57.826380+0000 front 2024-09-21T10:53:57.826464+0000 (oldest deadline 2024-09-21T10:54:21.926400+0000) 2024-09-21T10:54:25.397 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:25 smithi145 podman[246510]: 2024-09-21 10:54:25.116330526 +0000 UTC m=+5.818662034 container died 1a18c8e916547eb5e4e3725fbc35a1f16ade0bf78abd6764f48cc68b625fc32d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T10:54:25.764 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:25 smithi145 podman[246510]: 2024-09-21 10:54:25.487167393 +0000 UTC m=+6.189498901 container cleanup 1a18c8e916547eb5e4e3725fbc35a1f16ade0bf78abd6764f48cc68b625fc32d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-fbd1a6a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T10:54:25.765 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:25 smithi145 bash[246510]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7 2024-09-21T10:54:26.044 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:25 smithi145 podman[246543]: 2024-09-21 10:54:25.763388503 +0000 UTC m=+0.625418456 container remove 1a18c8e916547eb5e4e3725fbc35a1f16ade0bf78abd6764f48cc68b625fc32d (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=quincy-fbd1a6a) 2024-09-21T10:54:26.434 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:26 smithi145 podman[246638]: 2024-09-21 10:54:26.015492227 +0000 UTC m=+0.017158663 image pull fc3ec4a4f6be575045de0fd8a2fe84939e888152c447bdedb90add7841f7bef9 quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2 2024-09-21T10:54:26.435 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:26 smithi145 podman[246638]: 2024-09-21 10:54:26.139335154 +0000 UTC m=+0.141001617 container create d2a17b5685d7c0d5a872f086d6b4d7900dcc7e7aa8114f6bfe6ac61ebd130343 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2) 2024-09-21T10:54:26.435 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:26 smithi145 podman[246638]: 2024-09-21 10:54:26.34977731 +0000 UTC m=+0.351443743 container init d2a17b5685d7c0d5a872f086d6b4d7900dcc7e7aa8114f6bfe6ac61ebd130343 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=quincy-fbd1a6a, org.label-schema.vendor=CentOS) 2024-09-21T10:54:26.435 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:26 smithi145 podman[246638]: 2024-09-21 10:54:26.354999082 +0000 UTC m=+0.356665513 container start d2a17b5685d7c0d5a872f086d6b4d7900dcc7e7aa8114f6bfe6ac61ebd130343 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=quincy-fbd1a6a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T10:54:26.435 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:26 smithi145 podman[246638]: 2024-09-21 10:54:26.433034018 +0000 UTC m=+0.434700460 container attach d2a17b5685d7c0d5a872f086d6b4d7900dcc7e7aa8114f6bfe6ac61ebd130343 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-fbd1a6a, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T10:54:26.794 INFO:journalctl@ceph.osd.7.smithi145.stdout:Sep 21 10:54:26 smithi145 podman[246638]: 2024-09-21 10:54:26.530470488 +0000 UTC m=+0.532136910 container died d2a17b5685d7c0d5a872f086d6b4d7900dcc7e7aa8114f6bfe6ac61ebd130343 (image=quay.ceph.io/ceph-ci/ceph@sha256:ee510334bec3e52f2758802688c49d04ace3cbf9008a76cda46905f72653dab2, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=quincy-fbd1a6a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T10:54:27.170 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T10:54:27.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:54:27.208 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T10:54:27.209 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T10:54:27.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T10:54:27.544 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 systemd[1]: Stopping Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:27.858 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:54:27.858 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-09-21T10:54:27.858 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-09-21T10:54:27.858 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.549Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.552Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.552Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a[79532]: ts=2024-09-21T10:54:27.552Z caller=main.go:1273 level=info msg="See you next time!" 2024-09-21T10:54:27.859 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 podman[246765]: 2024-09-21 10:54:27.560203288 +0000 UTC m=+0.134245625 container died d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:28.183 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 podman[246765]: 2024-09-21 10:54:27.989034823 +0000 UTC m=+0.563077159 container cleanup d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:28.183 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:27 smithi145 bash[246765]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T10:54:28.411 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T10:54:28.434 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:28 smithi145 podman[246781]: 2024-09-21 10:54:28.181814586 +0000 UTC m=+0.619185016 container remove d2de6cb75d75e3c6c939a584cd4a84283a65c59f1531a6d6998b859e86356c34 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:28.434 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:28 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T10:54:28.435 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:28 smithi145 systemd[1]: Stopped Ceph prometheus.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:54:28.435 INFO:journalctl@ceph.prometheus.a.smithi145.stdout:Sep 21 10:54:28 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 11.284s CPU time. 2024-09-21T10:54:28.445 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T10:54:28.445 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T10:54:28.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c96f302a-77fe-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T10:54:30.117 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:30 smithi119 systemd[1]: Stopping Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:30.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:30 smithi119 ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a[90746]: ts=2024-09-21T10:54:30.285Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T10:54:30.617 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:30 smithi119 podman[375236]: 2024-09-21 10:54:30.336246528 +0000 UTC m=+0.153698715 container died e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:31.069 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:30 smithi119 podman[375236]: 2024-09-21 10:54:30.849831959 +0000 UTC m=+0.667284139 container cleanup e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:31.069 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:30 smithi119 bash[375236]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T10:54:31.366 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:31 smithi119 podman[375254]: 2024-09-21 10:54:31.067880603 +0000 UTC m=+0.731348710 container remove e4cf1f6d835543b195ff22879843f01a4be71d4893ec67aa1fa29a3f4ae71940 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:31.367 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:31 smithi119 podman[375254]: 2024-09-21 10:54:31.143542456 +0000 UTC m=+0.807010494 volume remove 3696cc076a32557327918e9d84820484bd812a46254b9134e990f1bf6e802be6 2024-09-21T10:54:31.867 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:31 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T10:54:31.867 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:31 smithi119 systemd[1]: Stopped Ceph alertmanager.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:54:31.867 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 21 10:54:31 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 4.409s CPU time. 2024-09-21T10:54:31.867 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:31 smithi119 systemd[1]: Stopping Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:32.247 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:31 smithi119 podman[375370]: 2024-09-21 10:54:31.89612237 +0000 UTC m=+0.118071579 container died 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:32.248 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 podman[375370]: 2024-09-21 10:54:32.246199547 +0000 UTC m=+0.468148798 container cleanup 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:32.524 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 bash[375370]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T10:54:32.858 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 podman[375388]: 2024-09-21 10:54:32.522588155 +0000 UTC m=+0.630040019 container remove 05912f78c0700984e06dfca333667da75fbe923c3bc039c16af808131a001f42 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T10:54:32.858 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:54:32.858 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T10:54:32.859 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 systemd[1]: Stopped Ceph node-exporter.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:54:32.859 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 21 10:54:32 smithi119 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 9.331s CPU time. 2024-09-21T10:54:41.859 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c96f302a-77fe-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T10:54:43.544 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:43 smithi145 systemd[1]: Stopping Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:43.919 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:43 smithi145 podman[247182]: 2024-09-21 10:54:43.646679882 +0000 UTC m=+0.109398500 container died e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.buildah.version=1.32.0) 2024-09-21T10:54:44.294 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:43 smithi145 podman[247182]: 2024-09-21 10:54:43.999814169 +0000 UTC m=+0.462532776 container cleanup e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, vendor=Red Hat, Inc., version=8.5, description=Ceph Grafana Container, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, name=ubi8, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.expose-services=) 2024-09-21T10:54:44.294 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:44 smithi145 bash[247182]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a 2024-09-21T10:54:44.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:44 smithi145 podman[247199]: 2024-09-21 10:54:44.376521324 +0000 UTC m=+0.732150051 container remove e033bb16a7cdb90183c9e874f9f1030ed115914ef4df082343a3b79bea94a41b (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.buildah.version=1.32.0, name=ubi8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, version=8.5, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-21T10:54:44.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:44 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:54:44.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:44 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T10:54:44.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:44 smithi145 systemd[1]: Stopped Ceph grafana.a for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:54:44.692 INFO:journalctl@ceph.grafana.a.smithi145.stdout:Sep 21 10:54:44 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@grafana.a.service: Consumed 33.732s CPU time. 2024-09-21T10:54:44.943 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:44 smithi145 systemd[1]: Stopping Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490... 2024-09-21T10:54:45.271 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:44 smithi145 podman[247312]: 2024-09-21 10:54:44.94217513 +0000 UTC m=+0.116602189 container died d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:54:45.271 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 podman[247312]: 2024-09-21 10:54:45.270349856 +0000 UTC m=+0.444776910 container cleanup d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:54:45.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 bash[247312]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T10:54:45.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 podman[247330]: 2024-09-21 10:54:45.446660979 +0000 UTC m=+0.508955240 container remove d906bafb8d867998e8dcc81d664627f23c4cd0e3ebb6d93212a4958a2f47fb02 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c96f302a-77fe-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T10:54:45.544 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T10:54:46.044 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T10:54:46.306 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 systemd[1]: Stopped Ceph node-exporter.b for c96f302a-77fe-11ef-baf6-efdc52797490. 2024-09-21T10:54:46.307 INFO:journalctl@ceph.node-exporter.b.smithi145.stdout:Sep 21 10:54:45 smithi145 systemd[1]: ceph-c96f302a-77fe-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 8.488s CPU time. 2024-09-21T10:54:50.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:54:50.313 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T10:54:50.340 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T10:54:50.342 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665/remote/smithi119/crash 2024-09-21T10:54:50.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/crash -- . 2024-09-21T10:54:50.398 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T10:54:50.398 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-09-21T10:54:50.400 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665/remote/smithi145/crash 2024-09-21T10:54:50.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/crash -- . 2024-09-21T10:54:50.435 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T10:54:50.436 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2024-09-21T10:54:50.437 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T10:54:50.437 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v OSD_DOWN | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v MON_DOWN | egrep -v 'out of quorum' | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v OSDMAP_FLAGS | egrep -v PG_AVAILABILITY | head -n 1 2024-09-21T10:54:50.520 INFO:teuthology.orchestra.run.smithi119.stdout:1726912800.0000906 mon.a (mon.0) 1014 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-21T10:54:50.522 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-21T10:54:50.522 INFO:tasks.cephadm:Compressing logs... 2024-09-21T10:54:50.523 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T10:54:50.525 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T10:54:50.564 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-21T10:54:50.565 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T10:54:50.565 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T10:54:50.565 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi119.oikglq.log 2024-09-21T10:54:50.565 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T10:54:50.565 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T10:54:50.566 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi119.oikglq.log: 81.6% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi119.oikglq.log.gz 2024-09-21T10:54:50.566 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T10:54:50.566 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T10:54:50.566 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T10:54:50.566 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T10:54:50.567 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T10:54:50.567 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log 2024-09-21T10:54:50.567 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T10:54:50.567 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T10:54:50.568 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-09-21T10:54:50.568 INFO:teuthology.orchestra.run.smithi145.stderr: /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T10:54:50.569 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi145.tjomfr.log 2024-09-21T10:54:50.570 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T10:54:50.570 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log: /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi145.tjomfr.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T10:54:50.570 INFO:teuthology.orchestra.run.smithi145.stderr: 62.5% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mds.foo.smithi145.tjomfr.log.gz 2024-09-21T10:54:50.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T10:54:50.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T10:54:50.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T10:54:50.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T10:54:50.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log 2024-09-21T10:54:50.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T10:54:50.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T10:54:50.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T10:54:50.575 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.1% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T10:54:50.846 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 84.0% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T10:54:50.875 INFO:teuthology.orchestra.run.smithi145.stderr: 91.3% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T10:54:50.881 INFO:teuthology.orchestra.run.smithi119.stderr: 91.1% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T10:54:50.892 INFO:teuthology.orchestra.run.smithi119.stderr: 90.9% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T10:54:51.059 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T10:54:51.176 INFO:teuthology.orchestra.run.smithi145.stderr: 90.0% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T10:54:51.306 INFO:teuthology.orchestra.run.smithi119.stderr: 95.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T10:54:51.385 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T10:54:51.643 INFO:teuthology.orchestra.run.smithi145.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T10:54:52.184 INFO:teuthology.orchestra.run.smithi145.stderr: 90.2% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T10:54:54.917 INFO:teuthology.orchestra.run.smithi119.stderr: 89.5% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T10:54:55.915 INFO:teuthology.orchestra.run.smithi145.stderr: 92.2% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T10:55:12.121 INFO:teuthology.orchestra.run.smithi119.stderr: 91.9% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T10:55:51.772 INFO:teuthology.orchestra.run.smithi119.stderr: 90.5% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T10:57:58.121 INFO:teuthology.orchestra.run.smithi145.stderr: 92.7% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T10:58:55.733 INFO:teuthology.orchestra.run.smithi145.stderr: 94.0% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T10:59:48.210 INFO:teuthology.orchestra.run.smithi145.stderr: 93.8% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T11:00:49.790 INFO:teuthology.orchestra.run.smithi145.stderr: 93.7% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T11:00:49.793 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-09-21T11:00:49.793 INFO:teuthology.orchestra.run.smithi145.stderr:real 5m59.253s 2024-09-21T11:00:49.793 INFO:teuthology.orchestra.run.smithi145.stderr:user 4m56.760s 2024-09-21T11:00:49.793 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m13.365s 2024-09-21T11:23:11.785 INFO:teuthology.orchestra.run.smithi119.stderr: 93.2% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T11:23:45.028 INFO:teuthology.orchestra.run.smithi119.stderr: 93.1% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T11:24:32.892 INFO:teuthology.orchestra.run.smithi119.stderr: 92.9% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T11:24:52.405 INFO:teuthology.orchestra.run.smithi119.stderr: 92.7% -- replaced with /var/log/ceph/c96f302a-77fe-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T11:24:52.448 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-09-21T11:24:52.449 INFO:teuthology.orchestra.run.smithi119.stderr:real 30m1.909s 2024-09-21T11:24:52.449 INFO:teuthology.orchestra.run.smithi119.stderr:user 10m5.857s 2024-09-21T11:24:52.449 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m30.311s 2024-09-21T11:24:52.449 INFO:tasks.cephadm:Archiving logs... 2024-09-21T11:24:52.450 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665/remote/smithi119/log 2024-09-21T11:24:52.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T11:26:23.527 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665/remote/smithi145/log 2024-09-21T11:26:23.528 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T11:27:07.715 INFO:tasks.cephadm:Removing cluster... 2024-09-21T11:27:07.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c96f302a-77fe-11ef-baf6-efdc52797490 --force 2024-09-21T11:27:09.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c96f302a-77fe-11ef-baf6-efdc52797490 --force 2024-09-21T11:27:10.077 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T11:27:10.077 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:27:10.094 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T11:27:10.109 INFO:tasks.cephadm:Teardown complete 2024-09-21T11:27:10.110 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T11:27:10.119 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T11:27:10.119 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:27:10.136 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T11:27:10.239 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T11:27:10.248 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T11:27:10.249 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:27:10.251 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T11:27:10.264 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:27:10.265 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-09-21T11:27:10.281 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:27:10.281 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-09-21T11:27:10.281 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 -220us[ -226us] +/- 47ms 2024-09-21T11:27:10.281 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 +269us[ +269us] +/- 66ms 2024-09-21T11:27:10.282 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +107us[ +107us] +/- 59ms 2024-09-21T11:27:10.282 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:27:10.340 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T11:27:10.340 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-21T11:27:10.341 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 3 6 377 56 -228us[ -233us] +/- 47ms 2024-09-21T11:27:10.341 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 52 +270us[ +270us] +/- 66ms 2024-09-21T11:27:10.341 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +105us[ +105us] +/- 59ms 2024-09-21T11:27:10.341 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T11:27:10.359 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T11:27:10.368 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T11:27:10.369 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T11:27:10.377 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:27:10.493 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T11:27:10.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T11:27:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:27:10.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T11:27:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:27:10.715 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T11:27:10.725 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T11:27:10.734 INFO:teuthology.task.internal:Duration was 6229.934146 seconds 2024-09-21T11:27:10.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T11:27:10.744 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T11:27:10.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:27:10.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T11:27:10.794 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:27:10.815 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T11:27:11.189 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T11:27:11.189 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-09-21T11:27:11.189 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:27:11.215 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-09-21T11:27:11.216 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T11:27:11.252 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T11:27:11.252 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:27:11.258 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T11:27:11.329 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T11:27:11.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:27:11.331 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T11:27:12.943 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T11:27:12.954 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T11:27:12.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:27:12.981 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T11:27:13.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T11:27:13.091 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:27:13.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:27:13.132 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-09-21T11:27:13.138 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-09-21T11:27:13.161 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:27:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:27:13.205 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T11:27:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:27:13.220 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T11:27:13.231 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T11:27:13.232 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665/remote/smithi119 2024-09-21T11:27:13.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:27:13.310 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_03:08:26-upgrade-main-distro-default-smithi/7913665/remote/smithi145 2024-09-21T11:27:13.311 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T11:27:13.356 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T11:27:13.356 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:27:13.359 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T11:27:13.404 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T11:27:13.420 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T11:27:13.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T11:27:13.455 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T11:27:13.455 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:27:13.458 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T11:27:13.473 INFO:teuthology.orchestra.run.smithi145.stdout: 266054 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 11:27 /home/ubuntu/cephtest 2024-09-21T11:27:13.484 INFO:teuthology.orchestra.run.smithi119.stdout: 265324 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 11:27 /home/ubuntu/cephtest 2024-09-21T11:27:13.484 INFO:teuthology.orchestra.run.smithi119.stdout: 5242881 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 10:50 /home/ubuntu/cephtest/mnt.0 2024-09-21T11:27:13.484 INFO:teuthology.orchestra.run.smithi119.stdout: 5242883 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 21 10:50 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T11:27:13.484 INFO:teuthology.orchestra.run.smithi119.stdout: 5256394 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 10:50 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T11:27:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T11:27:13.485 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-21T11:27:13.486 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi119 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:27:13.492 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T11:27:13.502 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T11:27:13.517 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi119 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-21T11:27:13.521 INFO:teuthology.run:Summary data: description: upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} duration: 6229.934146404266 failure_reason: 'Command failed (workunit test cls/test_cls_rbd.sh) on smithi119 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=408b126d58474820a3b2efa8dedbde51 status: fail success: false 2024-09-21T11:27:13.521 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T11:27:13.629 INFO:teuthology.run:FAIL